Method for training deep learning model for extracting road feature based on multi-tasks and computing device using the same

Abstract

To provide a method for training a deep learning model for extracting road features based on multi-tasks.SOLUTION: A method comprises: setting an initial value for learning; determining the number of learning steps required to complete one epoch for each of learning tasks; determining the total number of steps included in the entire epoch by referring to the number of learning steps; comparing the total sum of the current numbers of steps of the learning tasks with the total number of steps, if the total sum of the current numbers of steps of the learning tasks is less than the total number of steps, randomly selecting a specific learning task for learning out of the learning tasks by referring to each of initial selection probability values for each of the learning tasks; and calculating losses for the specific task by referring to a prediction result output in a specific learning task layer and Ground Truth corresponding to the specific learning task.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for training a deep learning model for extracting road features based on multitasking and a computing device using the same.

Background Art

[0002] When an autonomous vehicle travels, the technology for estimating the current position is very important.

[0003] However, the existing technology for an autonomous vehicle to estimate its position is a method of simply matching an HD map using lane detection information and only applying a lane detection model based on the edges of video processing. Therefore, due to noise such as changes in image quality, shadows, and hiding or damage of lanes, there are limitations in lane estimation performance. When matching the HD map and lane detection information, only lanes are considered, so the performance degradation due to incorrect lane estimation is very serious.

[0004] Therefore, the inventor of the present invention attempts to propose a road feature estimation model for performing HD map matching at a position where there are no lanes or the lanes are blocked on the road.

Summary of the Invention

Problems to be Solved by the Invention

[0005] An object of the present invention is to solve the above-mentioned problems.

[0006] Another object of the present invention is to perform HD map matching at a position where there are no lanes or the lanes are blocked on the road.

[0007] The present invention further aims to apply a multi-task model structure for minimizing overlapping operations for a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road.

[0008] The present invention further aims to propose a multi-task learning method based on heterogeneous data for minimizing the construction cost of learning data.

[0009] The present invention further aims to propose a road feature estimation model that is robust to image noise and can simplify post-processing.

Means for Solving the Problems

[0010] According to an embodiment of the present invention, in a method of learning a deep learning model for extracting road features based on multitasking, (a) a computing device sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road, sets initial selection probability values for each of the first to third learning tasks, sets first to third learning step numbers required to complete one epoch for each of the first to third learning tasks, and determines a total number of steps included in all epochs with reference to the first to third learning step numbers; (b) when it is determined that the total learning progress rate is equal to or lower than a first threshold progress rate or equal to or lower than a first threshold number of steps, the computing device randomly selects a specific learning task to perform learning from among the first to third learning tasks with reference to the initial selection probability values for each of the first to third learning tasks, and calculates a loss for the specific learning task; and (c) the computing device updates model parameters related to the specific learning task with reference to the loss value of the specific learning task. A method including these steps is provided.

[0011] In one example, in the step (b), when it is determined that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, the computing device calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the current specific learning performance to the highest specific learning performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate, calculates a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task is selected from among the first to third learning tasks as the specific learning selection probability adjustment value. In one example, in the step (b), the specific learning weight is represented by a value obtained by subtracting from 1 the ratio of the specific learning weight to the sum of the first to third performance achievement rates for the first to third learning tasks respectively, and the specific learning selection probability adjustment value is represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps of the specific learning task to the value obtained by summing all of the first to third learning step numbers.

[0012] In one example, when (d) the model parameters are updated, the computing device evaluates the specific current performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task with reference to the ratio of the specific current performance to the specific maximum learning performance of the specific learning task, calculates an overall learning progress rate with reference to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), and when the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total step number, supports repeating the steps after the (b) step with 1 added to the step number of the specific current step; is further included.

[0013] In one example, as the shared layer, the deep learning model includes an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer to which encoder output data output from the encoder layer is input. As individual layers, the deep learning model includes a first learning task layer including a first task header to which decoder output data output from the decoder layer is input and a first output function for applying an output function operation to first task header output data output from the first task header; a second learning task layer including a second task header to which decoder output data output from the decoder layer is input and a second output function for applying an output function operation to second task header output data output from the second task header; and a third learning task layer including a third task header to which decoder output data output from the decoder layer is input and a specific Softmax function for applying a classification operation to third task header output data output from the third task header.

[0014] In one example, in the step (c), when it is determined that the total learning progress rate is equal to or lower than a second threshold progress rate or equal to or lower than a second threshold number of steps, the computing device causes the shared layer and the individual layers included in the deep learning model to be learned. When it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, specific model parameters of a specific individual layer related to the specific learning task among the individual layers are updated in a state where the shared layer is frozen and learning is stopped.

[0015] In one example, the deep learning model includes, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first decoder corresponding to the first learning task and to which encoder output data output from the encoder layer is input, and a first learning task layer including a first output function for applying an output function operation to the first decoder output data output from the first decoder; a second decoder corresponding to the second learning task and to which the encoder output data output from the encoder layer is input, and a second learning task layer including a second output function for applying an output function operation to the second decoder output data output from the second decoder; and a third decoder corresponding to the third learning task and to which the encoder output data output from the encoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder.

[0016] In one example, in the step (c), when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or equal to or less than a second threshold number of steps, the computing device causes the shared layer and the individual layers included in the deep learning model to be learned, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, the specific model parameters of the specific individual layer related to the specific learning task among the individual layers are updated in a state where the shared layer is frozen and learning is stopped.

[0017] In one example, when the lane candidate group data corresponding to the first learning task output from the deep learning model is input, the computing device labels the lane candidate group data for each channel, selects the largest specific label blob (bLob), removes the remaining label blobs, determines the coordinates of the blob center point for the specific label blob, and then applies a moving average filter to the coordinates of the blob center point to determine prediction information regarding the lane; and (f) the computing device performs matching with reference to the prediction information regarding the lane and the HD map (map), thereby estimating the position and orientation of the autonomous vehicle; are further included.

[0018] In one example, after the step (e), when the prediction information regarding the lane and the information regarding the semantic segmentation are acquired, the computing device differentiates a specific overlapping portion between the prediction information regarding the lane and the information regarding the semantic segmentation as the left and right boundary lines of a specific lane, determines the coordinates of the overlapping center point between the left boundary line and the right boundary line as the center point of the exposed lane, restores the area for the non-exposed lane with reference to the left and right boundary lines of the exposed lane, and further includes the step of determining adjustment information regarding the lane. In the step (f), the computing device performs matching with reference to the adjustment information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous vehicle.

[0019] In one example, in the step (a), the computing device further sets a fourth learning task for simultaneous learning of the first to third learning tasks, further sets an initial selection probability value for the fourth learning task, further sets a number of fourth learning steps required to complete one epoch for the fourth learning task, determines the total number of steps included in the entire epoch with reference to the number of first to fourth learning steps, and in the step (b), when it is determined that the total learning progress rate is equal to or lower than the first threshold progress rate or equal to or lower than the first threshold number of steps, the computing device randomly selects a specific learning task to be learned from among the first to fourth learning tasks with reference to the initial selection probability value for each of the first to fourth learning tasks, and calculates a loss for the specific learning task.

[0020] In one example, (g) when the model parameters are updated, the computing device evaluates the specific current performance of the specific learning task with reference to a specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task with reference to a ratio of the specific current performance to the specific maximum performance of the specific learning task, calculates a total learning progress rate with reference to first to fourth learning performance achievement rates (any one of the first to fourth learning performance achievement rates is the specific performance achievement rate), and when a sum of first to third current step numbers in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, assists to perform the steps after the step (b) again with 1 added to the number of steps of the specific current step.

[0021] According to another embodiment of the present invention, in a computing device that learns a deep learning model for extracting road features based on multitasking, at least one memory for storing instructions; and at least one processor configured to execute the instructions, the processor (I) sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road, sets an initial selection probability value for each of the first to third learning tasks, sets a first to third number of learning steps required to complete one epoch for each of the first to third learning tasks, and determines a total number of steps included in all epochs with reference to the first to third number of learning steps, (II) when it is determined that the total learning progress rate is equal to or lower than a first threshold progress rate or equal to or lower than a first threshold number of steps, randomly selects a specific learning task to perform learning from among the first to third learning tasks with reference to the initial selection probability value for each of the first to third learning tasks, and calculates a loss for the specific learning task; and (III) updates model parameters related to the specific learning task with reference to the loss value of the specific learning task; A computing device that performs the above is provided.

[0022] In one example, in the (II) process, when it is determined that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, the processor calculates a specific performance achievement rate for the specific learning task with reference to the ratio of the specific current learning performance to the specific maximum learning performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task with reference to the specific performance achievement rate, calculates a specific learning selection probability adjustment value with reference to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task is selected from among the first to third learning tasks as the specific learning selection probability adjustment value.

[0023] In one example, in the (II) process, the specific learning weight is represented by a value obtained by subtracting from 1 the ratio of the specific learning weight to the sum of the first to third performance achievement rates for the first to third learning tasks respectively, and the specific learning selection probability adjustment value can be represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps of the specific learning task to the value obtained by adding up all of the first to third learning step numbers.

[0024] In one example, when the model parameters are updated, the processor evaluates the specific current performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current performance of the specific learning task to the specific maximum performance of the specific learning task, calculates a total learning progress rate by referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), and if the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, supports re - performing the process after the process (II) with 1 added to the step number of the specific current step; can be further performed.

[0025] In one example, as a shared layer, the deep learning model includes an encoder layer into which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer into which encoder output data output from the encoder layer is input. As individual layers, a first task header corresponding to the first learning task into which decoder output data output from the decoder layer is input, and a first learning task layer including a first output function for applying an output function operation to the first task header output data output from the first task header; a second task header corresponding to the second learning task into which the decoder output data output from the decoder layer is input, and a second learning task layer including a second output function for applying an output function operation to the second task header output data output from the second task header; and a third task header corresponding to the third learning task into which the decoder output data output from the decoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third task header output data output from the third task header can be performed.

[0026] In one example, in the (III) process, when it is determined that the total learning progress rate is equal to or lower than a second threshold progress rate, or equal to or less than a second threshold number of steps, the processor learns the shared layer and the individual layers included in the deep learning model. When it is determined that the total learning progress rate exceeds the second threshold progress rate, or exceeds the second threshold number of steps, with the shared layer frozen and learning stopped, the processor can update specific model parameters of specific individual layers related to the specific learning task among the individual layers.

[0027] In one example, the deep learning model includes, as a shared layer, an encoder layer into which image data acquired through at least one camera mounted on the autonomous vehicle is input, and as individual layers, a first decoder corresponding to the first learning task, into which encoder output data output from the encoder layer is input, and a first learning task layer including a first output function for applying an output function operation to the first decoder output data output from the first decoder; a second decoder corresponding to the second learning task, into which the encoder output data output from the encoder layer is input, and a second learning task layer including a second output function for applying an output function operation to the second decoder output data output from the second decoder; and a third decoder corresponding to the third learning task, into which the encoder output data output from the encoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder can be performed.

[0028] In one example, in the (III) process, when it is determined that the total learning progress rate is equal to or lower than a second threshold progress rate or equal to or lower than a second threshold number of steps, the processor causes the shared layer and the individual layers included in the deep learning model to be learned, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, the processor can update specific model parameters of a specific individual layer related to the specific learning task among the individual layers while freezing the shared layer and stopping the learning.

[0029] In one example, when the lane candidate group data corresponding to the first learning task output from the deep learning model is input to the processor, the processor labels the lane candidate group data for each channel, selects the largest specific label blob, removes the remaining label blobs, obtains the coordinates of the blob center point for the specific label blob, and then applies a moving average filter to the coordinates of the blob center point to determine prediction information regarding the lane; and (VI) a process of performing matching with reference to the prediction information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous driving vehicle; can be further performed.

[0030] In one example, after the (V) process, when the prediction information regarding the lane and the information regarding the semantic segmentation are obtained, the processor further performs a process of distinguishing a specific overlapping portion between the prediction information regarding the lane and the information regarding the semantic segmentation as the left and right boundary lines of a specific lane, determining the coordinates of the overlapping center point between the left boundary line and the right boundary line as the center point of the exposed lane, restoring the region for the non-exposed lane with reference to the left and right boundary lines of the exposed lane, and determining adjustment information regarding the lane. In the (VI) process, matching can be performed with reference to the adjustment information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous driving vehicle.

[0031] In one example, in the (I) process, the processor further sets a fourth learning task for simultaneous learning of the first to third learning tasks, further sets an initial selection probability value for the fourth learning task, and further sets a fourth number of learning steps required to complete one epoch for the fourth learning task. With reference to the first to fourth numbers of learning steps, the processor determines the total number of steps included in the entire epoch. In the (II) process, when it is determined that the total learning progress rate is equal to or lower than the first threshold progress rate or equal to or lower than the first threshold number of steps, the processor randomly selects a specific learning task to be learned from among the first to fourth learning tasks with reference to the initial selection probability value for each of the first to fourth learning tasks, and can calculate the loss for the specific learning task.

[0032] In one example, when the model parameters are updated, the processor evaluates the specific current performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task with reference to the ratio of the specific current performance to the specific maximum performance of the specific learning task, calculates the total learning progress rate with reference to the first to fourth learning performance achievement rates (any one of the first to fourth learning performance achievement rates is the specific performance achievement rate), and when the sum of the first to third current step numbers in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, assists in re - performing the process after the (II) process with 1 added to the step number of the specific current step.

Advantages of the Invention

[0033] According to the present invention, the following advantages are achieved.

[0034] The present invention has an effect that HD map matching can be performed at a position where there are no lanes or the like on the road or the lanes are blocked.

[0035] The present invention has an effect that a multi-task model structure for minimizing overlapping operations for a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road can be applied.

[0036] The present invention has an effect that a multi-task learning method based on heterogeneous data for minimizing the construction cost of learning data can be realized.

[0037] The present invention has an effect that a road feature estimation model that is robust to video noise and can simplify post-processing can be realized.

[0038] The following drawings attached for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and those with ordinary knowledge in the technical field to which the present invention belongs (hereinafter, "ordinary technicians") can obtain other drawings based on these drawings without performing inventive work.

Brief Description of the Drawings

[0039]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

MODE FOR CARRYING OUT THE INVENTION

[0040] The following detailed description of the present invention refers to the accompanying drawings that illustrate specific embodiments in which the present invention can be implemented, in order to clarify the objectives, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail so that an ordinary technician can implement the present invention.

[0041] Also, throughout the detailed description and claims of the present invention, the word "comprising" and its variations are not intended to exclude other technical features, additives, components, or steps. For an ordinary technician, other objectives, advantages, and characteristics of the present invention will be clarified partly from this specification and partly from the implementation of the present invention. The following examples and drawings are provided by way of illustration and are not intended to limit the present invention.

[0042] Furthermore, the present invention encompasses all possible combinations of the embodiments shown in this specification. It should be understood that the various embodiments of the present invention are different from each other but do not necessarily exclude each other. For example, the specific shapes, structures, and characteristics described in this specification can be realized as one embodiment without departing from the spirit and scope of the present invention in relation to one embodiment. Also, it should be understood that the position or arrangement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the present invention is limited only by the appended claims together with all ranges equivalent to what the claims claim, provided that the scope is appropriately described. In the drawings, like reference numerals refer to the same or similar functions throughout the various aspects.

[0043] Hereinafter, for the convenience of those having ordinary knowledge in the technical field to which the present invention pertains to easily implement the present invention, the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0044] FIG. 1 is a diagram showing a schematic configuration of a computing device that learns a deep learning model for extracting road features based on multitasking according to an embodiment of the present invention.

[0045] As shown in FIG. 1, a computing device 100 that learns a deep learning model for extracting road features based on multitasking according to the present invention can include a memory 110 and a processor 120.

[0046] The memory 110 of the computing device 100 that learns a deep learning model for extracting road features based on multitasking can store instructions performed by the processor 120. Specifically, the instructions are codes generated for the purpose of causing the computing device 100 for providing content to function in a specific manner, and can be stored in a computer-usable or computer-readable memory for a computer and other programmable data processing devices. The instructions can perform a process for executing the functions described in the specification of the present invention.

[0047] And the processor 120 of the computing device 100 can include hardware configurations such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit), a cache memory, and a data bus. Further, the computing device 100 can also further include a software configuration of an operating system and an application for performing a specific purpose.

[0048] In addition, the computing device 100 can be linked with a database. Here, the database can include at least one type of storage medium such as a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, and an optical disk, and is not limited thereto, and can include all media capable of storing data. Also, the database can be installed separately from the computing device 100, or alternatively, can be installed inside the computing device 100 to record data transmitted or received, and can be realized separately into two or more parts, which is different depending on the implementation conditions of the invention, unlike what is shown in the figure.

[0049] A method using the computing device 100 according to an embodiment of the present invention configured as described above will be described with reference to FIG. 2 as follows.

[0050] FIG. 2 is a flowchart schematically showing a process of learning a deep learning model for extracting road features based on multitasking according to an embodiment of the present invention.

[0051] Referring to FIG. 2, first, initial values for learning can be set in step S211. Specifically, in step S211, the computing device 100 registers a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road, sets selection probability initial values for each of the first to third learning tasks, and can set initial values of a performance achievement rate (details regarding the performance achievement rate will be described later in steps S215 and S242) and a total learning progress rate (details regarding the total learning progress rate will be described later in step S242) for each of the first to third learning tasks to 0.

[0052] Thereafter, proceeding to step S212, the number of first to third learning steps required to complete one epoch for each of the first to third learning tasks is determined, and with reference to the number of first to third learning steps, the total number of steps included in all epochs can be determined.

[0053] Thereafter, proceeding to step S213, the sum of the current step numbers of the first to third learning tasks is compared with the total number of steps. If the sum of the current step numbers of the first to third learning tasks is less than the total number of steps, proceed to step S214 described later; if the sum of the current step numbers of the first to third learning tasks is the same as the total number of steps, proceed to step S250 to complete the learning.

[0054] Since the current state is a state where only initial values are set as described in step S211, it is possible to proceed to step S214.

[0055] In step S214, when it is determined that the total learning progress rate is less than or equal to the first threshold progress rate or less than or equal to the first threshold number of steps, the computing device 100 proceeds to step S221 and randomly selects a specific learning task to perform learning from among the first to third learning tasks with reference to the initial selection probability values for each of the first to third learning tasks.

[0056] Here, the number of steps of the n-th learning task (where n is an integer from 1 to 3) = the number of data of the n-th learning task / the batch size corresponding to the n-th learning task, and the total number of steps = the number of epochs set by the user ×

Number

[0057] Explaining again by returning to step S221, after randomly selecting a specific learning task, it proceeds to step S222, and calculates the loss for a specific task with reference to the prediction result output by the specific learning task layer and the GT corresponding to the specific learning task, that is, the Ground Truth.

[0058] In step S214, contrary to the above example, when the total learning progress rate exceeds the first threshold progress rate or exceeds the first threshold number of steps, the computing device 100 proceeds to step S215, and refers to the ratio of the specific current learning performance (the current performance value of the specific learning task) to the specific maximum learning performance (the maximum performance value of the specific learning task calculated in advance through testing for a single learning task) of the specific learning task, calculates the specific performance achievement rate for the specific learning task, refers to the specific performance achievement rate, and calculates the specific learning weight corresponding to the specific learning task (the value obtained by subtracting 1 from the ratio of the specific performance achievement rate to the sum of the first to third performance achievement rates corresponding to the first to third learning tasks respectively), calculates a specific learning selection probability adjustment value with reference to the specific learning weight and the number of steps of the specific learning task, determines the probability that the specific learning task is selected from among the first to third learning tasks as the specific learning selection probability adjustment value, and then can proceed to steps S221 and S222 described above.

[0059] Here, the specific performance achievement rate can be expressed as the specific current learning performance / the specific maximum learning performance, the specific learning weight can be expressed as 1 - (the specific performance achievement rate / the sum of the first performance achievement rate + the second performance achievement rate + the third performance achievement rate), and the specific learning selection probability adjustment value can be expressed as the specific learning weight × (the number of steps of the specific learning task / the sum of the number of steps of the first learning task + the number of steps of the second learning task + the number of steps of the third learning task).

[0060] Explaining again by returning to step S222, step S222 calculates the loss for a specific learning task. To assist in understanding this, first, with reference to FIGS. 3 and 4, the process of calculating the loss will be schematically described.

[0061] FIG. 3 shows a configuration for estimating the characteristics of a driving road based on a multi-task deep learning model, and illustrates (i) a structure capable of simultaneously performing a lane estimation task (i.e., the first learning task), a road structure line estimation task (i.e., the second learning task), and a semantic segmentation task (i.e., the third learning task), and (ii) a process of estimating the position and orientation of an autonomous driving vehicle by matching the result of the lane estimation task and the result of the semantic segmentation task, and using adjustment information and an HD map obtained by matching the result of the road structure line estimation task and the result of the semantic segmentation task.

[0062] Referring to FIG. 3, after inputting image data 300 obtained through at least one camera mounted on an autonomous driving vehicle into an encoder layer 310, and then inputting the output of the encoder layer 310 into a decoder layer 320, as a result, lane candidate group data 321 for a first learning task for estimating lanes on a road, structure line candidate group data 322 for a second learning task for estimating lines of structures related to the road, and semantic segmentation 323 for a third learning task for semantic segmentation of the lanes and the structures of the road can be obtained as the output of the decoder layer 320.

[0063] First, in the deep learning model, the encoder layer 310 has a structure that shares operation information for all three tasks, and the decoder layer 320 can be configured such that the three tasks can commonly share the operation results according to the performance of the selected basic model, or can be configured to perform operations independently. This will be described later with reference to FIGS. 5 and 6.

[0064] Also, the first learning task is a task of a model that can predict by connecting lanes that are partially hidden or interrupted between the same lanes for a plurality of lanes. The second learning task is a task of a model that can predict by connecting a vertical structure (such as a utility pole or a traffic signal support) or a part of a road structure (such as a curb boundary) when it is hidden. The third learning task is a task of a model that can perform general segmentation on lanes, curbs, traffic signals, utility poles, automobiles, drivable roads, and the like.

[0065] On the other hand, post-processing 340 and the like in FIG. 3 will be described later as shown in FIGS. 7 and 8.

[0066] FIG. 4 is a diagram showing, as an example, lane learning GT data, road structure learning GT data, and semantic segmentation learning GT data corresponding to image data according to an embodiment of the present invention.

[0067] Specifically, the lane learning GT data 331 indicates the ground truth values for lines such as lanes, stop lines, guide lines, and speed bump lines that are obtained by adding left, right, and corresponding indexes (e.g., L3, L2, L1, R1, R2, R3...) centered on the autonomous vehicle to the image data 300 acquired through at least one camera mounted on the autonomous vehicle. The road structure learning GT data 332 indicates the ground truth values for lines such as utility poles, traffic lights, traffic sign supports, and curbs for displaying road structures like those in the image data 300. The semantic segmentation learning GT data 333 indicates the ground truth values obtained by displaying the visible states of lanes, drivable roads, curbs, traffic lights, vehicles, and pedestrians as surface information for the image data 300. Such ground truth values, namely, the lane learning GT data 331, the road structure learning GT data 332, and the semantic segmentation learning GT data 333, can calculate the loss through comparison with the estimated values of the decoder layer 320, that is, the lane candidate group data 321 for the first learning task, the structure line candidate group data 322 for the second learning task, and the semantic segmentation 323 for the third learning task, respectively.

[0068] Thereafter, it proceeds to step S223. When the total learning progress rate is compared with the second threshold progress rate and the second threshold step number, and it is determined that the total learning progress rate is less than or equal to the second threshold progress rate or less than or equal to the second threshold step number, the computing device 100 proceeds to step S230 and can update the model parameters related to a specific learning task by backpropagating the loss calculated in step S222.

[0069] In step S223, contrary to the above example, if it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, the process proceeds to step S224, where the shared layer is frozen and learning is stopped, and then the process proceeds to step S230, where specific model parameters of a specific individual layer related to the specific learning task among the individual layers can be updated.

[0070] Hereinafter, the shared layer and the individual layer will be examined more specifically.

[0071] FIG. 5 shows, as an example, a decoder-shared type structure divided into (i) a shared layer including an encoder layer and a decoder layer, and (ii) an individual layer including a first learning task layer to a third learning task layer, according to an embodiment of the present invention.

[0072] Specifically, the decoder - sharing type deep learning model includes, as a shared layer, an encoder layer 310 into which image data 300 obtained through at least one camera mounted on an autonomous vehicle is input, and a decoder layer 320 into which encoder output data output from the encoder layer 310 is input. As individual layers, it includes a first learning task layer 510 including a first task header 511 corresponding to the first learning task into which decoder output data output from the decoder layer is input, and a first output function 512 for applying an output function operation to the first task header output data output from the first task header 511; a second learning task layer 520 including a second task header 521 corresponding to the second learning task into which the decoder output data output from the decoder layer 320 is input, and a second output function 522 for applying an output function operation to the second task header output data output from the second task header 521; and a third learning task layer 530 including a third task header 531 corresponding to the third learning task into which the decoder output data output from the decoder layer 320 is input, and a specific softmax function 532 for applying a classification operation to the third task header output data output from the third task header 531, and has a structure including these layers.

[0073] Also, FIG. 6 shows, as an example, a decoder - independent structure including, as a shared layer, an encoder layer and, as individual layers, an nth learning task layer (where n is an integer from 1 to 3) each including an nth decoder according to an embodiment of the present invention.

[0074] Specifically, the decoder-independent deep learning model includes, as a shared layer, an encoder layer 310 into which image data 300 acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first learning task layer 610 corresponding to the first learning task, including a first decoder 611 into which encoder output data output from the encoder layer is input and a first output function 612 for applying an output function operation to the first decoder output data output from the first decoder 611; a second learning task layer 620 corresponding to the second learning task, including a second decoder 621 into which the encoder output data output from the encoder layer is input and a second output function 622 for applying an output function operation to the second decoder output data output from the second decoder 621; and a third learning task layer 630 corresponding to the third learning task, including a third decoder 631 into which the encoder output data output from the encoder layer is input and a specific softmax function 632 for applying a classification operation to the third decoder output data output from the third decoder 631, and has a structure including these layers.

[0075] Here, the first output function and the second output function of the decoder-independent deep learning model and the decoder-shared deep learning model may be, but are not limited to, sigmoid functions.

[0076] Next, in step S230, when the model parameters are updated, the computing device 100 proceeds to step S241. Based on the specific current step of the specific learning task, after evaluating the specific current performance of the specific learning task, it proceeds to step S242. By referring to the ratio of the specific current performance to the specific maximum performance of the specific learning task, it calculates a specific performance achievement rate for the specific learning task. By referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), after calculating the total learning progress rate (the performance average value obtained by dividing the sum of the first learning performance achievement rate to the third learning performance achievement rate by 3), it proceeds to step S243. If the sum of the first current step number to the third current step number in the state where 1 is added to the specific current step of the specific learning task (the value obtained by summing up all the step numbers in all the epochs accumulated so far) is less than the total step number, it can assist in performing the steps after step S213 again with 1 added to the step number of the specific current step.

[0077] Here, the total learning progress rate can be expressed as: the total learning progress rate = (the first performance achievement rate + the second performance achievement rate + the third performance achievement rate) / 3.

[0078] On the other hand, the learning process of the deep learning model from step S211 to step S243 is illustrated assuming the absence of integrated learning data. However, if integrated learning data exists, in addition to the first learning task to the third learning task, a fourth learning task that simultaneously learns the first learning task to the third learning task can be further set, and a process similar to steps S211 to S243 can be performed. That is, since all processes proceed through the same process except that the three learning tasks change to four learning tasks, the description thereof is omitted.

[0079] Next, the post - processing process will be described with reference to FIGS. 3, 7, and 8.

[0080] Describing from after the post - processing 340 in FIG. 3, when obtaining the lane candidate group data 321, the structure line candidate group data 322, and the semantic segmentation 323 from the deep learning model, line components are extracted from each of the lane candidate group data 321 and the structure line candidate group data 322, and this is fused with the result of the semantic segmentation 323, and the adjustment information 341 which is the result of post - processing to obtain the line information and / or surface information of each of the lane candidate group data 321 and the structure line candidate group data 322 is matched with the HD map 350, and the position and orientation of the autonomous driving vehicle can be estimated.

[0081] Hereinafter, with reference to FIGS. 7 and 8, exemplarily, the process of matching the result of the lane estimation task and the result of the semantic segmentation task to obtain the adjustment information 341, and performing post - processing using the adjustment information 341 and the HD map 350 will be described. Since the process of matching the result of the road structure line estimation task and the result of the semantic segmentation task to perform post - processing is similar, the description thereof will be omitted.

[0082] First, FIG. 7 is a diagram showing, as an example, the process of labeling data related to lanes for each channel, removing noise, and then applying a moving average filter to obtain prediction information related to lanes according to an embodiment of the present invention.

[0083] Specifically, when the lane candidate group data 321 is input from the deep learning model, the computing device 100 labels 710, 720 for each channel of the lane candidate group data 321, selects and retains the largest specific label blob 710, removes the remaining label blobs 720, and based on the y-axis on the screen of the lane candidate group data 321, after obtaining the minimum x-axis position 711 and the maximum x-axis position 713 corresponding to each of the y-axis values of the specific label blob 710, extracts the coordinates 712 of a plurality of blob center points as the center values of the minimum x-axis position 711 and the maximum x-axis position 713 respectively, and applies a moving average filter to the coordinates 712 of the blob center points to determine a point determined as prediction information regarding the lane.

[0084] Here, to explain the process of applying the moving average filter in more detail, in order to apply the moving average filter, among a plurality of blob center points 712, four center points 712a, 712b, 712c, 712d are selected, and among the four center points 712a to 712d, one can be set at a point 714 determined as prediction information regarding the lane at a position located in the middle of the two end points 712a, 712d. Similar to the above content, the moving average filter can be applied multiple times in pixel units of the four points, and a plurality of points 714 determined as prediction information regarding the lane can be set. The plurality of points 714 determined as prediction information regarding the lane selected in this way can be sampled at a value set by the user, for example, eight points, and finally determined as prediction information 715 regarding the lane.

[0085] In the process of explaining the moving average filter of FIG. 7, four center points 712a to 712d among the plurality of blob center points 712 are selected, and a portion located at the center of the two center points 712a and 712d located at both ends among the four center points 712a to 712d is set as a point 714 determined as prediction information regarding the lane, and the process of extracting the prediction information 715 regarding the lane is explained. However, the method of applying the moving average filter is not necessarily limited to selecting four points, and the user can proceed by setting different numbers of selected blob center points 712.

[0086] Similarly, in FIG. 7, the content of sampling eight points 714 determined as prediction information regarding the lane to determine prediction information 715 regarding the lane is illustrated and explained. However, the present invention is not limited to this, and it is also possible to sample M points 714 determined as prediction information regarding the lane to determine the prediction information 715 regarding the lane.

[0087] In the content of FIG. 7, the process of extracting the point components of the lane is specifically illustrated and explained. However, the process of extracting the point components is not limited to the lane, and it goes without saying that the point information of the line estimation data of structures related to the road such as stop lines, speed bumps included in the lane estimation data such as the lane, as well as curbs and utility poles can be extracted in the same manner.

[0088] Next, in order to explain the process in which the prediction information regarding the lane and the semantic segmentation are fused, reference will be made to FIG. 8 for explanation.

[0089] FIG. 8 is a diagram showing, as an example, a process of applying prediction information regarding a lane (for example, the result obtained in FIG. 7) to the result of semantic segmentation according to an embodiment of the present invention to determine an exposed lane, restoring a region for an unexposed lane with reference to the exposed lane, and determining adjustment information regarding the lane.

[0090] When, among the prediction information 715 regarding the lane and the result 323 of the semantic segmentation in FIG. 3, the result 323a of the semantic segmentation for a specific lane is acquired, the computing device 100 identifies a specific overlapping portion between the prediction information 715 regarding the lane and the result 323a of the semantic segmentation for the specific lane as the left boundary line 811 and the right boundary line 813 of the specific lane, determines the coordinates of the overlapping center point between the left boundary line 811 and the right boundary line 813 as the center point 812 of the exposed lane, and can restore the non-exposed lane (a region that exists in the prediction information 715 regarding the lane but is not visible in the result 323a of the semantic segmentation for the specific lane, which is considered a non-exposed lane region where there is no lane or the lane is hidden by a structure or the like) using the lane center point 812.

[0091] For example, the center point 812 of the previously predicted past exposed lane can be applied as the current lane center point 812 to restore the region for the non-exposed lane, and the adjustment information 341 regarding the lane can be calculated, and the user can determine whether to use the information for the non-exposed lane.

[0092] Next, when the adjustment information 341 regarding the lane is calculated, the computing device 100 performs matching with reference to the adjustment information 341 regarding the lane and the HD map 350, whereby the position and orientation of the autonomous driving vehicle can be estimated 360.

[0093] As described above, the embodiments according to the present invention can be realized in the form of program instructions that can be executed through various components of a computer, and can be recorded on a computer-readable recording medium. The computer-readable recording medium can include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be those known and usable to those skilled in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes created by compilers, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

[0094] As described above, the present invention has been described with specific matters such as specific components, limited embodiments, and drawings, but this is only provided to assist in a more general understanding of the present invention, and the present invention is not limited to the above embodiments. Those having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and deformations from such descriptions.

[0095] Therefore, the idea of the present invention should not be defined only by the described embodiments, and not only the following claims, but also all those equivalently or equivalently deformed to the claims belong to the scope of the idea of the present invention.

Claims

1. In a method of training a deep learning model for extracting road features based on multitasking, (a) a computing device sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road, sets initial selection probability values for each of the first to third learning tasks, sets first to third learning step numbers required to complete one epoch for each of the first to third learning tasks, and determines a total number of steps included in all epochs with reference to the first to third learning step numbers; (b) when it is determined that the total learning progress rate is equal to or lower than a first threshold progress rate or equal to or lower than a first threshold number of steps, the computing device randomly selects a specific learning task to be learned from among the first to third learning tasks with reference to the initial selection probability values for each of the first to third learning tasks, and calculates a loss for the specific learning task; and (c) the computing device updates model parameters related to the specific learning task with reference to the loss value of the specific learning task. A method comprising the above.

2. In the step (b), when it is determined that the total learning progress rate exceeds the first threshold progress rate or exceeds the first threshold number of steps, the computing device calculates a specific performance achievement rate for the specific learning task with reference to a ratio of the current performance of the specific learning task to the highest performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task with reference to the specific performance achievement rate, calculates a specific learning selection probability adjustment value with reference to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task is selected from among the first to third learning tasks as the specific learning selection probability adjustment value. The method according to claim 1.

3. In the step (b), The specific learning weight is represented by a value obtained by subtracting, from 1, the ratio of the specific learning weight to the sum of the first performance achievement rate to the third performance achievement rate for each of the first learning task to the third learning task, and the specific learning selection probability adjustment value is represented by a value obtained by multiplying the ratio of the specific number of learning steps of the specific learning task to the value obtained by summing all of the first number of learning steps to the third number of learning steps by the specific learning weight. The method according to claim 2, characterized in that.

4. (d) When the model parameters are updated, the computing device evaluates the specific current performance of the specific learning task based on the specific current step of the specific learning task, and calculates a specific performance achievement rate for the specific learning task with reference to the ratio of the specific current performance to the specific maximum performance of the specific learning task. A first learning performance achievement rate to a third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate) is referred to, and a total learning progress rate is calculated. When the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, the steps after the step (b) are performed again with 1 added to the step number of the specific current step. A step of assisting; The method according to claim 1, further comprising the following.

5. The deep learning model includes, as a shared layer, an encoder layer into which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer into which encoder output data output from the encoder layer is input. As individual layers, a first task header corresponding to the first learning task into which decoder output data output from the decoder layer is input, and a first output function for applying an output function operation to the first task header output data output from the first task header, a first learning task layer including a second task header corresponding to the second learning task into which the decoder output data output from the decoder layer is input, and a second output function for applying an output function operation to the second task header output data output from the second task header, a second learning task layer, and a third task header corresponding to the third learning task into which the decoder output data output from the decoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third task header output data output from the third task header. The method according to claim 1, characterized in that it comprises the above.

6. In the step (c), When it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or equal to or less than a second threshold number of steps, the computing device causes the shared layer and the individual layers included in the deep learning model to be learned. When it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, the specific model parameters of the specific individual layer related to the specific learning task among the individual layers are updated in a state where the shared layer is frozen and learning is stopped. The method according to claim 5, characterized in that it comprises the above.

7. The deep learning model includes, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first decoder corresponding to the first learning task and to which encoder output data output from the encoder layer is input, and a first learning task layer including a first output function for applying an output function operation to the first decoder output data output from the first decoder, a second decoder corresponding to the second learning task and to which the encoder output data output from the encoder layer is input, and a second learning task layer including a second output function for applying an output function operation to the second decoder output data output from the second decoder, and a third learning task layer corresponding to the third learning task and including a specific softmax function for applying a classification operation to the encoder output data output from the encoder layer and the third decoder output data output from the third decoder. The method according to claim 1, characterized in that it includes the above.

8. In the step (c), When it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or equal to or less than a second threshold number of steps, the computing device causes the shared layer and the individual layers included in the deep learning model to be learned. When it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, with the shared layer frozen and learning stopped, specific model parameters of a specific individual layer related to the specific learning task among the individual layers are updated. The method according to claim 7, characterized in that it is as described above.

9. (e)When the lane candidate group data corresponding to the first learning task output from the deep learning model is input, the computing device labels the lane candidate group data for each channel, selects the largest specific label blob (bBlob), removes the remaining label blobs, obtains the coordinates of the blob center point for the specific label blob, and then applies a moving average filter to the coordinates of the blob center point to determine prediction information regarding the lane; and (f)The computing device performs matching with reference to the prediction information regarding the lane and the HD map (map), thereby estimating the position and orientation of the autonomous driving vehicle; The method according to claim 1, further comprising.

10. After the step (e), When the prediction information regarding the lane and the information regarding the semantic segmentation are acquired, the computing device distinguishes a specific overlapping portion between the prediction information regarding the lane and the information regarding the semantic segmentation as the left and right boundary lines of a specific lane, determines the coordinates of the overlapping center point between the left boundary line and the right boundary line as the center point of the exposed lane, restores the area for the non-exposed lane with reference to the left and right boundary lines of the exposed lane, and further includes a step of determining adjustment information regarding the lane, In the step (f), the computing device performs matching with reference to the adjustment information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous driving vehicle. The method according to claim 9, characterized in that.

11. In the step (a), The computing device further sets a fourth learning task for simultaneous learning of the first learning task to the third learning task, further sets an initial selection probability value for the fourth learning task, further sets the number of fourth learning steps required to complete one epoch for the fourth learning task, and determines the total number of steps included in the entire epoch with reference to the number of the first learning steps to the number of the fourth learning steps, In the step (b), When it is determined that the total learning progress rate is less than or equal to the first threshold progress rate or less than or equal to the first threshold number of steps, the computing device refers to the initial selection probability values for each of the first learning task to the fourth learning task, and randomly selects the specific learning task for which learning is to be performed from among the first learning task to the fourth learning task, and calculates the loss for the specific learning task. The method according to claim 1, characterized in that.

12. (g) When the model parameters are updated, the computing device evaluates the specific current performance of the specific learning task based on the specific current step of the specific learning task, and refers to the ratio of the specific current performance to the specific maximum learning performance of the specific learning task, calculates a specific performance achievement rate for the specific learning task, refers to the first learning performance achievement rate to the fourth learning performance achievement rate (any one of the first learning performance achievement rate to the fourth learning performance achievement rate is the specific performance achievement rate), calculates the total learning progress rate, and when the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, assist to perform the steps after the step (b) again with 1 added to the step number of the specific current step; The method according to claim 11, further comprising the above.

13. In a computing device that learns a deep learning model for extracting road features based on multi-tasks, At least one memory for storing instructions; and Including at least one processor configured to execute the instructions, The processor sets (I) a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures related to the road, and a third learning task for semantic segmentation of the lanes and the structures of the road, sets initial selection probability values for each of the first to third learning tasks, sets the number of first to third learning steps required to complete one epoch for each of the first to third learning tasks, and determines the total number of steps included in all epochs with reference to the number of first to third learning steps; (II) when it is determined that the total learning progress rate is less than or equal to a first threshold progress rate or less than or equal to a first threshold number of steps, randomly selects a specific learning task to perform learning from among the first to third learning tasks with reference to the initial selection probability values for each of the first to third learning tasks, and calculates the loss for the specific learning task; and (III) updates the model parameters related to the specific learning task with reference to the loss value of the specific learning task; a computing device that performs the processes.

14. The processor is In the process of (II), when it is determined that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, calculates a specific performance achievement rate for the specific learning task with reference to the ratio of the specific current performance of the specific learning task to the specific maximum performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task with reference to the specific performance achievement rate, calculates a specific learning selection probability adjustment value with reference to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task is selected from among the first to third learning tasks as the specific learning selection probability adjustment value. The computing device according to claim 13, which performs the above.

15. In the process of (II), The specific learning weight is represented by a value obtained by subtracting, from 1, the ratio of the specific learning weight to the sum of the first performance achievement rate to the third performance achievement rate for each of the first learning task to the third learning task, and the specific learning selection probability adjustment value is obtained by multiplying the ratio of the specific number of learning steps of the specific learning task to the value obtained by adding up all of the first number of learning steps to the third number of learning steps by the specific learning weight. The computing device according to claim 14, wherein the computing device performs the above.

16. The processor (IV)When the model parameters are updated, an evaluation is made of the specific current performance of the specific learning task with reference to the specific current step of the specific learning task, and a specific performance achievement rate for the specific learning task is calculated by referring to the ratio of the specific current performance to the specific maximum performance of the specific learning task. A total learning progress rate is calculated by referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate). When the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, the process after the process (II) is performed again with 1 added to the number of steps of the specific current step. The computing device according to claim 13, further performing the process of assisting.

17. The deep learning model includes, as a shared layer, an encoder layer into which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer into which encoder output data output from the encoder layer is input. As individual layers, a first task header corresponding to the first learning task into which decoder output data output from the decoder layer is input, and a first output function for applying an output function operation to the first task header output data output from the first task header, a first learning task layer including a second task header corresponding to the second learning task into which the decoder output data output from the decoder layer is input, and a second output function for applying an output function operation to the second task header output data output from the second task header, a second learning task layer, and a third task header corresponding to the third learning task into which the decoder output data output from the decoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third task header output data output from the third task header. The computing device according to claim 13 performs the above.

18. The processor In the (III) process, When it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or equal to or less than a second threshold number of steps, the shared layer and the individual layers included in the deep learning model are learned. When it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, with the shared layer frozen and learning stopped, specific model parameters of specific individual layers related to the specific learning task among the individual layers are updated. The computing device according to claim 17 performs the above.

19. The deep learning model includes, as a shared layer, an encoder layer into which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first decoder corresponding to the first learning task, into which encoder output data output from the encoder layer is input, and a first learning task layer including a first output function for applying an output function operation to the first decoder output data output from the first decoder; a second decoder corresponding to the second learning task, into which the encoder output data output from the encoder layer is input, and a second learning task layer including a second output function for applying an output function operation to the second decoder output data output from the second decoder; and a third decoder corresponding to the third learning task, into which the encoder output data output from the encoder layer is input, and a third learning task layer including a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder. The computing device according to claim 13 performs the above operations.

20. The processor In the (III) process, When it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or equal to or less than a second threshold number of steps, the shared layer and the individual layers included in the deep learning model are learned. When it is determined that the total learning progress rate exceeds the second threshold progress rate or exceeds the second threshold number of steps, the specific model parameters of the specific individual layer related to the specific learning task among the individual layers are updated in a state where the shared layer is frozen and learning is stopped. The computing device according to claim 19 performs the above operations.

21. The processor When the lane candidate group data corresponding to the first learning task output from the deep learning model is input, the lane candidate group data is labeled for each channel, the largest specific label blob is selected, and the remaining label blobs are removed. After obtaining the coordinates of the blob center point for the specific label blob, a moving average filter is applied to the coordinates of the blob center point to determine prediction information regarding the lane; and (VI) a process of performing matching with reference to the prediction information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous driving vehicle; are further performed. The computing device according to claim 13.

22. The processor is After the process (V), (VI) When the prediction information regarding the lane and the information regarding the semantic segmentation are acquired, a specific overlapping portion between the prediction information regarding the lane and the information regarding the semantic segmentation is distinguished as the left and right boundary lines of a specific lane, the coordinates of the overlapping center point between the left boundary line and the right boundary line are determined as the center point of the exposed lane, and with reference to the left and right boundary lines of the exposed lane, the area for the non-exposed lane is restored, and a process of determining adjustment information regarding the lane is further performed. In the process (VI), matching is performed with reference to the adjustment information regarding the lane and the HD map, thereby estimating the position and orientation of the autonomous driving vehicle. The computing device according to claim 21.

23. The processor In the process (I), A fourth learning task for simultaneous learning of the first learning task to the third learning task is further set, an initial selection probability value for the fourth learning task is further set, the number of fourth learning steps required to complete one epoch for the fourth learning task is further set, and with reference to the number of first learning steps to the number of fourth learning steps, the total number of steps included in the entire epoch is determined. In the (II) process, when it is determined that the total learning progress rate is less than or equal to the first threshold progress rate or less than or equal to the first threshold step number, the processor refers to the initial selection probability values for each of the first learning task to the fourth learning task, randomly selects the specific learning task to perform learning from among the first learning task to the fourth learning task, and calculates the loss for the specific learning task. The computing device according to claim 13.

24. The processor (VII) When the model parameters are updated, based on the specific current step of the specific learning task, evaluate the specific current learning performance of the specific learning task, and refer to the ratio of the specific current learning performance to the specific maximum learning performance of the specific learning task to calculate the specific performance achievement rate for the specific learning task. Refer to the first learning performance achievement rate to the fourth learning performance achievement rate (any one of the first learning performance achievement rate to the fourth learning performance achievement rate is the specific performance achievement rate) to calculate the total learning progress rate. When the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, assist to perform the process after the (II) process again with 1 added to the step number of the specific current step. The computing device according to claim 23, further performing the process.

Images