A training method and learning device for updating a deep learning-based object detector of an autonomous vehicle that enables adaptation to a driving environment, and an updating method and updating device using the method

Abstract

Provided is a training method for updating a deep learning-based object detector of an autonomous vehicle to adapt to a driving environment, the method comprising (a)(i) inputting a training image corresponding to the driving environment into an environment-specific object detector to cause the kth environment-specific object detector to (i-1) generate an environment-specific feature map through convolution, (i-2) generate an environment-specific pooling feature map through ROI pooling, and (i-3) generate environment-specific object detection information by applying a full connection operation to the kth environment-specific pooling feature map, (ii) inputting the environment-specific feature map into an environment-specific ranking network, (ii-1) generating an environment-specific segmentation map through an environment-specific deconvolution layer, and (ii-2) generating an environment-specific ranking score through an environment-specific discriminator, and (b) training the environment-specific object detector, training the environment-specific deconvolution layer, and training the environment-specific discriminator.

Description

[0001] Related cross-references

[0002] This application claims priority to U.S. Patent Application No. 63 / 024,657, filed May 14, 2020, and U.S. Patent Application No. 17 / 229,264, filed April 13, 2021, the disclosures of which are incorporated herein by reference. Technical Field

[0003] This invention relates to a deep learning-based object detector for updating autonomous vehicles, specifically to a training method and learning device for updating a deep learning-based object detector for autonomous vehicles to adapt to various driving environments, and an updating method and updating device utilizing this method. Background Technology

[0004] Deep Convolutional Neural Networks (Deep CNNs) are the core of the rapid development of deep learning. CNNs were used to solve character recognition problems in the 1990s, but their current widespread application is due to the research results in recent years. This Deep CNN defeated other competitors in the ImageNet image classification competition in 2012. Since then, convolutional neural networks have become a very useful tool in the field of machine learning.

[0005] This type of convolutional neural network is currently used in various fields; for example, it can be used to implement object detectors installed on self-driving cars.

[0006] However, the driving environment of autonomous vehicles varies with region, time, weather, etc., and high-configuration computing resources are required to realize an object detector that optimizes various driving environments that autonomous vehicles may encounter.

[0007] However, autonomous vehicles typically have limited computing resources, making it nearly impossible to install a single object detector optimized for various driving environments on them.

[0008] Therefore, an improved solution is needed to address the above problems. Summary of the Invention

[0009] Technical issues

[0010] The purpose of this invention is to solve all the problems mentioned above.

[0011] Another objective of this invention is to enable autonomous vehicles to drive smoothly in various driving environments.

[0012] Another object of the present invention is to update the object detector of autonomous vehicles to be the best object detector suitable for every driving environment.

[0013] Technical solution

[0014] The objective of this invention is achieved through the following technical solution:

[0015] According to one aspect of the present invention, a training method for updating a deep learning-based object detector for an autonomous vehicle to adapt to a driving environment is provided, comprising the following steps: (a) when at least one k-th training image from a first training image to an n-th training image is acquired, the training device performs the following process, wherein the first training image corresponds to a first driving environment and includes a first object label and a first segmentation label, the n-th training image corresponds to an n-th driving environment and includes an n-th object label and an n-th segmentation label, and k is an integer greater than or equal to 1 and less than or equal to n; (i) inputting the k-th training image into the training environment corresponding to the first driving environment. The first object detector of the driving environment is selected from the nth object detectors corresponding to the nth driving environment to the kth environment-specific object detectors corresponding to the kth driving environment. The kth environment-specific object detector (i-1) applies at least one convolution operation to the kth training image through at least one kth environment-specific convolutional layer of the kth environment-specific object detector to generate a kth environment-specific feature map. (i-2) The kth environment-specific pooling layer of the kth environment-specific object detector applies at least one ROI pooling operation to the kth environment-specific feature map using kth environment-specific ROI information to generate a kth environment-specific pooling feature map. Wherein, the k-th environment-specific ROI information corresponds to at least one region of the predicted at least one object present in the k-th environment-specific feature map, (i-3) the k-th environment-specific pooling feature map is subjected to at least one fully connected operation through the k-th environment-specific fully connected layer of the k-th environment-specific object detector to generate the k-th environment-specific object detection information, (ii) the k-th environment-specific feature map is input to the k-th environment-specific ranking network corresponding to the k-th environment-specific object detector in the first ranking network corresponding to the first object detector to the n-th ranking network corresponding to the n-th object detector, so that the k-th environment-specific ROI information corresponds to the k-th environment-specific object detector. The environment-specific ranking network (ii-1) applies at least one deconvolution operation to the k-th environment-specific feature map through at least one k-th environment-specific deconvolution layer of the k-th environment-specific ranking network to generate a k-th environment-specific segmentation map. (ii-2) The k-th environment-specific ranking score is generated by the k-th environment-specific discriminator of the k-th environment-specific ranking network, wherein the k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false. The k-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and the first sub-region on the corresponding k-th environment-specific segmentation map.(b) The learning device performs the following processes: (i) training the k-th environment-specific object detector to minimize the k-th environment-specific object detection loss, and training the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer to maximize the k-th environment-specific ranking score, wherein the k-th environment-specific object detection loss is generated by referencing the k-th environment-specific object detection information and its corresponding k-th environment-specific object annotation; (ii) training the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score associated with the k-th environment-specific annotation ranking map and minimize the k-th environment-specific ranking score, wherein the k-th environment-specific annotation ranking map is generated by cropping and concatenating the main region on the k-th training image and the second sub-region on its corresponding k-th environment-specific segmentation annotation, and the k-th environment-specific annotation ranking score is generated by the k-th environment-specific discriminator.

[0016] In step (a), the learning device causes the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, wherein the j-th environment-specific ranking score represents information about whether at least one j-th environment-specific ranking map is true or false, the j-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and at least one third sub-region on the corresponding at least one j-th environment-specific segmentation map, the j-th environment-specific segmentation map is generated by applying the convolution operation to the k-th training image through at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer to generate at least one j-th environment-specific feature map, and then applying the deconvolution operation to the j-th environment-specific feature map through at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer, where j is an integer greater than or equal to 1 and less than or equal to n. In step (b), the learning device trains the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively.

[0017] In step (b), the learning device trains the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer respectively to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and its corresponding k-th environment-specific segmentation annotation.

[0018] The k-th environment-specific discriminator includes the convolutional PatchGAN classifier.

[0019] In step (a), the learning device enables the k-th environment-specific object detector to generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector. In step (b), the learning device further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and its corresponding k-th environment-specific object annotation.

[0020] According to another aspect of the present invention, a method is provided for updating a deep learning-based object detector for an autonomous vehicle to adapt to a driving environment, comprising the following steps: (a) when at least one k-th training image from a first training image to an n-th training image is acquired, a training device performs the following process, wherein the first training image corresponds to the first driving environment and includes the first object label and the first segmentation label, the n-th training image corresponds to the n-th driving environment and includes the n-th object label and the n-th segmentation label, and k is an integer greater than or equal to 1 and less than or equal to n; (i) inputting the k-th training image to the first object detector corresponding to the first driving environment to the deep learning-based object detector corresponding to the n-th driving environment. In the nth object detector of the nth driving environment, the kth environment-specific object detector corresponding to the kth driving environment is used to generate a kth environment-specific feature map by applying at least one convolution operation to the kth training image through at least one kth environment-specific convolutional layer of the kth environment-specific object detector (i-1) and applying at least one ROI pooling operation to the kth environment-specific feature map through the kth environment-specific pooling layer of the kth environment-specific object detector using the kth environment-specific ROI information. The kth environment-specific ROI information and the prediction include at least one ROI information present in the kth environment-specific feature map. For at least one region of an object, (i-3) the k-th environment-specific pooling feature map is subjected to at least one fully connected operation through the k-th environment-specific fully connected layer of the k-th environment-specific object detector to generate k-th environment-specific object detection information. (ii) The k-th environment-specific feature map is input to the k-th environment-specific ranking network corresponding to the k-th environment-specific object detector, from the first ranking network corresponding to the first object detector to the n-th ranking network corresponding to the n-th object detector, so that the k-th environment-specific ranking network (ii-1) applies at least one deconvolution layer to the k-th environment-specific feature map through at least one k-th environment-specific deconvolution layer of the k-th environment-specific ranking network. (ii-2) The k-th environment-specific segmentation map is generated by performing a product operation. The k-th environment-specific discriminator of the k-th environment-specific ranking network generates a k-th environment-specific ranking score, where the k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false. The k-th environment-specific ranking map is generated by plucking and concatenating the main region on the k-th training image and the corresponding first sub-region on the k-th environment-specific segmentation map. (iii) The k-th environment-specific object detector is trained to minimize the k-th environment-specific object detection loss, and the k-th environment-specific deconvolutional layer and the k-th environment-specific convolutional layer are trained to maximize the k-th environment-specific ranking score.The k-th environment-specific object detection loss is generated by referring to the k-th environment-specific object detection information and its corresponding k-th environment-specific object annotation. (iv) The k-th environment-specific discriminator is trained to maximize the k-th environment-specific annotation ranking score related to the k-th environment-specific annotation ranking map and minimize the k-th environment-specific ranking score. The k-th environment-specific annotation ranking map is generated by cropping and concatenating the main region on the k-th training image and the second sub-region on its corresponding k-th environment-specific segmentation annotation. The k-th environment-specific annotation ranking score is generated by the k-th environment-specific discriminator. In this state, more... (a) The new device acquires a driving environment image representing the driving environment of the autonomous vehicle; (b) The updating device (i) inputs the driving environment image to the first object detector to the nth object detector, respectively, so that the first object detector to the nth object detector applies at least one convolution operation to the driving environment image through the first convolutional layer to the nth convolutional layer, thereby generating a first driving environment feature map to the nth driving environment feature map respectively; (ii) inputs the first driving environment feature map to the nth driving environment feature map to the first sorting network to the nth sorting network, respectively, so that the first sorting network to the nth sorting network... (ii-1) At least one deconvolution operation is applied to each of the first to nth driving environment feature maps through the first to nth deconvolution layers to generate the first to nth driving environment segmentation maps respectively. (ii-2) A first to nth driving environment ranking score is generated through each of the first to nth discriminators, wherein the first to nth driving environment ranking scores respectively represent information about whether the first to nth driving environment ranking maps are true or false. The first driving environment ranking map is obtained by using the first driving environment feature map on the driving environment image. The nth driving environment ranking map is generated by cropping and connecting the nth main driving region on the driving environment image and the nth driving sub-region on the corresponding nth driving environment segmentation map; and (c) the updating device (i) selects a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score, the specific driving environment ranking score having the largest value among the first driving environment ranking score to the nth driving environment ranking score, and (ii) updates the current object detector of the autonomous vehicle to the specific object detector.

[0021] In step (a), the learning device causes the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, and trains the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively. The j-th environment-specific ranking score represents information about whether at least one j-th environment-specific ranking map is true or false. The j-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and at least one third sub-region on the corresponding at least one j-th environment-specific segmentation map. The j-th environment-specific segmentation map is generated by applying the convolution operation to the k-th training image using at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer to generate at least one j-th environment-specific feature map, and then applying the deconvolution operation to the j-th environment-specific feature map using at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer. Here, j is an integer greater than or equal to 1 and less than or equal to n.

[0022] In step (a), the learning device trains the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer respectively to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and its corresponding k-th environment-specific segmentation annotation.

[0023] The k-th environment-specific discriminator includes the convolutional PatchGAN classifier.

[0024] In step (a), the learning device enables the k-th environment-specific object detector to generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector, and further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and the corresponding k-th environment-specific object annotation.

[0025] According to another aspect of the present invention, a learning device for updating a deep learning-based object detector for an autonomous vehicle to adapt to a driving environment is provided, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein the processor performs the following process: (I) when at least one k-th training image from the first to the n-th training images is acquired, the following process is performed, wherein the first training image corresponds to the first driving environment and includes the first object label and the first segmentation label, and the n-th training image corresponds to the n-th driving environment and includes the n-th object label and the n-th segmentation label, wherein... k is an integer greater than or equal to 1 and less than or equal to n. (i) The k-th training image is input into the k-th environment-specific object detector corresponding to the k-th driving environment, from the first object detector to the n-th object detector corresponding to the first driving environment, so that the k-th environment-specific object detector (i-1) applies at least one convolution operation to the k-th training image through at least one k-th environment-specific convolutional layer of the k-th environment-specific object detector, thereby generating a k-th environment-specific feature map. (i-2) The k-th environment-specific feature map is processed by the k-th environment-specific pooling layer of the k-th environment-specific object detector using the k-th environment-specific ROI information. At least one ROI pooling operation is applied to generate a k-th environment-specific pooling feature map, wherein the k-th environment-specific ROI information corresponds to at least one region of the predicted at least one object present in the k-th environment-specific feature map. (i-3) At least one fully connected operation is applied to the k-th environment-specific pooling feature map through the k-th environment-specific fully connected layer of the k-th environment-specific object detector to generate k-th environment-specific object detection information. (ii) The k-th environment-specific feature map is input to the first to the nth ranking networks corresponding to the k-th object detector, from the first to the nth ranking networks corresponding to the k-th object detector. The k-th environment-specific ranking network (ii-1) applies at least one deconvolution operation to the k-th environment-specific feature map through at least one k-th environment-specific deconvolution layer of the k-th environment-specific ranking network to generate a k-th environment-specific segmentation map, and (ii-2) generates a k-th environment-specific ranking score through the k-th environment-specific discriminator of the k-th environment-specific ranking network, wherein the k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false, and the k-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and the corresponding first sub-region on the k-th environment-specific segmentation map;And (II) (i) train the k-th environment-specific object detector to minimize the k-th environment-specific object detection loss, and train the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer to maximize the k-th environment-specific ranking score, wherein the k-th environment-specific object detection loss is generated by referring to the k-th environment-specific object detection information and its corresponding k-th environment-specific object annotation; (ii) train the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score related to the k-th environment-specific annotation ranking map and minimize the k-th environment-specific ranking score, wherein the k-th environment-specific annotation ranking map is generated by cropping and concatenating the main region on the k-th training image and the second sub-region on its corresponding k-th environment-specific segmentation annotation, and the k-th environment-specific annotation ranking score is generated by the k-th environment-specific discriminator.

[0026] In process (I), the processor causes the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, wherein the j-th environment-specific ranking score represents information about whether at least one j-th environment-specific ranking map is true or false, the j-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and at least one third sub-region on the corresponding at least one j-th environment-specific segmentation map, the j-th environment-specific segmentation map is generated by applying the convolution operation to the k-th training image through at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer to generate at least one j-th environment-specific feature map, and then applying the deconvolution operation to the j-th environment-specific feature map through at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer, where j is an integer greater than or equal to 1 and less than or equal to n. In process (II), the k-th environment-specific discriminator is trained to maximize the k-th environment-specific annotation ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively.

[0027] In the process of (II), the processor trains the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer respectively to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and its corresponding k-th environment-specific segmentation annotation.

[0028] The k-th environment-specific discriminator includes the convolutional PatchGAN classifier.

[0029] In process (I), the processor causes the k-th environment-specific object detector to generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector. In process (II), the processor further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and the corresponding k-th environment-specific object label.

[0030] According to another aspect of the present invention, an updating device for updating a deep learning-based object detector for an autonomous vehicle to adapt to the driving environment is provided, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein the processor performs the following process: (I) when at least one k-th training image from the first to the nth training images is acquired, the learning device performs the following process, wherein the first training image corresponds to the first driving environment and includes the first object label and the first segmentation label, the n-th training image corresponds to the nth driving environment and includes the n-th object label and the n-th segmentation label, and k is greater than or equal to An integer less than or equal to n, (i) inputting the k-th training image into the k-th environment-specific object detector corresponding to the k-th driving environment from the first object detector to the n-th object detector corresponding to the first driving environment, so that the k-th environment-specific object detector (i-1) applies at least one convolution operation to the k-th training image through at least one k-th environment-specific convolutional layer of the k-th environment-specific object detector, thereby generating a k-th environment-specific feature map, (i-2) applying at least one ROI pooling operation to the k-th environment-specific feature map using the k-th environment-specific ROI information through the k-th environment-specific pooling layer of the k-th environment-specific object detector, thereby generating The k-th environment-specific pooling feature map, wherein the k-th environment-specific ROI information corresponds to at least one region of the predicted at least one object present in the k-th environment-specific feature map, (i-3) the k-th environment-specific pooling feature map is subjected to at least one fully connected operation through the k-th environment-specific fully connected layer of the k-th environment-specific object detector to generate k-th environment-specific object detection information, (ii) the k-th environment-specific feature map is input to the k-th environment-specific ranking network corresponding to the k-th environment-specific object detector in the first ranking network corresponding to the first object detector to the n-th ranking network corresponding to the n-th object detector, so that the k-th environment-specific ranking network (ii-1) (ii-2) At least one deconvolution operation is applied to the k-th environment-specific feature map by at least one k-th environment-specific deconvolution layer of the k-th environment-specific ranking network to generate a k-th environment-specific segmentation map. (ii-3) A k-th environment-specific ranking score is generated by the k-th environment-specific discriminator of the k-th environment-specific ranking network, wherein the k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false. The k-th environment-specific ranking map is generated by cropping and concatenating the main region on the k-th training image and the corresponding first sub-region on the k-th environment-specific segmentation map. (iii) The k-th environment-specific object detector is trained to minimize the k-th environment-specific object detection loss.The k-th environment-specific deconvolutional layer and the k-th environment-specific convolutional layer are trained to maximize the k-th environment-specific ranking score, wherein the k-th environment-specific object detection loss is generated by referring to the k-th environment-specific object detection information and its corresponding k-th environment-specific object annotation; (iv) the k-th environment-specific discriminator is trained to maximize and minimize the k-th environment-specific annotation ranking score related to the k-th environment-specific annotation ranking map, wherein the k-th environment-specific annotation ranking map is generated by cropping and concatenating the main region on the k-th training image and the second sub-region on its corresponding k-th environment-specific segmentation annotation. The k-th environment-specific label ranking score is generated by the k-th environment-specific discriminator. In this case, a driving environment image representing the driving environment of the autonomous vehicle is obtained; (II) (i) The driving environment image is input to the first object detector to the n-th object detector, so that the first object detector to the n-th object detector applies at least one convolution operation to the driving environment image through the first convolutional layer to the n-th convolutional layer, thereby generating the first driving environment feature map to the n-th driving environment feature map respectively; (ii) The first driving environment feature map to the n-th driving environment feature map is input to the first ranking network to the n-th ranking network respectively. The network is configured such that the first to the nth ranking networks (ii-1) apply at least one deconvolution operation to each of the first to the nth driving environment feature maps through the first to the nth deconvolution layers, thereby generating first to nth driving environment segmentation maps respectively; (ii-2) each of the first to the nth discriminators generates a first to nth driving environment ranking score, wherein the first to the nth driving environment ranking scores respectively represent information about whether the first to the nth driving environment ranking maps are true or false, and the first driving environment ranking map is generated by the first to the nth discriminator. The first driving main region on the driving environment image and the first driving sub-region on the corresponding first driving environment segmentation map are cropped and connected to generate the nth driving environment ranking map. The nth driving environment ranking map is generated by cropping and connecting the nth driving main region on the driving environment image and the nth driving sub-region on the corresponding nth driving environment segmentation map; and (III) (i) Select a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score, the specific driving environment ranking score having the largest value among the first driving environment ranking score to the nth driving environment ranking score, and (ii) Update the current object detector of the autonomous vehicle to the specific object detector.

[0031] In the process described in (I), the learning device enables the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, and trains the k-th environment-specific discriminator to maximize the k-th environment-specific ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively. The j-th environment-specific ranking score represents information about whether at least one j-th environment-specific ranking map is true or false. The j-th environment-specific ranking map is generated by cropping and connecting the main region on the k-th training image and at least one third sub-region on the corresponding at least one j-th environment-specific segmentation map. The j-th environment-specific segmentation map is generated by applying the convolution operation to the k-th training image through at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer to generate at least one j-th environment-specific feature map, and then applying the deconvolution operation to the j-th environment-specific feature map through at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer. The j is an integer greater than or equal to 1 and less than or equal to n.

[0032] In the process of (I), the learning device trains the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer respectively to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and its corresponding k-th environment-specific segmentation annotation.

[0033] The k-th environment-specific discriminator includes the convolutional PatchGAN classifier.

[0034] In the process of (I), the learning device enables the k-th environment-specific object detector to generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector, and further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and the corresponding k-th environment-specific object annotation.

[0035] Beneficial effects

[0036] This invention enables autonomous vehicles to drive smoothly in various driving environments.

[0037] The present invention has the ability to update the object detector of an autonomous vehicle to the optimal object detector suitable for every driving environment. Attached Figure Description

[0038] The following drawings, which are used to illustrate the embodiments of the present invention, are only a part of the embodiments of the present invention. Those skilled in the art (hereinafter referred to as "the skill person") can obtain other drawings based on the following drawings without making creative work.

[0039] Figure 1 This is a schematic diagram of a learning device according to an embodiment of the present invention, which trains each object detector corresponding to each driving environment and each ranking network thereon.

[0040] Figure 2 and Figure 3 This is a schematic diagram illustrating the process by which a learning device according to an embodiment of the present invention trains each object detector and its corresponding ranking network to update the deep learning-based object detector of an autonomous vehicle to adapt to the driving environment.

[0041] Figure 4 This is a schematic diagram of an updating device that uses a specific object detector to update the object detector of an autonomous vehicle to adapt to a specific driving environment, according to an embodiment of the present invention.

[0042] Figure 5 and Figure 6 This is a schematic diagram illustrating the process by which an updating device according to an embodiment of the present invention updates the object detector of an autonomous vehicle to a specific object detector.

[0043] Figure 7 This is a schematic diagram illustrating a method for updating the object detector of an autonomous vehicle to a specific object detector using an updating device according to an embodiment of the present invention. Detailed Implementation

[0044] The objectives, technical solutions, and advantages of the present invention will now be described in detail with reference to the accompanying drawings, which exemplarily illustrate specific embodiments in which the present invention can be implemented. These embodiments will be described in detail below to enable those skilled in the art to fully implement the present invention.

[0045] Furthermore, throughout the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and features of the invention will become apparent to those skilled in the art through this specification, while others will become apparent through embodiments of the invention. The following examples and figures are illustrative only and are not intended to limit the scope of the invention.

[0046] Furthermore, this invention encompasses all combinations of the embodiments described in this specification. It should be understood that while the various embodiments of the invention differ, they are not mutually exclusive. For example, specific shapes, structures, and characteristics relating to one embodiment of this specification may be implemented as other embodiments without departing from the technical concept and scope of the invention. Additionally, it should be understood that the position or configuration of the various components in each disclosed embodiment may be changed without departing from the technical concept and scope of the invention. Therefore, the following detailed description is not intended to be limiting, and if properly described, the scope of the invention is defined only by the full scope of the claims and their equivalents. Similar reference numerals in the drawings refer to the same or similar functions in several respects.

[0047] The various images mentioned in this invention may include images of paved or unpaved roads, in which case objects (e.g., cars, people, animals, plants, objects, buildings, aircraft or drones, and other obstacles) may be conceived in the road environment, but are not necessarily limited thereto. The various images mentioned in this invention may also be images unrelated to roads (e.g., images of unpaved roads, alleys, open spaces, seas, lakes, rivers, mountains, forests, deserts, skies, and interiors), in which case objects (e.g., cars, people, animals, plants, objects, buildings, aircraft or drones, and other obstacles) in unpaved roads, alleys, open spaces, seas, lakes, rivers, mountains, forests, deserts, skies, and interiors may be conceived in the environment, but are not necessarily limited thereto.

[0048] The titles or abstracts of the invention provided herein are for convenience only and do not limit or interpret the scope or meaning of the embodiments.

[0049] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the present invention.

[0050] The following description uses a car as an example, but the invention is not limited thereto. The invention can be applied to any field where at least one lens detects at least one object in at least one predetermined area, such as in the military, surveillance and other fields.

[0051] Figure 1This is a schematic diagram of a learning device 1000 according to an embodiment of the present invention, which trains each object detector and each corresponding ranking network for each driving environment. The learning device 1000 may include a memory 1001 and a processor 1002. The memory 1002 stores instructions for training each object detector and each corresponding ranking network for each driving environment. The processor 1002 responds to the instructions stored in the memory 1001 by performing the operation of training each object detector and each corresponding ranking network for each driving environment.

[0052] Specifically, the learning device 1000 can typically utilize a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific manner) to achieve the required system performance. For example, the computing device may be a device that includes a computer processor, memory, storage, input devices and output devices, as well as other existing computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).

[0053] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software components that execute applications for specific purposes.

[0054] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.

[0055] The following is for reference. Figure 2 and Figure 3 This paper describes a method for training a deep learning-based object detector for updating an autonomous vehicle using a learning device 1000 configured as described above, thereby adapting it to the driving environment.

[0056] For reference, even though each of the following constituent elements is described in singular form, the possibility that it may be plural is not excluded.

[0057] For example, the learning device of the present invention can train a plurality of object detectors (e.g., n object detectors) and a plurality of corresponding ranking networks (e.g., n ranking networks). For convenience, Figure 2The k-th environment-specific object detector 1100-k and its corresponding k-th environment-specific sorting network 1200-k are shown only representatively for the k-th driving environment. Here, n can be an integer greater than or equal to 1, and k can be an integer greater than or equal to 1 and less than or equal to n.

[0058] refer to Figure 2 The k-th environment-specific object detector 1100-k may include at least a portion of the k-th environment-specific convolutional layer, the k-th environment-specific pooling layer, the k-th environment-specific RPN (region proposal network) layer, and the k-th environment-specific fully connected layer.

[0059] Additionally, the k-th environment-specific ranking network 1200-k may include a k-th environment-specific deconvolution layer and a k-th environment-specific discriminator.

[0060] At this point, each discriminator, including the k-th environment-specific discriminator, can be a classifier that performs classification calculations; for example, the classifier can be a convolutional PatchGAN classifier.

[0061] For reference, the discriminator may be a discriminator for a generative adversarial network (GAN) model.

[0062] Typically, a GAN model can include a generator and a discriminator. (i) The generator can generate fake images (e.g., segmentation maps) by transforming the input, and (ii) The discriminator can output probability values ​​that indicate whether the acquired input (e.g., the segmentation map output by the generator) is a fake image or a real image.

[0063] At this point, the generator may include convolutional layers and deconvolutional layers. Therefore, the following description will assume that the k-th environment-specific generator includes a k-th environment-specific convolutional layer and a k-th environment-specific deconvolutional layer.

[0064] First, the operation of the learning device 1000 is described in detail. At least one k-th training image can be selected from all training images. This k-th training image corresponds to the k-th driving environment and includes a k-th object label and a k-th segmentation label. In this case, all training images can include training images 1 through n. Furthermore, each training image 1 can correspond to the k-th driving environment and include a k-th object label and a k-th segmentation label, and each training image n can correspond to the n-th driving environment and include an n-th object label and an n-th segmentation label. Here, n can be an integer greater than or equal to 1, and k can be an integer greater than or equal to 1 and less than or equal to n.

[0065] For example, the first to nth driving environments can be at least a portion of urban driving environments, suburban driving environments, mountain driving environments, coastal driving environments, rainy driving environments, and snowy driving environments. However, these are merely examples of various driving environments, and the first to nth driving environments can include at least a portion of any driving environment that an autonomous vehicle can encounter.

[0066] Additionally, the learning device 1000 can input the k-th training image into the k-th environment-specific object detector 1100-k, which is one of the first object detectors corresponding to the first driving environment to the n-th object detectors corresponding to the n-th driving environment, so that the k-th environment-specific object detector 1100-k performs object detection on at least one object included in the k-th training image, thereby generating k-th environment-specific object detection information.

[0067] For example, the learning device 1000 can input the k-th training image into the k-th environment-specific object detector 1100-k, so that the k-th environment-specific object detector 1100-k (i) applies at least one convolution operation to the k-th training image through at least one k-th environment-specific convolutional layer of the k-th environment-specific object detector 1100-k, thereby generating a k-th environment-specific feature map; (ii) applies ROI pooling operation to the k-th environment-specific feature map using the k-th environment-specific ROI information through the k-th environment-specific pooling layer of the k-th environment-specific object detector 1100-k, thereby generating a k-th environment-specific pooling feature map, wherein the k-th environment-specific ROI information corresponds to at least one region that predicts at least one object present in the k-th environment-specific feature map; and (iii) applies at least one fully connected operation to the k-th environment-specific pooling feature map through the k-th environment-specific fully connected layer of the k-th environment-specific object detector 1100-k, thereby generating k-th environment-specific object detection information.

[0068] At this time, the learning device 1000 can enable the k-th environment-specific object detector 1100-k to generate the k-th environment-specific ROI information through the k-th environment-specific RPN (region proposal network) layer of the k-th environment-specific object detector 1100-k.

[0069] Additionally, the learning device 1000 can input the k-th environment-specific feature map into the k-th environment-specific ranking network 1200-k, which corresponds to the k-th environment-specific object detector 1100-k, from the first ranking network corresponding to the first object detector to the n-th ranking network corresponding to the n-th object detector. This allows the k-th environment-specific ranking network 1200-k to apply at least one deconvolution operation to the k-th environment-specific feature map through at least one k-th environment-specific deconvolution layer, thereby generating the k-th environment-specific segmentation map.

[0070] For reference, the k-th environment-specific generator may include a k-th environment-specific convolutional layer and a k-th environment-specific deconvolutional layer. In this case, the k-th environment-specific generator corresponding to the k-th environment-specific discriminator may share the k-th environment-specific convolutional layer with the k-th environment-specific object detector 1100-k.

[0071] That is, the k-th environment-specific generator is the output corresponding to the k-th training image, which can generate the k-th environment-specific segmentation map.

[0072] Additionally, the learning device 1000 can enable the k-th environment-specific ranking network 1200-k to generate a k-th environment-specific ranking score through a k-th environment-specific discriminator, wherein the k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false, and the k-th environment-specific ranking map is generated by plucking and connecting the main region on the k-th training image and at least one first sub-region on the corresponding k-th environment-specific segmentation map.

[0073] Specifically, in addition to information about the k-th environment-specific segmentation map output by the k-th environment-specific generator, the k-th environment-specific sorting map input to the k-th environment-specific discriminator may also include information about the k-th training image. This is to enable the k-th environment-specific discriminator to make a judgment based on richer information when determining whether the k-th environment-specific sorting map input to the k-th environment-specific discriminator is true or false. For reference, (i) the main region on the k-th training image, (ii) the first sub-region on the k-th environment-specific segmentation map, (iii) the second sub-region on the k-th environment-specific segmentation annotation (described later), and (iv) the third sub-region on at least one j-th environment-specific segmentation map (described later) may each be a region corresponding to the same region.

[0074] Additionally, the learning device 1000 can perform the following processes: (i) training the k-th environment-specific object detector 1100-k to minimize the k-th environment-specific object detection loss, and training the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer to maximize the k-th environment-specific ranking score, wherein the k-th environment-specific object detection loss is generated by referring to the k-th environment-specific object detection information and its corresponding k-th environment-specific object annotation; (ii) training the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score related to the k-th environment-specific annotation ranking map and minimize the k-th environment-specific ranking score, wherein the k-th environment-specific annotation ranking map can be generated by cropping and connecting the main region on the k-th training image and the second sub-region on its corresponding k-th environment-specific segmentation annotation; and the k-th environment-specific annotation ranking score can be generated by the k-th environment-specific discriminator.

[0075] In addition, the learning device 1000 can train the k-th environment-specific deconvolution layer and the k-th environment-specific convolution layer respectively to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score. The k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and its corresponding k-th environment-specific segmentation annotation.

[0076] In addition, the learning device 1000 can further train the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and its corresponding k-th environment-specific object annotation.

[0077] The following describes a training method that can further improve the formation of the discriminator.

[0078] refer to Figure 3 The learning device 1000 can generate a k-th environment-specific ranking score through a k-th environment-specific discriminator. The k-th environment-specific ranking score represents information about whether the k-th environment-specific ranking map is true or false. In this case, the k-th environment-specific ranking map can be generated by cropping and connecting the main region on the k-th training image and the first sub-region on the corresponding k-th environment-specific segmentation map.

[0079] Additionally, the learning device 1000 can further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator. The j-th environment-specific ranking score represents information about whether at least one j-th environment-specific ranking map is true or false. In this case, the j-th environment-specific ranking map can be generated by cropping and connecting the main region on the k-th training image and the third sub-region on the corresponding j-th environment-specific segmentation map. Furthermore, j can be an integer greater than or equal to 1 and less than or equal to n, which is different from the value of k.

[0080] At this point, the j-th environment-specific segmentation map can be generated by at least one j-th environment-specific generator other than the k-th environment-specific generator corresponding to the k-th environment-specific discriminator.

[0081] For reference, the j-th environment-specific generator may include a j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer corresponding to the k-th environment-specific discriminator, and a corresponding j-th environment-specific deconvolutional layer.

[0082] That is, the j-th environment-specific segmentation map is generated by applying a convolution operation to the k-th training image through at least one j-th environment-specific convolutional layer to generate the j-th environment-specific feature map, and then applying a deconvolution operation to the j-th environment-specific feature map through a j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer.

[0083] For example, in order to further improve the performance of the first discriminator, the learning device 1000 may (i) output a third environment-specific segmentation map related to the first training image through a third generator among the second to nth generators other than the first generator, and (ii) cause the first ranking network to further generate a third environment-specific ranking score related to the third environment-specific ranking map through the first discriminator. At this time, the third environment-specific segmentation map corresponds to the first driving environment corresponding to the first discriminator. In addition, the third environment-specific ranking map may be generated by cropping and connecting the main region on the first training image and the third sub-region on the corresponding third environment-specific segmentation map.

[0084] For example, in order to improve the performance of the first discriminator, the learning device 1000 (i) outputs a fifth environment-specific segmentation map related to the first training image through the fifth generator (i-1), (i-2) outputs a seventh environment-specific segmentation map related to the first training image through the seventh generator (ii), and (ii) causes the first ranking network to further generate a fifth environment-specific ranking score related to the fifth environment-specific ranking map and a seventh environment-specific ranking score related to the seventh environment-specific ranking map through the first discriminator. At this time, the fifth environment-specific segmentation map and the seventh environment-specific segmentation map correspond to the first driving environment corresponding to the first discriminator. In addition, the fifth environment-specific ranking map can be generated by cropping and connecting the main region on the first training image and the third sub-region on the corresponding fifth environment-specific segmentation map. In addition, the seventh environment-specific ranking map can be generated by cropping and connecting the main region on the first training image and the third sub-region on the corresponding seventh environment-specific segmentation map.

[0085] In addition, the learning device 1000 can train the k-th environment-specific discriminator to maximize the k-th environment-specific label ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively.

[0086] The following describes the objective function used to train the object detector and its corresponding ranking network.

[0087] First, the GAN loss used to train (i) the discriminator and (ii) its corresponding generator, which includes convolutional and deconvolutional layers, can be calculated using the following mathematical formula 1.

[0088] <Mathematical Formula 1>

[0089] L GAN =E x，y [logD(x, y)] + E x [log(1-D(x, G(x)))]

[0090] Where X can be a training image; y can be a segmentation label; G(x) can be a segmentation map output by the generator; D(x, y) can be a label ranking score output by a discriminator related to the label ranking map, which is generated by cropping and concatenating the main region on the training image and the second sub-region on the corresponding segmentation label; D(x, G(x)) can be a ranking score output by a discriminator related to the ranking map, which is generated by cropping and concatenating the main region on the training image and the first sub-region on the corresponding segmentation map.

[0091] Alternatively, the segmentation loss can be calculated using the following mathematical formula 2.

[0092] <Mathematical Formula 2>

[0093] L seg =L ce (G(x), y)

[0094] Among them, L ce It could be cross-entropy loss.

[0095] Additionally, the object detection loss can be calculated using the following mathematical formula 3.

[0096] <Mathematical Formula 3>

[0097] L D =L rpn (R(x), y) rpn )+L det (O(x), y) det )

[0098] Where R(x) can be the ROI information output by the RPN; O(x) can be the object detection information output by the object detector; y rpn This can be a target annotation included in object annotations and output from the RPN; y det It can be a target label included in the object annotation and output from the object detector.

[0099] Alternatively, the discriminator loss used to train the discriminator can be calculated using the following mathematical formula 4. In this case, the discriminator loss can be calculated by referring to information about other generators besides the specific generator corresponding to the specific discriminator used as the training object.

[0100] <Mathematical Formula 4>

[0101] L comp =E n，x [log(1-D(x,G n (x)))]

[0102] Wherein, Gn(x) can be at least one other segmentation map output by at least one other generator besides the specific generator corresponding to the specific discriminator, and D(x, Gn(x)) can be the sorting score as the output of the specific discriminator associated with the sorting map, wherein the sorting map is generated by cropping and concatenating the main region on the training image and the third sub-region on at least one other segmentation map corresponding to it.

[0103] In summary, the objective function, referring to mathematical formulas 1 to 4, can be represented by the following mathematical formula 5.

[0104] <Mathematical Formula 5>

[0105]

[0106] For reference, λ can be any value that can be chosen to adjust the segmentation loss.

[0107] The learning device 1000 can train the object detector 1000 and its corresponding sorting network using the objective function of mathematical formula 5 in the following way.

[0108] First, while preventing the discriminator from updating its parameters, the learning device 1000 can utilize the arg min of the objective function in Formula 5. G，R，O L GAN (G,D)+λL seg (G)+L D (R, O) are used to train (i) convolutional and deconvolutional layers in the generator, (ii) an object detector, and (iii) an RPN layer.

[0109] That is, it can be based on arg min G，R，O L GAN (G,D)+λL seg (G)+L D (R, O) training includes (i) convolutional and deconvolutional layers in the generator, (ii) an object detector, and (iii) an RPN layer, to enable (i) L GAN (G, D), (ii)λL seg (G) and (iii)L D Minimize (R, O) respectively.

[0110] Specifically, in order to make L GAN (G, D)(i.e., E) x，y [logD(x, y)] + E x Minimizing [log(1-D(x, G(x)))] allows the generator to be trained while preventing the discriminator from updating its parameters, so that D(x, G(x)) (i.e., the ranking score) is maximized.

[0111] Furthermore, while preventing the generator's parameters from being updated, the learning device 1000 utilizes the arg max of the objective function in mathematical formula 5. D L GAN (G, D) + L comp (D) is used to train the discriminator.

[0112] That is, it can be determined based on arg max. D L GAN (G, D) + L comp (D) Train the discriminator so that L GAN (G, D) and L comp (D) Maximize each.

[0113] Specifically, in order to make L GAN (G, D)(i.e., E) x，y [logD(x, y)] + E x [log(1-D(x, G(x)))]) can be maximized by training the discriminator while preventing the generator from updating its parameters, so that (i) D(x, y) (i.e., the label ranking score) is maximized and (ii) D(x, G(x)) (i.e., the ranking score) is minimized.

[0114] As described above, after training (i) convolutional and deconvolutional layers for each generator, (ii) each object detector, and (iii) each RPN layer, the learning device 1000 may train each discriminator corresponding to each generator, but is not limited thereto. For example, after training each discriminator corresponding to each generator, the learning device 1000 may also train (i) convolutional and deconvolutional layers for each generator, (ii) each object detector, and (iii) each RPN layer corresponding to each discriminator.

[0115] pass Figures 4 to 7 The operation of the updating device when acquiring driving environment images under the training states of each generator, each discriminator, and each object detector as described above is described.

[0116] First, refer to Figure 4 The description describes an updating device that updates the object detector of an autonomous vehicle to a specific object detector suitable for a specific driving environment. In this case, the object detector suitable for the specific driving environment can be an object detector optimized for the specific driving environment.

[0117] The updating device 2000 may include a memory 2001 and a processor 2002. The memory 2001 stores instructions for updating the object detector of the autonomous vehicle to a specific object detector corresponding to a specific driving environment. The processor 2002 responds to the instructions stored in the memory 2001 and performs the operation of updating the object detector of the autonomous vehicle to a specific object detector corresponding to a specific driving environment.

[0118] Specifically, the updated device 2000 can typically utilize a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific manner) to achieve the required system performance. For example, the computing device may be a device that includes a computer processor, memory, storage, input devices and output devices, as well as other existing computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).

[0119] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software components that execute applications for specific purposes.

[0120] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.

[0121] At this time, updating device 2000 can be done with Figure 1 The learning devices shown are either identical to each other or are independent of each other.

[0122] refer to Figure 5 When the autonomous vehicle's vision system 3000 acquires a driving environment image displaying the autonomous vehicle's driving environment, the updating device 2000 can enable the first sorting network to the nth sorting network to generate a first driving environment sorting score to an nth driving environment sorting score corresponding to the driving environment image, respectively.

[0123] Additionally, the updating device 2000 may (i) select a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score, the specific driving environment ranking score having the largest value among the first driving environment ranking scores to the nth driving environment ranking scores, and (ii) update the current object detector of the autonomous vehicle to the specific object detector.

[0124] For example, the updating device 2000 can transmit the best object detector to the vision system 3000 of the autonomous vehicle via a wire, thereby updating the current object detector of the autonomous vehicle to a specific object detector.

[0125] For example, the update device 2000 can wirelessly transmit the best object detector to the vision system 3000 of the autonomous vehicle, thereby updating the current object detector of the autonomous vehicle to a specific object detector.

[0126] That is, the update device 2000 can use OTA (over-the-air) to transmit the best object detector to the vision system 3000 of the autonomous vehicle, thereby updating the current object detector of the autonomous vehicle to a specific object detector.

[0127] The following is for reference. Figure 6 The process of updating the current object detector of an autonomous vehicle to a specific object detector is described in further detail.

[0128] First, in the state S601, when the learning device 1000 trains the first to nth ranking networks and the first to nth object detectors using the first to nth training images obtained from the database, and the trained first to nth ranking networks and the trained first to nth object detectors are stored in a predetermined storage space, when the driving environment image is acquired from the vision system 3000 of the autonomous vehicle in the state S602, the updating device 2000 can make the first to nth ranking networks stored in the predetermined storage space generate the first driving environment ranking score to the nth driving environment ranking score corresponding to the driving environment image, respectively.

[0129] Additionally, the updating device 2000 can update the current object detector of the autonomous vehicle to the specific object detector by (i) selecting a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score, the specific driving environment ranking score having the largest value among the first driving environment ranking score to the nth driving environment ranking score, and (ii) transmitting the specific object detector S603 to the vision system 3000 of the autonomous vehicle.

[0130] The following is for reference. Figure 7 This describes a method for updating the object detector of an autonomous vehicle to a specific object detector.

[0131] For reference, the updating device 2000 of the present invention can perform operations on at least a portion of each of the first to nth object detectors and their corresponding first to nth sorting networks. For convenience, Figure 7 The k-th environment-specific object detector 1100-k and its corresponding k-th environment-specific sorting network 1200-k are shown only as representative examples of the k-th driving environment.

[0132] First, when a driving environment image is acquired, the updating device 2000 can (i) input the driving environment image to the first object detector to the nth object detector, so that the first object detector to the nth object detector applies at least one convolution operation to the driving environment image through the first convolutional layer to the nth convolutional layer, thereby generating the first driving environment feature map to the nth driving environment feature map respectively; (ii) input the first driving environment feature map to the nth driving environment feature map to the first sorting network to the nth sorting network, so that the first sorting network to the nth sorting network applies at least one deconvolution operation to the first driving environment feature map to the nth driving environment feature map through the first deconvolutional layer to the nth deconvolutional layer respectively, thereby generating the first driving environment segmentation map to the nth driving environment feature map respectively. The nth driving environment segmentation map (ii-2) generates a first driving environment ranking score to an nth driving environment ranking score through each of the first to nth discriminators. At this time, the first driving environment ranking score can represent information about whether the first driving environment ranking map is true or false. In addition, the first driving environment ranking map can be generated by cropping and connecting the first main driving region on the driving environment image and the first driving sub-region on the corresponding first driving environment segmentation map. In addition, the nth driving environment ranking score can represent information about whether the nth driving environment ranking map is true or false. In addition, the nth driving environment ranking map can be generated by cropping and connecting the nth main driving region on the driving environment image and the nth driving sub-region on the corresponding nth driving environment segmentation map.

[0133] Additionally, the updating device 2000 may (i) select a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score, wherein the specific driving environment ranking score has the largest value among the first driving environment ranking scores to the nth driving environment ranking scores, and (ii) update the current object detector of the autonomous vehicle to a specific object detector, wherein the specific object detector may be the k-th environment specific object detector 1100-k, the specific ranking network may be the k-th environment specific ranking network 1200-k, and the specific driving environment ranking score may be the k-th environment specific driving environment ranking score.

[0134] Therefore, the updated device 2000 can support autonomous vehicles to operate appropriately using object detectors optimized for various driving environments.

[0135] On the other hand, the above describes the use of object detectors in autonomous vehicles, but unlike that, object detectors can be applied to various devices such as autonomous aircraft and autonomous robots.

[0136] Furthermore, while the method for updating the object detector has been described above, the present invention is not limited thereto, and the same method can also be applied to updating various deep learning network models used in autonomous vehicles, autonomous aircraft, and autonomous robots.

[0137] Furthermore, the embodiments of the present invention described above can be implemented and recorded in the form of program commands executed by various computer components on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., individually or in combination. The program commands recorded on the computer-readable recording medium may be specifically designed or configured for the present invention, or may be program commands known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices such as ROMs, RAMs, and flash memory specifically configured to store and execute program commands. Examples of program commands include not only machine language code generated by a compiler, but also executable high-level language code such as that executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0138] The present invention has been described above with reference to specific components and other specific matters, limited embodiments, and accompanying drawings. However, these descriptions are intended to help to provide a more comprehensive understanding of the present invention. The present invention is not limited to the described embodiments, and those skilled in the art can attempt various modifications and variations based on these descriptions.

[0139] Therefore, the technical concept of the present invention should not be limited to the embodiments described above, and all contents of the claims of the present invention and their equivalent or equivalent transformations are within the technical concept of the present invention.

Claims

1. A training method for updating a deep learning-based object detector of an autonomous vehicle so as to adapt to a driving environment, comprising the following steps: (a) when at least one kth training image is acquired from a 1st training image to an nth training image, a learning device performs the following processes, wherein the 1st training image corresponds to a 1st driving environment and comprises a 1st object label and a 1st segmentation label, the nth training image corresponds to an nth driving environment and comprises an nth object label and an nth segmentation label, the k is an integer greater than or equal to 1 and less than or equal to n, (i) inputting the kth training image into a kth environment-specific object detector corresponding to a kth driving environment among a 1st object detector corresponding to the 1st driving environment to an nth object detector corresponding to the nth driving environment, so that the kth environment-specific object detector (i-1) generates a kth environment-specific feature map by applying at least one convolution operation on the kth training image through at least one kth environment-specific convolution layer of the kth environment-specific object detector, (i-2) generates a kth environment-specific pooling feature map by applying at least one ROI pooling operation on the kth environment-specific feature map through a kth environment-specific pooling layer of the kth environment-specific object detector using kth environment-specific ROI information corresponding to at least one region predicted to include at least one object present in the kth environment-specific feature map, (i-3) generates kth environment-specific object detection information by applying at least one fully connected operation on the kth environment-specific pooling feature map through a kth environment-specific fully connected layer of the kth environment-specific object detector, (ii) inputting the kth environment-specific feature map into a kth environment-specific ranking network corresponding to the kth environment-specific object detector among a 1st ranking network corresponding to the 1st object detector to an nth ranking network corresponding to the nth object detector, so that the kth environment-specific ranking network (ii-1) generates a kth environment-specific segmentation map by applying at least one deconvolution operation on the kth environment-specific feature map through at least one kth environment-specific deconvolution layer of the kth environment-specific ranking network, (ii-2) generates a kth environment-specific ranking score by a kth environment-specific discriminator of the kth environment-specific ranking network, wherein the kth environment-specific ranking score represents information about whether a kth environment-specific ranking map is true or false, the kth environment-specific ranking map is generated by connecting a main region on the kth training image and a 1st sub-region on the kth environment-specific segmentation map corresponding thereto after being cropped; and (b) the learning device performs the following processes, (i) trains the k-th environment-specific object detector to minimize the k-th environment-specific object detection loss generated by referring to the k-th environment-specific object detection information and the k-th environment-specific object annotation corresponding thereto, and trains the k-th environment-specific deconvolutional layer and the k-th environment-specific convolutional layer to maximize the k-th environment-specific ranking score, (ii) trains the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score related to a k-th environment-specific annotation ranking map generated by connecting the main region on the k-th training image and a second sub-region on the k-th environment-specific segmentation annotation corresponding thereto after being cropped, and minimize the k-th environment-specific ranking score generated by the k-th environment-specific discriminator. 2.The training method of claim 1, wherein: in the step (a), the learning device causes the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, the j-th environment-specific ranking score indicating information about whether at least one j-th environment-specific ranking map is true or false, the j-th environment-specific ranking map being generated by connecting the main region on the k-th training image and at least one third sub-region on at least one j-th environment-specific segmentation map corresponding thereto after being cropped, the j-th environment-specific segmentation map being generated by applying the deconvolutional operation to the j-th environment-specific feature map through at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer after generating the j-th environment-specific feature map by applying the convolutional operation to the k-th training image through at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer, the j being an integer equal to or greater than 1 and equal to or less than n, and in the step (b), the learning device trains the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively. 3.The training method of claim 1, wherein: in the step (b), the learning device trains the k-th environment-specific deconvolutional layer and the k-th environment-specific convolutional layer, respectively, to minimize a k-th environment-specific segmentation loss generated by referring to the k-th environment-specific segmentation map and the k-th environment-specific segmentation annotation corresponding thereto and maximize the k-th environment-specific ranking score. 4.The training method of claim 1, wherein: the k-th environment-specific discriminator includes a convolutional PatchGAN classifier. 5.The training method of claim 1, wherein: in the step (a), the learning device causes the k-th environment-specific ranking network to further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, the j-th environment-specific ranking score indicating information about whether at least one j-th environment-specific ranking map is true or false, the j-th environment-specific ranking map being generated by connecting the main region on the k-th training image and at least one third sub-region on at least one j-th environment-specific segmentation map corresponding thereto after being cropped, the j-th environment-specific segmentation map being generated by applying the deconvolutional operation to the j-th environment-specific feature map through at least one j-th environment-specific deconvolutional layer corresponding to the j-th environment-specific convolutional layer after generating the j-th environment-specific feature map by applying the convolutional operation to the k-th training image through at least one j-th environment-specific convolutional layer other than the k-th environment-specific convolutional layer, the j being an integer equal to or greater than 1 and equal to or less than n, and in the step (b), the learning device trains the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively. ​ ​ ​ ​ ​ ​ ​ ​ The learning device causes the kth environment-specific object detector to generate the kth environment-specific ROI information through a kth environment-specific RPN layer of the kth environment-specific object detector, In the step of (b), The learning device further trains the kth environment-specific RPN layer to minimize a kth environment-specific RPN loss, which is generated by referring to the kth environment-specific ROI information and the kth environment-specific object label corresponding thereto.

6. A method for updating a deep learning-based object detector of an autonomous vehicle so as to adapt to a driving environment, comprising the following steps: (a) when at least one kth training image is acquired from the first training image to the nth training image, the learning device performs the following processes, wherein the first training image corresponds to a first driving environment and includes a first object label and a first segmentation label, the nth training image corresponds to an nth driving environment and includes an nth object label and an nth segmentation label, the k is an integer greater than or equal to 1 and less than or equal to n, (i) inputting the kth training image into a kth environment-specific object detector corresponding to a kth driving environment from a first object detector corresponding to the first driving environment to an nth object detector corresponding to the nth driving environment, so that the kth environment-specific object detector (i-1) generates a kth environment-specific feature map by applying at least one convolution operation on the kth training image through at least one kth environment-specific convolution layer of the kth environment-specific object detector, (i-2) generates a kth environment-specific pooled feature map by applying at least one ROI pooling operation on the kth environment-specific feature map through a kth environment-specific pooling layer of the kth environment-specific object detector using kth environment-specific ROI information corresponding to at least one region predicted to include at least one object present in the kth environment-specific feature map, (i-3) generates kth environment-specific object detection information by applying at least one fully connected operation on the kth environment-specific pooled feature map through a kth environment-specific fully connected layer of the kth environment-specific object detector, (ii) inputting the kth environment-specific feature map into a kth environment-specific sorting network corresponding to the kth environment-specific object detector from a first sorting network corresponding to the first object detector to an nth sorting network corresponding to the nth object detector, so that the kth environment-specific sorting network (ii-1) generates a kth environment-specific segmentation map by applying at least one deconvolution operation on the kth environment-specific feature map through at least one kth environment-specific deconvolution layer of the kth environment-specific sorting network, (ii-2) generates a kth environment-specific sorting score by a kth environment-specific discriminator of the kth environment-specific sorting network, wherein the kth environment-specific sorting score represents information about whether a kth environment-specific sorting map is true or false, the kth environment-specific sorting map is generated by connecting a main region on the kth training image and a first sub-region on the kth environment-specific segmentation map corresponding thereto after being cropped, (iii) training the kth environment-specific object detector to minimize a kth environment-specific object detection loss and training the kth environment-specific deconvolution layer and the kth environment-specific convolution layer to maximize the kth environment-specific sorting score, wherein the kth environment-specific object detection loss is generated by referring to the kth environment-specific object detection information and a kth environment-specific object label corresponding thereto, (iv) training the kth environment-specific discriminator,maximize a kth environment-specific labeling ranking score related to a kth environment-specific labeling ranking graph generated by connecting a main region on the kth training image and a second sub-region on a kth environment-specific segmentation labeling corresponding to the main region after being cropped, and minimize the kth environment-specific ranking score generated by the kth environment-specific discriminator, in which state, the device updates obtains a driving environment image representing a driving environment of the autonomous vehicle; (b) The updating device (i) inputs the driving environment images into the 1st object detector to the nth object detector respectively, so that the 1st object detector to the nth object detector respectively applies at least one convolution operation to the driving environment images through the 1st convolution layer to the nth convolution layer, thereby respectively generating the 1st driving environment feature map to the nth driving environment feature map, (ii) inputs the 1st driving environment feature map to the nth driving environment feature map into the 1st ranking network to the nth ranking network, so that the 1st ranking network to the nth ranking network respectively (ii-1) applies at least one deconvolution operation to the 1st driving environment feature map to the nth driving environment feature map through the 1st deconvolution layer to the nth deconvolution layer, thereby respectively generating the 1st driving environment segmentation map to the nth driving environment segmentation map, (ii-2) generates the 1st driving environment ranking score to the nth driving environment ranking score through each of the 1st discriminator to the nth discriminator, respectively, wherein the 1st driving environment ranking score to the nth driving environment ranking score respectively represent information about whether the 1st driving environment ranking map to the nth driving environment ranking map is true or false, the 1st driving environment ranking map is generated by connecting the 1st driving main area on the driving environment image and the 1st driving sub-area on the 1st driving environment segmentation map corresponding thereto after being cropped, and the nth driving environment ranking map is generated by connecting the nth driving main area on the driving environment image and the nth driving sub-area on the nth driving environment segmentation map corresponding thereto after being cropped; and (c) The updating device (i) selects a specific object detector corresponding to a specific ranking network that generates a specific driving environment ranking score having the largest value among the 1st driving environment ranking score to the nth driving environment ranking score, (ii) updates the current object detector of the autonomous vehicle to the specific object detector.

7. The method of claim 6, wherein: in the step of (a), The learning device makes the k-th environment-specific ranking network further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, and trains the k-th environment-specific discriminator to maximize the k-th environment-specific labeled ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively, wherein the j-th environment-specific ranking score represents information about whether a j-th environment-specific ranking graph is true or false, the j-th environment-specific ranking graph is generated by connecting at least one third sub-region on at least one j-th environment-specific segmentation map corresponding to the main region on the k-th training image after being cropped, the j-th environment-specific segmentation map is generated by applying the de-convolution operation to the j-th environment-specific feature map through at least one j-th environment-specific de-convolution layer corresponding to the j-th environment-specific convolution layer after applying the convolution operation to the k-th training image through at least one j-th environment-specific convolution layer other than the k-th environment-specific convolution layer to generate at least one j-th environment-specific feature map, and j is an integer greater than or equal to 1 and less than or equal to n.

8. The method of claim 6, wherein: in the step of (a), the learning device trains the k-th environment-specific de-convolution layer and the k-th environment-specific convolution layer, respectively, to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation map and the k-th environment-specific segmentation label corresponding thereto.

9. The method of claim 6, wherein: the k-th environment-specific discriminator includes a convolutional PatchGAN classifier.

10. The method of claim 6, wherein: in the step of (a), the learning device makes the k-th environment-specific object detector generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector, and further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and the k-th environment-specific object label corresponding thereto.

11. A learning device for updating a deep learning-based object detector of an autonomous vehicle so as to adapt to a driving environment, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions. The processor performs the following processes: (I) when at least one kth training image is acquired from the first training image to the nth training image, the first training image corresponds to the first driving environment and includes the first object label and the first segmentation label, the nth training image corresponds to the nth driving environment and includes the nth object label and the nth segmentation label, k is an integer greater than or equal to 1 and less than or equal to n, (i) input the kth training image into the kth environment-specific object detector corresponding to the kth driving environment from the first object detector corresponding to the first driving environment to the nth object detector corresponding to the nth driving environment, so that the kth environment-specific object detector (i-1) applies at least one convolution operation to the kth training image through at least one kth environment-specific convolution layer of the kth environment-specific object detector, thereby generating a kth environment-specific feature map, (i-2) applying at least one ROI pooling operation to the kth environment-specific feature map through the kth environment-specific pooling layer of the kth environment-specific object detector using the kth environment-specific ROI information, thereby generating a kth environment-specific pooling feature map, wherein the kth environment-specific ROI information corresponds to at least one region predicted to include at least one object present in the kth environment-specific feature map, (i-3) applying at least one fully connected operation to the kth environment-specific pooling feature map through the kth environment-specific fully connected layer of the kth environment-specific object detector, thereby generating kth environment-specific object detection information, (ii) input the kth environment-specific feature map into the kth environment-specific ordering network corresponding to the kth environment-specific object detector from the first ordering network corresponding to the first object detector to the nth ordering network corresponding to the nth object detector, so that the kth environment-specific ordering network (ii-1) applies at least one deconvolution operation to the kth environment-specific feature map through at least one kth environment-specific deconvolution layer of the kth environment-specific ordering network, thereby generating a kth environment-specific segmentation map, (ii-2) generating a kth environment-specific ordering score through a kth environment-specific discriminator of the kth environment-specific ordering network, wherein the kth environment-specific ordering score represents information about whether the kth environment-specific ordering map is true or false, and the kth environment-specific ordering map is generated by connecting the main region on the kth training image and the first sub-region on the kth environment-specific segmentation map corresponding thereto after being cropped;and (II) (i) training the k-th environment-specific object detector to minimize the k-th environment-specific object detection loss and training the k-th environment-specific deconvolutional layer and the k-th environment-specific convolutional layer to maximize the k-th environment-specific ranking score, wherein the k-th environment-specific object detection loss is generated by referring to the k-th environment-specific object detection information and the k-th environment-specific object annotation corresponding thereto, (ii) training the k-th environment-specific discriminator to maximize the k-th environment-specific annotation ranking score related to the k-th environment-specific annotation ranking map and minimize the k-th environment-specific ranking score, wherein the k-th environment-specific annotation ranking map is generated by connecting the main region on the k-th training image and the second sub-region on the k-th environment-specific segmentation annotation corresponding thereto after being cropped, and the k-th environment-specific annotation ranking score is generated by the k-th environment-specific discriminator.

12. The learning device of claim 11, wherein: the processor, in the process of (I), The kth environment-specific discriminator is trained to maximize the kth environment-specific label ranking score and minimize the kth environment-specific ranking score and the jth environment-specific ranking score. In the process of (II), The kth environment-specific discriminator is trained to maximize the kth environment-specific label ranking score and minimize the kth environment-specific ranking score and the jth environment-specific ranking score.

13. The learning device of claim 11, wherein: In the process of (II), The kth environment-specific de-convolutional layer and the kth environment-specific convolutional layer are trained to minimize the kth environment-specific segmentation loss and maximize the kth environment-specific ranking score, respectively, wherein the kth environment-specific segmentation loss is generated by referring to the kth environment-specific segmentation map and the kth environment-specific segmentation label corresponding thereto.

14. The learning device of claim 11, wherein: The kth environment-specific discriminator comprises a convolutional PatchGAN classifier.

15. The learning device of claim 11, wherein: In the process of (I), The kth environment-specific object detector is caused to generate the kth environment-specific ROI information by a kth environment-specific RPN layer of the kth environment-specific object detector, In the process of (II), The kth environment-specific RPN layer is further trained to minimize a kth environment-specific RPN loss, wherein the kth environment-specific RPN loss is generated by referring to the kth environment-specific ROI information and the kth environment-specific object label corresponding thereto.

16. An updating device for updating a deep learning-based object detector of an autonomous vehicle so as to adapt to a driving environment, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein the processor performs the following processes, (I) when at least one kth training image is acquired from the first training image to the nth training image, the learning device performs the following processes, wherein the first training image corresponds to a first driving environment and includes a first object label and a first segmentation label, the nth training image corresponds to an nth driving environment and includes an nth object label and an nth segmentation label, the k is an integer greater than or equal to 1 and less than or equal to n, (i) inputting the kth training image into a kth environment-specific object detector corresponding to a kth driving environment from a first object detector corresponding to the first driving environment to an nth object detector corresponding to the nth driving environment, so that the kth environment-specific object detector (i-1) generates a kth environment-specific feature map by applying at least one convolution operation on the kth training image through at least one kth environment-specific convolution layer of the kth environment-specific object detector, (i-2) generates a kth environment-specific pooled feature map by applying at least one ROI pooling operation on the kth environment-specific feature map through a kth environment-specific pooling layer of the kth environment-specific object detector using kth environment-specific ROI information corresponding to at least one region predicted to include at least one object present in the kth environment-specific feature map, (i-3) generates kth environment-specific object detection information by applying at least one fully connected operation on the kth environment-specific pooled feature map through a kth environment-specific fully connected layer of the kth environment-specific object detector, (ii) inputting the kth environment-specific feature map into a kth environment-specific sorting network corresponding to the kth environment-specific object detector from a first sorting network corresponding to the first object detector to an nth sorting network corresponding to the nth object detector, so that the kth environment-specific sorting network (ii-1) generates a kth environment-specific segmentation map by applying at least one deconvolution operation on the kth environment-specific feature map through at least one kth environment-specific deconvolution layer of the kth environment-specific sorting network, (ii-2) generates a kth environment-specific sorting score by a kth environment-specific discriminator of the kth environment-specific sorting network, wherein the kth environment-specific sorting score represents information about whether a kth environment-specific sorting map is true or false, the kth environment-specific sorting map being generated by connecting a main region on the kth training image and a first sub-region on the kth environment-specific segmentation map corresponding thereto after being cropped, (iii) training the kth environment-specific object detector to minimize a kth environment-specific object detection loss and training the kth environment-specific deconvolution layer and the kth environment-specific convolution layer to maximize the kth environment-specific sorting score, wherein the kth environment-specific object detection loss is generated by referring to the kth environment-specific object detection information and a kth environment-specific object label corresponding thereto, (iv) training the kth environment-specific discriminator,maximizing a kth environment-specific labeling ranking score related to a kth environment-specific labeling ranking graph generated by connecting a main region on the kth training image and a second sub-region on a kth environment-specific segmentation labeling corresponding to the main region after being cropped, and minimizing the kth environment-specific ranking score generated by the kth environment-specific discriminator, in which case, an environment image representing an environment in which the autonomous vehicle travels is obtained; (II) (i) inputting the environment image into the first object detector to the nth object detector, respectively, to cause the first object detector to the nth object detector to apply at least one convolution operation to the environment image through a first convolution layer to an nth convolution layer, respectively, to thereby generate a first environment feature map to an nth environment feature map, respectively, (ii) inputting the first environment feature map to the nth environment feature map into the first ranking network to the nth ranking network, respectively, to cause the first ranking network to the nth ranking network to respectively (ii-1) apply at least one deconvolution operation to the first environment feature map to the nth environment feature map through a first deconvolution layer to an nth deconvolution layer, respectively, to thereby generate a first environment segmentation map to an nth environment segmentation map, respectively, (ii-2) generate a first environment ranking score to an nth environment ranking score through each of a first discriminator to an nth discriminator, respectively, wherein the first environment ranking score to the nth environment ranking score respectively represent information about whether a first environment ranking graph to an nth environment ranking graph is true or false, the first environment ranking graph being generated by connecting a first main region on the environment image and a first sub-region on the first environment segmentation map corresponding to the first main region after being cropped, the nth environment ranking graph being generated by connecting an nth main region on the environment image and an nth sub-region on the nth environment segmentation map corresponding to the nth main region after being cropped; and (III) (i) selecting a specific object detector corresponding to a specific ranking network that generates a specific environment ranking score having a largest value among the first environment ranking score to the nth environment ranking score, (ii) updating a current object detector of the autonomous vehicle to the specific object detector.

17. The updating device of claim 16, wherein: In the process of (I), The learning device makes the k-th environment-specific ranking network further generate at least one j-th environment-specific ranking score through the k-th environment-specific discriminator, and trains the k-th environment-specific discriminator to maximize the k-th environment-specific label ranking score and minimize the k-th environment-specific ranking score and the j-th environment-specific ranking score, respectively, wherein the j-th environment-specific ranking score represents information about whether a j-th environment-specific ranking graph is true or false, the j-th environment-specific ranking graph being generated by connecting at least one third sub-region on at least one j-th environment-specific segmentation graph corresponding to the main region on the k-th training image after being cropped, the j-th environment-specific segmentation graph being generated by applying a de-convolution operation to the j-th environment-specific feature map through at least one j-th environment-specific de-convolution layer corresponding to the j-th environment-specific convolution layer after generating the j-th environment-specific feature map by applying the convolution operation to the k-th training image through at least one j-th environment-specific convolution layer other than the k-th environment-specific convolution layer, and the j is an integer greater than or equal to 1 and less than or equal to n.

18. The updating device of claim 16, wherein: in the process of (I), the learning device trains the k-th environment-specific de-convolution layer and the k-th environment-specific convolution layer, respectively, to minimize the k-th environment-specific segmentation loss and maximize the k-th environment-specific ranking score, wherein the k-th environment-specific segmentation loss is generated by referring to the k-th environment-specific segmentation graph and the k-th environment-specific segmentation label corresponding thereto.

19. The updating device of claim 16, wherein: the k-th environment-specific discriminator includes a convolutional PatchGAN classifier.

20. The updating device of claim 16, wherein: in the process of (I), the learning device makes the k-th environment-specific object detector generate the k-th environment-specific ROI information through the k-th environment-specific RPN layer of the k-th environment-specific object detector, and further trains the k-th environment-specific RPN layer to minimize the k-th environment-specific RPN loss, wherein the k-th environment-specific RPN loss is generated by referring to the k-th environment-specific ROI information and the k-th environment-specific object label corresponding thereto.

Images