Apparatus for collecting training data for image learning and method thereof

Abstract

The invention discloses an apparatus for collecting training data for image learning and a method thereof. The apparatus may identify, via an interest network, an object of interest corresponding to a predetermined category by learning an image provided from a vehicle, obtain, via the interest network, one or more reliability scores indicating a reliability of identifying the object of interest, a learning process associated with the image is performed via the auxiliary network and an occluded object is detected in the image that affects a learning result of the network of interest, and whether to mark the image is determined based on whether the occluded object is detected and the one or more reliability scores.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2023-0114013 filed on August 29, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a device and method for collecting training data for image learning. Background Art

[0004] An autonomous vehicle is a vehicle that can operate autonomously without the need for a driver or passenger, and an autonomous driving system is a system that monitors and controls the autonomous operation of such an autonomous vehicle. Generally speaking, an autonomous vehicle may refer to a vehicle that monitors the vehicle's external environment to assist the driver in driving and is equipped with various driving assistance devices based on the monitored vehicle's external environment.

[0005] An autonomous vehicle or a vehicle equipped with a driving assistance device monitors the exterior of the vehicle to detect objects and controls the vehicle based on a scene determined from the detected objects. In other words, autonomous driving or assisted driving (e.g., using a driving assistance device) is generally premised on a process of determining the type of an object outside the vehicle.

[0006] To recognize objects outside the vehicle, deep learning schemes that leverage information obtained from sensors are often used. The network's object recognition performance can be deteriorated by occluded objects such as rain, dust, camera contamination, etc. contained in the image, which can negatively impact object recognition. Therefore, to improve object recognition performance, it is necessary to learn images that contain occluded objects.

[0007] The task of selecting images including occluded objects is usually performed by humans. The presence rate of images containing occluded objects is very low compared to all images acquired by the vehicle. Therefore, in order to select occluded objects from images acquired by the vehicle, a large number of images must be checked, which may require a lot of manpower and time.

[0008] In addition, images containing occluded objects may be artificially created, but in this case, it is difficult to fully reflect the characteristics of images obtained during actual vehicle driving, so there is a limitation on improving the learning performance of the network. Summary of the invention

[0009] The present disclosure is directed to solving the above-mentioned problems in some embodiments while maintaining the advantages achieved by these embodiments.

[0010] An aspect of the present disclosure provides an apparatus and method for collecting training data for image learning, which is capable of selecting training data that directly affects the improvement of recognition performance of an image learning network.

[0011] Another aspect of the present disclosure provides an apparatus and method for collecting training data for image learning, which can reduce the manpower and time in the process of selecting image training data.

[0012] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and those skilled in the art to which the present disclosure belongs will clearly understand any other technical problems not mentioned herein through the following description.

[0013] According to one or more example embodiments of the present disclosure, a device may include: one or more processors; and a memory. The memory may store instructions that, when executed by the one or more processors, cause the device to: identify an object of interest corresponding to a predetermined category by learning an image provided from a vehicle via an interest network; and obtain one or more reliability scores indicating the reliability of identifying the object of interest via the interest network; perform a learning process associated with the image via an auxiliary network and detect an occlusion object in the image that affects the learning result of the interest network; and determine whether to mark the image based on whether the occlusion object is detected and the one or more reliability scores.

[0014] When the instructions are executed by one or more processors, the device may determine whether to label an image based on one or more reliability scores in the following manner: determining whether to label an image based on a first reliability score indicating the accuracy of image segmentation and a second reliability score indicating the accuracy of detecting an object of interest output into a bounding box.

[0015] When the instructions are executed by one or more processors, the apparatus may further: obtain entropy of each pixel of the image; determine a representative entropy value based on the entropy of each pixel of the image; and determine a first reliability score. The first reliability score may be inversely proportional to the representative entropy value.

[0016] The instructions, when executed by one or more processors, may further cause the apparatus to: obtain entropy of at least one bounding box in the image; and determine a second reliability score. The second reliability score may be inversely proportional to the entropy of the at least one bounding box.

[0017] When the instructions are executed by one or more processors, the device may determine whether to mark an image in the following manner: determining a first reliability score and a second reliability score based on the recognition of an occluded object through learning of an auxiliary network; and determining to mark the image for learning of the auxiliary network based on at least one of the first reliability score or the second reliability score being less than a threshold.

[0018] When the instructions are executed by one or more processors, the apparatus may determine whether to mark an image by excluding the image from training data of the interest network and the auxiliary network based on the first reliability score and the second reliability score being greater than or equal to a threshold.

[0019] When the instructions are executed by one or more processors, the device may determine whether to mark an image in the following manner: determining a first reliability score based on the absence of detected occluded objects through learning of an auxiliary network; and determining to mark the image for learning of the auxiliary network based on the first reliability score being less than a threshold.

[0020] When executed by one or more processors, the instructions may further cause the apparatus to: determine a second reliability score based on the first reliability score being greater than or equal to a threshold; and determine to add a new category of the network of interest based on the second reliability score being less than the threshold.

[0021] When the instructions are executed by one or more processors, the apparatus may further determine whether to mark the image in the following manner: based on the first reliability score and the second reliability score being greater than or equal to a threshold, determine to mark the image for learning the interest network.

[0022] When executed by one or more processors, the instructions may further cause the apparatus to determine whether an occluding object is detected based on the first reliability score and the second reliability score being less than a threshold value.

[0023] According to one or more example embodiments of the present disclosure, a method may include: identifying an object of interest corresponding to a predetermined category by learning an image from a vehicle via an interest network; and obtaining one or more reliability scores for identifying the object of interest via the interest network; performing a learning process associated with the image via an auxiliary network and detecting an occluding object in the image that affects the learning result of the interest network; and determining whether to label the image based on whether the occluding object is detected and the one or more reliability scores.

[0024] Identifying the object of interest may include: determining a first reliability score indicating accuracy of the segmentation; and determining a second reliability score indicating accuracy of detecting the object of interest outputted into the bounding box.

[0025] Determining the first reliability score may include: obtaining entropy of each pixel of the image; determining a representative entropy value based on the entropy of each pixel of the image; and determining the first reliability score. The first reliability score may be inversely proportional to the representative entropy value.

[0026] Determining the second reliability score may include: obtaining an entropy of at least one bounding box in the image; and determining the second reliability score. The second reliability score may be inversely proportional to the entropy of the at least one bounding box.

[0027] Determining whether to mark an image may include: determining a first reliability score and a second reliability score based on recognition of an occluded object through learning of an auxiliary network; and determining to mark the image for learning of the auxiliary network based on at least one of the first reliability score or the second reliability score being less than a threshold.

[0028] Determining whether to tag the image may include excluding the image from training data for the interest network and the auxiliary network based on the first reliability score and the second reliability score being greater than or equal to a threshold.

[0029] Determining whether to mark the image may include: determining a first reliability score based on no detection of the occluded object through learning of the auxiliary network; and determining to mark the image for learning of the auxiliary network based on the first reliability score being less than a threshold.

[0030] Identifying the object of interest may include determining a second reliability score based on the first reliability score being greater than or equal to a threshold. The method may further include determining to add a new category of the network of interest based on the second reliability score being less than the threshold.

[0031] Determining whether to tag the image may include determining to tag the image for learning the interest network based on the first reliability score and the second reliability score being greater than or equal to a threshold.

[0032] The determination of the first reliability score and the second reliability score may be performed prior to detecting the occluding object.Detecting the occluding object may include detecting the occluding object based on the first reliability score and the second reliability score being less than a threshold value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings:

[0034] Figure 1 is a schematic diagram illustrating a method of collecting training data for image learning;

[0035] Figure 2 is a diagram showing a vehicle capable of obtaining an image outside the vehicle during driving;

[0036] Figure 3 is a diagram showing a method of constructing a learning model for image learning;

[0037] Figure 4 is a block diagram showing the configuration of a management device;

[0038] Figure 5 is a diagram illustrating image segmentation performed by a first network;

[0039] Figure 6is a diagram showing a bounding box output by the second network;

[0040] Figure 7 is a diagram showing types of objects detected in an auxiliary network that are susceptible to recognition;

[0041] Figure 8 is a flow chart illustrating a method of collecting training data;

[0042] Fig. 9 is a flow chart illustrating a method of determining whether to continue marking an image;

[0043] Fig.10 is a diagram showing an example of an image excluded from training data;

[0044] Fig.11 is a flow chart illustrating a method of determining whether to continue marking an image; and

[0045] Fig.12 is a diagram illustrating a method of determining whether to continue marking an image. DETAILED DESCRIPTION

[0046] Hereinafter, one or more example embodiments of the present disclosure will be described in detail with reference to the exemplary drawings. When adding reference numerals to components in the drawings, it should be noted that the same or equivalent components are represented by the same reference numerals even when they are shown in other drawings. In addition, when describing the example embodiments of the present disclosure, when it is determined that the detailed description of the relevant known configuration or function interferes with the understanding of the example embodiments of the present disclosure, the detailed description will be omitted.

[0047] In addition, when describing the components of the present disclosure, terms such as first, second, A, B, (a), (b) and the like may be used herein. These terms are provided only to distinguish an element from other elements, and the nature, order, sequence and quantity of the elements are not limited by these terms. In addition, unless otherwise defined, all terms (including technical or scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs. The terms defined in the general dictionary should be interpreted as having a meaning consistent with the meaning of the context of the relevant technology, and unless clearly defined in the specification of the present disclosure, should not be interpreted as an ideal or overly formal meaning.

[0048] In the following, reference will be made to Figures 1 to 12 One or more example embodiments of the present disclosure are described in detail.

[0049] Figure 1 is a schematic diagram illustrating a method of collecting training data for image learning. Figure 2 is a diagram showing a vehicle capable of obtaining an image outside the vehicle during traveling.

[0050] Reference Figure 1 , a server SV that collects training data for image learning may receive images from vehicles.

[0051] like Figure 2 As shown, each of the vehicles VEH1, VEH2 and VEH3 may include at least one of a camera 11, light imaging detection and ranging (lidar) 12, and radio detection and ranging (radar) 13 to detect objects outside the vehicle VEH.

[0052] The camera 11 for obtaining the external image of the vehicle VEH can obtain the front image or the front and side images of the vehicle VEH. For example, the camera 11 can be arranged around the front windshield to obtain the front image of the vehicle VEH.

[0053] The laser radar 12, which is set to emit laser light and determine the object by using the reflected wave of the laser light reflected from the object, can be implemented in a time-of-flight (TOF) scheme or a phase shift scheme. The laser radar 12 can be installed to be exposed to the outside of the vehicle and can be arranged around the front bumper or the front grille.

[0054] The radar 13 may include an electromagnetic wave transmitting module and a receiving module. The radar 13 may be implemented in a pulse radar scheme or a continuous wave radar scheme based on the principle of transmitting radio waves. According to the signal waveform in the continuous wave radar scheme, the radar 13 may be implemented in a frequency modulated continuous wave (FMCW) scheme or a frequency shift keying (FSK) scheme. The radar 13 may include a front radar 13-1 located at the center of the front part of the vehicle VEH, a front side radar 13-2 located at both ends of the front bumper, and a rear radar 13-3 located at the rear part of the vehicle VEH.

[0055] The positions of the camera 11, the laser radar 12 and the radar 13 may not be limited to Figure 2 Position shown.

[0056] In addition, the sensors of the vehicle VEH may include ultrasonic sensors and infrared sensors.

[0057] The images provided to the server SV from the vehicles VEH1 , VEH2 , and VEH3 may be data obtained by fusing the images acquired by the camera 11 and the information acquired by the laser radar 12 or the information acquired by the radar 13 .

[0058] The server SV may perform deep learning on the images provided from the vehicles VEH1 , VEH2 , and VEH3 by utilizing an interest network.

[0059] In addition, the server SV can learn the image and detect occluded objects in the image by utilizing the auxiliary network. As used herein, an occluded object in an image can be any object or artifact within the image that negatively affects (e.g., hinders, prevents, obstructs, etc.) object recognition. An occluded object can be a factor that may reduce the accuracy of the deep learning of the interest network, and can include dust, rain, liquid, light blur, pollutants, etc.

[0060] According to the learning results of the interest network and the auxiliary network, the server SV can determine whether to use the image as training data for the interest network or the auxiliary network.

[0061] The server SV may provide the images determined as training data to the labeler LB.

[0062] Figure 3 is a diagram illustrating a method of constructing a learning model for image learning.

[0063] Reference Figure 3 , a learning model for image learning can be built based on machine learning operations (Mlops).

[0064] In S310, data collection may be a process of obtaining surrounding information. The surrounding information may be an image obtained when the vehicle is traveling, or may be obtained in a simulated environment.

[0065] In S320, data preprocessing may be a process of processing an image. Data preprocessing may include a process for converting and processing the size of an image, and may include a process for adding metadata to the image. In addition, data preprocessing may include a process for fusing lidar information or radar information into the image.

[0066] In S330 , data management may be a process of selecting data for network learning.

[0067] In S340 , data labeling may be a process of labeling images determined to be training data, and may be a task performed by a human.

[0068] In S350 , network training may be a process of learning images that complete labeling work.

[0069] In S360, the network evaluation may be a process of evaluating the network model based on the image learning result.

[0070] In S370 , model monitoring may be a process of monitoring the prediction performance of the network model.

[0071] As described below Figure 4As shown, the server SV may include the management device 100 and select the training data by using the management device 100. Figure 4 The management device 100 will be described.

[0072] Figure 4 is a block diagram showing the configuration of a management device.

[0073] Reference Figure 1 and Figure 4 The management device 100 may include a communication device 101 , an interest network 110 , an auxiliary network 120 , and a processor 130 .

[0074] The communication device 101 may receive images from the vehicles VEH1 , VEH2 , and VEH3 or the outside. The communication device 101 may provide images to the interest network 110 and the auxiliary network 120 .

[0075] The communication device 101 can send and receive radio signals between the vehicles VEH1, VEH2 and VEH3 on a mobile communication network constructed according to a technical standard or communication scheme for mobile communication. For example, the communication device 101 can perform communication based on Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Code Division Multiple Access 2000 (CDMA2000), Enhanced Voice Data Optimized or Enhanced Voice Data Only (EV-DO), Wideband CDMA (WCDMA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Advanced Long Term Evolution (LTEA), etc.

[0076] The interest network 110 may detect an object (eg, an object of interest) or classify pixel categories through deep learning images. To this end, the interest network 110 may include at least one of a first network 111 and a second network 112. Figure 5 and Figure 6 The learning result of the interest network 110 is described.

[0077] Figure 5 is a diagram illustrating image segmentation performed by the first network. Figure 6 is a diagram showing the bounding box output by the second network.

[0078] The first network 111 of the interest network 110 may be a network model for performing image segmentation. Figure 5 As shown, the first network 111 can perform image segmentation and assign a category to each pixel of the image. The label can be used to classify the predetermined category.

[0079] like Figure 6As shown, the second network 112 of the interest network 110 can learn the image and express the location of the object detected in the image as a box.

[0080] In addition, the interest network 110 may obtain a reliability score based on the image deep learning result. For example, the first network 111 may obtain a first reliability score based on the deep learning result, and the second network 112 may obtain a second reliability score. The first reliability score may indicate the accuracy of the segmentation. In addition, the second reliability score may indicate the accuracy of the object detection output as a bounding box.

[0081] The auxiliary network 120 can detect occluded objects that affect the learning results of the interest network.

[0082] Figure 7 is a diagram illustrating types of occluding objects detected in the auxiliary network.

[0083] Reference Figure 7 , the auxiliary network 120 can detect dust as in case 1. In addition, the auxiliary network 120 can detect water drops as in case 2, and detect light blur as in case 3.

[0084] The auxiliary network 120 may detect occluded objects in units of pixels in an image.

[0085] The processor 130 may determine whether to continue labeling the image based on whether an occluding object is detected and the reliability score. Image labeling may be a process of assigning specific values ​​to training data before learning an image. Image labeling may be a necessary process for generating correct answer data for supervised learning.

[0086] To this end, the processor 130 may include an artificial intelligence (AI) processor for image learning. The AI ​​processor may learn a neural network by utilizing a pre-stored program. The neural network for detecting target vehicles and dangerous vehicles may be designed to simulate the human brain structure on a computer, and may include multiple network nodes with weights that simulate neurons of a human neural network. Multiple network nodes may send and receive data according to a connection relationship to allow neurons to simulate the synaptic activity of neurons that send and receive signals through synapses. The neural network may include a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes may send and receive data according to a convolutional connection relationship when located at different layers. For example, a neural network model may include various deep learning schemes, such as a deep neural network (DNN), a convolutional deep neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a deep Q network, etc. The interest network may be a neural network implemented with one or more processors. The auxiliary network may be a neural network implemented with one or more processors.

[0087] The processor 130 may include a memory (not shown) for storing the AI ​​processor, algorithms, etc. The memory may use a hard disk drive, a flash memory, an electrically erasable programmable read-only memory (EEPROM), a static RAM (SRAM), a ferroelectric RAM (FRAM), a phase change RAM (PRAM), a magnetic RAM (MRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR-SDRAM), etc.

[0088] Image labeling may be performed on images determined to be training data.

[0089] Figure 8 is a flow chart illustrating a method of collecting training data. Figure 8 The process shown can be Figure 3 The data management process shown below includes the process. Figure 8 Describe the method for collecting training data.

[0090] In S810 , the interest network 110 may learn the image and obtain a reliability score according to the learning result.

[0091] The reliability score may include at least one of a first reliability score output from the first network 111 and a second reliability score output from the second network 112 .

[0092] The first reliability score may indicate the accuracy of the segmentation.

[0093] To obtain the first reliability score, the first network may obtain the entropy of each pixel of the image. Entropy may represent the uncertainty of the label given to the pixel. For example, as a result of the segmentation, the first pixel may have a 25% probability of being a person and a 20% probability of being a dog. When the second pixel contains an 85% probability of being a person and a 2% probability of being a dog, the entropy of the first pixel may be calculated to be higher than the entropy of the second pixel.

[0094] The first network 111 may obtain a representative entropy value of the image based on the entropy of each pixel. For example, the representative entropy value may be an average entropy of pixels included in the image. Alternatively, the representative entropy value may be a total entropy of the pixel.

[0095] A second reliability score may be calculated based on the entropy of the bounding box.

[0096] The entropy of the bounding box may be calculated based on the uncertainty of the bounding box. For example, the second network 112 may obtain a confidence score indicating the probability that the object exists inside the bounding box. The entropy of the bounding box may be inversely proportional to the confidence score, and the second reliability score may be determined to be proportional to the confidence score.

[0097] In S820 , the auxiliary network 120 may learn the image and detect occluded objects.

[0098] In S830 , the processor 130 may determine whether to continue labeling the image based on whether an occluding object is detected and a reliability score output by the interest network.

[0099] As described in this article, the image learning results of the interest network and the auxiliary network can be used to collect training data for the deep learning network during the Mlops process. The training data can be selected from images obtained when the vehicle is actually driving. In particular, because the training data is selected based on the reliability score obtained by image learning, images that affect the learning of the interest network or the auxiliary network can be selected. Therefore, training data that has a greater impact on object recognition performance can be selected. In addition, since the images used to learn occluded objects are not manually created by humans, the time and cost of preparing training data can be reduced.

[0100] Next, an example of determining whether to continue marking an image will be described.

[0101] Fig. 9 is a flow chart illustrating a method of determining whether to continue marking an image. Fig. 9 The following will refer to the process executed by the processor. Fig. 9 Describes a method for determining whether to continue labeling an image.

[0102] In operation S901 , after completing the learning of the interest network for the image, the processor 130 may determine whether both the first reliability score and the second reliability score are equal to or greater than a threshold reliability score.

[0103] The threshold reliability score may be determined as the degree to which the accuracy of the learning result of the first network and the learning result of the second network may be doubted. For example, although the category is classified through the learning of the first network, the accuracy may not be guaranteed. In addition, although the bounding box and the category of the bounding box are matched through the learning of the second network, the accuracy may not be guaranteed. The threshold reliability score may be set at around 50%.

[0104] In operation S902 , the processor 130 may determine whether an occluding object is detected in an image.

[0105] In operation S903 , the processor 130 may determine whether a first reliability score is less than a first threshold based on the detected occluding object.

[0106] like Figure 7 As shown, when the auxiliary network 120 detects an occluding object such as dust, water droplets, light blur, etc., the processor 130 may identify a first reliability score.

[0107] In S904, the processor 130 may determine the image as training data for the auxiliary network based on the fact that the first reliability score is less than the first threshold. In addition, the processor 130 may determine to perform image labeling.

[0108] The first threshold may be set to a degree to which the class matched to the pixel is determined based on the segmentation result and may be set to a degree below a threshold reliability score.

[0109] Although the auxiliary network detects the occluding object, when the first reliability score of the first network is low, it can be determined that the segmentation cannot be performed smoothly due to the occluding object. Therefore, since the image classified in operation S904 causes the recognition performance of the first network to decrease, the image can be selected as a target for learning the occluding object.

[0110] In S905 , the processor 130 may identify a second reliability score based on the fact that the first reliability score is greater than or equal to the first threshold.

[0111] In S906, the processor 130 may determine the image as training data for the auxiliary network based on the fact that the second reliability score is less than the second threshold. In addition, the processor 130 may determine whether to perform image labeling.

[0112] The second threshold may be set to determine the degree to which the bounding box detected by the second network is uncertain, and may be set to a degree below a threshold reliability score.

[0113] When the auxiliary network detects the occluding object and the second reliability score of the second network is low, it can be determined that the bounding box cannot be successfully created due to the occluding object. Therefore, since the image classified in operation S906 causes the recognition performance of the second network to decrease, the occluding object can be selected as the target of learning the occluding object.

[0114] In S907 , the processor 130 may exclude the image from the training data based on the fact that the second reliability score is greater than or equal to the second threshold.

[0115] Fig.10 is a diagram showing an example of images excluded from training data.

[0116] Reference Fig.10 , the original image obtained by the vehicle may be an image obtained by capturing water droplets.

[0117] In S907, the classified images classified by the interest network do not have high learning reliability, but it can be determined that the water droplets do not affect the category classification, such as Fig.10 shown.

[0118] As described herein, processor 130 may decide to exclude an image from the training data even when an occluding object is detected in the image and when it does not significantly affect the recognition performance of the interest network.

[0119] Fig.11 is a flow chart illustrating a method of determining whether to continue marking an image. Fig.11 is a diagram illustrating a process performed on an image having a first reliability score and a second reliability score greater than or equal to a threshold reliability score after learning of an interest network is completed. Fig.12 is a diagram illustrating a method of determining whether to continue marking an image. Fig.11 and Fig.12 , the method of determining whether to continue marking an image will be described below.

[0120] In operations S1101 and S1102 , the processor 130 may determine whether the first reliability score is less than a first threshold based on the fact that the occluding object is not detected in the image.

[0121] In S1103, the processor 130 may determine the image as training data for the auxiliary network based on the fact that the first reliability score is less than the first threshold. In addition, the processor 130 may determine whether to perform image labeling.

[0122] Even if the auxiliary network does not recognize the occluding object, when the first reliability score of the first network is low, it can be determined that the segmentation cannot be performed smoothly due to the occluding object. Fig.12 As shown, even if no occluding object is detected, the first network may not be able to recognize the actual vehicle VEH_g included in the image. This phenomenon may occur when the entire image is blurred by drizzle or dust, or when no occluding object is detected and a distant object is not recognized. Therefore, since the image classified in operation S1103 causes the recognition performance of the first network to decrease, the image can be selected as a target for learning occluding objects.

[0123] In operation S1104 , the processor 130 may determine whether the second reliability score is less than a second threshold based on the fact that the first reliability score is greater than or equal to the first threshold.

[0124] In operation S1105 , the processor 130 may determine to add a new category based on the fact that the second reliability score is less than the second threshold value.

[0125] When the bounding box reliability score is low compared to the segmentation result, the classification of the detected object may be uncertain. Fig.12As shown, although the second network detects the vehicle and creates the first bounding box Bbox1, it may not be able to accurately determine the type of the vehicle. Therefore, it can be determined that the category of the vehicle divided by the first bounding box Bbox1 is not predefined. Therefore, in operation S1105, the processor 130 can determine that a new category needs to be added to the classified image.

[0126] In S1106 , the processor 130 may determine the image as training data of the network of interest based on the fact that the second reliability score is greater than or equal to the second threshold.

[0127] When no occluded object is detected in the image and it is determined that the recognition performance of the interest network is acceptable, the image may be determined as training data for the interest network 110, thereby improving the object recognition performance of the interest network 110. Operation S1106 may include an operation of determining a labeling task to utilize the image as training data for the interest network 110.

[0128] Since the training data is selected based on the results of detecting occluded objects in images obtained while the vehicle is traveling, it is possible to select training data that directly affects the improvement of the recognition performance of the image learning network.

[0129] In addition, since training data for the image learning network can be selected without generating images containing occluded objects, manpower and time can be significantly reduced.

[0130] In addition, various effects directly or indirectly understood by the present disclosure can be provided.

[0131] Although one or more exemplary embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the disclosure.

[0132] Therefore, the exemplary embodiments disclosed in the present disclosure are provided for description, rather than for limiting the technical concept of the present disclosure, and it should be understood that these exemplary embodiments are not intended to limit the scope of the technical concept of the present disclosure. The protection scope of the present disclosure should be understood by the attached claims, and all technical concepts within the equivalent scope should be interpreted as falling within the scope of the rights of the present disclosure.

Claims

1. A device for collecting training data for image learning, comprising: one or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the device to: identifying an object of interest corresponding to a predetermined category by learning from an image provided from a vehicle via an interest network; obtaining, via the interest network, one or more reliability scores indicating reliability of identifying the object of interest; performing a learning process associated with the image via an auxiliary network and detecting an occluded object in the image that affects a learning result of the interest network; as well as Whether to tag the image is determined based on the following conditions: whether the occluding object is detected, and The one or more reliability scores.

2. The apparatus of claim 1 , wherein the instructions, when executed by the one or more processors, cause the apparatus to determine whether to label the image based on the one or more reliability scores by: Whether to tag the image is determined based on the following conditions: a first reliability score indicating segmentation accuracy of the image, and A second reliability score indicating the accuracy of detecting the object of interest output into the bounding box.

3. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: Obtaining the entropy of each pixel of the image; determining a representative entropy value based on the entropy of each pixel of the image; and The first reliability score is determined, the first reliability score being inversely proportional to the representative entropy value.

4. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: obtaining entropy of at least one bounding box in the image; and The second reliability score is determined, the second reliability score being inversely proportional to the entropy of the at least one bounding box.

5. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, cause the apparatus to determine whether to tag the image by: determining the first reliability score and the second reliability score based on recognition of the occluded object through learning of the auxiliary network; and Determining to mark the image for learning of the auxiliary network based on at least one of the first reliability score and the second reliability score being less than a threshold.

6. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, cause the apparatus to determine whether to tag the image by: Based on the first reliability score and the second reliability score being greater than or equal to a threshold, excluding the image from training data of the interest network and the auxiliary network.

7. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, cause the apparatus to determine whether to tag the image by: determining the first reliability score based on non-detection of the occluding object through learning of the auxiliary network; and Based on the first reliability score being less than a threshold, determining to mark the image for learning of the auxiliary network.

8. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: determining the second reliability score based on the first reliability score being greater than or equal to a threshold; and Adding a new category of the interest network is determined based on the second reliability score being less than the threshold.

9. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to determine whether to tag the image by: Based on the first reliability score and the second reliability score being greater than or equal to a threshold, determining to tag the image for learning the interest network.

10. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to determine whether the occluding object is detected based on the first reliability score and the second reliability score being less than a threshold.

11. A method for collecting training data for image learning, comprising: identifying an object of interest corresponding to a predetermined category by learning from an image from a vehicle via an interest network; as well as obtaining, via the interest network, one or more reliability scores for identifying the object of interest; performing a learning process associated with the image via an auxiliary network and detecting an occluded object in the image that affects a learning result of the interest network; as well as Whether to tag the image is determined based on the following conditions: whether the occluding object is detected, and The one or more reliability scores.

12. The method of claim 11, wherein identifying the object of interest comprises: determining a first reliability score indicative of segmentation accuracy; as well as A second reliability score is determined that indicates the accuracy of detecting the object of interest output into the bounding box.

13. The method of claim 12, wherein determining the first reliability score comprises: Obtaining the entropy of each pixel of the image; Determining a representative entropy value based on the entropy of each pixel of the image; as well as The first reliability score is determined, the first reliability score being inversely proportional to the representative entropy value.

14. The method of claim 12, wherein determining the second reliability score comprises: obtaining entropy of at least one bounding box in the image; as well as The second reliability score is determined, the second reliability score being inversely proportional to the entropy of the at least one bounding box.

15. The method of claim 12, wherein determining whether to tag the image comprises: determining the first reliability score and the second reliability score based on the recognition of the occluded object through learning of the auxiliary network; as well as Determining to mark the image for learning of the auxiliary network based on at least one of the first reliability score and the second reliability score being less than a threshold.

16. The method of claim 12, wherein determining whether to tag the image comprises: Based on the first reliability score and the second reliability score being greater than or equal to a threshold, excluding the image from training data of the interest network and the auxiliary network.

17. The method of claim 12, wherein determining whether to tag the image comprises: determining the first reliability score based on failure to detect the occluding object through learning of the auxiliary network; as well as Based on the first reliability score being less than a threshold, determining to mark the image for learning of the auxiliary network.

18. The method of claim 12, wherein identifying the object of interest comprises: determining the second reliability score based on the first reliability score being greater than or equal to a threshold, and The method further comprises: Adding a new category of the interest network is determined based on the second reliability score being less than the threshold.

19. The method of claim 12, wherein determining whether to tag the image comprises: Based on the first reliability score and the second reliability score being greater than or equal to a threshold, determining to tag the image for learning the interest network.

20. The method of claim 12, wherein determining the first reliability score and the second reliability score is performed before detecting the occluding object, and Detecting the occluding object includes detecting the occluding object based on the first reliability score and the second reliability score being less than a threshold value.

Images