Apparatus for collecting training data for image learning and method therefor

Abstract

An apparatus for collecting training data for image learning and a corresponding method are provided. The apparatus can, by means of an interest network (110), recognize an object of interest corresponding to a predetermined class by learning an image provided by a vehicle; obtain, by means of the interest network (110), one or more confidence scores indicating the confidence with which the object of interest is recognized; perform a learning operation associated with the image by means of an auxiliary network (120), detect an occluding object in the image that interferes with a learning result of the interest network (110); and determine, based on the one or more confidence scores, whether to label the image when the occluding object is detected.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus for collecting training data for image learning or training from images and a method therefor. BACKGROUND

[0002] An autonomous vehicle refers to a vehicle that can be operated or driven by itself without the intervention of a driver or passenger, and an autonomous driving system refers to a system that monitors and controls such an autonomous vehicle to drive itself. In general, an autonomous vehicle may refer to a vehicle that monitors the vehicle's external environment to assist the driver in driving and is equipped with various driving assistance devices based on the monitored external environment of the vehicle.

[0003] An autonomous vehicle or a vehicle equipped with a driving assistance device monitors the vehicle's external environment to detect an object and controls the vehicle based on a scenario determined according to the detected object. In other words, autonomous driving or assisted driving (e.g., with a driving assistance device) is generally based on the process of determining the type of object outside the vehicle.

[0004] To detect an object outside a vehicle, a deep learning scheme is typically used that leverages information from sensors. The object detection performance of a network may degrade due to occluding objects that are dangerous for detecting and negatively impact object detection, such as rainwater present in an image, dust, dirt on the camera, and the like. Therefore, to improve object detection performance, it is necessary to learn from images containing occluding objects.

[0005] The task of selecting images containing occluding objects is generally performed by a single person. The number of images containing occluding objects is very small compared to all the images captured by a vehicle. Therefore, selecting occluding objects from the images captured by a vehicle requires reviewing a large number of images, which requires a significant amount of labor and time.

[0006] In addition, an image containing an occluding object can be artificially generated, but in this case, it is difficult to fully reproduce the characteristics of the image captured during the actual driving of the vehicle, so there is a limitation to improving the learning performance of a network. BRIEF EXPLANATION

[0007] The present disclosure has been made to solve the above-mentioned problems in some embodiments while retaining the advantages achieved by these embodiments.

[0008] One aspect of the present disclosure provides an apparatus and method for collecting training data for image learning or training from images, which are capable of selecting training data that directly impacts the improvement of the recognition performance of an image learning network (e.g., deep learning network).

[0009] Another aspect of the present disclosure provides an apparatus and method for collecting training data for image learning or training from images, which are capable of reducing the labor and time required for the process of selecting image training data.

[0010] The technical problems to be solved by the present disclosure are not limited to the aforementioned problems, and any other technical problems not mentioned here will be clearly understood by those skilled in the art to which the present disclosure pertains from the following description.

[0011] According to one or more example embodiments of the present disclosure, a device may comprise: one or more processors, and a memory. The memory may store instructions that, when executed by the one or more processors, cause the device to: detect, via an interest network, an object of interest corresponding to a predetermined class by learning an image provided by a vehicle (e.g., captured, e.g., by a camera), and obtain, via the interest network, one or more confidence scores indicative of the confidence with which the object of interest is detected, perform, via an auxiliary network, a learning operation associated with the image, and detect an obscuring object in the image (e.g., detection-threatening object, which e.g.,a correct recognition of the object of interest based on the image is difficult, e.g. due to dust, water drops, etc.) that influences a learning outcome of the interest network, and determining whether to label the image (e.g., associate information with the image, e.g., in the form of a label) based on whether the occluding object is detected and the one or more confidence scores.

[0012] For example, the instructions, when executed by the one or more processors, may cause the device to: determine whether to label the image based on the one or more confidence scores by determining whether to label the image based on: a first confidence score indicative of an accuracy of segmenting the image (e.g., classifying pixels of the image); and a second confidence score indicative of an accuracy of detecting the object of interest output into a minimal bounding box.

[0013] For example, the instructions, when executed by the one or more processors, may further cause the device to: obtain (e.g., determine) the entropy of each pixel of the image (e.g., the confidence with which the pixel can be assigned to a class); determine, based on the entropy of each pixel of the image, a representative entropy value; and determine the first confidence score. The first confidence score may, for example, be inversely proportional to the representative entropy value.

[0014] For example, the instructions, when executed by the one or more processors, may further cause the device to: obtain the entropy of at least one bounding box in the image; and determine the second reliability score. The second reliability score may, for example, be inversely proportional to the entropy of the at least one bounding box.

[0015] For example, the instructions, when executed by the one or more processors, may cause the device to determine whether to label the image by: determining the first reliability score and the second reliability score based on the occluding object being detected by learning of the auxiliary network and the auxiliary network, respectively; and determining to label the image for the auxiliary network's learning based on at least one of the first reliability score and the second reliability score being less than a (e.g., associated) threshold.

[0016] For example, the instructions, when executed by the one or more processors, may cause the device to exclude the image from the training data of the interest network and the auxiliary network based on the first reliability score and the second reliability score being greater than or equal to a (e.g., associated) threshold.

[0017] For example, the instructions, when executed by the one or more processors, may cause the device to determine whether to label the image by: determining the first confidence score based on the occluding object not being detected or recognized by learning the auxiliary network (e.g., no occluding object is present); and determining to label the image for learning the auxiliary network based on the first confidence score being less than a (e.g., associated) threshold.

[0018] For example, the instructions, when executed by the one or more processors, may further cause the device to: determine the second reliability score based on the first reliability score being greater than or equal to a (e.g., associated) threshold; and determine to add a new class of the network of interest based on the second reliability score being less than the threshold.

[0019] For example, the instructions, when executed by the one or more processors, may further cause the device to determine whether to label the image by: determining to label the image for training the network of interest based on the first reliability score and the second reliability score being greater than or equal to a threshold (e.g., when the first reliability score is greater than or equal to a first threshold and when the second reliability score is greater than or equal to a second threshold).

[0020] For example, the instructions, when executed by the one or more processors, may further cause the device to determine whether the occluding object is detected based on the first confidence score and the second confidence score being less than a (e.g., associated) threshold.

[0021] According to one or more example embodiments of the present disclosure, a method may include: detecting, via an interest network, an object of interest corresponding to a predetermined class by learning an image of a vehicle, obtaining, via the interest network, one or more confidence scores with which the object of interest is detected, performing, via an auxiliary network, a learning operation associated with the image and detecting an occluding object in the image that influences a learning result of the interest network, and determining whether to label the image based on: whether the occluding object is detected and the one or more confidence scores.

[0022] For example, detecting the object of interest may comprise determining a first confidence score indicative of an accuracy of segmentation and determining a second confidence score indicative of an accuracy of detecting the object of interest output in a bounding box.

[0023] Determining the first reliability score may, for example, comprise obtaining the entropy of each pixel of the image, determining a representative entropy value based on the entropy of each pixel of the image, and determining the first reliability score. The first reliability score may, for example, be inversely proportional to the representative entropy value.

[0024] Determining the second reliability score may, for example, comprise obtaining the entropy of the at least one bounding box in the image and determining the second reliability score. The second reliability score may, for example, be inversely proportional to the entropy of the at least one bounding box.

[0025] Determining whether the image should be labeled may, for example, comprise: determining the first reliability score and the second reliability score based on the fact that the occluding object is detected by means of learning of the auxiliary network, and determining that the image should be labeled for the learning of the auxiliary network based on at least one of the first reliability score and the second reliability score being smaller than a (eg associated) threshold.

[0026] The method may comprise, for example: excluding the image from the training data of the interest network and the auxiliary network based on the first reliability score and the second reliability score being greater than or equal to a (e.g., associated) threshold.

[0027] Determining whether to label the image may, for example, comprise determining the first confidence score based on the occluding object not being detected by learning the auxiliary network, and determining that the image should be labeled for learning the auxiliary network based on the first confidence score e being smaller than a (e.g., associated) threshold.

[0028] Detecting the object of interest may, for example, comprise: determining the second reliability score based on the first reliability score being greater than or equal to a (e.g., associated) threshold. The method may, for example, further comprise: determining whether a new class of the interest network should be added to the interest network based on the second reliability score being less than the threshold.

[0029] Determining whether to label the image may, for example, comprise: determining to label the image to train the interest network based on the first confidence value and the second confidence value being greater than or equal to a (e.g., associated) threshold.

[0030] Determining the first reliability score and the second reliability score may, for example, be performed before detecting the occluding object. Detecting the occluding object may, for example, comprise detecting the occluding object based on the first reliability value and the second reliability value being smaller than a (e.g., associated) threshold value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Fig. 1 is a schematic diagram showing a method for collecting training data for image learning, Fig. 2 is a diagram showing a vehicle capable of obtaining an image from outside a vehicle while driving, Fig. 3 is a diagram showing a method for constructing a learning model for image learning, Fig. 4 is a block diagram showing the configuration of a curating device, Fig. 5 is a diagram showing image segmentation performed by the first network, Fig. 6 is a diagram showing a bounding box output by the second network, Fig. 7 is a diagram showing a type of detection-threatening object detected in the auxiliary network, Fig. 8 is a flowchart showing a method for collecting training data, Fig. 9 is a flowchart showing a method for determining whether to proceed with labeling an image, Fig. 10 is a diagram showing an example of an image that was excluded from the training data, Fig. 11 is a flowchart showing a method for determining whether to proceed with labeling an image, and Fig. 12 is a diagram showing a method for determining whether to proceed with labeling an image. DETAILED DESCRIPTION

[0032] Hereinafter, one or more embodiments of the present disclosure will be described in detail with reference to the example drawings. When adding reference numerals to the components of each drawing, it should be noted that the identical or equivalent component will be denoted by the identical reference numeral even if they are illustrated in other drawings. Furthermore, in describing the example embodiment of the present disclosure, a detailed description of the related known configuration or function will be omitted if it is determined that it would interfere with the understanding of the example embodiment of the present disclosure.

[0033] In addition, terms such as first, second, A, B, (a), (b), or the like may be used herein when describing components of the present disclosure. The terms are used only to distinguish the elements from other elements, and the contents, sequences, orders, and numbers of the elements are not limited by the terms. Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure relates.The terms defined in commonly used dictionaries should be interpreted to have meanings consistent with the meanings of the related art contexts and should not be interpreted as having ideal or overly formal meanings unless they are clearly defined in the description of the present disclosure.

[0034] Hereinafter, one or more exemplary embodiments of the present disclosure will be described with reference to the Fig. 1 to 12 described in detail.

[0035] Fig. Figure 1 is a schematic diagram showing a method for collecting training data for image learning. Fig. Figure 2 is an illustration showing a vehicle capable of taking an image from outside the vehicle while driving.

[0036] Referring to Fig. 1, a server SV can receive images of vehicles to collect training data for image learning or training based on images.

[0037] As in Fig. 2, each of the vehicles VEH1, VEH2 and VEH3 may have at least one of a camera 11, a lidar (Light Imaging Detection and Ranging) 12 and a radar (Radio Detection and Ranging) 13 to detect an object outside a vehicle VEH.

[0038] The camera 11 used to capture an external image of the vehicle VEH can capture a front image or front and side images from the vehicle VEH. For example, the camera 11 can be arranged near / on the windshield to obtain a front image of the vehicle VEH.

[0039] The lidar 12, which is designed to emit a laser and detect an object based on the laser wave reflected from the object, can be implemented in a time-of-flight (TOF) or phase-shift scheme. The lidar 12 can be mounted so that it is exposed to the exterior of the vehicle and can be located near / on the front bumper and / or front grille.

[0040] The radar 13 can have a transmitter module for electromagnetic waves and a receiver module. The radar 13 can be designed as a pulse radar or as a continuous wave radar, which are based on the principle of transmitting radio waves. Depending on the signal shape among the continuous wave radar methods, the radar 13 can be implemented as a frequency-modulated continuous wave radar (FMCW) or a frequency-shift keying radar (FSK). The radar 13 can have a front radar 13-1 located in the front center of the vehicle VEH, a front side radar 13-2 located at both (e.g., transverse) ends of the front bumper, and a rear radar 13-3 located at the rear of the vehicle VEH.

[0041] The arrangement of the camera 11, the lidar 12 and the radar 13 does not have to be based on the Fig. 2 positions shown.

[0042] In addition, the vehicle VEH sensors may include an ultrasonic sensor and an infrared sensor.

[0043] The images provided by the vehicles VEH1, VEH2 and VEH3 to the server SV may, for example, be data obtained by merging the image recorded by the camera 11 and the information recorded by the lidar 12 and / or the radar 13.

[0044] The server SV can perform deep learning on images provided by the vehicles VEH1, VEH2 and VEH3 by using an interest network.

[0045] Furthermore, the server SV can learn images using an auxiliary network and detect an occluding object in the images. As used herein, an occluding object in an image can be any object or artifact that negatively impacts object detection within the image (e.g., obstructs, prevents, obscures, etc.). Occluding objects can be factors that reduce the accuracy of deep learning (e.g., training) of the interest network and can include dust, rainwater, liquid, light blur / brightness spots, contaminants, and the like.

[0046] Depending on the learning results of the interest network and the auxiliary network, the server SV can determine whether the image should be used as training data for the interest network or the auxiliary network.

[0047] The server SV can forward the image determined as training data to an identifier LB (e.g. device that provides the image with a mark, e.g. identification information such as a label, a tag, etc.).

[0048] Fig. Figure 3 is a diagram showing a method for constructing a learning model for image learning.

[0049] With reference to Fig. 3, a learning model for image learning based on machine learning operations (MLOPs) can be designed accordingly.

[0050] In S310, data acquisition or collection may be a method for obtaining environmental information. The environmental information may, for example, be images captured while the vehicle is moving or in a simulation environment.

[0051] In S320, the data preprocessing may be an operation for processing an image. The data preprocessing may include operations for converting and processing the dimensions of the image and may include operations for adding metadata to the image. Furthermore, the data preprocessing may include a method for inserting / merging LIDAR or radar information into / with the image.

[0052] In S330, data curation may be a process of selecting data for network learning.

[0053] In S340, the data labeling may be a method for labeling images that have been determined as training data (e.g., determined as such by labeling), and may be an operation that is performed, for example, by a person.

[0054] In S350, network training may be a method for learning from or using images (e.g., training the network) for which a labeling process has been completed.

[0055] In S360, network evaluation can be a method for evaluating a network model based on an image learning result.

[0056] In S370, model monitoring may be a method for monitoring the predictive performance of a network model.

[0057] As in Fig. 4 and described below, the server SV may comprise a curation device 100 and select training data using the curation device 100. The curation device 100 is described below with reference to Fig. 4 described.

[0058] Fig. Figure 4 is a block diagram showing the configuration of a curation device.

[0059] With reference to Fig. 1 and Fig. 4, the curation device 100 may include a communication device 101, an interest network 110, an auxiliary network 120, and a processor 130.

[0060] The communication device 101 can receive images from the vehicles VEH1, VEH2, and VEH3 or from outside. The communication device 101 can deliver images to the interest network 110 and the auxiliary network 120.

[0061] The communication device 101 can transmit and receive radio signals with the vehicles VEH1, VEH2, and VEH3 via a mobile communication network constructed according to technical standards or communication schemes for mobile communication. For example, the communication device 101 can implement communication based on the global system for mobile communication (GSM), code division multiple access (CDMA), code division multi-access 2000 (CDMA2000), enhanced voice-data optimized or enhanced voice-data only (EV-DO), wideband CDMA (WCDMA), high-speed downlink packet access (HPE), and other technologies.high speed downlink packet access (HSDPA for short), high speed uplink packet access (HSUPA for short), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA) and the like.

[0062] The interest network 110 can detect objects (e.g., objects of interest) or classify classes of pixels through deep learning images. For this purpose, the interest network 110 can comprise at least one of a first network 111 and a second network 112. The learning or training results of the interest network 110 are described below with reference to Fig. 5 and Fig. 6 described.

[0063] Fig. Figure 5 is a diagram showing the image segmentation performed by the first network. Fig. Figure 6 is a diagram showing a bounding box output by the second network.

[0064] The first network 111 of the interest network 110 may be a network model that performs image segmentation. As in Fig. As shown in Figure 5, the first network 111 can perform image segmentation and assign a class to each pixel of an image. The marking (e.g., a label, tag, etc.) can be used to classify a predetermined class.

[0065] As in Fig. 6, the second network 112 of the interest network 110 can learn an image and express the position of the object detected in the image as a rectangular box.

[0066] Furthermore, the interest network 110 may obtain (e.g., determine) a reliability score according to the deep learning results of images. For example, the first network 111 may obtain or determine a first reliability score according to the deep learning result, and the second network 112 may obtain or determine a second reliability score. The first reliability score may indicate the accuracy of the segmentation. Furthermore, the second reliability score may indicate the accuracy of the object detection output in the form of a bounding box.

[0067] The auxiliary network 120 can detect an occluding object that influences the learning or training result of the interest network.

[0068] Fig. Figure 7 is a diagram showing (by way of example) the type of occluding object detected in the auxiliary network.

[0069] Referring to Fig. 7, the auxiliary network 120 can detect dust as in Case 1. In addition, the auxiliary network 120 can detect water droplets as in Case 2 and brightness blur or brightness spots as in Case 3.

[0070] The auxiliary network 120 can detect occluding objects in units (e.g., pixel groups) of pixel units in the image.

[0071] Processor 130 may determine whether to proceed with labeling an image based on whether an occluding object is detected and the confidence score. Image labeling may be a process of assigning a specific value to the training data prior to image learning. Image labeling may be a necessary procedure to generate correct response data for supervised learning.

[0072] For this purpose, the processor 130 may include an artificial intelligence (AI) processor for learning from or training with images. The AI ​​processor may learn or train a neural network using a pre-stored program. A neural network for detecting a target vehicle (e.g., the vehicle's own vehicle) and a dangerous vehicle (e.g., a vehicle other than the vehicle's own vehicle) may be designed to simulate the structure of a human brain on a computer and may include a plurality of network nodes whose weights simulate the neurons of a human neural network. A plurality of network nodes may send and receive data according to a connection relationship to enable a neuron to simulate the synaptic activity of the neuron that sends and receives a signal across a synapse.The neural network may include a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes may transmit and receive data according to a convolutional connection relationship while located in different layers. For example, a neural network model may include various deep learning schemes, such as a deep neural network (DNN), a convolutional deep neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a deep Q-network, and the like. An interest network (e.g., a network for objects of interest such as other vehicles, pedestrians, etc.) may be a neural network implemented with one or more processors. An auxiliary network (e.g., a network for occluding objects such as dust, water droplets, etc.) may be a neural network implemented with one or more processors.) can be a neural network implemented with one or more processors.

[0073] The processor 130 may include a memory (not shown) for storing an AI processor, an algorithm, and the like. The memory may use a hard disk drive, flash memory, electrically erasable programmable read-only memory (EEPROM), static RAM (SRAM), ferroelectric RAM (FRAM), phase-change RAM (PRAM), magnetic RAM (MRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double date rate SDRAM (DDR-SDRAM), and the like.

[0074] Image labeling can be performed on images that were determined as training data.

[0075] Fig. Figure 8 is a flowchart showing a method for collecting training data. The Fig. The processes shown in Figure 8 may be processes that are described in the Fig. 3. Below, a method for collecting training data is described with reference to Fig. 8 described.

[0076] In S810, the interest network 110 may learn or train an image and obtain or determine a reliability score according to the learning result.

[0077] The reliability score may include at least one of the first reliability score output by the first network 111 and the second reliability score output by the second network 112.

[0078] The first reliability assessment can indicate the accuracy of the segmentation.

[0079] To obtain the initial confidence score, the first network can obtain or determine the entropy of each pixel in the image. Entropy can represent uncertainty in labeling the pixels. For example, as a result of segmentation, a first pixel may have a 25% probability of being a human and a 20% probability of being a dog. If a second pixel has an 85% probability of being a human and a 2% probability of being a dog, the entropy of the first pixel can be calculated to be higher than that of the second pixel.

[0080] The first network 111 can obtain a representative entropy value of an image based on the entropy of each pixel. For example, the representative entropy value e can be the average entropy of the pixels present in the image. Alternatively, the representative entropy value can be the total entropy of the pixels.

[0081] The second reliability score can be calculated based on the entropy of the bounding box.

[0082] The entropy of the bounding box may be calculated based on an uncertainty of the bounding box. For example, the second network 112 may obtain or determine a confidence value indicating the probability that an object exists within the bounding box. The entropy of the bounding box may be inversely proportional to the confidence value, and the second confidence score may be determined to be proportional to the confidence value.

[0083] In S820, the auxiliary network 120 can learn an image or train based on the image and detect an occluding object (e.g., an object that jeopardizes recognition).

[0084] In S830, the processor 130 may determine whether to proceed with labeling the image based on whether an occluding object is detected and the confidence score output by the interest network.

[0085] As described herein, in the Mlops process, it is possible to use the image learning results of the interest network and the auxiliary network to collect training data of the deep learning network. The training data can be selected from images taken while vehicles are driving. In particular, since the training data is selected based on the reliability score obtained by image learning, it is possible to select images that influence the learning of the interest network or the auxiliary network. Therefore, it is possible to select training data that has a greater impact on object detection performance. Furthermore, since the images for learning occluding objects are not manually created by humans, it is also possible to reduce the time and cost of preparing the training data.

[0086] The following is an example of determining whether to proceed with tagging an image.

[0087] Fig. Figure 9 is a flowchart showing a method for determining whether to proceed with labeling an image. Fig. 9 shows the methods performed by a processor. With reference to Fig. 9, a method for determining whether to proceed with labeling an image is described below.

[0088] In operation S901, after learning the interest network for an image is completed, the processor 130 may determine whether both the first confidence score and the second confidence score are equal to or greater than a threshold confidence score.

[0089] The threshold reliability score can be determined or set as the degree at which the accuracy of the learning result of the first network and the learning result of the second network can be questioned. For example, even if the class is classified by the learning of the first network, the accuracy cannot be guaranteed. Furthermore, even if the bounding box and the class of the bounding box are consistent by the learning of the second network, the accuracy cannot be guaranteed. The threshold reliability score can be set, for example, to approximately 50%.

[0090] In operation S902, the processor 130 may determine whether an obscuring object (e.g., one that jeopardizes image recognition) is detected in an image.

[0091] In operation S903, the processor 130 may determine whether the first confidence score is less than the first threshold based on the detected occluding object.

[0092] As in Fig. 7, the processor 130 may determine the first reliability score when the auxiliary network 120 detects an obscuring object such as dust, water droplets, brightness blur or spot(s), and the like.

[0093] In S904, the processor 130 may determine the image as training data for the auxiliary network based on the first reliability score being less than the first threshold. Furthermore, the processor 130 may determine that labeling of an image should be performed.

[0094] The first threshold may be set to a level at which the class assigned to the pixel is determined to be uncertain based on the segmentation result, and may be set to a level lower than the threshold confidence score.

[0095] Although an occluding object is detected by the auxiliary network, if the first reliability evaluation of the first network is low, it may be determined that segmentation is not performed smoothly due to the occluding object. Therefore, since the image classified in operation S904 causes a decrease in the recognition performance of the first network, the image may be selected as a target for occluding object learning.

[0096] In S905, the processor 130 may identify the second reliability score based on the first reliability score being greater than or equal to the first threshold.

[0097] In S906, the processor 130 may determine the image as training data for the auxiliary network based on the second reliability score being less than the second threshold. Furthermore, the processor 130 may determine whether to perform labeling of the image.

[0098] The second threshold may be set to a degree at which the bounding box detected by the second network can be determined to be uncertain and may be set to a degree lower than the threshold reliability score.

[0099] When an occluding object is detected by the auxiliary network and the second confidence score of the second network is low, it may be determined that the bounding box is not uniformly constructed due to the occluding object. Therefore, since the image classified in operation S906 causes a decrease in the detection performance of the second network, the occluding object may be selected as the target for occluding object learning.

[0100] In S907, the processor 130 may exclude an image from the training data based on the second reliability score being greater than or equal to the second threshold.

[0101] Fig. Figure 10 is a diagram showing an example of an image excluded from the training data.

[0102] In Fig. 10, an original image taken by a vehicle may be an image obtained by capturing water drops (in the image).

[0103] In S907, the classified image does not have a very high learning reliability by the interest network, but it can be determined that the water drops do not affect the classification (e.g. segmentation), as in Fig. 10 shown.

[0104] As described herein, even if an occluding object is detected in the image and if it does not significantly affect the recognition performance of the interest network, the processor 130 may decide to exclude the image from the training data.

[0105] Fig. Figure 11 is a flowchart showing a method for determining whether to proceed with labeling an image. Fig. 11 is a diagram showing the operations performed on an image whose first reliability score and second reliability score are greater than or equal to the threshold reliability score after the learning or training of the interest network is completed. Fig. 12 shows a method for determining whether to proceed with labeling an image. Referring to Fig. 11 and Fig. 12, a method for determining whether to proceed with labeling an image is described below.

[0106] In operations S1101 and S1102, the processor 130 may determine whether the first confidence score is less than the first threshold based on the occluding object not being detected in the image.

[0107] In S1103, the processor 130 may determine the image as training data of the auxiliary network based on the fact that the first reliability score is less than the first threshold. Furthermore, the processor 130 may determine whether to perform labeling of the image.

[0108] Even if an occluding object is not detected by the auxiliary network, if the initial reliability score of the first network is low, it can be determined that the segmentation is not performed evenly due to the occluding object. As shown in Fig. For example, as shown in Figure 12, a vehicle VEH_g included in the image may not be recognized by the first network even though no occluding objects are detected. Such a phenomenon may occur when the entire image is blurred due to light rain or dust, or when occluding objects are not detected and distant objects are not recognized. Since the image classified in operation S1103 causes a decrease in the recognition performance of the first network, the image may be selected as a target for occluding object learning.

[0109] In operation S1104, the processor 130 may determine whether the second reliability score is less than the second threshold based on the first reliability score being greater than or equal to the first threshold.

[0110] In operation S1105, the processor 130 may determine whether to add a new class based on the second reliability score being less than the second threshold.

[0111] If the confidence score of the bounding box is low compared to the segmentation result, the classification of the detected object may be uncertain. For example, as in Fig. As shown in Figure 12, although the second network detects a vehicle and creates a first bounding box Bbox1, it may not be able to accurately determine the type of the vehicle. As a result, it may be determined that the class of the vehicle categorized by the first bounding box is not predefined. Accordingly, the processor 130 may determine that a new class needs to be added to the classified image in operation S1105.

[0112] In S1106, the processor 130 may determine the image as training data of the interest network based on the second reliability score being greater than or equal to the second threshold.

[0113] If no occluding objects are detected in the image and the detection performance of the interest network is determined to be acceptable, the image may be determined as training data of the interest network 110, thereby improving the object detection performance of the interest network 110. Operation S1106 may include an operation for determining a labeling task to use an image as training data for the interest network 110.

[0114] Since the training data is selected based on the results of detecting occluding objects in images taken while a vehicle is moving, it is possible to select training data that directly affects the improvement of the detection performance of the image learning network.

[0115] Furthermore, since the training data of the image learning network can be selected without having to generate images with occluding objects, the effort and time required can be significantly reduced.

[0116] In addition, various effects directly or indirectly understood by the present disclosure can be provided.

[0117] Although one or more embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope of the disclosure.

[0118] Therefore, the embodiments disclosed in the present disclosure are for descriptive purposes only and do not limit the technical concepts of the present disclosure, and it should be understood that such embodiments are not intended to limit the scope of the technical concepts of the present disclosure. The scope of the present disclosure should be understood by the following claims, and all technical concepts within the corresponding ranges should be construed as being within the scope of the present disclosure. LIST OF REFERENCE SYMBOLS 100 Curation Device 101 Communication device 110 Interest Network 111 first network 112 second network 120 support network 130 processor VEH (VEH1, VEH2, VEH3) vehicle Bbox1, Bbox2 bounding box

Claims

[1] A device comprising: one or more processors (130) and a memory storing instructions that, when executed by the one or more processors (130), cause the device to: Recognizing, by means of an interest network (110), an object of interest corresponding to a predetermined class by learning an image provided by a vehicle (VEH; VEH1, VEH2, VEH3), and Obtaining, by means of the interest network (110), one or more reliability scores indicating the reliability with which the object of interest is detected, Performing, by means of an auxiliary network (120), a learning process associated with the image and detecting an occluding object in the image that influences a learning result of the interest network (110), and Determine whether the image should be labeled based on: whether the obscuring object is detected and one or more reliability assessments. [2] The apparatus of claim 1, wherein the instructions, when executed by the one or more processors (130), cause the apparatus to determine whether to label the image based on the one or more reliability scores by determining whether to label the image based on: an initial reliability assessment indicating the accuracy of segmentation of the image, and a second confidence score indicating an accuracy of detecting the object of interest output in a bounding box (Bbox 1, Bbox2). [3] The apparatus of claim 2, wherein the instructions, when executed by the one or more processors (130), further cause the apparatus to: Obtaining an entropy of each pixel of the image, Determining, based on the entropy of each pixel of the image, a representative entropy value, and Determining the first reliability score, wherein the first reliability score is inversely proportional to the representative entropy value. [4] The apparatus of claim 2 or 3, wherein the instructions, when executed by the one or more processors (130), further cause the apparatus to: Obtaining an entropy of at least one bounding box (Bbox1, Bbox2) in the image and Determining the second reliability score, wherein the second reliability score is inversely proportional to the entropy of the at least one bounding box (Bbox1, Bbox2). [5] Apparatus according to any one of claims 2 to 4, wherein the instructions, when executed by the one or more processors (130), cause the apparatus to determine whether the image is to be labelled by Determining the first reliability score and the second reliability score on the basis that the occluding object is detected by learning the auxiliary network (120), and Determining to label the image for learning of the auxiliary network (120) based on at least one of the first reliability score and the second reliability score being less than a threshold. [6] The apparatus of any one of claims 2 to 5, wherein the instructions, when executed by the one or more processors (130), cause the apparatus to: Excluding the image from the training data of the interest network (110) and the auxiliary network (120) based on the first reliability score and the second reliability score being greater than or equal to a threshold. [7] Apparatus according to any one of claims 2 to 6, wherein the instructions, when executed by the one or more processors (130), cause the apparatus to determine whether the image is to be labelled by Determining the first reliability assessment based on the fact that the occluding object is not detected by learning the auxiliary network (120), and Determine to label the image for learning of the auxiliary network (120) based on the first reliability score being less than a threshold. [8] The apparatus of any one of claims 2 to 7, wherein the instructions, when executed by the one or more processors (130), further cause the apparatus to: Determining the second reliability rating based on the first reliability rating being greater than or equal to a threshold, and Determine to add a new class of the interest network (110) based on the second reliability score being less than the threshold. [9] The apparatus of any one of claims 2 to 8, wherein the instructions, when executed by the one or more processors (130), further cause the apparatus to determine whether the image is to be labeled by: Determine to label the image for learning the interest network (110) based on the first confidence score and the second confidence score being greater than or equal to a threshold. [10] The apparatus of any one of claims 2 to 9, wherein the instructions, when executed by the one or more processors (130), further cause the apparatus to determine whether the occluding object is detected based on the first reliability score and the second reliability score being less than a threshold. [11] A method comprising: Recognizing (S810), by means of an interest network (110), an object of interest corresponding to a predetermined class by learning an image of a vehicle (VEH; VEH1, VEH2, VEH3), Obtaining, by means of the interest network (110), one or more reliability ratings with which the object of interest is recognized, Performing, by means of an auxiliary network (120), a learning process associated with the image and detecting an occluding object in the image that affects a learning result of the interest network (110), and Determine (S810) whether the image is to be marked based on: whether the obscuring object is detected and one or more reliability assessments. [12] The method of claim 11, wherein detecting (S810) the object of interest comprises: Determining an initial reliability score indicating the accuracy of a segmentation, and Determining a second confidence score indicating an accuracy of detecting the object of interest output in a bounding box (Bbox1, Bbox2). [13] The method of claim 12, wherein determining the first reliability rating comprises: Obtaining an entropy of each pixel of the image, Determining a representative entropy value based on the entropy of each pixel of the image and Determining the first reliability score, wherein the first reliability score is inversely proportional to the representative entropy value. [14] The method of claim 12 or 13, wherein determining the second reliability rating comprises: Obtaining an entropy of at least one bounding box (Bbox1, Bbox2) in the image and Determining the second reliability score, wherein the second reliability score is inversely proportional to the entropy of the at least one bounding box (Bbox1, Bbox2). [15] A method according to any one of claims 12 to 14, wherein determining whether the image is to be labeled comprises: Determining the first reliability score and the second reliability score on the basis that the occluding object is detected by learning the auxiliary network (120), and Determining to label the image for learning of the auxiliary network (120) based on at least one of the first reliability score and the second reliability score being less than a threshold. [16] A method according to any one of claims 12 to 15, further comprising: Excluding the image from the training data of the interest network (110) and the auxiliary network (120) based on the first confidence score and the second confidence score being greater than or equal to a threshold. [17] The method of any one of claims 12 to 16, wherein determining whether the image is to be labeled comprises Determining the first reliability assessment based on the fact that the occluding object is not detected by learning the auxiliary network (120), and Determine to label the image for learning of the auxiliary network (120) based on the first reliability score being less than a threshold. [18] A method according to any one of claims 12 to 17, wherein detecting the object of interest comprises: Determining the second reliability rating based on the first reliability rating being greater than or equal to a threshold, and the method further comprising: Determine to add a new class of the interest network (110) based on the second reliability score being less than the threshold. [19] A method according to any one of claims 12 to 18, wherein determining whether the image is to be labeled comprises: Determine to label the image for learning the interest network (110) based on the first confidence score and the second confidence score being greater than or equal to a threshold. [20] A method according to any one of claims 12 to 19, wherein determining the first reliability score and the second reliability score is performed before detecting the occluding object, and wherein detecting the occluding object comprises detecting the occluding object based on the first reliability score and the second reliability score being less than a threshold.

Images