Object detection method and apparatus

Abstract

To disclose an object detection method and apparatus.SOLUTION: A method includes the steps of: updating a pre-trained model based on sensing data of an image sensor; performing pseudo labeling using an interim model which provides an individual training set; determining confidence thresholds based on an evaluation of the interim model; performing multiple trainings using the interim model and a pseudo labeled data generated through the pseudo labeling, by applying the determined confidence thresholds to the multiple trainings; and generating an object detection model dependent on the performance of the multiple trainings: The generation step includes a step of generating an initial candidate object detection model when the interim model is the updated model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The following disclosure relates to an object detection method and apparatus. [Background technology]

[0002] Object detection models can be generated through a labeling process that uses deep learning to assign labels to objects within an image, and then by training the model using the labeled images. Such labeling processes are time-consuming and resource-intensive, and it takes a long time for an object detection model to be generated.

[0003] When a new image sensor is introduced, an object detection model corresponding to the new image sensor is required, and the labeling and training operations described above are necessary to generate such an object detection model. [Overview of the project] [Problems that the invention aims to solve]

[0004] The object of the present invention is to provide an object detection method and apparatus. [Means for solving the problem]

[0005] A method performed by a processor according to one embodiment includes the steps of updating a pre-trained model based on detection data from an image sensor; performing pseudo-labeling using an intermediate model that provides individual training sets; determining a confidence threshold based on an evaluation of the intermediate model; applying the determined confidence threshold to multiple training sessions and performing multiple training sessions using pseudo-label data generated via pseudo-labeling and the intermediate model; and generating an object detection model that depends on the execution of multiple training sessions, the generation step including, if the intermediate model is an updated model, generating an initial candidate object detection model.

[0006] The update step may include updating the first layer of a model that has been pre-trained using the detection data.

[0007] The first layer may be a batch normalization layer.

[0008] The update step may include a step to perform image adjustments on the detected data.

[0009] The image adjustment step may include a step to adjust the brightness distribution of the detected data.

[0010] A pre-trained model is based on corresponding detection data in a different format than the detection data, and the update step may include a step to convert the detection data to another format.

[0011] Each of the training sessions may include running an intermediate model that provides generated pseudo-labeled data to obtain individual labeling results for the intermediate model, and each of the training sessions may include additional training based on at least the individual labeling results.

[0012] A pre-trained model can be based on detection data from other image sensors that have different characteristics from the image sensor.

[0013] Multiple training iterations are repeated collectively multiple times after the initial iteration, when the intermediate model is an updated model. In the generation step in the immediately preceding iteration, the intermediate model is the previous candidate object detection model. The generation step further includes a step in the last iteration to generate another candidate object detection model, and the previous candidate object detection model in the iteration immediately following the initial iteration may be the initial object candidate detection model.

[0014] The generation step includes selecting an object detection model from among multiple candidate object detection models based on a performance comparison of multiple candidate object detection models, and the multiple candidate object detection models may include the initial candidate object detection model, previous candidate object detection models generated in multiple iterations excluding the iteration immediately following the initial iteration, and other candidate object detection models.

[0015] The method may further include the step of performing object detection using the generated object detection model by the vehicle.

[0016] The evaluation of the intermediate model includes a step of determining multiple evaluation scores from individual runs of the intermediate model using multiple thresholds and individual validation sets, and the step of determining confidence thresholds may include a step of determining the threshold used to determine the highest evaluation score among the determined evaluation scores as the first confidence threshold, a step of determining a second confidence threshold greater than the determined first confidence threshold, and a step of determining a third confidence threshold less than the determined first confidence threshold.

[0017] Each of the multiple trainings has a trained model result, and in each of the multiple trainings, the generating step can generate a corresponding candidate object detection model by performing an ensemble on the individually trained model results.

[0018] The generating step can include the step of performing an ensemble on the individual model results of the multiple trainings to generate an initial candidate object model when the intermediate model is the updated model.

[0019] The evaluation of the intermediate model includes the step of determining a plurality of evaluation scores from the individual executions of the intermediate model using a plurality of thresholds and individual verification sets. The step of determining the confidence threshold includes the step of determining, as the first confidence threshold, the threshold used for determining the maximum evaluation score among the determined evaluation scores, the step of determining a second confidence threshold greater than the determined first confidence threshold, and the step of determining a third confidence threshold smaller than the determined first confidence threshold.

[0020] The step of performing multiple trainings includes the step of performing a first training to which the determined first confidence threshold is applied using the intermediate model and the generated pseudo-label data, the step of performing a second training to which the determined second confidence threshold is applied using the intermediate model and the generated pseudo-label data, and the step of performing a third training to which the determined third confidence threshold is applied using the intermediate model and the generated pseudo-label data.

[0021] The step of performing pseudo-labeling includes a step of performing pseudo-labeling based on the first unlabeled training set as an individual training set and the updated model to generate first pseudo-labeled data, and a step of performing pseudo-labeling based on the second unlabeled training set as an individual training set and the initial candidate object detection model to generate second pseudo-labeled data. The method further includes a step of evaluating the initial candidate object detection model and a step of determining a confidence threshold for the second pseudo-labeled data generated based on the evaluation result of the initial candidate object detection model. The step of performing multiple trainings uses the initial candidate object detection model and the generated second pseudo-labeled data to perform multiple second trainings among the multiple trainings, and includes a step of applying each of the confidence thresholds for the generated second pseudo-labeled data to each of the multiple second trainings to perform the multiple second trainings. The step of generating can include a step of generating a second candidate object detection model using the results of each of the multiple second trainings.

[0022] The first unlabeled training set and the second unlabeled training set may be the same or different training sets from each other.

[0023] Among multiple times, the steps of repeating multiple times the execution of pseudo-labeling using the previous candidate object detection model in the corresponding round, the evaluation of the intermediate model in the corresponding round, the execution of multiple trainings in the corresponding round related to the intermediate model, and the generation of another candidate object detection model based on the results of the multiple trainings in the corresponding round, and further including a step of generating an object detection model by selecting an object detection model from among the multiple candidate object detection models based on the performance comparison of the multiple candidate object detection models can be included.

[0024] Multiple candidate object detection models may include an initial candidate object detection model, previous candidate object detection models from previous iterations, and other object detection models from the last of the previous iterations.

[0025] An apparatus according to one embodiment includes a memory for storing an object detection model and a processor that receives an image from an image sensor and performs object detection using the received image and the object detection model. For the generation of the object detection model, the processor performs operations to update a pre-trained model and to perform pseudo-labeling using an intermediate model that provides individual training sets in order to generate pseudo-label data.

[0026] The system can perform the following operations: determining multiple confidence thresholds based on the evaluation of an intermediate model; applying each of the determined confidence thresholds to each of the multiple training iterations; performing multiple training iterations using the generated pseudo-label data and the intermediate model; and generating an object detection model that depends on the execution of multiple training iterations.

[0027] The generation process includes generating an initial candidate object detection model if the intermediate model is an updated model.

[0028] The update operation may include updating the first layer of a pre-trained model using detection data from another image sensor having the same characteristics as the image sensor.

[0029] The first layer may be a batch normalization layer.

[0030] The update operation may include operations that perform image adjustments on the detected data.

[0031] The evaluation operation includes determining multiple evaluation scores for an intermediate model using multiple thresholds and individual validation sets, and the operation to determine confidence thresholds may include determining the threshold used to determine the highest evaluation score among the determined evaluation scores as the first confidence threshold, determining a second confidence threshold greater than the determined first confidence threshold, and determining a third confidence threshold less than the determined first confidence threshold.

[0032] The operation of performing multiple training steps may include: an operation to perform training using a first confidence threshold determined using the intermediate model and the generated pseudo-label data; an operation to perform training using a second confidence threshold determined using the intermediate model and the generated pseudo-label data; and an operation to perform training using a third confidence threshold determined using the intermediate model and the generated pseudo-label data.

[0033] The process of generating a pre-object detection model may include the process of ensembling the results of multiple training sessions to generate an initial candidate object detection model.

[0034] The processor can adjust the brightness distribution of the received image using an object detection model.

[0035] The device may further include an image sensor.

[0036] The device in question may be a vehicle. [Effects of the Invention]

[0037] According to the present invention, an object detection method and apparatus can be provided. [Brief explanation of the drawing]

[0038] [Figure 1] This figure illustrates an object detection model generation method according to one embodiment. [Figure 2]This figure illustrates a model update in an object detection model generation method according to one embodiment. [Figure 3] This figure illustrates pseudo-labeling in an object detection model generation method according to one embodiment. [Figure 4] This figure illustrates the evaluation process in an object detection model generation method according to one embodiment. [Figure 5] This figure illustrates the training process in an object detection model generation method according to one embodiment. [Figure 6] This figure illustrates the training process in an object detection model generation method according to one embodiment. [Figure 7] This is a flowchart of an object detection model generation method according to one embodiment. [Figure 8] This is a block diagram illustrating a computing device according to one embodiment. [Figure 9] This is a block diagram illustrating a computing device according to one embodiment. [Figure 10] This is a diagram illustrating the operation of a vehicle with object detection according to one embodiment. [Figure 11] This is a diagram illustrating a vehicle according to one embodiment. [Modes for carrying out the invention]

[0039] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be modified in various ways. Therefore, the embodiments are not limited to any particular disclosure, and the scope of this specification includes modifications, equivalents, or substitutions that are part of the technical idea.

[0040] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0041] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0042] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0044] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations will be omitted.

[0045] An "object" refers to the target of image detection, such as a person or a thing. The type of "object" to be detected may be predefined to a specific type (e.g., a person, a vehicle, an animal), or there may be no restrictions on the type.

[0046] The object detection model may be based on a Deep Convolutional Neural Network (DCNN). A DCNN includes one or more convolutional layers, one or more pooling layers, and one or more fully connected layers. Here, DCNN is just one embodiment, and the object detection model may be based on a neural network of a different structure than DCNN.

[0047] Figure 1 is a diagram illustrating an object detection model generation method according to one embodiment.

[0048] Referring to Figure 1, the computing device 100 performs, for example, model updates 110, pseudo-labeling 120, evaluation 130, and training 140.

[0049] In Figure 1, the image sensor 101 may be a new image sensor. The detection data 101-1 from the image sensor 101 may be unlabeled data. The detection data 101-1 may be represented as an image or a difference from image data.

[0050] The computing device 100 can perform a model update 110 using the detection data 101-1 from the image sensor 101 and a pre-trained model 102. The pre-trained model 102 may be a model trained based on existing label data from an existing image sensor, or it may be a model that detects objects from the detection data of an existing image sensor. The computing device 100 can update the pre-trained model 102 using the detection data 101-1 from the image sensor 101. The model update 110 will be explained in detail with reference to Figure 2. For example, if the computing device 100 is a vehicle, the existing image sensor is mounted on the vehicle, and the vehicle's memory can store the pre-trained model 102. The method may include replacing the existing image sensor with image sensor 101, generating and storing a newly trained object detection model based on the pre-trained model 101, and the vehicle can perform object detection using image sensor 101 and the newly trained object detection model.

[0051] The computing device 100 performs pseudo-labeling 120 using the model updated via model update 110 and a training set (for example, an individual training set for performing pseudo-labeling 120 using the updated model). The training set may be a dataset obtained from the image sensor 101. As will be described later with reference to Figure 3, the updated model can be fed the training set to generate individual labeling results (i.e., pseudo-label data) for the training set, and inference operations can be performed on the training set.

[0052] The computing device 100 can evaluate the updated model via the model update 110 130 and determine a confidence threshold for the pseudo-labeled data based on the evaluation result. The confidence thresholds may have different values ​​from each other. The evaluation 130 will be described later with reference to Figure 4.

[0053] The computing device 100 can perform multiple training sessions using the updated model and pseudo-labeled data. Here, the multiple training sessions may involve intermediate models (for example, in cases where multiple training sessions are performed on the updated model, the intermediate model may be the updated model; in other cases, the intermediate model may be a model generated based on the model results of multiple training sessions). The model generated in relation to the updated model is referred to as the initial candidate object detection model, which will be described later with reference to Figures 5 and 6. Subsequent models generated by collectively repeating multiple training sessions multiple times are referred to as individual candidate object detection models (for example, candidate object detection models in each of the multiple iterations after the generation of the initial candidate object detection model (for example, object detection model 1 in Figure 5)). Similarly, as will be further described below, for each collective iteration of multiple training sessions, for example, in multiple corresponding times, each of the pseudo-labeling, evaluation of the intermediate model, and determination of multiple confidence thresholds can be repeated with respect to the model represented by the intermediate model in the corresponding time (for example, a previous candidate object detection model generated in a previous time). As an example, as explained with reference to Figure 5, in the next time after the generation of the initial candidate object detection model (e.g., object detection model 1 in Figure 5), the intermediate model may be the previous candidate object detection model (e.g., object detection model 1 in Figure 5). For example, in relation to Figure 5, the previous candidate object detection model may be the initial candidate object detection model, and in Figure 6, the previous candidate object detection model (e.g., object detection model 1) may be the candidate object detection model generated in relation to Figure 5.

[0054] Returning to the case of performing multiple training sessions on the updated model, the computing device 100 performs multiple training sessions by applying each of the confidence thresholds to each of the training sessions. Using the results of the multiple training sessions, the computing device 100 can generate an object detection model (hereinafter referred to as "object detection model 1" for convenience of explanation).

[0055] The computing device 100 generates object detection model 2 by performing pseudo-labeling 120, evaluation 130, and training 140 on object detection model 1, and generates object detection model 3 by performing pseudo-labeling 120, evaluation 130, and training 140 on object detection model 2. In this manner, the computing device 100 can generate multiple object detection models and determine the object detection model with the best performance among the multiple object detection models as the object detection model for the image sensor 101 (for example, preferably the main object detection model for the image sensor 101).

[0056] Figure 2 is a diagram illustrating a model update in an object detection model generation method according to one embodiment.

[0057] Referring to Figure 2, the computing device 100 can perform image adaptation 210 and batch normalization updates 220.

[0058] The computing device 100 can adjust (or change) the distribution (e.g., brightness distribution) of the detected data 101-1 by performing image adjustment 210 on the detected data 101-1. The distribution of the detected data 101-1 may differ from the distribution of the dataset used to train the pre-trained model 102. For example, the detected data 101-1 may be RA1B RAW data (16-bit, 1-channel, 7.4 megapixels), and the dataset used to train the pre-trained model 102 may be RGB data (8-bit, 3-channel, 1 megapixel) processed via an ISP (Image Signal Processor).

[0059] If the distribution of the detected data 101-1 is similar to the distribution of the dataset used to train the pre-trained model 102, the pseudo-labeling 120 can operate smoothly. To reduce the difference between the distribution of the detected data 101-1 and the distribution of the dataset used to train the pre-trained model 102, the computing device 100 can adjust the distribution of the detected data 101-1. As an example, the computing device 100 can adjust the brightness distribution of the detected data 101-1 via the following equation 1.

number

[0060] In the above formula 1, JPEG0007893549000002.jpg1723 shows the average brightness of detected data 101-1. JPEG0007893549000003.jpg1823 shows the lower threshold, JPEG0007893549000004.jpg1926 indicates the upper threshold.

[0061] If the image sensor 101 collects detection data 101-1 in low-light conditions (for example, when a vehicle is driving through a tunnel or at night), the computing device 100 can adjust the brightness distribution of the detection data 101-1 to be relatively higher via (1) of equation 1 above.

[0062] If the image sensor 101 collects detection data 101-1 in a bright condition (for example, when a vehicle is driving in the daytime), the computing device 100 can adjust the brightness distribution of the detection data 101-1 to a relatively lower level via equation 1(2) above.

[0063] The computing device 100 can perform a batch normalization update 220 on the pre-trained model 102 based on the adjusted detection data 210-1. More specifically, the computing device 100 can update the batch normalization layer of the pre-trained model 102 via the adjusted detection data 210-1. As an example, the computing device 100 can update the mean and variance of the placement via the following equation 2. This allows the computing device 100 to update the batch normalization layer of the pre-trained model 102 to match the adjusted detection data 210-1.

number

[0064] In this step, when the computing device 100 updates the batch normalization layer, it does not need to update values ​​other than the mean and variance of the placement (for example, weights in the pre-trained model 102). For example, the computing device may update only the batch normalization layer of the pre-trained model 102 so that it matches the adjusted detection data 210-1.

[0065] In one embodiment, the detection data 101-1 may be converted to the format of the dataset used to update the pre-trained model 102. The batch normalization layer of the pre-trained model 102 can be updated via the converted detection data. For example, the detection data 101-1 may be raw data, and the dataset used to train the pre-trained model 102 may be in RGB format. The detection data 101-1 may be converted to RGB format, and the batch normalization layer of the pre-trained model 102 can be updated via the converted detection data.

[0066] The computing device 100 can perform the pseudo-labeling 120 described later using the updated model 230.

[0067] Figure 3 is a diagram illustrating pseudo-labeling in an object detection model generation method according to one embodiment.

[0068] Referring to Figure 3, the computing device 100 inputs the training set 301 into the updated model 230. The training set 301 may be a dataset acquired by the image sensor 110. The training set 301 may also be an unlabeled dataset. In other words, the training set 301 includes unlabeled images.

[0069] The updated model 230 performs inference operations based on the input training set to generate first pseudo-label data 310. The updated model 230 can determine at least one bounding box for each image in the input training set 301 and calculate probability values ​​for each class for each bounding box of each image. For example, the updated model 230 can determine the image in the input training set 301 A bounding box A Determine the bounding box A We calculate the probability for each class (for example, the probability of belonging to the vehicle class, the probability of belonging to the human class, the probability of belonging to the animal class, etc.) and create a bounding box. A It can be assigned to a pseudo-label.

[0070] Figure 4 is a diagram illustrating the evaluation in an object detection model generation method according to one embodiment.

[0071] Referring to Figure 4, the computing device 100 can evaluate 130 the updated model 230. As an example shown in Figure 4, the computing device 100 can determine multiple evaluation scores 420 (e.g., F1 score) for the updated model 230 using multiple thresholds 410 and a validation set 401. The validation set 401 includes a dataset acquired via the image sensor 101. The validation set 401 may also be a labeled dataset (e.g., a pre-labeled dataset). In other words, the validation set 401 may include labeled images. The evaluation 130 shown in Figure 4 will be described in detail.

[0072] The computing device 100 can apply the threshold th1 to the updated model 230. The updated model 230 to which the threshold th1 is applied can generate an object detection result when the verification set 401 is input. As an example, the updated model 230 determines the bounding box 1 in the image 1 within the verification set 401, calculates the probability for each class for the bounding box 1, and if the maximum probability among the calculated probabilities for each class is greater than or equal to the threshold th1, the object within the bounding box 1 can be selected as the final detection result. When the threshold is th1, the computing device 100 classifies the object detection result of the updated model 230 into FP (false positive), FN (false negative), TP (true positive), and TN (true negative). The computing device 100 can calculate the precision and recall when the threshold is th1 based on the classification results, and calculate the score 1 using the precision and recall. The score 1 may be, for example, an F1 score, but is not limited thereto.

[0073] Similarly, the computing device 100 can calculate the evaluation score for the updated model 230 when each of the other thresholds th2,..., th n is applied. As shown in FIG. 4, the computing device 100 calculates the score 2 (for example, F1 score 2) when the threshold is th2, and the score m when the threshold is th m (for example, F1 score m ), and the score n when the threshold is th n (for example, F1 score n ) can be calculated.

[0074] The computing device 100 can determine the threshold used to determine the highest evaluation score from among multiple evaluation scores 420 as the first confidence threshold β1, determine the second confidence threshold α1 which is greater than the first confidence threshold, and determine the third confidence threshold γ1 which is less than the first confidence threshold. As an example, the computing device 100 can determine the first to third confidence thresholds β1, α1, and γ1 430 via the following equation 3.

number

[0075] In equation 3 above, the optimal detection threshold can be the threshold used to determine the maximum evaluation score described above. ε represents a constant.

[0076] Of the first, second, and third confidence thresholds β1, α1, and γ1, the second confidence threshold α1 is the largest, and the third confidence threshold γ1 is the smallest.

[0077] Figures 5 and 6 illustrate the training process in an object detection model generation method according to one embodiment.

[0078] Referring to Figure 5, the computing device 100 performs multiple training sessions 510, 520, and 530 using the updated model 230 and the first pseudo-label data 310. Here, the computing device 100 can execute each of the training sessions 510, 520, and 530 independently, and can apply confidence thresholds β1, α1, and γ1 to each of the training sessions 510, 520, and 530. The computing device 100 ensembles 540 the results of each of the training sessions 510, 520, and 530 to generate an object detection model 1550. The object detection model 1550 represents the initial candidate object detection model described above. In the example described with reference to Figure 6, the object detection model 1550 may also be referred to as the immediately preceding candidate object detection model or an intermediate model. The training session 140 in Figure 5 will be described in detail below.

[0079] The computing device 100 can perform training 510 with a first confidence threshold β1 applied using the updated model 230 and the first pseudo-label data 310. For example, the computing device 100 can train the updated model 230 520 through an image in the first pseudo-label data 310 that has a pseudo-label equal to or greater than the first confidence threshold β1. Of the confidence thresholds β1, α1, and γ1, the first confidence threshold β1 is intermediate. In training 510 with the first confidence threshold β1 applied, more FPs may occur than in training 510 with the second confidence threshold α1 applied, and more FNs may occur than in training 510 with the third confidence threshold β1 applied.

[0080] The computing device 100 can perform training 520 using the updated model 230 and the first pseudo-label data 310, with the second confidence threshold α1 applied. For example, the computing device 100 can train the updated model 230 510 through an image in the first pseudo-label data 310 that has a pseudo-label greater than or equal to the second confidence threshold α1. Of the confidence thresholds β1, α1, and γ1, the second confidence threshold α1 is the largest, and no false positives (FPs) may occur in the training results 520. In other words, the computing device 100 can perform training 520 with the maximum confidence threshold α1 applied so that no FPs occur.

[0081] The computing device 100 can perform training 530 using the updated model 230 and the first pseudo-label data 310, with the third confidence threshold γ1 applied. For example, the computing device 100 can train the updated model 230 520 through images in the first pseudo-label data 310 that have pseudo-labels equal to or greater than the third confidence threshold γ1. Of the confidence thresholds β1, α1, and γ1, the third confidence threshold γ1 is the smallest, and FN may not occur in the training results 530. In other words, the computing device 100 can perform training 530 with the smallest confidence threshold γ1 applied so that FN does not occur.

[0082] The computing device 100 can generate an object detection model 1550 by ensembling 540 the results of multiple training sessions 510, 520, and 530. For example, the computing device 100 can ensemble 540 the results of multiple training sessions 510, 520, and 530 via the following equation 4.

number

[0083] In the above formula 4, M1 corresponds to the result of training 510, M2 corresponds to the result of training 520, M3 corresponds to the result of training 530, and M E This corresponds to object detection model 1550.

[0084] Depending on the implementation, the computing device 100 can generate object detection model 1 by averaging the weighted results of training 510, 520, and 530, respectively. As an example, Table 1 below shows an example of the weighted values ​​of each model trained through training 510, 520, and 530, respectively. [Table 1]

[0085] The computing device 100 can generate an object detection model 1550 using the corresponding weight values ​​in Table 1 above. For example, the computing device 100 can average the corresponding weight values ​​in Table 1 above. Table 2 below shows the average results of the corresponding weight values ​​w1, w2, ... w n Here is an example. [Table 2]

[0086] The computing device 100 uses the weighted values ​​w1, w2, ..., w shown in Table 2 above. n An object detection model 1550 having can be generated. As a different example, the computing device 100 may apply a ratio to each of the corresponding weight values ​​in Table 1 above. The computing device 100, w 1_1 Apply ratio a to w 1_2 Apply ratio b to w 1_3 The ratio c can be applied to this. Table 3 below shows an example of the result of applying the ratio to each of the corresponding weighted values. [Table 3]

[0087] In Table 3 above, the sum of a, b, and c is 1. Computing device 100 uses the weighted values ​​w1, w2, ..., w in Table 3 above. n An object detection model 1550 having the following characteristics can be generated.

[0088] The computing device 100 can generate an object detection model 2 by performing pseudo-labeling 120, evaluation 130, and training 140 on the object detection model 1550.

[0089] The computing device 100 can generate second pseudo-labeled data by performing pseudo-labeling 120 based on the second training set and the object detection model 1550. The second training set is a dataset acquired via the image sensor 110, and may be an unlabeled dataset. The second training set may be the same as or different from the training set described with reference to Figure 3. Since the matters described with reference to Figure 3 can be applied to the generation of the second pseudo-labeled data, a detailed explanation is omitted.

[0090] The computing device 100 performs an evaluation 130 on the object detection model 1550 and can determine confidence thresholds β2, α2, and γ2 for the second pseudo-label data based on the evaluation results for the object detection model 1550. Since the methods described in Figure 4 can be applied to determining the confidence thresholds β2, α2, and γ2, a detailed explanation is omitted.

[0091] As shown in Figure 6, the computing device 100 can perform multiple training cycles 610, 620, and 630 using the object detection model 1550 and the second pseudo-label data. Here, the computing device 100 applies confidence thresholds β2, α2, and γ2 to each of the multiple training cycles 610, 620, and 630, respectively, and performs each of the multiple training cycles 610, 620, and 630. The computing device 100 ensembles the results of each of the multiple training cycles 610, 620, and 630 640 to generate the object detection model 2650. Since the explanation for multiple training cycles 510, 520, and 530 can be applied to multiple training cycles 610, 620, and 630, a detailed explanation is omitted.

[0092] The computing device 100 can generate object detection model 3 by performing pseudo-labeling 120, evaluation 130, and training 140 on object detection model 2650. In this method, the computing device 100 can generate multiple object detection models and select the object detection model with the best performance among them. The selected object detection model can be stored in various devices (e.g., vehicles) and perform object detection by receiving detection data from image sensors mounted on those devices.

[0093] In one embodiment, the image sensor 101 may be a newly released image sensor and may have different characteristics from existing image sensors. In this case, the computing device 100 can quickly and efficiently generate an object detection model for the image sensor 101 without ground truth (GT) operation (or data labeling) on ​​the image sensor 101. Furthermore, the computing device 100 can quickly and efficiently generate an object detection model for the image sensor 101 even if the domain of the dataset used to train the pre-trained model 102 (e.g., RGB) and the domain of the detection data 101-1 of the image sensor 101 (e.g., raw Bayer) are different. Moreover, even if the domain of the dataset used to train the pre-trained model 102 and the domain of the detection data 101-1 of the image sensor 101 are the same, the computing device 100 can quickly and efficiently generate an object detection model for the image sensor 101 even if the image sensor 101 is of a different type from the existing image sensor used to train the pre-trained model 102 (e.g., different distribution and / or resolution characteristics, as an unrestrictive example).

[0094] Figure 7 is a flowchart of an object detection model generation method according to one embodiment.

[0095] Referring to Figure 7, in step S710, the computing device 100 updates a pre-trained model 102 based on detection data 101-1 from the image sensor 101. The computing device 100 can update the first layer of the pre-trained model 102 using the detection data 101-1. In one embodiment, the computing device 100 can adjust the brightness distribution of the detection data 101-1 and update the first layer of the pre-trained model 102 using the detection data with the adjusted brightness distribution. The first layer of the pre-trained model 102 may include a batch normalization layer.

[0096] In step S720, the computing device 100 performs pseudo-labeling 120 based on the updated model 230 and training set 301 to generate pseudo-label data.

[0097] In step S730, the computing device 100 evaluates the updated model 230.

[0098] In step S740, the computing device 100 determines a confidence threshold for the pseudo-labeled data based on the evaluation results of the updated model 230.

[0099] In step S750, the computing device 100 performs multiple training sessions using the updated model 230 and pseudo-labeled data, applying each of the confidence thresholds to each of the training sessions.

[0100] As an example, as explained with reference to Figure 5, the computing device 100 may perform training (e.g., training 510) in which a first confidence threshold is applied using the updated model 230 and the first pseudo-label data 310. The computing device 100 can then perform other training (e.g., training 520) in which a second confidence threshold is applied using the updated model 230 and the first pseudo-label data 310, and other training (e.g., training 530) in which a third confidence threshold is applied using the updated model 230 and the first pseudo-label data 310.

[0101] In step S760, the computing device 100 generates an object detection model based on the results of multiple training sessions.

[0102] Since the matters described with reference to Figures 1 to 6 can also be applied to the matters described with reference to Figure 7, a detailed explanation is omitted.

[0103] Figure 8 is a block diagram illustrating a computing device according to one embodiment.

[0104] Referring to Figure 8, the computing device 800 includes a processor 810 and memory 820.

[0105] The computing device 800 may be the same as the computing device 100 described with reference to Figures 1 to 7.

[0106] The processor 810 performs model updates 110, pseudo-labeling 120, evaluation 130, and training 140, as described with reference to Figures 1 to 7.

[0107] Memory 820 can store information necessary for the processor 810 to perform processing operations. For example, memory 820 can store instructions executed by the processor 810, detection data 101-1, a pre-trained model 102, a training set 301, and a verification set 401.

[0108] Memory 820 can store the processing results of processor 810. For example, memory 820 can store the updated model 230, the first pseudo-label data 310, the second pseudo-label data, and so on.

[0109] Since the points explained with reference to Figures 1 to 7 can also be applied to the points explained with reference to Figure 8, a detailed explanation will be omitted.

[0110] Figure 9 is a block diagram illustrating a computing device according to one embodiment.

[0111] Referring to Figure 9, the computing device 900 includes a processor 910 and memory 920.

[0112] Memory 920 stores the object detection models. The processor 910 can generate one or more object detection models by performing the operations described with reference to Figures 1 to 6 above (e.g., model update 110, pseudo-labeling 120, evaluation 130, training 140). The generated object detection models may be stored in memory 920. Alternatively, the best-performing object detection model among the generated models may be stored in memory 920.

[0113] The processor 910 receives an image from the image sensor and performs object detection using the received image and an object detection model called up in the memory 920. The processor 910 can provide object detection results including the position of the bounding box on the received image (e.g., reference position such as the center or corner), size (width and height), and object class.

[0114] In one embodiment, the processor 910 can adjust the brightness distribution of the image using an object detection model. For example, the processor 910 may receive raw detection data from an image sensor. The object detection model may perform image adjustments on the raw detection data. More specifically, the input layer of the object detection model may be configured to adjust the brightness distribution of the input data via Equation 1 above. However, without limitation, the input layer of the object detection model may be configured to perform various operations of an ISP (e.g., contrast adjustment, distortion correction). Subsequent layers of the object detection model can detect objects based on the raw detection data from which image adjustments have been performed. Therefore, the processor 910 can perform object detection based on raw detection data even without an ISP that processes the raw detection data from the image sensor.

[0115] The computing device 900 can be applied in a variety of fields. For example, the computing device 900 (or a combination of the computing device 800 and the computing device 900) may be included in a vehicle's ADAS (Advanced Driver Assistance Systems) / AD (Autonomous Driving) system. However, it is not limited to this, and the computing device 900 (or a combination of the computing device 800 and the computing device 900) may be used in fields such as surveillance systems like CCTV surveillance or military border surveillance, sports competition analysis systems, smart campus systems, and video conferencing systems. In addition, the computing device 900 (or a combination of the computing device 800 and the computing device 900) may be used in fields where it is necessary to detect objects in images.

[0116] Figure 10 is a diagram illustrating the operation of a vehicle with object detection according to one embodiment, and Figure 11 is a diagram illustrating the vehicle.

[0117] As an example shown in Figures 10 and 11, a computing device 900, a combination of computing device 800 and computing device 900, an ADAS / AD system mounted on vehicle 1000, one or more processors 1110 and / or ADAS / AD system 1150 of vehicle 1100 can perform one or more operations described in relation to Figures 1 to 9. For example, the computing device may be one or more processors (e.g., one or more processors 1110) and / or one or more ADAS / AD systems (e.g., vehicle 1000 or vehicle 1100 having one or more ADAS / AD systems 1150). As a different example, the ADAS / AD system of vehicle 1000 or the ADAS / AD system 1150 of vehicle 1100 may include, as an example not limited to, computing device 900 or a combination of computing device 800 and computing device 900. The ADAS / AD system of vehicle 1000 may further include an image sensor, otherwise the image sensor may be mounted on vehicle 1000. For example, the image sensor 1130 of vehicle 1100 may be separate from the ADAS / AD system 1150, or it may communicate with the ADAS / AD system 1150. One or more processors 1110 of vehicle 1100 shown in Figure 11 may perform some or all of the same operations or functions as the ADAS / AD system of vehicle 1000. Some or all of such ADAS / AD operations or functions may be performed by the ADAS / AD system 1150 shown in Figure 11.

[0118] The memory 1120 of the vehicle 1100 can store one or more object detection models. For example, one or more of the processors 1110 can generate one or more object detection models through the execution of operations (e.g., model update 110, pseudo-labeling 120, evaluation 130, and training 140). As an example, previously generated object detection models may be stored in memory 1120. In addition, memory 1120 and / or the memory of the ADAS / AD system 1150 may store instruction words executed by the processors (e.g., one or more processors 1110 or one or more processors of the ADAS / AD system 1150). Furthermore, memory 1120 may store additional instruction words for additional operations for driving and / or other functions of the vehicle 1100.

[0119] The ADAS / AD system of vehicle 1000 and the ADAS / AD system 1150 of vehicle 1100 can each generate information regarding the driving of vehicle 1000 and vehicle 1100, respectively. Information regarding the driving of vehicle 1000 may include, for example, route guidance information, hazard warning information (e.g., accident information such as collisions), road condition information (e.g., road congestion), and surrounding environment information, as data to assist in or be used for the driving of vehicle 1000. Such information may be provided or generated by the ADAS / AD system 1150 and / or the information output 1170 (e.g., a speaker) shown in Figure 11. The ADAS / AD system of vehicle 1000 can visually display the driving information via the display 1020 of vehicle 1000. Furthermore, one or more processors 1110, ADAS / AD systems 1150, and / or information outputs 1170 can be controlled to visually display information related to driving via the vehicle's display 1160.

[0120] The image sensors of vehicle 1000 and one or more image sensors 1130 of vehicle 1100 can capture the surrounding scene 1010 and generate an input image corresponding to the surrounding scene 1010. The computing device 900 can detect objects from the input image.

[0121] The ADAS / AD system of vehicle 1000 and one or more processors 1100 of vehicle 1100 can perform or control autonomous driving based on the object detection results of the computing device 900. For example, the ADAS / AD system of vehicle 1000 may perform one or more combinations of speed control, acceleration control, and steering control of vehicle 1000. Similarly, the ADAS / AD system 1150 of vehicle 1100 may, for example, perform one or more combinations of speed control, acceleration control, and steering control of vehicle 1100 based on corresponding control or command from the ADAS / AD system 1150 in vehicle operation / function 1140. As an example, the ADAS / AD system of vehicle 1000 or the ADAS / AD system 1150 of vehicle 1100 can calculate the distance to surrounding objects and perform one or more combinations of speed changes (e.g., increase or decrease), acceleration changes (e.g., increase or decrease), and steering changes of vehicle 1000 based on the distance to the objects.

[0122] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

[0123] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by the processing unit or for providing instructions or data to the processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0124] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0125] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0126] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0127] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims. [Explanation of Symbols]

[0128] 800 Computing Devices 810 processor 820 memory 900 Computing Devices 910 Processor 920 memory 1100 vehicles 1110 processor 1120 memory 1130 Image Sensor 1140 Vehicle Operation / Function 1150 ADAS / AD System 1160 displays 1170 Information Output

Claims

1. A method that is executed by a processor, The steps include updating a pre-trained model based on detection data from an image sensor, The steps include performing pseudo-labeling using an intermediate model that provides individual training sets, The steps include determining a confidence threshold based on the evaluation of the aforementioned intermediate model, The steps include applying the determined confidence threshold to multiple training sessions and performing the multiple training sessions using the pseudo-label data generated via pseudo-labeling and the intermediate model, The step of generating an object detection model that depends on the execution of the aforementioned multiple training sessions, The generation step includes, if the intermediate model is the updated model, the step of generating an initial candidate object detection model. The method by which it is executed by the processor.

2. The method performed by the processor according to claim 1, wherein the updating step includes updating the first layer of the pre-trained model using the detection data.

3. The method, performed by the processor according to claim 2, wherein the first layer is a batch normalization layer.

4. The method performed by the processor according to claim 1, wherein the updating step includes a step of performing image adjustment on the detected data.

5. The method performed by the processor according to claim 4, wherein the step of performing the image adjustment includes the step of adjusting the brightness distribution of the detected data.

6. The aforementioned pre-trained model is based on corresponding detection data in a different format from the aforementioned detection data, The updating step includes the step of converting the detected data into another format. A method performed by the processor described in claim 1.

7. Each of the aforementioned trainings includes running the intermediate model to provide the generated pseudo-label data in order to obtain individual labeling results for the intermediate model, Each of the aforementioned training sessions includes at least additional training based on the individual labeling results, A method performed by the processor described in claim 1.

8. The method, executed by the processor according to claim 7, wherein the pre-trained model is based on detection data from another image sensor having different characteristics from the image sensor.

9. The aforementioned multiple training steps are repeated collectively a number of times after the initial step, when the intermediate model is the updated model, and in the step of generating the intermediate model in the immediately preceding step, the intermediate model is the previous candidate object detection model. The generation step further includes the step of generating other candidate object detection models in the last of the multiple iterations, The previous candidate object detection model in the run immediately following the initial run is the initial object candidate detection model. A method performed by the processor described in claim 7.

10. The generation step includes selecting an object detection model from among the multiple candidate object detection models based on a performance comparison of the multiple candidate object detection models, The plurality of candidate object detection models include the initial candidate object detection model, the previous candidate object detection models generated in each of the plurality of iterations excluding the iteration immediately following the initial iteration, and the other candidate object detection models. A method performed by the processor described in claim 9.

11. A method performed by the processor according to claim 10, further comprising the step of performing object detection using the generated object detection model on a vehicle.

12. The evaluation of the intermediate model includes the step of determining multiple evaluation scores from individual runs of the intermediate model using multiple thresholds and individual validation sets, The step of determining the confidence threshold is: The steps include determining the threshold used to determine the maximum evaluation score among the determined evaluation scores as the first confidence threshold, The steps include determining a second confidence threshold that is greater than the first confidence threshold determined above, The step of determining a third confidence threshold that is smaller than the first confidence threshold determined above, A method performed by the processor described in claim 9.

13. Each of the aforementioned training sessions has individually trained model results. In each of the aforementioned multiple steps, the generation step generates a corresponding candidate object detection model by ensembling the individually trained model results. A method performed by the processor described in claim 9.

14. A method performed by the processor according to claim 9, wherein the generating step includes, when the intermediate model is the updated model, the step of ensembling the individual model results of the plurality of trainings to generate an initial candidate object model.

15. The evaluation of the intermediate model includes the step of determining multiple evaluation scores from individual runs of the intermediate model using multiple thresholds and individual validation sets, The step of determining the confidence threshold is: The steps include determining the threshold used to determine the maximum evaluation score among the determined evaluation scores as the first confidence threshold, The steps include determining a second confidence threshold that is greater than the first confidence threshold determined above, The step of determining a third confidence threshold that is smaller than the first confidence threshold determined above, A method performed by the processor described in claim 1.

16. The steps of performing the aforementioned multiple training exercises are: The steps include: performing a first training using the intermediate model and the generated pseudo-label data, to which the determined first confidence threshold is applied; The steps include: performing a second training using the intermediate model and the generated pseudo-label data, to which the determined second confidence threshold is applied; The step includes performing a third training using the intermediate model and the generated pseudo-label data, to which the determined third confidence threshold is applied. A method performed by the processor according to claim 15.

17. The step of performing the aforementioned pseudo-labeling is: The steps include generating first pseudo-labeled data by performing pseudo-labeling based on the first unlabeled training set as the individual training set and the updated model, The step includes generating second pseudo-label data by performing pseudo-labeling based on the second unlabeled training set as the individual training set and the initial candidate object detection model, The method performed by the processor is: The steps include evaluating the initial candidate object detection model, The method further includes the step of determining a confidence threshold for the generated second pseudo-label data based on the evaluation results of the initial candidate object detection model, The step of performing the plurality of training includes performing a plurality of second training sessions using the initial candidate object detection model and the generated second pseudo-label data, and applying each of the confidence thresholds for the generated second pseudo-label data to each of the plurality of second training sessions. The generation step includes generating a second candidate object detection model using the results of each of the plurality of second trainings, The first unravel training set and the second unravel training set are either identical or different training sets. A method performed by the processor described in claim 1.

18. The process involves repeating several steps in which, in a given run, pseudo-labeling is performed using a previous candidate object detection model; the intermediate model is evaluated in the given run; the multiple training sessions are performed in relation to the intermediate model in the given run; and other candidate object detection models are generated in the given run based on the results of the multiple training sessions. The method further includes the step of generating an object detection model by selecting one of the multiple candidate object detection models based on a performance comparison of the multiple candidate object detection models, The plurality of candidate object detection models include the initial candidate object detection model, the previous candidate object detection model in the corresponding iteration, and the other object detection model in the last of the plurality of iterations. A method performed by the processor according to claim 17.

19. Memory for storing the object detection model, A processor that receives an image from an image sensor and performs object detection using the received image and the object detection model, It is a device, For generating the object detection model, the processor: The operation of updating a pre-trained model, To generate pseudo-labeled data, the process involves performing pseudo-labeling using an intermediate model that provides individual training sets, and The operation of determining multiple confidence thresholds based on the evaluation of the aforementioned intermediate model, The operation involves applying each of the determined confidence thresholds to each of the multiple training models, and performing the multiple training models using the generated pseudo-label data and the intermediate model. The operation of generating an object detection model that depends on the execution of the aforementioned multiple training sessions, and the operation of performing the above, The aforementioned generation operation includes, if the intermediate model is the updated model, the operation of generating an initial candidate object detection model. Device.

20. The apparatus according to claim 19, wherein the updating operation includes updating the first layer of the pre-trained model using detection data from another image sensor having the same characteristics as the image sensor.

21. The apparatus according to claim 20, wherein the first layer is a batch normalization layer.

22. The apparatus according to claim 20, wherein the updating operation includes an operation to perform image adjustment on the detected data.

23. The evaluation operation includes determining multiple evaluation scores for the intermediate model using multiple thresholds and individual validation sets. The operation for determining the aforementioned confidence threshold is: The operation involves determining the threshold used to determine the maximum evaluation score among the determined evaluation scores as the first confidence threshold, The operation of determining a second confidence threshold that is greater than the first confidence threshold determined above, The operation includes determining a third confidence threshold that is smaller than the first confidence threshold determined above, The apparatus according to claim 19.

24. The aforementioned actions for performing multiple training exercises are: The operation involves performing training using the intermediate model and the generated pseudo-label data to which the determined first confidence threshold is applied, The operation involves performing training using the intermediate model and the generated pseudo-label data to which the determined second confidence threshold is applied, The operation includes performing training using the intermediate model and the generated pseudo-label data to which the determined third confidence threshold is applied, The apparatus according to claim 23.

25. The apparatus according to claim 19, wherein the operation to generate the object detection model includes the operation to generate the initial candidate object detection model by ensembling the results of the plurality of trainings.

26. The apparatus according to claim 19, wherein the processor adjusts the brightness distribution of the received image using the object detection model.

27. The apparatus according to claim 26, further comprising the image sensor.

28. The apparatus according to claim 27, wherein the apparatus is a vehicle.

Images