Explains active learning methods and active learning devices that utilize these methods.

By combining a depth autoencoder and an object detector, the problem of selecting rare images in existing technologies is solved, enabling efficient acquisition of rare samples and improving detection accuracy in object detection.

CN115427976BActive Publication Date: 2026-11-17STRADVISION
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202180026892.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-24
Filing Date
2021-04-15
Publication Date
2026-11-17
Estimated Expiration
2041-04-15

AI Technical Summary

Technical Problem

Existing active learning techniques struggle to identify rare images in object detection, especially in similar street scenes where they cannot distinguish whether there are pedestrians, and they cannot effectively map small areas such as stop signs, leading to reduced training effectiveness.

Method used

By combining a deep autoencoder with an object detector, specific data codes are generated by cropping and resizing image regions. Rare samples are selected using the data codebook and the training images are updated to retrain the object detector.

Benefits of technology

It enables easy acquisition of rare samples based on object type during active learning, reflecting whether specific types of objects appear and the overall scene characteristics, confirming the reasons for selecting rare samples, and improving the accuracy of object detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115427976B_ABST
    Figure CN115427976B_ABST
Patent Text Reader

Abstract

This invention discloses an interpretable active learning method and an active learning device utilizing this method. It relates to an interpretable active learning method for an object detector using a depth encoder, comprising the following steps: a) i) inputting an acquired test image into the object detector to detect objects and output bounding boxes; ii) cropping regions corresponding to the bounding boxes in the test image; iii) resizing the test image and the cropped image to the same size; iv) inputting the resized image into the data encoder of the depth autoencoder to output data codes; and b) i) referring to a data codebook to confirm reference data codes corresponding to the number of resized images less than a counter threshold; ii) extracting specific data codes from the data codes; iii) selecting specific test images as rare samples; iv) updating the data codebook with reference to the specific data codes. This invention can easily acquire rare samples based on object type.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-referencing related applications

[0002] This application claims priority to U.S. Patent Application No. 63 / 020,106, filed May 5, 2020, and U.S. Patent Application No. 17 / 211,123, filed March 24, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This invention relates to an active learning method for object detection and an active learning device using the same, and more specifically, to an explainable active learning method for an object detector using a deep autoencoder and an active learning device using the same, wherein the object detector reflects information about whether a particular type of object appears in an image, thereby confirming the reason for actively learning to select new images, and thereby allowing the sampling of images of a particular type. Background Technology

[0004] In recent years, research has been conducted on methods that utilize machine learning to perform object recognition and other tasks.

[0005] Deep learning is a type of machine learning that utilizes neural networks with multiple hidden layers between the input and output layers, and it has high recognition performance.

[0006] In addition, neural networks utilizing deep learning typically learn by backpropagating the loss.

[0007] To train such a deep learning network, existing techniques collect raw data according to a data collection strategy. Human annotators then label the collected raw data to generate new training data. After training the deep learning network using the new and existing training data, human engineers modify and improve the training algorithm based on performance analysis results. Additionally, the data collection strategy is changed based on the analysis results, and any erroneous annotations are re-checked and corrected.

[0008] However, as the performance of deep learning networks improves, hard examples that are useful for training become scarce. As a result, the effectiveness of using new training data to improve the performance of deep learning networks decreases. Furthermore, as the labeled image database grows larger, the effectiveness of improving the performance of deep learning networks also decreases, leading to a decline in the return on investment for data annotation by human annotators.

[0009] On the other hand, in order to solve the above problems, the existing technology does not label all the collected unlabeled images, but uses active learning technology, that is, only selects unlabeled images that are expected to have a greater effect under the current circumstances for labeling, such as the size of the database storing labeled images or the performance of the model to be trained.

[0010] In particular, assuming the acquired unlabeled images are useful, existing active learning methods acquire unlabeled images as rare samples by dividing the feature space into subspaces or partitions and selecting unlabeled images belonging to the less abundant subspaces from a database storing labeled images.

[0011] However, existing active learning techniques are limited by classification problems.

[0012] That is, existing active learning techniques map a given H*W-sized image as a whole to a feature space so that similar scenes belong to the same subspace.

[0013] However, existing active learning techniques struggle to identify rare images in object detection problems.

[0014] For example, in similar street scenes, they should be mapped to different subspaces depending on whether there are pedestrians, but existing active learning techniques have the problem of classifying them into the same subspace.

[0015] In addition, existing active learning techniques have a drawback: in images with parking signs, they cannot map from a very small region in the overall image to different subspaces.

[0016] That is, in a 1920*1080 image, the area with a stop sign is only a tiny 50*50 or 100*100. Therefore, when selecting an image with a stop sign, all other parts except the corresponding area should be ignored. However, existing active learning techniques have the drawback of not being able to achieve the above process. Summary of the Invention

[0017] Technical issues

[0018] The purpose of this invention is to solve all the problems mentioned above.

[0019] Another objective of this invention is to easily acquire rare samples based on object type in active learning.

[0020] Another objective of this invention is to obtain rare samples by reflecting whether a specific type of object appears and the overall scene characteristics in active learning.

[0021] Another object of the present invention is to easily identify the reasons for selecting rare samples in active learning.

[0022] Another object of the present invention is to easily select only specific types of unlabeled images from rare samples acquired through active learning.

[0023] Technical solution

[0024] According to the present invention, an interpretable active learning method for an object detector is provided, which utilizes a deep autoencoder, comprising the following steps: a) after a test image is acquired, an active learning device inputs the test image to an object detector, so that the object detector detects a test object from each test image and outputs a test bounding box corresponding to the test object; generating test cropped images 1 to n by cropping regions corresponding to the test bounding boxes in each test image; generating test resized images 1 to n+1 by resizing the test image and the test cropped images 1 to n+1 by resizing them to the same size; and inputting the test cropped images 1 to n+1 to the data encoder of the deep autoencoder to encode the test cropped images 1 to n+1 by the data encoder. The process involves: a) adjusting the image to output test data codes from the first size-adjusted test image to the (n+1)th size-adjusted test image, where n is an integer greater than or equal to 1; b) the active learning device reference data book confirms reference data codes corresponding to the number of size-adjusted images less than or equal to a counter threshold; extracting test-specific data codes matching the reference data codes from the first to the (n+1)th size-adjusted test images; selecting test-specific images corresponding to the test-specific data codes in the test images as rare samples for retraining the object detector; updating the data book with reference to the test-specific data codes corresponding to the rare samples, wherein the data book is generated or updated using previously learned images previously used to train the object detector, and the data book lists the number of size-adjusted images for each data code in the previously learned images.

[0025] The method may further include the following step: c) The active learning device retrains the object detector using previously learned images and the rare samples.

[0026] In step a), the active learning device is able to map the test bounding box that detects errors in the test bounding box to the background data code.

[0027] In step b), the counter threshold may be a preset number of resized images, or the number of resized images corresponding to the qth threshold data code in the data codebook arranged in descending order of the number of resized images, where q is an integer greater than or equal to 1.

[0028] In step a), the active learning device may set the detection threshold of the object detector to be lower than when the object detector was trained.

[0029] Prior to step a), the active learning device may further include the following steps: a01) The active learning device crops training object regions from each training image to generate training first cropped object images to training t-th cropped object images; crops background regions from each training image to generate training first cropped background images to training u-th cropped background images; and resizes the training images, the training first cropped object images to training t-th cropped object images corresponding to each training image, and the training first cropped background images to training u-th cropped background images corresponding to each training image by the same size, thereby generating... Training objects resized from the first size to the (t+1)th size and background images resized from the first size to the uth size are used for training. These training objects resized from the first size to the (t+1)th size and background images resized from the first size to the uth size are input to the data encoder of the depth autoencoder, so that the data encoder encodes these training objects resized from the first size to the (t+1)th size and background images resized from the first size to the uth size. Output training object data codes from the first resized training object image to the (t+1)th resized training object image, and training background data codes from the first resized training background image to the u-th resized training background image, wherein each training image is sampled from a training image database used to train the object detector, and t and u are integers greater than or equal to 1; and a02) the active learning device inputs the training object data codes from the first resized training object image to the (t+1)th resized training object image into... The data decoder of the depth autoencoder decodes each of the training first object data codes to the training (t+1)th object data codes to output training first key images to training (t+1)th key images; trains the data decoder and the data encoder using the first object loss to the (t+1)th object loss referencing the training first resized object images to the training (t+1)th resized object images and the training first key images to the training (t+1)th key images; and trains the data encoder using the background loss referencing the training first background data codes to the training (u)th background data codes.

[0030] The method may further include the following steps: a03) The active learning device crops the evaluation object region in each evaluation image sampled from the training image database to generate evaluation first cropped object images to evaluation vth cropped object images; the evaluation images and the evaluation first cropped object images to evaluation vth cropped object images corresponding to each evaluation image are resized by the same size to generate evaluation first resized object images to evaluation v+1 resized object images; the evaluation first resized object images to evaluation v+1 resized object images are resized by the same size. An image is input to the data encoder, which encodes the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image and outputs evaluation first-object data codes to evaluation (v+1)th-object data codes corresponding to the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image; the evaluation first-object data codes to the evaluation (v+1)th-object data codes are classified by data codes, and the number of resize-adjusted images corresponding to each data code is counted to generate the data codebook, wherein v is an integer greater than or equal to 1.

[0031] In step a01), the active learning device may generate a training cropped background image from a single training image that is one to three times the number of training cropped object images.

[0032] Additionally, according to another embodiment of the present invention, an active learning device for an object detector is provided, which performs interpretable active learning using a deep autoencoder, comprising: a memory storing instructions for performing interpretable active learning using a deep autoencoder to sample training images for training the object detector; and a processor performing the interpretable active learning using the deep autoencoder to sample the training images for training the object detector according to the instructions stored in the memory, wherein the processor performs the following process: I) after a test image is acquired, the test image is input to the object detector so that the object detector detects a test object from each test image, thereby outputting a test bounding box corresponding to the test object; generating test cropped images from the first to the nth cropped images by cropping the region corresponding to the test bounding box in each test image; generating test resized images from the first to the n+1th resized images by resizing the test images and the test cropped images from the first to the nth cropped images to the same size; and generating test resized images from the test image to the (n+1)th resized images by resizing the test images and the test cropped images from the first to the nth cropped images to the same size; and further resizing the test images by cropping the test images from the test image to the n+1th cropped images to the n ... The data encoder of the depth autoencoder is input from the first resized image to the (n+1)th resized image for test, so that the data encoder encodes the first resized image to the (n+1)th resized image for test and outputs the first data code to the (n+1)th data code for test corresponding to the first resized image to the (n+1)th resized image for test, where n is an integer greater than or equal to 1; and II) a reference data codebook confirms the reference data code corresponding to the number of resized images less than or equal to a counter threshold; from the test Extract test-specific data codes that match the reference data codes from the first data code to the (n+1)th test data code; select test-specific images in the test images corresponding to the test-specific data codes as rare samples for retraining the object detector; update the data codebook with reference to the test-specific data codes corresponding to the rare samples, wherein the data codebook is generated or updated using previously learned images used to train the object detector, and the data codebook lists the number of size-adjusted images for each data code in the previously learned images.

[0033] The processor can further execute the process of retraining the object detector using previously learned images and the rare samples.

[0034] In the process described in I), the processor can map the test bounding box that detects errors in the test bounding box to the background data code.

[0035] In the process described in II), the counter threshold may be a preset number of resized images, or the number of resized images corresponding to the qth threshold data code in the data codebook arranged in descending order of the number of resized images, wherein q is an integer greater than or equal to 1.

[0036] During the process described in I), the processor may set the detection threshold of the object detector to be lower than that used when training the object detector.

[0037] Prior to process I), the processor may further execute the following process: I01) cropping the training object region from each training image to generate training cropped object images 1 to t; cropping the background region from each training image to generate training cropped background images 1 to u; and resizing the training cropped object images 1 to t and the training cropped background images 1 to u for each training image by the same size, thereby generating... Training images of objects resized from the first size to the (t+1)th size and background images resized from the first size to the uth size are used. These images are then input into the data encoder of the depth autoencoder, causing the data encoder to encode the images of the objects resized from the first size to the (t+1)th size and background images resized from the first size to the uth size. The image outputs training object data codes from the first resized object image to the (t+1)th resized object image, and training background data codes from the first resized background image to the u-th resized background image, wherein the training images are sampled from a training image database used to train the object detector, and t and u are integers greater than or equal to 1; and I02) inputs the training object data codes from the first resized object image to the (t+1)th resized object image into the depth... The data decoder of the autoencoder is configured to decode each of the training first object data codes to the training (t+1)th object data codes and output training first key images to training (t+1)th key images; the data decoder and the data encoder are trained using the first object loss to the (t+1)th object loss referencing the training first resized object images to the training (t+1)th resized object images and the training first key images to the training (t+1)th key images; the data encoder is trained using the background loss referencing the training first background data codes to the training (u)th background data codes.

[0038] The processor may further execute the following process, I03): cropping the evaluation object region in each evaluation image sampled from the training image database to generate evaluation first cropped object images to evaluation vth cropped object images; resizing the evaluation image and the evaluation first cropped object images to evaluation vth cropped object images corresponding to each evaluation image by the same size to generate evaluation first resized object images to evaluation v+1 resized object images; and outputting the evaluation first resized object images to evaluation v+1 resized object images. The data is fed into the data encoder to encode the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image and output evaluation first-object data code to evaluation (v+1)th-object data code corresponding to the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image; the evaluation first-object data code to the evaluation (v+1)th-object data code are classified by data code, and the number of resize-adjusted images corresponding to each data code is counted to generate the data codebook, wherein v is an integer greater than or equal to 1.

[0039] During the I01 process, the processor can generate one to three times the number of training cropped background images from a single training image.

[0040] Technical effect

[0041] In this invention, rare samples based on object type can be easily obtained by using an object detector and a deep autoencoder to map the feature space together in active learning.

[0042] In this invention, by using an object detector and a deep autoencoder to map the feature space together in active learning, rare samples can be obtained that reflect whether a specific type of object appears and the characteristics of the overall scene.

[0043] In this invention, by using an object detector and a deep autoencoder together to map and code the feature space in active learning, the reasons for selecting rare samples can be easily identified.

[0044] In this invention, by using an object detector and a deep autoencoder together to map and code the feature space in active learning, it is easy to select only specific types of unlabeled images. Attached Figure Description

[0045] The following drawings, which are used to illustrate the embodiments of the present invention, are only a part of the embodiments of the present invention. Those skilled in the art (hereinafter referred to as "the skill person") can obtain other drawings based on the following drawings without making creative work.

[0046] Figure 1 This is a schematic diagram illustrating an active learning device according to an embodiment of the present invention, which performs interpretable active learning by using a depth autoencoder for an object detector.

[0047] Figure 2 This diagram schematically illustrates an interpretable active learning method performed using a depth autoencoder for an object detector, according to an embodiment of the present invention.

[0048] Figure 3 This diagram schematically illustrates the process of selecting training images for training the object detector using a codebook in an interpretable active learning method for an object detector utilizing a depth encoder according to an embodiment of the present invention.

[0049] Figure 4 This is an illustrative diagram illustrating the reason for selecting training images based on a codebook in an interpretable active learning method for object detectors utilizing a depth encoder according to an embodiment of the present invention.

[0050] Figure 5 This diagram schematically illustrates the process of training a deep autoencoder in an interpretable active learning method for object detectors utilizing a deep encoder, according to an embodiment of the present invention.

[0051] Figure 6 This is a schematic diagram illustrating the process of generating a data codebook in an interpretable active learning method for an object detector utilizing a depth encoder according to an embodiment of the present invention. Detailed Implementation

[0052] The objectives, technical solutions, and advantages of the present invention will now be described in detail with reference to the accompanying drawings, which exemplarily illustrate specific embodiments in which the present invention can be implemented. These embodiments will be described in detail below to enable those skilled in the art to fully implement the present invention.

[0053] Furthermore, throughout the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and features of the invention will become apparent to those skilled in the art through this specification, while others will become apparent through embodiments of the invention. The following examples and figures are illustrative only and are not intended to limit the scope of the invention.

[0054] Furthermore, this invention encompasses all combinations of the embodiments described in this specification. It should be understood that while the various embodiments of the invention differ, they are not mutually exclusive. For example, specific shapes, structures, and characteristics relating to one embodiment of this specification may be implemented as other embodiments without departing from the technical concept and scope of the invention. Additionally, it should be understood that the position or configuration of the various components in each disclosed embodiment may be changed without departing from the technical concept and scope of the invention. Therefore, the following detailed description is not intended to be limiting, and if properly described, the scope of the invention is defined only by the full scope of the claims and their equivalents. Similar reference numerals in the drawings refer to the same or similar functions in several respects.

[0055] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the present invention.

[0056] Figure 1 This is a schematic diagram of an active learning device according to an embodiment of the present invention, which performs interpretable active learning using a depth autoencoder for an object detector. (Reference) Figure 1 The active learning device 1000 may include a memory 1001 and a processor 1002. The memory 1001 stores instructions for performing interpretable active learning using a depth autoencoder to sample training images for training an object detector. The processor 1002 performs interpretable active learning using a depth autoencoder to sample training images for training an object detector according to the instructions stored in the memory 1001.

[0057] Specifically, the active learning device 1000 can achieve the required system performance by utilizing a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific manner). For example, the computing devices may be devices including computer processors, memory, storage devices, input devices and output devices, as well as other existing computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).

[0058] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software components that execute applications for specific purposes.

[0059] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.

[0060] The following is for reference. Figure 2 A method for interpretable active learning using an active learning device 1000 configured as described above is described, the method utilizing a depth autoencoder for an object detector.

[0061] First, after the test images are acquired, the active learning device 1000 inputs the test images into the object detector 200 so that the object detector 200 detects the test objects from each test image and outputs test bounding boxes corresponding to the test objects. Additionally, the object detector 200 can output test category information corresponding to the test objects, as well as various detection information related to the test objects.

[0062] At this point, the active learning device 1000 can set the detection threshold of the object detector 200 to a low value, thereby enabling the object detector 200 to also include bounding boxes with lower foreground category scores in the test image. This allows the object detector 200 to detect as many test bounding boxes as possible; for example, by setting the threshold lower than when the object detector 200 was trained. This maximizes the prevention of incorrectly classifying actual objects as absent, and in particular, enables the detection of objects belonging to difficult-to-detect categories.

[0063] Additionally, the object detector 200 can be a pre-trained state to detect objects on the input image.

[0064] In addition, the test images may be images acquired by an image capture device linked to a device on which the object detector 200 is installed, or images collected for a task that the object detector 200 is to perform, or unlabeled images that are not labeled for use as training images.

[0065] For example, test images can be driving images acquired by cameras, lidar, radar, etc., during vehicle operation.

[0066] Then, the active learning device 1000 can generate test cropped images 1 to n by cropping the region corresponding to the test bounding box in each test image. n can be an integer greater than or equal to 1.

[0067] In addition, the active learning device 1000 can generate test first resized image 210_1 to test (n+1) resized image 210_n+1 by resizing the test image and the test first cropped image to the test nth cropped image corresponding to the test image to the same size.

[0068] Then, the active learning device 1000 inputs the test first-size adjustment images 210_1 to the test (n+1)th size adjustment images 210_n+1 to the data encoder 310 of the depth autoencoder, which enables the data encoder 310 to encode the test first-size adjustment images 210_1 to the test (n+1)th size adjustment images 210_n+1 and output the test first data code to the test (n+1)th data code corresponding to the test first-size adjustment images 210_1 to the test (n+1)th size adjustment images 210_n+1.

[0069] At this time, the active learning device 1000 can obtain test data codes 1 to n+1 by applying activation functions such as sigmoid to the output of the data encoder 310.

[0070] Furthermore, the data code can be represented by a range of [0, 1] in dimension d, and can become a subspace associated with each resized image. d can be an integer greater than or equal to 1. Additionally, the data code corresponding to the background region can be represented by all zeros. Furthermore, the data code associated with the resized image corresponding to the test bounding box for detecting errors can be mapped to the background code, i.e., all zeros.

[0071] Then, refer to Figure 3 The active learning device 1000 can refer to the data codebook 410 to confirm the reference data code corresponding to the number of size-adjusted images less than or equal to a counter threshold; extract test-specific data codes that match the reference data codes from the test data codebook 1 to the test data codebook n+1; select the test-specific images corresponding to the test-specific data codes in the test images as rare samples for retraining the object detector 200; update the data codebook 410 with reference to the test-specific data codes corresponding to the rare samples, wherein the data codebook 410 is generated or updated using previously learned images used to train the object detector 200. The data codebook 410 can list the number of size-adjusted images for each data code in the previously learned images, and the process of generating the data codebook 410 will be described later.

[0072] At this time, the counter threshold can be a preset number of resized images, or the number of resized images corresponding to the q-th threshold data code in the data codebook 410 arranged in descending order of the number of resized images. q can be an integer greater than or equal to 1.

[0073] That is, the fewer the number of images corresponding to the subspace that serves as the data code, the more likely it is to be identified as a rare subspace that was not adequately trained during the previous training of the object detector 200. Thus, the test-specific images corresponding to the test-specific data codes that were not adequately trained during the previous training of the object detector 200 can be sampled as rare samples (i.e., rare images) using the data codebook 410. The data codebook 410 is then updated with the test-specific data codes corresponding to the sampled test-specific images to retrain the object detector 200 and used as a standard for sampling new rare images from the new test images.

[0074] Additionally, the data codebook 410 can be continuously updated each time the object detector 200 is retrained.

[0075] Then refer to Figure 4 The reason why a specific image was selected as a rare sample for testing can be confirmed by the data codebook formed by the method described above.

[0076] That is, test images can be described using scene and object levels through the data codebook, and the reasons for being selected as rare samples can be confirmed through scene and object levels.

[0077] For example, it can be confirmed that stop signs appear less frequently when vehicles are driving on roads during the day, i.e., when the test image is a "day road" scenario.

[0078] Additionally, it can be confirmed that even the same vehicle will show spots and smudges due to the rain when detected in the "heavy rain" scene, making it rarer at the object level compared to the vehicle detected in the "day highway" scene.

[0079] Then, the active learning device 1000 retrains the object detector 200 using previously learned images and rare samples.

[0080] On the other hand, the method for training a deep autoencoder and the process for generating a data codebook are described below.

[0081] First, refer to Figure 5 Describe a method for training a deep autoencoder.

[0082] The active learning device 1000 samples training images from a training image database for training the deep autoencoders 310 and 320, the training image database storing training images for training the target model (i.e., the object detector).

[0083] Additionally, the active learning device 1000 can crop training object regions from each training image to generate training cropped object images 1 to t; and crop background regions from each training image to generate training cropped background images 1 to u, wherein each training image is sampled from a training image database used to train the object detector. t and u can be integers greater than or equal to 1.

[0084] At this point, the background area can be randomly cropped, so that the number of background images cropped is one to three times the number of object images cropped.

[0085] Then, the active learning device 1000 generates training first-sized object images 20_1 to training t+1 sized object images 20_t+1 and training first-sized background images 30_1 to training t-1 sized background images 30_u by resizing the training images, corresponding to the training first cropped object images to the training t+1 cropped object images for each training image, and the training first cropped background images to the training u-th cropped background images for each training image, by resizing them to the same size.

[0086] Then, the active learning device 1000 can input the training first-size resized object images 20_1 to t+1, and the training first-size resized background images 30_1 to u-size resized background images 30_u into the data encoder 310 of the depth autoencoder, so that the data encoder 310 encodes each training first-size resized object image 20_1 to t+1, and the training first-size resized background images 30_1 to u-size resized background images 30_u, and outputs training first-object data codes to training t+1 object data codes corresponding to the training first-size resized object images 20_1 to t+1, and training first-background data codes to training u-background data codes corresponding to the training first-size resized background images 30_1 to u-size resized background images 30_u.

[0087] Then, the active learning device 1000 can input the training object data code from the first training object data code to the training object data code from the t+1th training object data code to the data decoder 320 of the depth autoencoder, so that the data decoder 320 decodes each training object data code from the first training object data code to the training object data code from the t+1th training object data code and outputs the training reconstructed image from the first training key to the training t+1th training key image.

[0088] Additionally, the active learning device 1000 can train the data decoder 320 and the data encoder 310 using the first object loss to the t+1th object loss from the first size-adjusted object image 20_1 to the t+1th size-adjusted object image 20_t+1 and the first weighted key image to the t+1th weighted key image; and train the data encoder 310 using the background loss from the first background data code to the uth background data code.

[0089] At this point, the object loss can be expressed as follows.

[0090]

[0091] Where I can be a resized image and R can be a keyed image.

[0092] In addition, the background loss can be represented as follows.

[0093]

[0094] That is, the active learning device 1000 reflects all images of the training images and cropped object images of each object into the code loss and reconstruction loss for training the data decoder 320 and the data encoder 310, and reflects the cropped background image for the background only into the code loss, such that the output code is all zeros for training the data encoder 310.

[0095] On the other hand, in the above-mentioned active learning device 1000 trains the data encoder 310, but unlike this, the data encoder 310 can also be trained by another learning device.

[0096] Then, refer to Figure 6 Describe the method for generating the data codebook.

[0097] The active learning device 1000 samples evaluation images from a training image database used to generate a data codebook, the training image database storing training images used to train the target model (i.e., the object detector).

[0098] At this point, the active learning device 1000 can sample training images from the training image database and divide the sampled training images into training images for training the data encoder 310 and evaluation images for generating the data codebook. Alternatively, the division of training and evaluation images can be performed using uniform random sampling, thereby avoiding the possibility that the distribution characteristics (i.e., the sparseness of the subspaces) of the training and evaluation images are different from each other.

[0099] Furthermore, the active learning device 1000 can generate evaluation first cropped object images to evaluation v-th cropped object images by cropping evaluation object regions from each evaluation image sampled from the training image database; and can generate evaluation first resized object images 40_1 to evaluation (v+1) resized object images 40_(v+1) by resizing the evaluation images and the evaluation first cropped object images to evaluation v-th cropped object images corresponding to each evaluation image to the same size. The v can be an integer greater than or equal to 1.

[0100] Then, the active learning device 1000 can input the evaluation first-size resized object images 40_1 to the evaluation (v+1)-size resized object images 40_(v+1) into the data encoder 310 that is being trained, so that the data encoder 310 encodes each evaluation first-size resized object image 40_1 to the evaluation (v+1)-size resized object image 40_(v+1) and outputs the evaluation first object data code to the evaluation (v+1) object data code corresponding to each evaluation first-size resized object image 40_1 to the evaluation (v+1)-size resized object image 40_(v+1). At this time, the number of data codes output by the data encoder 310 can be (number of objects + 1) / one evaluation image.

[0101] Furthermore, the active learning device 1000 can classify and evaluate object data codes from the first object data code to the (v+1)th object data code, and count the number of resized images corresponding to each data code to generate a data codebook 410. That is, the number of resized images for each data code can be mapped to each data code, thereby generating the data codebook 410. At this time, the image ID can be further mapped to the data codebook 410. At this time, the data codes in the data codebook 410 can become subspaces, and the fewer the number of resized images corresponding to a subspace, the more likely it is to be identified as a rare subspace, that is, a rare sample.

[0102] On the other hand, the active learning device 1000 described above generates a data codebook 410, but unlike this, the data codebook 410 can also be generated by another learning device or data codebook generation device.

[0103] Furthermore, the embodiments of the present invention described above can be implemented and recorded in the form of program commands executed by various computer components on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., individually or in combination. The program commands recorded on the computer-readable recording medium may be specifically designed or configured for the present invention, or may be program commands known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices such as ROMs, RAMs, and flash memory specifically configured to store and execute program commands. Examples of program commands include not only machine language code generated by a compiler, but also executable high-level language code such as that executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0104] The present invention has been described above with reference to specific components and other specific matters, limited embodiments, and accompanying drawings. However, these descriptions are intended to help to provide a more comprehensive understanding of the present invention. The present invention is not limited to the described embodiments, and those skilled in the art can attempt various modifications and variations based on these descriptions.

[0105] Therefore, the technical concept of the present invention should not be limited to the embodiments described above, and all contents of the claims of the present invention and their equivalent or equivalent transformations are within the technical concept of the present invention.

Claims

1. An interpretable active learning method for object detectors, which utilizes a deep autoencoder, wherein, Includes the following steps: a) After the test images are acquired, the active learning device inputs the test images into an object detector, so that the object detector detects test objects from each test image and outputs test bounding boxes corresponding to the test objects; by cropping the regions corresponding to the test bounding boxes in each test image, test first-cropped images to test n-cropped images are generated; by resizing the test images and test first-cropped images to test n-cropped images by the same size, test first-resized images to test n+1-resized images are generated; by inputting test first-resized images to test n+1-resized images into the data encoder of a depth autoencoder, the data encoder encodes test first-resized images to test n+1-resized images and outputs test first-data codes to test n+1-data codes corresponding to test first-resized images to test n+1-resized images, where n is an integer greater than or equal to 1; and b) The active learning device reference data codebook identifies reference data codes corresponding to the number of size adjustment images less than or equal to a counter threshold; extracts test-specific data codes matching the reference data codes from the first test data code to the (n+1)th test data code; selects the test-specific images corresponding to the test-specific data codes in the test images as rare samples for retraining the object detector; updates the data codebook with reference to the test-specific data codes corresponding to the rare samples, wherein the data codebook is generated or updated using previously learned images used to train the object detector, and the data codebook lists the number of size adjustment images for each data code in the previously learned images; In step b), The counter threshold is a preset number of resized images, or the number of resized images corresponding to the qth threshold data code in the data codebook arranged in descending order of the number of resized images, where q is an integer greater than or equal to 1.

2. The method according to claim 1, wherein, Further steps include: c) The active learning device retrains the object detector using previously learned images and the rare samples.

3. The method according to claim 1, wherein: In step a), The active learning device maps the test bounding boxes that detect errors in the test bounding boxes to background data codes.

4. The method according to claim 1, wherein: In step a), The active learning device sets the detection threshold of the object detector to be lower than when the object detector was trained.

5. The method according to claim 1, wherein: Further before step a) Includes the following steps, (a01) The active learning device crops a training object region from each training image to generate training first cropped object images to training t-th cropped object images; crops a background region from each training image to generate training first cropped background images to training u-th cropped background images; resizes the training images, the training first cropped object images to training t-th cropped object images corresponding to each training image, and the training first cropped background images to training u-th cropped background images corresponding to each training image by the same size, thereby generating training first resized object images to training t+1 resized object images and training first resized background images to training u-th resized background images; and combines the training first resized object images to training t+1 resized object images and the training... The first resized background image to the training u-th resized background image is input to the data encoder of the depth autoencoder, so that the data encoder encodes the training first resized object image to the training (t+1)-th resized object image and the training first resized background image to the training u-th resized background image to output training first object data code to training (t+1)-th object data code and training first background data code to training u-th background data code corresponding to the training first resized object image to the training (t+1)-th resized object image and training first background data code to training u-th background data code corresponding to the training first resized background image to the training u-th resized background image, wherein each training image is sampled from a training image database used to train the object detector, and t and u are integers greater than or equal to 1; and (a02) The active learning device inputs the training first object data code to the training (t+1)th object data code into the data decoder of the depth autoencoder, so that the data decoder decodes each of the training first object data code to the training (t+1)th object data code and outputs the training first key image to the training (t+1)th key image; trains the data decoder and the data encoder using the first object loss to the (t+1)th object loss with reference to the training first resized object image to the training (t+1)th resized object image and the training first key image to the training (t+1)th key image; and trains the data encoder using the background loss with reference to the training first background data code to the training (u)th background data code.

6. The method according to claim 5, wherein, Further steps include: (a03) The active learning device crops evaluation object regions from each evaluation image sampled from the training image database to generate evaluation first cropped object images to evaluation vth cropped object images; the evaluation images and the evaluation first cropped object images to evaluation vth cropped object images corresponding to each evaluation image are resized by the same size to generate evaluation first resized object images to evaluation v+1 resized object images; the evaluation first resized object images to evaluation v+1 resized object images are input into the database. According to the encoder, the data encoder encodes the evaluation first-size-adjusted object image to the evaluation v+1-size-adjusted object image to output evaluation first-object data code to evaluation v+1-object data code corresponding to the evaluation first-size-adjusted object image to the evaluation v+1-size-adjusted object image; classifies the evaluation first-object data code to the evaluation v+1-object data code by data code, and counts the number of resize-adjusted images corresponding to each data code to generate the data codebook, wherein v is an integer greater than or equal to 1.

7. The method according to claim 5, wherein: In step a01), The active learning device generates one to three times the number of training cropped background images from a single training image.

8. An active learning device for an object detector, which utilizes a deep autoencoder to perform interpretable active learning, wherein, include: A memory storing instructions for performing interpretable active learning using a deep autoencoder to sample training images for training an object detector; as well as A processor that performs the interpretable active learning using the deep autoencoder to sample the training images for training the object detector according to instructions stored in the memory, wherein: The processor executes the following process: I) After a test image is acquired, the test image is input to the object detector, so that the object detector detects a test object from each test image, thereby outputting a test bounding box corresponding to the test object; by cropping the region corresponding to the test bounding box in each test image, test first cropped images to test nth cropped images are generated; by resizing the test image and the test first cropped images to test nth cropped images by the same size, test first resized images to test n+1 resized images are generated; by inputting the test first resized images to test n+1 resized images to the data encoder of a depth autoencoder, so that the data encoder encodes the test first resized images to test n+1 resized images and outputs test first data codes to test n+1 data codes corresponding to the test first resized images to test n+1 resized images, where n is an integer greater than or equal to 1; and II) The reference data codebook identifies the reference data codes corresponding to the number of size-adjusted images less than or equal to a counter threshold; test-specific data codes matching the reference data codes are extracted from the test data codes 1 to 1 (n+1); test-specific images corresponding to the test-specific data codes in the test images are selected as rare samples for retraining the object detector; the data codebook is updated with reference to the test-specific data codes corresponding to the rare samples, wherein the data codebook is generated or updated using previously learned images used to train the object detector, and the data codebook lists the number of size-adjusted images for each data code in the previously learned images; In process II), the counter threshold is a preset number of resized images, or the number of resized images corresponding to the qth threshold data code in the data codebook arranged in descending order of the number of resized images, where q is an integer greater than or equal to 1.

9. The active learning device according to claim 8, wherein: The processor further executes (III) the process of retraining the object detector using previously learned images and the rare samples.

10. The active learning device according to claim 8, wherein: In the process described in I), the processor maps the test bounding box that detects errors in the test bounding box to the background data code.

11. The active learning device according to claim 8, wherein: In the process described in I), the processor sets the detection threshold of the object detector to be lower than that used when training the object detector.

12. The active learning device according to claim 8, wherein: The processor further performs the following process before the I) process: I01) cropping the training object region from each training image to generate the first cropped training object image to the t-th cropped training object image; cropping the background region from each training image to generate the first cropped training background image to the u-th cropped training background image; For the training images, the training first cropped object images to the training t-th cropped object images corresponding to each training image, and the training first cropped background images to the training u-th cropped background images corresponding to each training image, are resized by the same size to generate training first resized object images to training t+1 resized object images and training first resized background images to training u-th resized background images; the training first resized object images to the training t+1 resized object images and the training first resized background images to the training u-th resized background images are input into the data encoder of the depth autoencoder, so that the training... The data encoder encodes the training object images from the first resized object image to the (t+1)th resized object image and the training background images from the first resized background image to the uth resized background image to output training object data codes from the first resized object image to the (t+1)th resized object image, and training background data codes from the first resized background image to the uth resized background image, wherein each training image is sampled from a training image database used to train the object detector, and t and u are integers greater than or equal to 1; and (I02) Input the training first object data code to the training (t+1)th object data code into the data decoder of the depth autoencoder, so that the data decoder decodes each of the training first object data code to the training (t+1)th object data code and outputs the training first key image to the training (t+1)th key image; train the data decoder and the data encoder using the first object loss to the (t+1)th object loss with reference to the training first resized object image to the training (t+1)th resized object image and the training first key image to the training (t+1)th key image; train the data encoder using the background loss with reference to the training first background data code to the training (u)th background data code.

13. The active learning device according to claim 12, wherein: The processor further executes the following process, I03): Cropping evaluation object regions from each evaluation image sampled from the training image database to generate evaluation first cropped object images to evaluation vth cropped object images; resizing the evaluation images and the evaluation first cropped object images to evaluation vth cropped object images corresponding to each evaluation image by the same size to generate evaluation first resized object images to evaluation v+1 resized object images; inputting the evaluation first resized object images to evaluation v+1 resized object images. The data encoder encodes the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image to output evaluation first-object data code to evaluation (v+1)th-object data code corresponding to the evaluation first-size-adjusted object image to the evaluation (v+1)th-size-adjusted object image; the evaluation first-object data code to the evaluation (v+1)th-object data code are classified by data code, and the number of resize-adjusted images corresponding to each data code is counted to generate the data codebook, wherein v is an integer greater than or equal to 1.

14. The active learning device according to claim 12, wherein: During the I01 process, the processor generates a training cropped background image from a single training image that is one to three times the number of training cropped object images.

Citation Information

Patent Citations

  • System and method for active learning

    CN111052146A