OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

Abstract

An estimation unit (41) estimates a density map for a target image that is subject to object detection using a density estimation model (31) that has been trained using correct answer data in the form of density maps that indicate the probability density of an object being at respective locations in an image, the density maps being created by assigning a normal distribution that has a dispersion that corresponds to the size of an object to locations that correspond to the region of the object in a training image for which the region of the object is known, an extraction unit (42) extracts a region that corresponds to the locations on the density map at which the probability density is at or above a threshold value as a region to be subject to object detection, and a detection unit (43) applies an object detection model (32) to a region of the target image that corresponds to the extracted region to detect an object.

Description

[Technical Field]

[0001] The disclosed technology relates to an object detection device, an object detection method, and an object detection program. [Background technology]

[0002] An object detection device is a device that estimates the class of an object, such as a person or a vehicle, contained in an input image, the coordinate information of a rectangular bounding box that surrounds the object's area in the image, and the reliability of the detection result. In recent years, several object detection models using deep learning have been proposed. As object detection models based on deep learning, YOLO (You Only Look Once) and RetinaNet, which infer bounding boxes and object classes simultaneously, have been proposed (Non-Patent Documents 1 and 2). In addition, R-CNN, which detects candidate object regions and classifies objects separately, and an improved version of Faster R-CNN have also been proposed (Non-Patent Documents 3 and 4).

[0003] Furthermore, when performing object detection from high-resolution images or videos such as 4K or 8K, the input image size to the object detection model is limited to a maximum of 1536 x 1536 in YOLO v5, making it impossible to input high-resolution images at their original size. Therefore, a method has been proposed in which the input high-resolution image is divided to match the input image size of the object detection model, the object detection results from each divided image are aggregated, and the result of object detection from the entire image is output. Several methods have been proposed for this purpose, depending on the image division method. For example, a method has been proposed in which a group of images equally divided to match the input image size of the object detection model and a reduced image of the entire image are input to the object detection model. In this method, the coordinate information of the obtained bounding box is scaled, and the detection results of each divided image and the reduced image are combined to output the final result (Non-Patent Document 5).

[0004] In addition, a method has been proposed in which probability density estimation or cluster detection is used to estimate the distribution of areas in an image where objects exist, and then only areas where objects are predicted to exist are extracted and an object detection model is applied (Non-Patent Documents 6 and 7). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788. [Non-patent document 2] T. -Y. Lin et al., "Focal Loss for Dense Object Detection," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999-3007. [Non-patent document 3] R. Girshick et al., "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587. [Non-patent document 4] S. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017. [Non-Patent Document 5] H. Uzawa et al., "High-definition object detection technology based on AI inference scheme and its implementation", IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323. [Non-patent document 6] C. Li et al., "Density Map Guided Object Detection in Aerial Images," 2020 IEEE / CVF CVPRW, 2020, pp. 737-746. [Non-Patent Document 7] F. Yang et al., "Clustered Object Detection in Aerial Images," 2019 IEEE / CVF ICCV, 2019, pp. 8310-8319. Summary of the Invention [Problem to be solved by the invention]

[0006] In a method such as the method described in Non-Patent Document 5, which divides a high-resolution image equally and performs object detection, it is not possible to selectively apply object detection to each divided image. Therefore, as the number of divided images increases with the resolution of the input image, the amount of calculation required for object detection increases. On the other hand, in methods based on cluster detection or probability density estimation such as the methods described in Non-Patent Documents 6 and 7, the area in the image to which object detection is applied is narrowed down, which may enable a significant reduction in the amount of calculation depending on the image.

[0007] However, conventional methods that perform object detection based on the results of probability density estimation have the problem that they cannot detect relatively large objects in the input image.

[0008] The disclosed technology has been made in consideration of the above points, and aims to reduce the number of missed detections of relatively large objects in an input image. [Means for solving the problem]

[0009] A first aspect of the present disclosure is an object detection device including: an estimation unit that estimates a density map of a target image to be subjected to object detection using a density estimation model trained using as ground truth data a density map that indicates a probability density of an object being present at each position in an image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the object's region in a learning image where the object's region is known; an extraction unit that extracts, as a region to be subjected to object detection, a region in the density map estimated by the estimation unit corresponding to a position where the probability density is equal to or greater than a threshold; and a detection unit that detects an object by applying an object detection model that has been generated in advance by machine learning to detect an object from an image to a region of the target image corresponding to the region extracted by the extraction unit.

[0010] A second aspect of the present disclosure is an object detection method, in which an estimation unit estimates a density map of a target image to be subjected to object detection using a density estimation model trained using as ground truth data a density map that indicates the probability density of an object being present at each position in an image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the area of ​​the object in a learning image where the area of ​​the object is known; an extraction unit extracts, as an area to be subjected to object detection, an area in the density map estimated by the estimation unit corresponding to a position where the probability density is equal to or greater than a threshold; and a detection unit detects the object by applying an object detection model that has been generated in advance by machine learning to detect objects from images to an area of ​​the target image corresponding to the area extracted by the extraction unit.

[0011] A third aspect of the present disclosure is an object detection program that causes a computer to function as each unit of the object detection device. [Effects of the Invention]

[0012] According to the disclosed technology, it is possible to reduce the number of missed detections of relatively large objects in an input image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the object detection device. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the object detection device. [Figure 3] FIG. 10 is a diagram for explaining the creation of correct data for a density map. [Figure 4] 10A and 10B are diagrams for explaining improvement in accuracy of generating correct data for a density map. [Figure 5] 10A and 10B are diagrams for explaining improvement in accuracy of the estimation result of the density map. [Figure 6] 10 is a flowchart illustrating an example of a creation process. [Figure 7] 10 is a flowchart illustrating an example of a model learning process. [Figure 8]10 is a flowchart illustrating an example of a detection process. [Figure 9] 10 is a flowchart showing an example of rectangular area extraction processing in the first embodiment. [Figure 10] 10 is a flowchart showing an example of rectangular area extraction processing in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0015] <Problems with conventional methods based on probability density estimation> Before describing each embodiment in detail, a problem that a conventional method based on probability density estimation cannot detect a relatively large object in an input image will be described.

[0016] The density estimation model, which estimates a density map showing the probability density of an object at each position in an image, uses ground truth data of a probability density map created based on annotations from an object detection dataset. The object detection annotations include the class of the object in the image, the coordinate information of the bounding box (hereinafter also referred to as "Bbox"), and the vertical and horizontal lengths of the Bbox.

[0017] In the conventional technology, the average vertical length H and average horizontal length W of the Bbox are calculated for each object class, and these values ​​are used to create a normal distribution with variance V' shown in the following equation (1). In the conventional technology, this normal distribution is considered to be the probability density distribution of the object centered on the central coordinates of the Bbox, and these are superimposed to create ground truth data for the density map.

[0018]

number

[0019] For example, in an image captured at an oblique angle, such as with a surveillance camera, objects farther from the camera appear smaller, while objects closer to the camera appear larger, even if they are of the same class. Therefore, the conventional density map ground truth data generation method described above assigns only a probability density distribution much smaller than the size of objects close to the camera. A density estimation model trained using ground truth data from a density map created in this way cannot estimate an appropriate probability density distribution for relatively large objects in the image. As a result, it is not possible to properly extract the area where an object is predicted to exist, resulting in relatively large objects being overlooked in the image and resulting in missed detections during object detection.

[0020] In the following embodiments, a method for reducing the number of missed detections of relatively large objects in an input image is proposed.

[0021] First Embodiment Fig. 1 is a block diagram showing the hardware configuration of an object detection device 10 according to the first embodiment. As shown in Fig. 1, the object detection device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0022] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program for executing a creation process, a model learning process, and a detection process, which will be described later.

[0023] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including the operating system and various data.

[0024] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs. The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may employ a touch panel system and function as the input unit 15. The communication I / F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0025] Next, the functional configuration of the object detection device 10 according to the first embodiment will be described. Fig. 2 is a block diagram showing an example of the functional configuration of the object detection device 10. As shown in Fig. 2, the object detection device 10 includes, as its functional configuration, a learning unit 20, an estimation unit 41, an extraction unit 42, and a detection unit 43. The learning unit 20 further includes a creation unit 21 and a model learning unit 22. Each functional configuration is realized when the CPU 11 reads out an object detection program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

[0026] The creation unit 21 acquires training images stored in the training image DB 25, each having a known object region. As shown in the upper diagram of FIG. 3, each training image 51 is annotated with an object. The annotation includes coordinate information (x, y) of predetermined coordinates (e.g., center coordinates) of a Bbox, which is an area surrounding an object in the image, and the vertical length h and horizontal length w of the Bbox. Note that the example of FIG. 3 shows only a Bbox indicating the region of one object in the training image 51, but in reality, annotations are attached to all objects in the image.

[0027] The creation unit 21 creates ground truth data of a density map that indicates the probability density of an object being present at each position in the image from the training image 51. Specifically, the creation unit 21 creates ground truth data of the density map by allocating a predetermined density distribution having a variance according to the size of the object to a position in the training image 51 that corresponds to the area of ​​the object.

[0028] More specifically, the creation unit 21 prepares a density map 52 having the same size as the training image 51, with all pixels initialized to 0. Furthermore, as an example of a density distribution, the creation unit 21 prepares multiple normal distributions with different variances, as shown in the middle diagram of FIG. 3 . In the example of FIG. 3 , the numbers 32, 128, and 512 correspond to the variances of the respective normal distributions. That is, each normal distribution is a normal distribution that is distributed within a range corresponding to the respective sizes of 32×32 pixels, 128×128 pixels, and 512×512 pixels, with the center of the range being the average. Note that the above ranges are merely examples and can be set arbitrarily. Furthermore, any number of types of normal distributions may be prepared. The creation unit 21 treats this normal distribution as a probability density distribution for each object.

[0029] 3, the creation unit 21 selects, from among a plurality of normal distributions, a normal distribution that minimizes the difference between the value corresponding to the size of the object and the variance value of each normal distribution, and assigns the selected normal distribution to the density map 52, thereby creating ground truth data 53 for the density map. Specifically, the creation unit 21 obtains the vertical length h and horizontal length w of the Bbox for each object from the annotations added to the training image 51, and calculates V shown in the following equation (2).

[0030]

number

[0031] The creation unit 21 compares the calculated V with the variance of each of the multiple normal distributions prepared, and selects a normal distribution whose variance minimizes the difference from V. The creation unit 21 acquires the center coordinates (x, y) of the Bbox from the annotations added to the training image 51, sets the acquired (x, y) as the mean of the normal distribution, and considers the normal distribution to be the probability density distribution of the object and assigns it to the density map 52. As a result, each pixel in the density map 52 to which a normal distribution is assigned has a probability density value corresponding to the assigned normal distribution. In the example of FIG. 3, the second normal distribution from the left among the multiple normal distributions is selected, and the selected normal distribution is assigned to the corresponding position in the density map 52. The creation unit 21 repeatedly selects a normal distribution and assigns it to the density map 52 for all objects included in the training image 51, i.e., all Bboxes, to create final ground truth data 53 for the density map.

[0032] The creation unit 21 creates density map correct answer data 53 for each of the multiple training images 51 stored in the training image DB 25, and stores the pair of the training image 51 and the density map correct answer data 53 as training data in the training data DB 26.

[0033] 4 shows a comparison between the correct answer data 53 of the density map created by the creation unit 21 of this embodiment and the correct answer data 353 of the density map created by the conventional method. In the correct answer data 353 of the density map created by the conventional method, the same probability density distribution is assigned to the large car in the foreground and the small car in the background. On the other hand, in the correct answer data 53 of the density map created by the creation unit 21 of this embodiment, large objects are assigned a probability density distribution with a large variance that covers the entire object.

[0034] The model learning unit 22 uses the learning data stored in the learning data DB 26 to learn a density estimation model 31 that estimates a density map for an input image. Specifically, the model learning unit 22 updates the parameters of the density estimation model 31 so as to minimize the error between the density map estimation result obtained by inputting a learning image 51 to the density estimation model 31 and the ground truth data 53 of the density map paired with the learning image 51. The model learning unit 22 stores the learned density estimation model 31 in a predetermined storage area of ​​the object detection device 10.

[0035] The estimation unit 41 acquires a target image to be subjected to object detection. The estimation unit 41 estimates a density map of the target image using a density estimation model 31 trained by the learning unit 20. FIG. 5 shows a comparison between a density map 55 estimated by the estimation unit 41 of this embodiment and a density map 355 estimated by a conventional method for a target image 54. The conventional method is unable to estimate the probability density distribution of buses and cars in the foreground (the dotted white frame in the lower left diagram of FIG. 5). On the other hand, in this embodiment, by using the density estimation model 31 trained using the correct answer data 53 created by the creation unit 21, it is possible to estimate the probability density distribution of buses and cars in the foreground, which could not be estimated by the conventional method (the solid white frame in the lower right diagram of FIG. 5). As a result, this embodiment makes it possible to prevent the overlooking of relatively large objects in the image.

[0036] The extraction unit 42 extracts, as a target region for object detection, a region corresponding to a position where the probability density is equal to or greater than a threshold in the density map 55 estimated by the estimation unit 41 for the target image 54. Specifically, the extraction unit 42 generates a mask image in which the pixel values ​​whose probability density is equal to or greater than the threshold in the density map 55 estimated by the estimation unit 41 are set to 1 and the pixel values ​​whose probability density is less than the threshold are set to 0. The extraction unit 42 then extracts, as a target region for object detection, a rectangular region in the generated mask image that includes a portion where pixels with a value of 1 are consecutive and has a size corresponding to the input size to the object detection model 32 (described later). The extraction unit 42 assigns an ID, which is identification information, to each extracted rectangular region and passes rectangular information including the ID, the center coordinates of the rectangular region, and the vertical and horizontal lengths of the rectangular region to the detection unit 43.

[0037] Based on the rectangle information passed from the extraction unit 42, the detection unit 43 detects an object by applying the object detection model 32 to an area of ​​the target image 54 corresponding to the rectangular area extracted by the extraction unit 42. The object detection model 32 is generated in advance by machine learning to detect objects from images, and is stored in a predetermined storage area of ​​the object detection device 10. The object detection model 32 outputs metadata including at least the class of the object included in the input image, a Bbox indicating the area of ​​the object, and the reliability of the detection. The detection unit 43 outputs the metadata output by the object detection model 32 as the detection result.

[0038] Next, the operation of the object detection device 10 according to the first embodiment will be described. Fig. 6 is a flowchart showing the flow of a creation process performed by the object detection device 10. Fig. 7 is a flowchart showing the flow of a model learning process performed by the object detection device 10. Fig. 8 is a flowchart showing the flow of a detection process performed by the object detection device 10. The creation process, model learning process, and detection process are each performed by the CPU 11 reading an object detection program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0039] First, the creation process shown in FIG. 6 will be described.

[0040] In step S101, the CPU 11, functioning as the creation unit 21, acquires one training image 51 from the training image DB 25. Next, in step S102, the CPU 11, functioning as the creation unit 21, prepares a density map that is the same size as the training image 51 and in which all pixels are initialized to 0, and a plurality of normal distributions with different variances.

[0041] Next, in step S103, the CPU 11 as the creation unit 21 selects one Bbox that has not been subjected to the following processing from among the Bboxes included in the learning image 51. Next, in step S104, the CPU 11 as the creation unit 21 calculates V in the above equation (2) from the vertical length h and horizontal length w of the selected Bbox.

[0042] Next, in step S105, the CPU 11, as the creation unit 21, selects a normal distribution having a variance that minimizes the difference between the calculated V and the variance of each of the prepared normal distributions. Next, in step S106, the CPU 11, as the creation unit 21, assigns the pixel of the density map 52 corresponding to the center coordinates (x, y) of the Bbox to the density map 52 as the mean of the selected normal distribution.

[0043] Next, in step S107, the CPU 11, functioning as the creation unit 21, determines whether all B boxes included in the training image 51 have been processed. If all B boxes have been processed, the process proceeds to step S108, and if there are any unprocessed B boxes, the process returns to step S103. In step S108, the CPU 11, functioning as the creation unit 21, stores a pair of the created supervised data 53 of the density map and the training image 51 as training data in the training data DB 26, and the creation process ends.

[0044] The CPU 11 functions as the creating unit 21 and executes the above-described creating process for each of the plurality of learning images 51 stored in the learning image DB 25.

[0045] Next, the model learning process shown in FIG. 7 will be described.

[0046] In step S201, the CPU 11, functioning as the model learning unit 22, acquires a learning data set including a plurality of learning data from the learning data DB 26. Next, in step S202, the CPU 11, functioning as the model learning unit 22, inputs the learning images 51 to the density estimation model 31 and estimates a density map. Next, in step S203, the CPU 11, functioning as the model learning unit 22, updates the parameters of the density estimation model 31 so as to minimize the error between the estimated density map and the ground truth data 53 of the density map.

[0047] Next, in step S204, the CPU 11, functioning as the model learning unit 22, determines whether a learning termination condition is satisfied. For example, the CPU 11 determines that the learning termination condition is satisfied when the parameter update is repeated a predetermined number of times, when the error is equal to or less than a predetermined value, or when the difference between the previous error and the current error is equal to or less than a predetermined value. If the termination condition is satisfied, the process proceeds to step S205; if not, the process returns to step S201. In step S205, the CPU 11, functioning as the model learning unit 22, stores the learned density estimation model 31 in a predetermined storage area of ​​the object detection device 10, and the model learning process ends.

[0048] Next, the detection process shown in FIG. 8 will be described.

[0049] In step S301, the CPU 11 functions as the estimation unit 41 to acquire a target image 54 that is a target for object detection. Next, in step S302, the CPU 11 functions as the estimation unit 41 to estimate a density map 55 of the target image 54 using the density estimation model 31 learned by the learning unit 20.

[0050] Next, in step S310, the CPU 11 executes rectangular area extraction processing as the extraction unit 42. Here, the rectangular area extraction processing will be described with reference to FIG.

[0051] In step S311, the CPU 11 functions as the extraction unit 42 to select one pixel that has not yet been subjected to the following processing from among the pixels of the estimated density map 55. The CPU 11 functions as the extraction unit 42 to select pixels in the order from left to right and top to bottom, for example, starting from the top left pixel of the density map 55.

[0052] Next, in step S312, the CPU 11, functioning as the extraction unit 42, determines whether the probability density of the selected pixel is equal to or greater than a predetermined threshold. If the probability density is equal to or greater than the threshold, the process proceeds to step S313, and if it is less than the threshold, the process proceeds to step S314. In step S313, the CPU 11, functioning as the extraction unit 42, sets the value of the pixel to 1, and in step S314, the CPU 11, functioning as the extraction unit 42, sets the value of the pixel to 0.

[0053] Next, in step S315, the CPU 11, functioning as the extraction unit 42, determines whether or not all pixels in the density map 55 have been processed. If all pixels have been processed, the process proceeds to step S316, and if there are any unprocessed pixels, the process returns to step S311.

[0054] In step S316, the CPU 11, functioning as the extraction unit 42, detects a portion of consecutive pixels with a value of 1 in the mask image generated by the processes of steps S313 and S314. Next, in step S317, the CPU 11, functioning as the extraction unit 42, extracts rectangular regions that include the detected portion and have a size according to the input image size to the object detection model 32. Then, the CPU 11, functioning as the extraction unit 42, assigns an ID to each of the extracted rectangular regions, and passes rectangle information including the ID, the center coordinates of the rectangular region, and the vertical and horizontal lengths of the rectangular region to the detection unit 43, and returns to the detection process (FIG. 8).

[0055] Next, in step S320, CPU 11, functioning as detection unit 43, cuts out an area of ​​target image 54 corresponding to the rectangular area extracted by extraction unit 42, based on the rectangle information passed from extraction unit 42. Next, in step S321, CPU 11, functioning as detection unit 43, applies object detection model 32 to the image of the cut-out area to detect an object. Next, in step S322, CPU 11, functioning as detection unit 43, outputs the detection result, and the detection process ends.

[0056] As described above, the object detection device according to the first embodiment estimates a density map of a target image to be subjected to object detection using a density estimation model trained using ground truth data, the density map being a density map indicating the probability density of an object's presence at each position in an image. The density map is created by assigning a predetermined density distribution with a variance according to the size of the object to positions corresponding to object regions in a training image where the object regions are known. The object detection device extracts, as a target region for object detection, regions in the estimated density map corresponding to positions where the probability density is equal to or greater than a threshold, and detects the object by applying an object detection model, which has been generated in advance by machine learning to detect objects in images, to the regions of the target image corresponding to the extracted regions. This makes it possible to reduce the number of relatively large objects in the input image that go undetected.

[0057] Second Embodiment Next, a second embodiment will be described. In the object detection device according to the second embodiment, the same components as those in the object detection device 10 according to the first embodiment will be denoted by the same reference numerals, and detailed description thereof will be omitted. Furthermore, the hardware configuration of the object detection device according to the second embodiment is similar to the hardware configuration of the object detection device 10 according to the first embodiment shown in FIG. 1, and therefore description thereof will be omitted.

[0058] In the first embodiment, the extraction unit 42 extracts a rectangular area where an object is predicted to exist by applying a single threshold to the probability density of each pixel in the density map 55. In this case, setting the threshold high results in the extraction of a narrow rectangular area with a high probability density, whereas setting the threshold low results in the extraction of a wide rectangular area with a low probability density. Therefore, in order to be able to extract a rectangular area surrounding a large object, it is possible to set the threshold low. However, when setting the threshold low to extract a wide rectangular area, in order to simultaneously detect small objects and large objects, it is necessary to divide the area cut out from the target image 54 corresponding to the rectangular area according to the input image size of the object detection model 32. When the input image to the object detection model 32 is divided in this way, the area representing the large object may be cut off, potentially resulting in the object being detected as multiple different objects or not being detected at all.

[0059] In the second embodiment, taking this into consideration, a method is proposed in which a plurality of thresholds are set to more appropriately reduce the number of missed detections of relatively large objects in an input image.

[0060] Next, a functional configuration of the object detection device 210 according to the second embodiment will be described. As shown in Fig. 2, the object detection device 210 includes, as its functional configuration, a learning unit 20, an estimation unit 41, an extraction unit 242, and a detection unit 243. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0061] Similar to the extraction unit 42 of the first embodiment, the extraction unit 242 extracts, as a target region for object detection, a region corresponding to a position where the probability density is equal to or greater than a threshold in the density map 55 estimated by the estimation unit 41 for the target image 54. In this case, the extraction unit 242 sets a plurality of thresholds in multiple stages as thresholds for extracting the target region for object detection.

[0062] The number of threshold levels and the values ​​of each threshold level can be set arbitrarily. A trial run can be conducted in advance on a plurality of images, and the most accurate combination and number of threshold levels can be adopted. Alternatively, when performing object detection from video, the number of threshold levels and values ​​can be adaptively changed depending on the frame. In the latter case, the number of threshold levels and values ​​can be set based on the results of extracting a rectangular area and detecting an object in a frame prior to the target image in the video. For example, the number of threshold levels and values ​​can be set so that a rectangular area larger in size than the object detected in the previous frame is extracted.

[0063] In either case, the extraction unit 242 applies the set thresholds in order from the largest to the smallest. Due to the characteristics of the ground truth data used for training the density estimation model 31, when a large threshold is applied, a region where small objects are densely packed is extracted, and when a small threshold is applied, a region where large objects exist is extracted. For this reason, the extraction unit 242 extracts small rectangular regions when the threshold is large, and extracts large rectangular regions when the threshold is small.

[0064] Here, as in the first embodiment, it is desirable to set the size of the rectangular area to be extracted based on the input image size of the object detection model 32. For example, suppose that YOLO v3 with an input image size of 608 x 608 pixels is used as the object detection model 32 and there are three threshold levels. In this case, the extraction unit 242 may extract three types of rectangular areas, 608 x 608 pixels, 1216 x 1216 pixels, and 1824 x 1824 pixels, in descending order of threshold level.

[0065] The extraction unit 242 repeatedly extracts rectangular areas for each of the multiple thresholds in ascending order of threshold value for all threshold values. The extraction unit 242 then assigns an ID to each of the extracted rectangular areas and passes rectangle information including the ID, the center coordinates of the rectangular area, and the vertical and horizontal lengths of the rectangular area to the detection unit 243.

[0066] As in the first embodiment, the detection unit 243 detects an object by applying the object detection model 32 to the region of the target image 54 corresponding to the rectangular region extracted by the extraction unit 242, based on the rectangle information passed from the extraction unit 242. At this time, the detection unit 243 preferentially adopts and outputs the detection result of an object detected from a small rectangular region, that is, a region of the target image cut out according to a rectangular region extracted using a larger threshold. This makes it possible to suppress a decrease in detection accuracy for large objects due to object segmentation, and enables high-accuracy object detection even for large objects.

[0067] Next, the operation of the object detection device 210 according to the second embodiment will be described. In the second embodiment, the creation process shown in Fig. 6, the model learning process shown in Fig. 7, and the detection process shown in Fig. 8 are also performed. In the second embodiment, the rectangular area extraction process executed in step S310 of the detection process differs from that in the first embodiment. The rectangular area extraction process in the second embodiment will be described with reference to Fig. 10.

[0068] In step S401, the CPU 11, functioning as the extraction unit 242, sets the number of stages of multi-stage thresholds and their respective values. Next, in step S402, the CPU 11, functioning as the extraction unit 242, selects one threshold from the multiple thresholds for which the following processing has not yet been performed. Next, in step S403, the CPU 11, functioning as the extraction unit 242, executes selected threshold rectangular area extraction processing for the selected threshold. The selected threshold rectangular area extraction processing is similar to the rectangular area extraction processing of the first embodiment (FIG. 9). Next, in step S404, the CPU 11, functioning as the extraction unit 242, determines whether all set thresholds have been processed. If an unprocessed threshold exists, the process returns to step S402; if all thresholds have been processed, the process returns to the detection processing (FIG. 8).

[0069] As described above, the object detection device according to the second embodiment sets multiple thresholds in multiple stages as thresholds for extracting regions to be subjected to object detection from a density map estimated for a target image. This enables extraction of rectangular regions that are more in line with the size of the object, thereby improving the accuracy of object detection in high-resolution images and videos.

[0070] In the above embodiments, the creation process, model learning process, and detection process executed by the CPU after reading software (programs) may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after fabrication, and application-specific integrated circuits (ASICs) that are processors with circuit configurations specifically designed to execute specific processes. The creation process, model learning process, and detection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0071] In addition, in each of the above embodiments, the object detection processing program is described as being pre-stored (installed) in the ROM 12 or the storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0072] The following additional notes are provided regarding the above-described embodiments.

[0073] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: a density estimation model trained using a density map indicating the probability density of an object being present at each position in the image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the object's area in a training image where the object's area is known, as ground truth data; and extracting an area corresponding to a position where the probability density is equal to or greater than a threshold in the estimated density map as an area to be subjected to object detection; An object is detected by applying an object detection model that has been generated in advance by machine learning to an area of ​​the target image corresponding to the extracted area. The object detection device is configured as follows.

[0074] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to perform object detection processing, The object detection process includes: a density estimation model trained using a density map indicating the probability density of an object being present at each position in the image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the object's area in a training image where the object's area is known, as ground truth data; and extracting an area corresponding to a position where the probability density is equal to or greater than a threshold in the estimated density map as an area to be subjected to object detection; An object is detected by applying an object detection model that has been generated in advance by machine learning to an area of ​​the target image corresponding to the extracted area. Non-transitory recording media, including [Explanation of symbols]

[0075] 10, 210 Object detection device 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 Display 17 Communication I / F 19 Bus 20 Learning Department 21 Creation Department 22 Model Learning Section 25 Image database for learning 26 Learning Data DB 31 Density Estimation Model 32 Object Detection Model 41 Estimation part 42, 242 Extraction part 43, 243 Detector 51 training images 52 Density map of training images 53 Correct data 54 Target images 55 Density map of the target image

Claims

1. an estimation unit that estimates a density map of a target image that is a target for object detection using a density estimation model that has been trained using, as ground truth data, a density map that indicates a probability density of an object being present at each position in an image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the object's region in a learning image where the object's region is known; an extraction unit that extracts, as a target area for object detection, an area corresponding to a position where the probability density is equal to or greater than a threshold in the density map estimated by the estimation unit; a detection unit that detects an object by applying an object detection model that has been generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit, in order to detect an object from an image; An object detection device comprising:

2. The object detection device according to claim 1 , wherein the extraction unit sets a plurality of thresholds in multiple stages as the threshold value, and extracts a region to be subjected to the object detection, the region having a size corresponding to each of the plurality of threshold values.

3. 3. The object detection device according to claim 2, wherein the extraction unit sets a value of 1 for pixels in the density map estimated by the estimation unit whose probability density is equal to or greater than the threshold and a value of 0 for pixels whose probability density is less than the threshold, and extracts, as the region to be targeted for object detection, a region that includes a portion where pixels with a value of 1 are consecutive and has a size corresponding to the input size to the object detection model.

4. 3. The object detection device of claim 2, wherein the extraction unit, when using each frame of an input video as the target image, sets each value of the multi-stage threshold and the number of stages of the multi-stage threshold when extracting the area to be subjected to object detection from the target image based on the results of extracting the area and detecting the object for frames in the video that precede the target image.

5. The object detection device according to claim 1 , further comprising a learning unit that creates the density map that serves as the ground truth data from the learning image, and learns the density estimation model using the created density map.

6. 6. The object detection device according to claim 5, wherein the learning unit creates the correct answer data by treating the density distribution as a normal distribution and assigning to the density map, from among a plurality of normal distributions with different variances, the normal distribution that has the smallest difference between the value corresponding to the size of the object and the variance value.

7. the estimation unit estimates a density map of a target image that is a target for object detection using a density estimation model trained using, as ground truth data, a density map that indicates a probability density of an object being present at each position in the image, the density map being created by assigning a predetermined density distribution having a variance according to the size of the object to positions corresponding to the object's region in a learning image where the object's region is known; an extraction unit extracts, as a target area for object detection, an area corresponding to a position where the probability density is equal to or greater than a threshold in the density map estimated by the estimation unit; A detection unit detects an object by applying an object detection model that has been generated in advance by machine learning to a region of the target image corresponding to the region extracted by the extraction unit. Object detection methods.

8. An object detection program for causing a computer to function as each of the components constituting the object detection device according to any one of claims 1 to 6.

Images