Object scale estimation device, object scale estimation method, and object scale estimation system
The system addresses the challenge of estimating object numbers at varying distances by generating depth maps and using density calculations, ensuring accurate object counting in aerial imagery.
Patent Information
- Application Number
- JP2023083264
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
AI Technical Summary
Existing object detection and region extraction technologies struggle to accurately estimate the number of objects at varying distances in images, particularly when the distance between the imaging device and the objects is not constant, such as in aerial footage of disaster areas, leading to incomplete detection of objects far away.
The system generates a depth map of the target image, detects target objects, extracts estimated regions, estimates object density based on the depth map, and calculates the number of objects using density and area, incorporating methods like monocular depth estimation, stereo cameras, and laser-based depth measurement.
Enables accurate estimation of the number of objects at both near and far distances, providing reliable data for disaster assessment and reconstruction planning.
Smart Images

Figure 2024166873000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an object scale estimation device, an object scale estimation method, and an object scale estimation system. [Background technology]
[0002] In recent years, rapid estimation of the scale of damage has become necessary for planning rescue immediately after a disaster occurs and for formulating plans for reconstruction, and there are high expectations for advanced use of drones, which are capable of mobile wide-area photography. For example, by using video recognition (or image recognition) technology, it is possible to quickly estimate the amount of damaged houses, driftwood, and sediment.
[0003] Techniques for estimating the scale of an object from a captured image include those disclosed in Patent Documents 1 to 3.
[0004] Patent Document 1 discloses a technique for estimating the density distribution of an object in a specific region in an image. The image processing device in Patent Document 1 has an image acquisition function, a calculation function, a region acquisition function, and an estimation function. The image acquisition function acquires a target image. The calculation function calculates the density distribution of an object (person) included in the target image. The region acquisition function acquires a first region set in the target image. The estimation function estimates the density distribution of the first region in the target image based on the density distribution of a surrounding region of the first region in the target image.
[0005] Patent Document 2 provides a system that uses a surveillance camera to visualize the congestion level of an entire facility. The system detects crowds from surveillance camera footage, and estimates the congestion status within the camera footage by deriving the size (number of people) of the crowd from the edge amount. It also calculates the flow of people (the amount of people moving) based on a passer-by counter, the size and moving speed of the crowd, etc., and estimates the number of people in areas outside the camera footage based on a modeled crowd behavior pattern. The model is constructed by selecting a distribution function and its parameters using a statistical model selection method such as BIC (Bayesian Information Criteria). The congestion status is plotted on a map, allowing the congestion status of the area to be immediately grasped.
[0006] In addition, the object counting system 40 in Patent Document 3 includes an acquisition means 41 for acquiring information about an estimated area, which is a partial area of an image associated with predetermined conditions related to the objects to be counted shown in the image and is a unit of area for estimating the number of objects to be counted, a setting means 42 for setting an estimated area within the image so that the estimated area indicated by the acquired information about the estimated area includes objects to be counted that satisfy predetermined conditions and are not included in the objects to be counted in other estimated areas, an estimation means 43 for estimating, for each estimated area, the number of objects to be counted shown in the estimated area set in the image, and a calculation means 44 for calculating the density of the objects to be counted in areas where predetermined areas within the estimated area overlap, using the number of objects to be counted estimated in each estimated area. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] JP 2018-022343 A [Patent Document 2] International Publication No. 2017 / 122258 [Patent Document 3] International Publication No. 2017 / 043002 Summary of the Invention [Problem to be solved by the invention]
[0008] Object detection and region extraction technologies are becoming more and more accurate with the introduction of deep learning technology. Here, when the distance to the object is constant, such as in satellite image analysis, relatively stable object counting is possible. However, when the distance between the imaging device and the object is not constant, such as when estimating the scale of a disaster from aerial footage, objects in close range can be detected on an object instance basis, but objects in far range cannot be detected on an object instance basis. Measures for such cases are not anticipated in Patent Documents 1 to 3. Therefore, an object of the present invention is to provide a technique for estimating the number of target objects with high accuracy. [Means for solving the problem]
[0009] In order to solve the above problems, one representative object scale estimation device of the present invention includes a depth estimation unit that generates a depth map of a target image, which is an image that is the subject of an object scale estimation process, an object detection unit that detects a target object, which is an object to be detected, in the target image, an area extraction unit that extracts an estimation area that is an area estimated to contain the target object, a density estimation unit that estimates the density of the target objects based on the target objects and the depth map, and an object number estimation unit that estimates the area of the estimation area based on the estimation area and the depth map, and calculates a first object number, which is the number of target objects in the estimation area, based on the density and area of the target objects. Effect of the Invention
[0010] According to the present invention, the number of target objects can be estimated with high accuracy. Problems, configurations and effects other than those described above will become apparent from the description of the following embodiments. [Brief description of the drawings]
[0011] [Figure 1]FIG. 1 is a diagram showing a case where object detection and area extraction are performed in the process of estimating the scale of an object. [Diagram 2] FIG. 2 is a diagram showing the configuration of an object scale estimation system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of the configuration of the object scale estimation system according to the first embodiment as hardware. [Figure 4] FIG. 4 is a diagram showing a schematic diagram of the data flow in the processing of the object scale estimation system. [Diagram 5] FIG. 5 is a flowchart of an object scale estimation process in the object scale estimation system. [Figure 6] FIG. 6 is a diagram illustrating an example of an output of a result of an object-scale estimation process in the object-scale estimation system. [Figure 7] FIG. 7 is a diagram showing the configuration of an object scale estimation system according to the second embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of data stored in the image database. [Figure 9] FIG. 9 is a diagram showing a schematic diagram of the data flow in the processing of the object scale estimation system. [Figure 10] FIG. 10 is a flowchart illustrating the density estimation process. [Figure 11] FIG. 11 is a diagram illustrating an example of an output of a result of an object scale estimation process in the object scale estimation system. [Figure 12] FIG. 12 is a sequence diagram showing the process of estimating the object scale. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.
[0013] This disclosure includes a description of image analysis. The image to be analyzed is an image of the ground viewed from above with an imaging device placed in the sky. Therefore, the image may include a subject that is close to the imaging device and a subject that is far from the imaging device. In the following description, a "close distance" in an image refers to a subject that is close to the imaging device, and a "long distance" in an image refers to a subject that is far from the imaging device. Also, when we say object size, we mean the number of objects. Sometimes we also use the expression "number of objects."
[0014] [Conventional example] A conventional example of object scale estimation processing using image recognition will be described with reference to Fig. 1. Fig. 1 is a diagram showing a case where object detection and area extraction are performed in the object scale estimation processing. Fig. 1(a) shows a case where object detection is performed, and Fig. 1(b) shows a case where area extraction (semantic segmentation) is performed.
[0015] Object detection will be described with reference to FIG. 1(a). A captured image 10, which is the subject of object detection, is an example of an image captured by an imaging device mounted on a drone. A drone is a remotely controlled unmanned aerial vehicle. Since the image is captured by the imaging device mounted on the drone, the captured image 10 is associated with information such as the depression angle, which is the direction in which the imaging device faces from the horizontal direction to the vertical downward direction, and the altitude, which is the height at which the imaging device is located.
[0016] In the captured image 10, a road is shown extending from the center of the screen toward the back. The lower part of the captured image 10 corresponds to an area close to the imaging device, and is a mixture of houses and green spaces with lush trees. The information of the captured image 10 corresponds to an area far from the imaging device, and although the boundaries between the houses are not clearly shown, a residential area where multiple houses are gathered is formed. The upper end of the captured image 10 shows the boundary between the ground and the sky.
[0017] Object detection in an image can be performed using, for example, a CNN (Convolutional Neural Network) model, a YOLO (You Look Only Once) model, etc.
[0018] The post-object detection image 11 is a diagram that shows a schematic representation of a detected object. In the post-object detection image 11, a rectangular portion 12 is shown. The rectangular portion 12 indicates an area that has the characteristics of a house. Since 16 rectangular portions are shown in the post-object detection image 11, it has been detected that the captured image 10 contains 16 houses.
[0019] Region extraction will be described with reference to FIG. 1(b). Region extraction involves, for example, determining the feature amount of pixels, and extracting parts with similar features as the same region. Post-region extraction image 13 is a diagram showing a schematic representation of the extracted region. In post-region extraction image 13, the hatched portion Sa indicates the region that includes the house.
[0020] Here, the characteristics of object detection and region extraction will be described. As shown in the post-object detection image 11, it is possible to detect objects below the captured image 10, i.e., objects that are close to the imaging device. However, it is not possible to detect objects above the captured image 10, i.e., objects that are far from the imaging device. On the other hand, when extracting an area, it is possible to classify the area above the captured image 10, that is, the area far away from the imaging device, as an area that includes houses. However, it is not possible to count the number of houses included in the area.
[0021] For example, if we could determine the number of houses in a disaster-stricken area from images captured by a drone immediately after the disaster occurs, it would contribute to creating an effective support plan. However, when using conventional image recognition methods, it is not possible to determine the number of houses.
[0022] [First embodiment] (System Configuration) The configuration of the object scale estimation system according to the first embodiment will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing the configuration of the object scale estimation system according to the first embodiment. The object scale estimation system 100 detects objects to be detected (hereinafter also referred to as "target objects") in a target image, which is an image that is the subject of object scale estimation processing, and estimates the number of target objects (object scale).
[0023] The object scale estimation system 100 includes a storage device 101, an input device 102, a display device 103, and a main functional unit 104. The main functional unit 104 performs the main functions of the object scale estimation system. The main functional unit 104 can also be configured as an object scale estimation device.
[0024] The storage device 101 stores images captured by an imaging device. The stored images are subjected to object scale estimation processing. The storage device 101 is configured with any type of storage medium. The storage device 101 may be configured with a combination of multiple storage media, for example, a combination of storage devices such as semiconductor memories and hard disk drives. The storage device 101 may be a storage area on the cloud. The storage device 101 may be an imaging device, for example, an imaging device mounted on a drone. In this case, the main function unit 104 may directly acquire images from the imaging device. Alternatively, the storage device 101 may be configured to acquire images from a plurality of imaging devices, for example, a system in which a plurality of drone-mounted imaging devices and the main function unit 104 are connected to each other may be configured.
[0025] The input device 102 accepts input operations by a user of the object scale estimation system 100. The display device 103 displays the results of the object scale estimation.
[0026] 2 shows the main functional unit 104 mainly as a functional block. The main functional unit 104 has an image input unit 105, a region extraction unit 106, a depth estimation unit 107, an object detection unit 108, a density estimation unit 109, an image database 110, an object number estimation unit 111, and a display unit 112.
[0027] The image input unit 105 has a function as an interface between the storage device 101 and the main function unit 104. The image input unit 105 acquires, from the storage device 101, a target image that is an image on which an object scale estimation process is to be performed, and related information associated with the target image.
[0028] The region extraction unit 106 extracts an estimated region that is an area that is estimated to include a target object. The region extraction unit 106 performs a process of classifying (grouping) a target image into regions having common characteristics. For example, the region extraction unit 106 determines the feature amount of pixels, and extracts parts with similar features as the same region. The region extraction unit 106 can, for example, learn in advance teacher data that indicates the relationship between the feature amount and the object, and distinguish the region for each object. It is also possible to classify the region using feature amounts such as spectrum and texture without using teacher data. In the case of the first embodiment, it is possible to distinguish and extract regions into residential areas, forests, roads, waterside, etc.
[0029] The depth estimation unit 107 generates a depth map of the target image. The depth map represents the distance between the object to be captured (subject) and the imaging device for each pixel in the target image. As a method of generating a depth map, for example, it is possible to estimate the depth and generate the depth map based on the depression angle and altitude when the target image is captured. In addition, when the target image is an image captured by a monocular camera, there is a method of generating a depth map by performing monocular depth estimation (for example, a method of collecting teacher data associated with a depth map corresponding to an image, learning, and generating a depth map of the target image), a method of generating a depth map using parallax between cameras in a stereo camera, a method of combining an image with information from a distance measuring sensor using a laser, infrared, or the like, and the like. In addition, when the position information of the imaging device is associated with the target image, it is also possible to generate a depth map using the position information. The depth estimation unit 107 can select from these methods based on the image and related information.
[0030] The object detection unit 108 detects a target object, which is an object to be detected, in a target image. The object detection unit 108 detects the target object from the target image using, for example, a CNN model, a YOLO model, or the like.
[0031] The density estimation unit 109 estimates the density of the target object based on the target object and the depth map.
[0032] The image database 110 stores data relating to target images and features associated with the target images.
[0033] The object number estimation unit 111 estimates the area of the estimation region based on the estimation region and the depth map, and calculates the number of target objects in the estimation region based on the density and area of the target objects.
[0034] The display unit 112 outputs information relating to the calculated number of objects to the display device 103.
[0035] 3 is a diagram showing an example of the configuration of an object scale estimation system according to the first embodiment as hardware. The object scale estimation system 100 includes a storage device 101, an input device 102, a display device 103, a processor 201, a main storage device 202, and a NIC (Network Interface Card) 204.
[0036] The processor 201 is a processing device that converts, calculates, and processes data based on a certain processing procedure. The processor 201 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or the like, or may be configured by a combination of multiple processing devices.
[0037] The primary storage device 202 is configured by any type of storage medium, similar to the storage device 101. The primary storage device 202 may be configured by a combination of multiple storage media, for example, a combination of storage devices such as semiconductor memories and hard disk drives.
[0038] A processing program 203 stored in the main memory device 202 defines the processing procedures of each functional block (i.e., the image input unit 105, the area extraction unit 106, the depth estimation unit 107, the object detection unit 108, the density estimation unit 109, the object number estimation unit 111, and the display unit 112). The processor 201 executes the processing program 203 to realize the function of each functional block.
[0039] The NIC 204 is an interface for connecting devices constituting the main function unit 104 to a network. The storage device 101 is connected to the main function unit 104 via the NIC 204. The input device 102 and the display device 103 are connected to the processor 201 and the main storage device 202 via the network provided by the NIC 204.
[0040] The above hardware configuration is an example, and the present disclosure is not limited thereto. For example, since the storage device 101 is connected to a network via the NIC 204, a NAS (Network Attached Storage) or a SAN (Storage Area Network) is assumed. The storage device 101 may be a DAS (Direct Attached Storage). Furthermore, although the functions of the functional blocks are realized by a combination of the processor 201 and the main storage device 202, in order to distribute the processing load, the processing program 203 may be executed by using a combination of another processor and a main storage device in addition. Furthermore, the storage device 101, the input device 102, and the display device 103 function as an interface between the user of the object scale estimation system 100 and the main functional unit 104. As long as they function as an interface between the user and the main functional unit 104, they do not need to be separate devices directly connected to the main functional unit 104. For example, the display device 103 may be a mobile terminal capable of communicating with the main functional unit 104. The storage device 101, the input device 102, and the display device 103 can be appropriately set according to the usage mode of the user.
[0041] (Data flow for object scale estimation process) The process of estimating the scale of an object using depth and density will be described with reference to Fig. 4. Fig. 4 is a diagram showing a schematic diagram of the flow of data in the process of the object scale estimation system.
[0042] First, information indicating a target image 301 is loaded into the object scale estimation system 100. The target image 301 is captured, for example, by a drone, and is associated with information regarding the depression angle and altitude. The image is not limited to being captured by an aerial device such as a drone, and other images can also be applied to the present disclosure. For example, an image captured by an imaging device installed at a high place such as a watchtower may be used. Note that the image of the target image 301 itself is the same as the captured image 10 described above.
[0043] Next, in a depth estimation process 302, a depth map of the target image 301 is created. A depth map 303 is a diagram showing an example of a result when depth estimation is performed on the target image 301. The distance (depth) between the imaging device and the object to be imaged is extracted for each pixel, and the shade of color corresponds to the magnitude of the depth.
[0044] In area extraction processing 304, an area including a house is extracted from target image 301. A diagonally shaded portion Sa in post-area extraction image 305 indicates an area including a house.
[0045] In object detection processing 306, houses (objects) are detected from the target image 301. In a post-object detection image 307, 16 rectangular portions are included, and 16 houses are detected.
[0046] The accuracy of object detection is determined by the reliability of detection when a model is used for detection and the pixel size (size of pixels) included in the rectangular portion. For example, it is possible to set a reference value for the reliability and pixel size, and determine that an object has been detected for a rectangular portion whose reliability and pixel size exceed the reference value, and not count the number of objects that have been detected if the reliability and pixel size are equal to or less than the reference value. In addition, it is also possible to set conditions for performing the region extraction process and the object detection process, such as performing object detection for a portion in an image that is more likely to exceed the reference value as a close distance, and performing region extraction for a portion in a target image that is more likely to be equal to or less than the reference value as a long distance.
[0047] Next, density estimation processing 308 is performed. The object density in the short distance where the object detection processing was performed is estimated. Specifically, first, the area of the range in which the rectangular part of the object detection image 307 is detected is calculated using the depth map 303. The number of objects per unit area (density) is calculated from the calculated area and the number of objects. Density information 309, which is information indicating the density, is generated.
[0048] In area estimation processing 310, the area of the hatched portion Sa is estimated in the post-region extraction image 305. Area information 311 that is information indicating the area of the hatched portion Sa is generated.
[0049] In object number estimation processing 312, the number of objects (number of houses) included in the hatched portion Sa is estimated based on density information 309 of the number of objects in the short distance and area information of the hatched portion Sa. Specifically, even if it is assumed that the density of houses in the short distance and the density of houses in the long distance are the same, the number of houses included in the hatched portion Sa is estimated by multiplying the density of houses by the area of the hatched portion Sa. For example, if the density of houses is 0.0125 m -2 The area of the shaded area Sa is 7200 m 2 In this case, 0.0125 x 7200 = 90, and it is estimated that 90 buildings are included in the shaded area Sa.
[0050] The order of the processes may be set as appropriate. For example, the depth estimation process 302, the region extraction process 304, and the object detection process 306 may be performed in parallel or in series. Even in the case of performing the processes in series, the order of the processes may be set as appropriate.
[0051] (Procedure for estimating the scale of an object) The procedure for estimating the scale of an object using density and depth will be described with reference to Fig. 5. Fig. 5 is a diagram showing a flowchart of the process for estimating the scale of an object in the object scale estimation system.
[0052] First, the object scale estimation system 100 reads an image for which an object scale estimation process is to be performed (step S601). Information on a target image is read into the object scale estimation system 100 by the image input unit 105.
[0053] Next, the depth estimation unit 107 creates a depth map of the read target image (step S602). The depth estimation unit 107 estimates the depth using information associated with the target image (for example, the depression angle and altitude of the imaging device, etc.) and creates the depth map.
[0054] Next, the region extraction unit 106 extracts a region from the read image (step S603). The region extraction unit 106 extracts a region (estimated region) that is estimated to include a target object for the object scale estimation process. The estimation of the region is also performed on a part of the target image that is far away from the imaging device and cannot perform object detection, which will be described later.
[0055] Next, the object detection unit 108 detects objects from the read image (step S604). The object detection unit 108 detects target objects, which are objects to be detected, in the target image. In the detection of objects, objects that are generally close to the imaging device are detected in the target image. The object detection unit 108 counts the number of detected objects.
[0056] Next, the density estimation unit 109 estimates the density of objects in the short distance (step S605). The density estimation unit 109 estimates the density of the target objects based on the target objects detected by the object detection unit 108 and the depth map generated by the depth estimation unit 107. The estimated density is the density of objects in the short distance from the imaging device.
[0057] Next, the object number estimation unit 111 estimates the area of the long distance (step S606). Specifically, the object number estimation unit 111 estimates the area of the estimation area based on the estimation area extracted by the area extraction unit 106 and the depth map generated by the depth estimation unit 107.
[0058] Furthermore, the object number estimation unit 111 estimates the number of objects at a long distance from the density and area (step S607). Specifically, the object number estimation unit 111 calculates the number of target objects in the estimation region based on the density of the target objects estimated by the density estimation unit 109 and the area of the estimation region estimated in step S606.
[0059] Next, the display unit 112 outputs the number of objects at close range and long range (step S608). A specific output example will be described later.
[0060] The image database 110 stores information indicating the result of object detection by the object detection unit 108, the result of area extraction by the area extraction unit 106, and the result of object number estimation by the object number estimation unit 111. Storing the object number estimation result in association with each other makes it easy for the user to access past estimation results.
[0061] (Example of output of object scale estimation process) FIG. 6 is a diagram showing a schematic example of an output result of the object scale estimation process in the object scale estimation system 100. As shown in FIG.
[0062] In FIG. 6, the input device 102 is shown as a mouse and a keyboard. Similarly, the display device 103 is shown as a display, and the main function unit 104 is shown as a tower-type desktop computer. Also, on the display screen of the display device 103, a mouse cursor 700, an image reading button 701, an input image display section 702, an image recognition result display section 704, a scale estimation result display section 705, and a scale estimation execution button 703 are shown. The object scale estimation system 100 performs processing based on a user request inputted via the mouse and keyboard. For example, when the image reading button 701 is selected and a corresponding process is performed, the user performs an input operation via the keyboard or mouse with the mouse cursor 700 positioned on the image reading button 701.
[0063] When the image load button 701 is selected, a process of loading an image that is to be the target of the object scale estimation process is performed. The target image to be loaded may be a predetermined one. After the image load button 701 is selected, it is also possible to display the image data stored in the storage device 101 in a list format or the like to allow the user to select an image.
[0064] The input image display area 702 shows the loaded image. The user can confirm whether the image shown in the input image display area 702 is the target image for which the object scale estimation process is to be performed. If the image is not the desired image, the user can select the image load button 701 again and select the image to load.
[0065] When the scale estimation execution button 703 is selected, an object scale estimation process is performed on the loaded target image. The image recognition result display unit 704 displays a rectangular portion (first figure) indicating the target object detected by the object detection process and a diagonally shaded portion (second figure) indicating the estimated area extracted by the area extraction process in a manner superimposed on the loaded target image. The scale estimation result display unit 705 displays the number of objects in the near distance estimated by the object detection process and the number of objects in the far distance included in the estimated area estimated by the object number estimation process. Specifically, as shown by "106 houses", the number of houses estimated to be included in the image displayed on the input image display unit 702 is 106. The breakdown of the 106 houses is 16 houses in the near distance of the target image as shown by "near distance: 16 houses" and 90 houses in the far distance of the target image as shown by "far distance: 90 houses (estimated)". Note that FIG. 6 shows an example of a display mode, and the present disclosure is not limited thereto. For example, although a rectangular portion is displayed to display the target object, other polygonal shapes may be used, or a display mode such as hatching may be changed. Also, for the hatched portion, a display mode other than the hatched display may be used as long as the extracted area can be distinguished.
[0066] (Action and effect) When a captured image contains objects that are close to the image capture device and objects that are far from the image capture device, it is possible to grasp the number of objects of interest for objects that are close by using object detection processing. On the other hand, it is difficult to perform highly reliable object detection for objects that are far away. In addition, although it is possible to perform area extraction processing, it is difficult to obtain more information than this, as it is only possible to recognize that the area contains the object of interest.
[0067] In contrast, in the present invention, a depth map of the target image is generated, and the number of objects at a long distance is estimated by using the density of objects at a short distance. In this way, it is possible to estimate the size of objects at a long distance that are difficult to recognize by object detection processing. In addition, by displaying the estimation results, it is possible to provide information to the user. The results of the object detection process and the region extraction process are superimposed on the target image, and the number of objects detected by the object detection process is compared with the number of objects estimated using the depth map. This allows the user to intuitively grasp the accuracy of the object scale estimation process.
[0068] [Second embodiment] (System Configuration) The second embodiment differs from the first embodiment in that images in an image database are used for the object scale estimation process. In the following description, the same or equivalent components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be simplified or omitted.
[0069] 7 is a diagram showing the configuration of an object scale estimation system according to the second embodiment. An object scale estimation system 100a according to the second embodiment differs from the object scale estimation system 100 according to the first embodiment in that information stored in an image database 110a is used by a density estimation unit 109.
[0070] FIG. 8 is a diagram showing an example of data stored in the image database 110a. The image database 110a includes an image table that associates a template image with a template depth map feature that is a feature included in the depth map of the template image, an object table that includes objects and object types included in the template image, and a region table that indicates regions and region types extracted from the template image. Note that the template image is referred to as a template image in order to distinguish it from a target image that is the subject of an object scale estimation process. The template image is an image that is collected in advance before performing an object scale estimation process or an image that is acquired during the object scale estimation process. As will be described later, the template image is associated with a depth map generated during the object scale estimation process, a result of an object detection process, a result of a region extraction process, and the like. The data in the three tables will be described below.
[0071] FIG. 8(a) is a diagram showing an example of an image table. The image table manages images loaded into the object scale estimation system. The image table also manages depth maps and features for search. The image table includes columns of ID, image (hereinafter also referred to as "template image"), depth map, image features, and depth map features (hereinafter also referred to as "template depth map features"). Rows (records) of the table include template image data, depth maps generated from the template image, template image features, and depth map features. As the features of the template image, edge patterns, colors, Histograms of Oriented Gradients (HOG), Scaled Invariance Feature Transform (SIFT), etc. are detected for each pixel. As the features of the depth map, for example, an index indicating the distance from the imaging device is used. For example, an image and a depth map are shown in the row corresponding to ID=1. The image features are [0.627, 0.055, 0.479, 0.882, 0.728, …], and the depth map features are [0.550, 0.046, 0.540, 0.459, 0.098, …].
[0072] FIG. 8(b) is a diagram showing an example of an object table. The object table manages the results of the object detection process. The object table includes columns of ID, image ID, coordinates, type, and reliability. The rows of the table include image ID data, coordinate data indicating the area where the object is detected in the template image, label data indicating the type, and reliability data of the object detection. The object table and the image table are linked by the image ID. Specifically, image ID=1 indicates the template image of ID=1 in the object table. Regarding the data (records) included in the table, for example, the row corresponding to ID=1 indicates that when the object detection process is performed on the template image of image ID=1, a house is extracted as the type of object in the area surrounded by coordinates [0.42, 0.63, 0.6, 0.8]. The reliability of the object detection at the above coordinates is 0.95. In the object table shown in FIG. 8(b), multiple objects are detected for the template image of image ID=1. Although FIG. 8B shows a case where a house is detected as the type, the object table may also include results of detecting objects other than houses.
[0073] FIG. 8(c) is a diagram showing an example of a region table. The region table manages the results of region extraction processing. The region table includes columns of ID, image ID, region, and type. The rows of the table include image ID data, data indicating the range of the extracted region, and label data indicating the type. As in the case of the object table, the object table and the region table are linked by the image ID in the region table. For example, when region extraction processing is performed on a template image with image ID=1 for data included in the table, a diagonally shaded portion Sb indicating the extracted region is displayed as the region. The type of the extracted region is a residential area. Note that FIG. 8(c) shows a case where a residential area is detected as the type, but the region table may also include the results of detecting areas other than residential areas.
[0074] (Data flow for object scale estimation process) The process of estimating the scale of an object using a search for similar images will now be described with reference to Fig. 9. Fig. 9 is a diagram showing a schematic diagram of the flow of data in the process of the object scale estimation system.
[0075] Target image 801 is an image that is read into object scale estimation system 100a. In target image 801, an area where trees are growing extends from the front to the back on the right side of target image 801. Also, land with no vegetation and no buildings extends in the central part. There are a few buildings on the lower left side of target image 801, and multiple buildings on the back left side of target image 801.
[0076] In object detection processing 802, a house is detected from a target image 801. In a post-object detection image 803, one rectangular portion is included and one house is detected. Since only one house is detected, density estimation processing cannot be performed and it is determined that density estimation is impossible 804.
[0077] In this way, the density estimation process of the first embodiment cannot be performed when a sufficient number of objects are not captured in a close-range area, as in the target image 801. In contrast, in the second embodiment, a similar depth map is searched for among template images stored in the image database 110a, and density is estimated from the object detection results associated with the searched depth map.
[0078] In a depth estimation process 805, a depth map 806 of the target image 801 is formed.
[0079] Next, in a similar image search process 807, the image database 110a is searched for depth maps and related information similar to the depth map 806. Specifically, a depth map feature extracted from the depth map 806 (hereinafter also referred to as a "target depth map feature") is used as a query.
[0080] As shown in similar image search result 808, first, the depth map feature of the depth map 806 is compared with the depth map feature of the depth table, and the similarity between the depth map 806 and the depth map of the depth table is calculated. The depth map of image ID=2 has a similarity of 0.90. The depth map of the template image of image ID=1 has a similarity of 0.80. The depth map of the template image of image ID=3 has a similarity of 0.50. Since the image table contains three or more records, the similar image search process extracts template images (hereinafter also referred to as "similar template images") whose similarity is a predetermined value (here, 0.50) or more. Alternatively, it is possible to allow the user to select a similar template image.
[0081] Next, the number of objects in the similar template images extracted from the image database 110a is obtained. Specifically, the object table is referenced for the extracted similar template images, and the number of objects detected as the same type as the target object is calculated. The template image with image ID=2 includes 16 houses. The template image with image ID=1 includes 11 houses. The template image with image ID=3 includes 12 houses. When counting the number of objects, only those with a predetermined reliability or higher may be counted.
[0082] Next, the average density of the objects is calculated in density estimation processing 809. Specifically, the area of the hatched portion Sb shown in the region table is calculated, and the density is calculated based on the number and area of the objects.
[0083] The calculated density is applied to the target image 801 to estimate the number of objects contained in the target image 801.
[0084] (Density estimation process procedure) With reference to FIG. 10, a processing procedure of density estimation using a search for similar images will be described. FIG. 10 is a diagram showing a flowchart of density estimation processing. Among the object scale estimation processing, a part relating to density estimation will be described in particular. The density estimation unit 109 extracts a similar template image, which is a template image having a template depth map feature similar to a target depth map feature which is a feature of the depth map of the target image, from the image table, calculates a second object number which is the number of target objects included in the similar template image based on the object table, calculates a second density which is the density of target objects included in the region of the similar template image based on the second object number and the region table, and the object number estimation unit 109 calculates a first object number using the second density.
[0085] The object detection unit 108 detects objects from the read target image and counts the number of detected objects. The density estimation unit 109 determines whether the number of detected objects is equal to or greater than a predetermined number (step S901). If the number is equal to or greater than the predetermined number (YES in step S901), the density estimation unit 109 performs density estimation as shown in the first embodiment (step S909). On the other hand, if the number is less than the predetermined number (NO in step S901), the density estimation unit 109 searches for a similar template image, which is a template image similar to the target image to be subjected to the object scale estimation process, using the depth map feature as a query (step S902).
[0086] The search process for similar template images is repeatedly performed on records included in the image table of the image database 110a (steps S903 to S908). The density estimation unit 109 extracts template images with a similarity equal to or greater than a predetermined value from the image table of the image database 110a. Alternatively, the similarity calculated by the density estimation unit 109 and the template images in the image table can be presented to the user, allowing the user to select a similar template image. Extraction based on similarity and user selection may be combined. By employing user selection, the appearance characteristics of the image itself that is the target of the object scale estimation process are taken into consideration, and it is expected that the result of the object scale estimation process will be more realistic.
[0087] The density estimation unit 109 acquires objects of the template image extracted from the image database 110a (step S905). Specifically, the density estimation unit 109 counts the number of objects included in a template image of a certain image ID extracted from the image table of the image database 110a by referring to the object table. When counting the number of objects, those with a predetermined reliability or higher may be counted. If the number of objects is not the predetermined number or more (NO in step S906), the density estimation unit 109 performs processing on the next image.
[0088] If the number of objects is equal to or greater than the predetermined number (YES in step S906), the density estimation unit 109 calculates the density and updates the average density. Specifically, the density estimation unit 109 calculates the area of the region shown in the region table, and calculates the density based on the number and area of the objects. It performs the processes from step S903 to step S907 on the template image extracted from the image table, and calculates the average value of the calculated densities.
[0089] By using the average density here, multiple search results are taken into consideration, and it is expected that the estimated density will be a more accurate value. As a method for calculating the average value, in addition to the method using a simple average, various methods can be applied, such as a method of calculating the average by weighting according to the similarity.
[0090] (Example of object scale estimation output) 11 is a diagram showing a schematic output example of the result of the object scale estimation process in object scale estimation system 100. The difference from the output example in the first embodiment is that a similar image search result display section 1001 is displayed.
[0091] The similar image search result display section 1001 displays the image ID, the similarity of the depth map features (similarity between the target depth map features and the template depth map features), the number of objects in the template image corresponding to the image ID, and the depth map and template image in the image table.
[0092] (Procedure for estimating the scale of an object) The processing procedure for density estimation will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing the object scale estimation processing. User 1100 is a user of object scale estimation system 100a. Also, computer 1120 represents one aspect of object scale estimation system 100a realized as hardware, and is, for example, an electronic computer including input device 102, display device 103, and main function unit 104.
[0093] First, the user 1100 inputs a request to estimate the number of objects to the computer 1120 (step S1101). The electronic computer 1120 sends a request for an input image, which is an image for which the number of objects is to be estimated, to the storage device 101 (step S1102), and the storage device 101 sends the input image to the electronic computer 1120 (step S1103).
[0094] The computer 1120 performs a depth estimation process (step S1104), a region extraction process (step S1105), and an object detection process (detection of the number of objects in the vicinity) (step S1106) on the target image, which is an input image. Note that the process shown in FIG. 11 is an example, and the order of these three processes can be changed.
[0095] The computer 1120 registers the image recognition result, including values obtained by the depth estimation process, the area estimation process, and the object detection process, in the image database 110a (S1107). The image database 110a notifies the completion of registration (step S1108).
[0096] The computer 1120 shows the image recognition result display to the user 1100 (step S1109). The image recognition result display shows a rectangular portion indicating an object detected by the object detection process and a diagonally shaded portion indicating an estimated area extracted by the area extraction process, superimposed on the loaded target image, as shown in the image recognition result display unit 704 in Fig. 10 .
[0097] If the number of objects is not equal to or greater than the predetermined number in the object detection process (step S1106), the computer 1120 performs a density estimation process (step S1110). The density estimation process (step S1110) is a process from the similar image search (step S1111) to the selection of a template image to be used for density estimation (step S1114), as shown in FIG.
[0098] The computer 1120 performs a similar image search (step S1111) and extracts search results from the image database 110a (step S1112).
[0099] The calculator 1120 shows the search result display to the user 1100 (step S1113). The result display of the similar image search (and density estimation) includes the image ID, the similarity of the depth map feature (similarity between the target depth map feature and the template depth map feature), the number of objects in the template image corresponding to the image ID, and the depth map and images in the image table, as shown in the similar image search result display section 1001 in FIG.
[0100] The user 1100 selects a template image to be used for density estimation from the results of the similar image search (step S1114). The user's selection is input to the computer 1120.
[0101] The calculator 1120 estimates the number of objects at a long distance based on the selected template image (step S1115). The calculator 1120 displays the number of objects to the user 1100 (step S1116). The number of objects is displayed in the manner shown in the scale estimation result display section 705 of FIG.
[0102] (Action and effect) Depending on the subject being imaged, the captured image may contain only a few objects at close range, making it impossible to perform density estimation by object detection. Even in such cases, density estimation can be performed based on similar images, making it possible to perform object scale estimation processing.
[0103] [Other embodiments] In the present disclosure, a house is taken as the target of the object-scale estimation process, but the present disclosure is not limited thereto. For example, when flooding occurs in mountainous areas, driftwood may flow into a river. Driftwood may also be applied as the target of the object-scale estimation process. In addition, in resource mining facilities, mined resources may be left to pile up. Sediments may also be applied as the target of the object-scale estimation process.
[0104] In the present disclosure, a case has been described in which the object scale estimation process is performed on a single image, but the present disclosure is not limited to this. When capturing an object divided into multiple images, it is also possible to perform object scale estimation process on the object formed by combining the multiple images.
[0105] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention.
[0106] The following are some possible embodiments of the present invention, but the present invention is not limited to these. (Aspect 1) a depth estimation unit for generating a depth map of a target image, the target image being an image that is the subject of an object scale estimation process; an object detection unit that detects a target object that is a detection target in the target image; a region extraction unit that extracts an estimated region that is an area that is estimated to include the target object; a density estimation unit that estimates a density of the target object based on the target object and the depth map; an object number estimation unit that estimates an area of the estimation region based on the estimation region and the depth map, and calculates a first object number that is the number of target objects in the estimation region based on the density and the area of the target objects; An object scale estimation device comprising: (Aspect 2) The depth estimation unit generates the depth map based on a depression angle and an altitude when the target image is captured. 2. An object scale estimation device according to embodiment 1. (Aspect 3) the target image is an image captured by a monocular camera, The depth estimation unit performs monocular depth estimation to generate the depth map. 3. An object scale estimation device according to claim 1 or 2. (Aspect 4) an image table that associates a template image with a template depth map feature that is a feature included in a depth map of the template image; an object table including objects included in the template image and types of the objects; an image database including an area table indicating an area to be extracted from the template image and a type of the area; When the density estimation unit is unable to estimate the density, Extracting a similar template image from the image table, the similar template image being a template image having a template depth map feature similar to a target depth map feature that is a feature of the depth map of the target image; Calculating a second object number, which is the number of target objects included in the similar template image, based on the object table; calculating a second density, which is a density of the target objects included in the region of the similar template image, based on the second object count and the region table; The object number estimation unit calculating the first number of objects using the second density; 4. An object scale estimation device according to any one of aspects 1 to 3. (Aspect 5) The similar template image is selected by a user. 5. An object scale estimation device according to any one of aspects 1 to 4. (Aspect 6) a depth estimation step for generating a depth map of a target image, the target image being an image that is the subject of an object scale estimation process; an object detection step of detecting a target object that is an object to be detected in the target image; a region extraction step of extracting an estimated region which is a region estimated to include the target object; a density estimation step of estimating a density of the target object based on the target object and the depth map; an object number estimation step of estimating an area of the estimation region based on the estimation region and the depth map, and calculating a first object number, which is the number of target objects in the estimation region, based on the density and the area of the target objects; The object scale estimation method comprises: (Aspect 7) an imaging device that captures a target image that is a target image for object scale estimation processing; a depth estimation unit for generating a depth map of the imaging device; an object detection unit that detects a target object that is a detection target in the target image; a region extraction unit that extracts an estimated region that is an area that is estimated to include the target object; a density estimation unit that estimates a density of the target object based on the target object and the depth map; an object number estimation unit that estimates an area of the estimation region based on the estimation region and the depth map, and calculates a first object number that is the number of target objects in the estimation region based on the density and the area of the target objects; An object scale estimation system comprising: (Aspect 8) The object scale estimation system of embodiment 7, wherein the imaging device is mounted on a drone. (Aspect 9) The object scale estimation system according to embodiment 7 or 8, further comprising a display device that displays an image recognition result in which a first graphic representing the target object and a second graphic representing the estimated area are superimposed on the target image. [Explanation of symbols]
[0107] 10 captured image, 11 image after object detection, 12 rectangular part, 13 image after region extraction, 100, 100a object scale estimation system, 101 storage device, 102 input device, 103 display device, 104 main function unit, 105 image input unit, 106 region extraction unit, 107 depth estimation unit, 108 object detection unit, 109 object number estimation unit, 109 density estimation unit, 110, 110a image database, 111 object number estimation unit, 112 Display section, 201 processor, 202 main memory device, 203 processing program 301 target image, 302 depth estimation process, 303 depth map, 304 region extraction process, 305 Image after object detection, 306 Object detection processing, 307 Image after object detection, 308 Density estimation process, 309 Density information, 310 Area estimation process, 311 Area information, 312 Object number estimation process, 700 mouse cursor, 701 button, 702 input image display unit, 703: Size estimation execution button; 704: Image recognition result display section; 705: Size estimation result display section; 801 target image, 802 object detection process, 803 image after object detection, 804 density estimation impossible, 805 depth estimation process, 806 depth map, 807 Similar image search processing, 808 Similar image search results, 809 Density estimation processing, 1001 Similar image search result display unit, 1100 User, 1120 Computer, 1120 Calculator
Claims
1. a depth estimation unit for generating a depth map of a target image, the target image being an image that is the subject of an object scale estimation process; an object detection unit that detects a target object that is a detection target in the target image; a region extraction unit that extracts an estimated region that is an area that is estimated to include the target object; a density estimation unit that estimates a density of the target object based on the target object and the depth map; an object number estimation unit that estimates an area of the estimation region based on the estimation region and the depth map, and calculates a first object number that is the number of target objects in the estimation region based on the density and the area of the target objects; An object scale estimation device comprising:
2. The depth estimation unit generates the depth map based on a depression angle and an altitude when the target image is captured. The object scale estimation device according to claim 1 .
3. the target image is an image captured by a monocular camera, The depth estimation unit performs monocular depth estimation to generate the depth map. The object scale estimation device according to claim 1 .
4. an image table that associates a template image with a template depth map feature that is a feature included in a depth map of the template image; an object table including objects included in the template image and types of the objects; an image database including an area table indicating an area to be extracted from the template image and a type of the area; When the density estimation unit is unable to estimate the density, Extracting a similar template image from the image table, the similar template image being a template image having a template depth map feature similar to a target depth map feature that is a feature of the depth map of the target image; Calculating a second object number, which is the number of target objects included in the similar template image, based on the object table; calculating a second density, which is a density of the target objects included in the region of the similar template image, based on the second object count and the region table; The object number estimation unit calculating the first number of objects using the second density; The object scale estimation device according to claim 1 .
5. The similar template image is selected by a user. The object scale estimation device according to claim 4.
6. a depth estimation step for generating a depth map of a target image, the target image being an image that is the subject of an object scale estimation process; an object detection step of detecting a target object that is an object to be detected in the target image; a region extraction step of extracting an estimated region which is a region estimated to include the target object; a density estimation step of estimating a density of the target object based on the target object and the depth map; an object number estimating step of estimating an area of the estimation area based on the estimation area and the depth map, and calculating a first object number, which is the number of target objects in the estimation area, based on the density and the area of the target objects; The object scale estimation method comprises:
7. an imaging device that captures a target image that is a target image for object scale estimation processing; a depth estimation unit for generating a depth map of the imaging device; an object detection unit that detects a target object that is a detection target in the target image; a region extraction unit that extracts an estimated region that is an area that is estimated to include the target object; a density estimation unit that estimates a density of the target object based on the target object and the depth map; an object number estimation unit that estimates an area of the estimation region based on the estimation region and the depth map, and calculates a first object number that is the number of target objects in the estimation region based on the density and the area of the target objects; An object scale estimation system comprising:
8. The object scale estimation system of claim 7 , wherein the imaging device is mounted on a drone.
9. 8. The object scale estimation system according to claim 7, further comprising a display device that displays the image recognition result in which a first figure indicating the target object and a second figure indicating the estimated area are superimposed on the target image.
Citation Information
Patent Citations
Image processing system and image processing method
JP2018022343A
Object counting device, object counting method, object counting program, and object counting system
WO2017043002A1
Congestion-state-monitoring system
WO2017122258A1
