Object scale estimation device, object scale estimation method, and object scale estimation system

The object size estimation system addresses the challenge of estimating object numbers at varying distances by using depth maps and density calculations, enhancing the accuracy of damage assessment and support planning post-disasters.

WO2024241739A1PCT designated stage expired Publication Date: 2024-11-28HITACHI LTD

Patent Information

Application Number
PCT/JP2024/014500
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-04-10
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional image recognition techniques struggle to accurately estimate the number of objects in images captured from varying distances, such as aerial footage after a disaster, as they can detect close objects but fail to count distant ones effectively.

Method used

An object size estimation system that includes a depth estimation unit to generate a depth map, an object detection unit to identify target objects, a density estimation unit to calculate object density, and an object number estimation unit to calculate the number of objects based on density and area, using the depth map and region extraction.

Benefits of technology

Enables accurate estimation of the number of objects at both close and distant distances, improving the reliability of damage assessment and support planning after disasters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014500_28112024_PF_FP_ABST
    Figure JP2024014500_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The purpose of the present invention is to provide a technology for estimating with high precision the number of objects serving as a target. An object scale estimation device according to the present invention comprises: a depth estimation unit that generates a depth map of a target image which is an image to be subjected to a process for estimating object scale; an object detection unit that detects, in the target image, a target object which is an object to be detected; a region extraction unit that extracts an estimation region which is a region estimated to include the target object; a density estimation unit that estimates the density of the target objects on the basis of the target object and the depth map; and an object number estimation unit that estimates the area of the estimation region on the basis of the estimation region and the depth map and that calculates a first object number, which is the number of the target objects in the estimation region, on the basis of the area and the density of the target objects.
Need to check novelty before this filing date? Find Prior Art

Description

Object scale estimation device, object scale estimation method, and object scale estimation system

[0001] The present invention relates to an object scale estimation device, an object scale estimation method, and an object scale estimation system.

[0002] In recent years, rapid estimation of the scale of damage has become necessary for the formulation of rescue plans immediately after a disaster occurs and for the formulation of reconstruction plans, and expectations are rising for advanced uses of drones, which are capable of mobile wide-area photography. For example, by utilizing video recognition (or image recognition) technology, it is possible to quickly estimate the amount of damaged houses, driftwood, and sediment.

[0003] Techniques for estimating the scale of an object from a captured image include those disclosed in Patent Documents 1 to 3.

[0004] Patent Document 1 discloses a technology for estimating the density distribution of an object in a specific region of an image. The image processing device in Patent Document 1 includes an image acquisition function, a calculation function, a region acquisition function, and an estimation function. The image acquisition function acquires a target image. The calculation function calculates the density distribution of objects (people) included in the target image. The region acquisition function acquires a first region set within the target image. The estimation function estimates the density distribution of the first region within the target image based on the density distribution of a region surrounding the first region within the target image.

[0005] Patent Literature 2 provides a system that visualizes the congestion level of an entire facility using surveillance cameras. The system detects crowds from surveillance camera footage and derives the size (number of people) of the crowd from the edge amount, thereby estimating the congestion level within the camera's image range. It also calculates the flow of people (the amount of people moving) based on a passerby counter and the size and movement speed of the crowd, and estimates the number of people in areas outside the camera's image range based on modeled crowd behavior patterns. The model is constructed by selecting distribution functions and their parameters using statistical model selection methods such as BIC (Bayesian Information Criteria). The congestion level is plotted on a map, allowing users to instantly grasp the congestion level of the area.

[0006] In addition, the object counting system 40 in Patent Document 3 includes an acquisition means 41 that acquires information about an estimation area, which is a partial area of ​​the image associated with predetermined conditions related to the objects to be counted shown in the image and is a unit of area for estimating the number of objects to be counted; a setting means 42 that sets an estimation area within the image so that the estimation area indicated by the acquired information about the estimation area includes objects to be counted that satisfy the predetermined conditions and are not included in objects to be counted in other estimation areas; an estimation means 43 that estimates the number of objects to be counted shown in the estimation area set in the image for each estimation area; and a calculation means 44 that calculates the density of the objects to be counted in areas where predetermined areas within the estimation area overlap using the number of objects to be counted estimated in each estimation area.

[0007] JP 2018-022343 A International Publication No. 2017 / 122258 International Publication No. 2017 / 043002

[0008] The introduction of deep learning technology has led to improvements in the accuracy of object detection and region extraction technology. Here, when the distance to the object is constant, as in satellite image analysis, relatively stable object counting is possible. However, when the distance between the imaging device and the object is not constant, as in estimating the scale of a disaster from aerial footage, objects at close range can be detected on an object instance basis, but objects at long range cannot be detected on an object instance basis. Patent Documents 1 to 3 do not anticipate measures for such cases. Therefore, the present invention aims to provide a technology for estimating the number of target objects with high accuracy.

[0009] In order to solve the above problem, one representative object scale estimation device of the present invention includes a depth estimation unit that generates a depth map of a target image, which is an image that is the subject of object scale estimation processing; an object detection unit that detects a target object, which is an object to be detected, in the target image; a region extraction unit that extracts an estimation region, which is a region that is estimated to include the target object; a density estimation unit that estimates the density of the target object based on the target object and the depth map; and an object number estimation unit that estimates the area of ​​the estimation region based on the estimation region and the depth map, and calculates a first object number, which is the number of target objects in the estimation region, based on the density and area of ​​the target object.

[0010] According to the present invention, the number of target objects can be estimated with high accuracy. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments.

[0011] FIG. 1 is a diagram illustrating object detection and region extraction in an object scale estimation process. FIG. 2 is a diagram illustrating the configuration of an object scale estimation system according to a first embodiment. FIG. 3 is a diagram illustrating an example of the configuration of the object scale estimation system according to the first embodiment as hardware. FIG. 4 is a diagram illustrating a data flow in the processing performed by the object scale estimation system. FIG. 5 is a diagram illustrating a flowchart of the object scale estimation process performed by the object scale estimation system. FIG. 6 is a diagram illustrating an example of an output of the result of the object scale estimation process performed by the object scale estimation system. FIG. 7 is a diagram illustrating the configuration of an object scale estimation system according to a second embodiment. FIG. 8 is a diagram illustrating an example of data stored in an image database. FIG. 9 is a diagram illustrating a data flow in the processing performed by the object scale estimation system. FIG. 10 is a diagram illustrating a flowchart of density estimation processing. FIG. 11 is a diagram illustrating an example of an output of the result of the object scale estimation process performed by the object scale estimation system. FIG. 12 is a sequence diagram illustrating the object scale estimation process.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0013] This disclosure includes a description of image analysis. The image to be analyzed is an image of the ground captured from an imaging device placed in the sky looking down. Therefore, the image may include both subjects that are close to the imaging device and subjects that are far from the imaging device. In the following description, the term "close distance" in an image refers to a subject that is close to the imaging device, and the term "long distance" in an image refers to a subject that is far from the imaging device. Furthermore, the term "object scale" refers to the counting of the number of objects. The term "number of objects" may also be used.

[0014] [Prior Art] A conventional example of object scale estimation processing using image recognition will be described with reference to Fig. 1. Fig. 1 is a diagram showing a case where object detection and region extraction are performed in the object scale estimation processing. Fig. 1(a) shows a case where object detection is performed, and Fig. 1(b) shows a case where region extraction (semantic segmentation) is performed.

[0015] Object detection will be described with reference to FIG. 1( a). A captured image 10, which is the target of object detection, is an example of an image captured by an imaging device mounted on a drone. A drone is a remotely controlled unmanned aerial vehicle. Because the image is captured by the imaging device mounted on the drone, the captured image 10 is associated with information such as the depression angle, which is the direction in which the imaging device faces from the horizontal to the vertical downward direction, and the altitude, which is the height at which the imaging device is located.

[0016] The captured image 10 shows a road extending from the center of the screen toward the back. The lower part of the captured image 10 corresponds to an area near the imaging device, where houses and green spaces with lush trees are mixed together. The information in the captured image 10 corresponds to an area far from the imaging device, and although the boundaries between houses are not clearly shown, a residential area where multiple houses are clustered together is formed. The upper end of the captured image 10 shows the boundary between the ground and the sky.

[0017] Object detection in an image can be performed using, for example, a CNN (Convolutional Neural Network) model, a YOLO (You Look Only Once) model, or the like.

[0018] The post-object detection image 11 is a diagram that schematically illustrates a detected object. The post-object detection image 11 shows rectangular portions 12. The rectangular portions 12 indicate areas that have the characteristics of houses. Since the post-object detection image 11 shows 16 rectangular portions, it has been detected that the captured image 10 contains 16 houses.

[0019] Region extraction will be described with reference to Figure 1(b). Region extraction involves, for example, determining the feature values ​​of pixels and extracting parts with similar feature values ​​as the same region. Post-region extraction image 13 is a diagram showing the extracted region. In post-region extraction image 13, the hatched portion Sa indicates the region that includes the house.

[0020] Here, the characteristics of object detection and region extraction will be explained. As shown in the post-object detection image 11, it is possible to detect objects below the captured image 10, i.e., objects that are close to the imaging device. However, it is not possible to detect objects above the captured image 10, i.e., objects that are far from the imaging device. On the other hand, with regard to region extraction, it is possible to classify the area above the captured image 10, i.e., areas that are far from the imaging device, as areas that include houses. However, it is not possible to count the number of houses included in the area.

[0021] For example, if it were possible to determine the number of houses in a disaster-stricken area from images taken by a drone immediately after a disaster occurs, it would contribute to the formulation of an effective relief plan.However, when using conventional image recognition methods, it is not possible to determine the number of houses.

[0022] [First Embodiment] (System Configuration) The configuration of an object scale estimation system according to the first embodiment will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing the configuration of an object scale estimation system according to the first embodiment. The object scale estimation system 100 detects objects to be detected (hereinafter also referred to as "target objects") in a target image, which is an image that is the subject of object scale estimation processing, and estimates the number of target objects (object scale).

[0023] The object scale estimation system 100 includes a storage device 101, an input device 102, a display device 103, and a main function unit 104. The main function unit 104 performs the main functions of the object scale estimation system. The main function unit 104 can also be configured as an object scale estimation device.

[0024] The storage device 101 stores images acquired by an imaging device. The stored images are subjected to object scale estimation processing. The storage device 101 is configured with any type of storage medium. The storage device 101 may be configured with a combination of multiple storage media, for example, a combination of storage devices such as semiconductor memories and hard disk drives. The storage device 101 may be a storage area on the cloud. The storage device 101 may also be an imaging device, for example, an imaging device mounted on a drone. In this case, the main function unit 104 can also acquire images directly from the imaging device. The storage device 101 may also be configured to acquire images from multiple imaging devices, for example, a system in which multiple imaging devices mounted on drones are connected to the main function unit 104 can be configured.

[0025] The input device 102 accepts input operations from a user of the object scale estimation system 100. The display device 103 displays the results of the object scale estimation.

[0026] 2 shows the main functional unit 104 mainly as a functional block. The main functional unit 104 has an image input unit 105, a region extraction unit 106, a depth estimation unit 107, an object detection unit 108, a density estimation unit 109, an image database 110, an object number estimation unit 111, and a display unit 112.

[0027] The image input unit 105 functions as an interface between the storage device 101 and the main function unit 104. The image input unit 105 acquires, from the storage device 101, a target image that is an image on which an object scale estimation process is to be performed, and related information associated with the target image.

[0028] The region extraction unit 106 extracts an estimated region, which is a region estimated to include a target object. The region extraction unit 106 performs a process of classifying (grouping) the target image into regions with common characteristics. For example, the region extraction unit 106 determines the feature amounts of pixels and extracts parts with similar feature amounts as the same region. The region extraction unit 106 can, for example, learn in advance training data indicating the relationship between feature amounts and objects, and distinguish regions for each object. It is also possible to classify regions using feature amounts such as spectrum and texture without using training data. In the first embodiment, it is possible to distinguish and extract regions into residential areas, forests, roads, watersides, etc.

[0029] The depth estimation unit 107 generates a depth map for the target image. The depth map represents the distance between the captured object (subject) and the imaging device for each pixel in the target image. For example, a depth map can be generated by estimating depth based on the depression angle and altitude at the time the target image was captured. Other methods include generating a depth map by performing monocular depth estimation when the target image is captured using a monocular camera (e.g., a method of generating a depth map for the target image by collecting training data associated with a depth map corresponding to the image and learning the data), generating a depth map using the parallax between cameras in a stereo camera, and combining an image with information from a ranging sensor using a laser, infrared, or the like. Furthermore, if the location information of the imaging device is associated with the target image, the depth map can also be generated using the location information. The depth estimation unit 107 can select from these methods based on the image and related information.

[0030] The object detection unit 108 detects a target object, which is an object to be detected, in a target image. The object detection unit 108 detects the target object from the target image using, for example, a CNN model, a YOLO model, or the like.

[0031] The density estimation unit 109 estimates the density of the target object based on the target object and the depth map.

[0032] The image database 110 stores data relating to target images and feature amounts associated with the target images.

[0033] The object number estimation unit 111 estimates the area of ​​the estimation area based on the estimation area and the depth map, and calculates the number of target objects in the estimation area based on the density and area of ​​the target objects.

[0034] The display unit 112 outputs information relating to the calculated number of objects to the display device 103 .

[0035] 3 is a diagram showing an example of the configuration of the object scale estimation system according to the first embodiment as hardware. The object scale estimation system 100 includes a storage device 101, an input device 102, a display device 103, a processor 201, a main storage device 202, and a NIC (Network Interface Card) 204.

[0036] The processor 201 is a processing device that converts, calculates, and processes data based on a certain processing procedure. The processor 201 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or the like, or may be configured by a combination of multiple processing devices.

[0037] The main memory device 202 is configured by any type of storage medium, similar to the storage device 101. The main memory device 202 may be configured by a combination of multiple storage media, for example, a combination of storage devices such as semiconductor memories and hard disk drives.

[0038] A processing program 203 stored in the main memory device 202 defines the processing procedures of each functional block (i.e., the image input unit 105, the area extraction unit 106, the depth estimation unit 107, the object detection unit 108, the density estimation unit 109, the object number estimation unit 111, and the display unit 112). The processor 201 executes the processing program 203 to thereby realize the functions of each functional block.

[0039] The NIC 204 is an interface for connecting the devices that make up the main function unit 104 to a network. The storage device 101 is connected to the main function unit 104 via the NIC 204. The input device 102 and the display device 103 are connected to the processor 201 and the main storage device 202 via the network provided by the NIC 204.

[0040] The above hardware configuration is merely an example, and the present disclosure is not limited thereto. For example, since the storage device 101 is connected to a network via the NIC 204, a network-attached storage (NAS) or a storage area network (SAN) is assumed. The storage device 101 may also be a direct-attached storage (DAS). Although the functions of the functional blocks are realized by a combination of the processor 201 and the main storage device 202, the processing program 203 may be executed using a combination of another processor and a main storage device to distribute the processing load. The storage device 101, the input device 102, and the display device 103 function as interfaces between the user and the main function unit 104 of the object scale estimation system 100. As long as they function as an interface between the user and the main function unit 104, they do not need to be separate devices directly connected to the main function unit 104. For example, the display device 103 may be a mobile terminal capable of communicating with the main function unit 104. The storage device 101, the input device 102, and the display device 103 can be configured appropriately to suit the user's usage patterns.

[0041] (Data Flow in Object Scale Estimation Processing) The object scale estimation processing using depth and density will be described with reference to Fig. 4. Fig. 4 is a diagram schematically showing the data flow in the processing of the object scale estimation system.

[0042] First, information indicating a target image 301 is loaded into the object scale estimation system 100. The target image 301 is captured using, for example, a drone, and is associated with information regarding the depression angle and altitude. Images captured using aircraft equipment such as a drone are not limited to images captured using aircraft equipment, and other images may also be applied to the present disclosure. For example, an image captured by an imaging device installed at a high location such as a watchtower may also be used. Note that the image of the target image 301 itself is the same as the captured image 10 described above.

[0043] Next, in depth estimation processing 302, a depth map is created for the target image 301. Depth map 303 is a diagram schematically illustrating an example of the results when depth estimation is performed on the target image 301. The distance (depth) between the image capture device and the object being captured is extracted for each pixel, and the shade of color corresponds to the magnitude of the depth.

[0044] In region extraction processing 304, a region including a house is extracted from target image 301. A diagonally shaded portion Sa in post-region extraction image 305 indicates a region including a house.

[0045] In object detection processing 306, houses (objects) are detected from the target image 301. In a post-object detection image 307, 16 rectangular portions are included, and 16 houses are detected.

[0046] The accuracy of object detection is determined by the reliability of detection when a model is used and the pixel size (size of pixels) included in the rectangular portion. For example, it is possible to set reference values ​​for the reliability and pixel size, and determine that an object has been detected for a rectangular portion whose reliability and pixel size exceed the reference values, and not count objects whose reliability and pixel size are equal to or less than the reference values ​​as having been detected. It is also possible to set conditions for the region extraction process and object detection process, such as detecting objects in a close distance when a high percentage of the image exceeds the reference value, and detecting objects in a long distance when a high percentage of the target image is equal to or less than the reference value.

[0047] Next, density estimation processing 308 is performed. The density of objects at close ranges where object detection processing has been performed is estimated. Specifically, first, the depth map 303 is used to calculate the area of ​​the range in which a rectangular portion of the post-object detection image 307 is detected. From the calculated area and the number of objects, the number of objects per unit area (density) is calculated. Density information 309, which is information indicating the density, is generated.

[0048] In area estimation processing 310, the area of ​​the hatched portion Sa in the post-region extraction image 305 is estimated, and area information 311 indicating the area of ​​the hatched portion Sa is generated.

[0049] In object number estimation processing 312, the number of objects (number of houses) included in the hatched portion Sa is estimated based on the density information 309 of the number of objects in the short distance and the area information of the hatched portion Sa. Specifically, even if it is assumed that the density of houses in the short distance and the density of houses in the long distance are the same, the number of houses included in the hatched portion Sa is estimated by multiplying the density of houses by the area of ​​the hatched portion Sa. For example, if the density of houses is 0.0125 m -2 The area of ​​the shaded area Sa is 7200 m 2 In this case, 0.0125 x 7200 = 90, and it is estimated that 90 buildings are included in the shaded area Sa.

[0050] The order of the processes can be set as appropriate. For example, the depth estimation process 302, the region extraction process 304, and the object detection process 306 can be performed in parallel or serially. Even when performed serially, the order in which the processes are executed can be set as appropriate.

[0051] (Procedure for Estimating Object Scale) The procedure for estimating object scale using density and depth will be described with reference to Fig. 5. Fig. 5 is a diagram showing a flowchart of the object scale estimation process in the object scale estimation system.

[0052] First, the object scale estimation system 100 reads an image for which the object scale estimation process is to be performed (step S601). Information on the target image is read into the object scale estimation system 100 by the image input unit 105.

[0053] Next, the depth estimation unit 107 creates a depth map of the read target image (step S602). The depth estimation unit 107 estimates the depth using information associated with the target image (for example, the depression angle and altitude of the imaging device) and creates the depth map.

[0054] Next, the region extraction unit 106 extracts a region from the read image (step S603). The region extraction unit 106 extracts a region (estimated region) that is estimated to include a target object for object scale estimation processing. The estimation region is also performed on a portion of the target image that is far away from the image capture device and cannot be subjected to object detection (described later).

[0055] Next, the object detection unit 108 detects objects from the read image (step S604). The object detection unit 108 detects target objects, which are objects to be detected, in the target image. The object detection generally involves detecting objects that are close to the imaging device in the target image. The object detection unit 108 counts the number of detected objects.

[0056] Next, the density estimation unit 109 estimates the density of objects at a short distance (step S605). The density estimation unit 109 estimates the density of the target objects based on the target objects detected by the object detection unit 108 and the depth map generated by the depth estimation unit 107. The estimated density is the density of objects at a short distance from the imaging device.

[0057] Next, the object number estimation unit 111 estimates the area of ​​the long distance (step S606). Specifically, the object number estimation unit 111 estimates the area of ​​the estimation area based on the estimation area extracted by the area extraction unit 106 and the depth map generated by the depth estimation unit 107.

[0058] Furthermore, the object number estimation unit 111 estimates the number of objects at a long distance from the density and area (step S607). Specifically, the object number estimation unit 111 calculates the number of target objects in the estimation area based on the density of the target objects estimated by the density estimation unit 109 and the area of ​​the estimation area estimated in step S606.

[0059] Next, the display unit 112 outputs the number of objects at close range and at long range (step S608). Specific output examples will be described later.

[0060] The image database 110 stores information indicating the results of object detection by the object detection unit 108, the results of area extraction by the area extraction unit 106, and the results of object number estimation by the object number estimation unit 111. Storing the results of object number estimation in association with each other makes it easier for the user to access past estimation results.

[0061] (Example of Output of Object Scale Estimation Processing Result) FIG. 6 is a diagram schematically showing an example of output of the object scale estimation processing result in the object scale estimation system 100. In FIG.

[0062] 6 , the input device 102 is shown as a mouse and keyboard. Similarly, the display device 103 is shown as a display, and the main function unit 104 is shown as a tower-type desktop computer. Also shown on the display screen of the display device 103 are a mouse cursor 700, an image reading button 701, an input image display unit 702, an image recognition result display unit 704, a scale estimation result display unit 705, and a scale estimation execution button 703. The object scale estimation system 100 performs processing based on a user request input via the mouse and keyboard. For example, when the image reading button 701 is selected to perform the corresponding processing, the user performs an input operation via the keyboard or mouse with the mouse cursor 700 positioned on the image reading button 701.

[0063] When the image read button 701 is selected, a process of reading an image that is to be the target of the object scale estimation process is performed. The target image to be read may be a predetermined image. After the image read button 701 is selected, it is also possible to display the image data stored in the storage device 101 in a list format or the like, and allow the user to select an image.

[0064] The input image display section 702 shows the loaded image. The user can confirm whether the image displayed in the input image display section 702 is the target image for which the object scale estimation process is to be performed. If the image is not the desired image, the user can select the image load button 701 again and select the image to load.

[0065] When the scale estimation execution button 703 is selected, an object scale estimation process is performed on the loaded target image. The image recognition result display unit 704 displays a rectangular portion (first figure) indicating the target object detected by the object detection process and a diagonally shaded portion (second figure) indicating the estimated area extracted by the area extraction process, superimposed on the loaded target image. The scale estimation result display unit 705 also displays the number of close-range objects estimated by the object detection process and the number of long-range objects included in the estimated area estimated by the object number estimation process. Specifically, as indicated by "106 houses," the number of houses estimated to be included in the image displayed in the input image display unit 702 is 106. The 106 houses are broken down into 16 houses in the close range of the target image, as indicated by "Close range: 16 houses," and 90 houses in the long range of the target image, as indicated by "Long-range: 90 houses (estimated)." Note that FIG. 6 illustrates an example of a display mode, and the present disclosure is not limited thereto. For example, although a rectangular portion is displayed to display the target object, other polygonal shapes may be used, or a different display mode such as hatching may be used. Also, for the hatched portion, a display mode other than the hatched display may be used as long as the extracted area can be distinguished.

[0066] (Actions and Effects) When a captured image contains both objects close to the image capture device and objects far from the image capture device, it is possible to determine the number of objects of interest for objects close to the image capture device through object detection processing. On the other hand, it is difficult to perform highly reliable object detection for objects far from the image capture device. Furthermore, while it is possible to perform area extraction processing, it is only able to recognize that the area contains the object of interest, and it is difficult to obtain any more information than this.

[0067] In contrast, in the present invention, a depth map of the target image is generated and the density of objects at close range is used to estimate the number of objects at long range. This makes it possible to estimate the scale of objects at long range that are difficult to recognize using object detection processing. Furthermore, by displaying the estimation results, it is possible to provide information to the user. The results of the object detection processing and the region extraction processing are superimposed on the target image, and a display comparing the number of objects obtained by the object detection processing with the number of objects estimated using the depth map is presented to the user at once, allowing the user to intuitively grasp the accuracy of the object scale estimation processing.

[0068] [Second Embodiment] (System Configuration) The second embodiment differs from the first embodiment in that images in an image database are used for object scale estimation processing. In the following description, components that are the same as or equivalent to those in the first embodiment described above are denoted by the same reference numerals, and their description will be simplified or omitted.

[0069] 7 is a diagram showing the configuration of an object scale estimation system according to the second embodiment. The object scale estimation system 100a according to the second embodiment differs from the object scale estimation system 100 according to the first embodiment in that information stored in an image database 110a is used by a density estimation unit 109.

[0070] FIG. 8 is a diagram illustrating an example of data stored in the image database 110a. The image database 110a includes an image table that associates template images with template depth map features, which are features included in the depth map of the template images; an object table that includes objects and object types included in the template images; and a region table that indicates regions and region types extracted from the template images. Note that the term "template image" is used here to distinguish it from the target image that is the subject of the object scale estimation process. The template image refers to an image collected in advance before the object scale estimation process or an image acquired during the object scale estimation process. As will be described later, the template image is associated with the depth map generated during the object scale estimation process, the result of the object detection process, the result of the region extraction process, and the like. The data in the three tables is described below.

[0071] FIG. 8A shows an example of an image table. The image table manages images loaded into the object scale estimation system. The image table also manages depth maps and search features. The image table includes columns for ID, image (hereinafter also referred to as "template image"), depth map, image features, and depth map features (hereinafter also referred to as "template depth map features"). Each row (record) of the table includes template image data, a depth map generated from the template image, template image features, and depth map features. Features of the template image include edge patterns, color, histograms of oriented gradients (HOG), and scaled invariance feature transform (SIFT), etc., detected for each pixel. Features of the depth map include, for example, an index indicating the distance from the imaging device. For example, the row corresponding to ID=1 shows an image and a depth map. The image features are [0.627, 0.055, 0.479, 0.882, 0.728, ...], and the depth map features are [0.550, 0.046, 0.540, 0.459, 0.098, ...].

[0072] FIG. 8B shows an example of an object table. The object table manages the results of object detection processing. The object table includes columns for ID, image ID, coordinates, type, and confidence level. Each row of the table includes image ID data, coordinate data indicating the area where an object was detected in the template image, label data indicating the type, and object detection confidence level data. The object table and image table are linked by image ID. Specifically, image ID = 1 refers to the template image with ID = 1 in the object table. Regarding the data (records) included in the table, for example, the row corresponding to ID = 1 indicates that when object detection processing was performed on the template image with image ID = 1, a house was extracted as the object type in the area surrounded by coordinates [0.42, 0.63, 0.6, 0.8]. The object detection confidence level at the above coordinates is 0.95. In the object table shown in FIG. 8B, multiple objects were detected for the template image with image ID = 1. Although FIG. 8B shows a case where a house is detected as the type, the object table may also include results of detecting objects other than houses.

[0073] FIG. 8( c) is a diagram showing an example of a region table. The region table manages the results of region extraction processing. The region table includes columns for ID, image ID, region, and type. The rows of the table include image ID data, data indicating the range of the extracted region, and label data indicating the type. As with the object table, the object table and region table are linked by the image ID. For example, when region extraction processing is performed on a template image with image ID = 1 for the data included in the table, a diagonally shaded portion Sb indicating the extracted region is displayed as the region. The type of the extracted region is residential area. Note that FIG. 8( c) shows a case where a residential area is detected as the type, but the region table may also include results of detecting areas other than residential areas.

[0074] (Data Flow in Object Scale Estimation Processing) The object scale estimation processing using similar image search will be described with reference to Fig. 9. Fig. 9 is a diagram schematically showing the data flow in the processing of the object scale estimation system.

[0075] The target image 801 is an image that is loaded into the object scale estimation system 100a. In the target image 801, an area where trees grow extends from the front to the back on the right side of the target image 801. In addition, a large area of ​​land with no vegetation or buildings extends in the central part. There are a few buildings on the lower left side of the target image 801, and multiple buildings on the back left side of the target image 801.

[0076] In object detection processing 802, a house is detected from target image 801. In post-object detection image 803, one rectangular portion is included and one house is detected. Because only one house is detected, density estimation processing cannot be performed and it is determined that density estimation is impossible 804.

[0077] In this way, the density estimation process of the first embodiment cannot be performed when a sufficient number of objects are not captured in a close-range area, as in the target image 801. In contrast, in the second embodiment, a similar depth map is searched for among template images stored in the image database 110a, and density is estimated from the object detection results associated with the searched depth map.

[0078] In a depth estimation process 805, a depth map 806 of the target image 801 is formed.

[0079] Next, in a similar image search process 807, the image database 110a is searched for depth maps and related information similar to the depth map 806. Specifically, a depth map feature extracted from the depth map 806 (hereinafter also referred to as a "target depth map feature") is used as a query.

[0080] As shown in similar image search results 808, first, the depth map feature of the depth map 806 is compared with the depth map feature of the depth table, and the similarity between the depth map 806 and the depth map of the depth table is calculated. The depth map of image ID = 2 has a similarity of 0.90. The depth map of the template image of image ID = 1 has a similarity of 0.80. The depth map of the template image of image ID = 3 has a similarity of 0.50. Since the image table contains three or more records, the similar image search process extracts template images (hereinafter also referred to as "similar template images") whose similarity is greater than or equal to a predetermined value (here, 0.50). Alternatively, it is also possible to have the user select a similar template image.

[0081] Next, the number of objects in the similar template images extracted from the image database 110a is obtained. Specifically, the object table is referenced for the extracted similar template images, and the number of objects detected as the same type as the target object is calculated. The template image with image ID = 2 includes 16 houses. The template image with image ID = 1 includes 11 houses. The template image with image ID = 3 includes 12 houses. When counting the number of objects, only those with a predetermined reliability or higher may be counted.

[0082] Next, the average density of the objects is calculated in density estimation processing 809. Specifically, the area of ​​the hatched portion Sb shown in the region table is calculated, and the density is calculated based on the number and area of ​​the objects.

[0083] The calculated density is applied to the target image 801 to estimate the number of objects contained in the target image 801.

[0084] (Processing Procedure for Density Estimation Processing) With reference to FIG. 10 , a processing procedure for density estimation using a search for similar images will be described. FIG. 10 is a diagram showing a flowchart of the density estimation processing. The density estimation portion of the object scale estimation processing will be described in particular. The density estimation unit 109 extracts, from the image table, a similar template image, which is a template image having template depth map features similar to target depth map features, which are features of the depth map of the target image, calculates a second object number, which is the number of target objects included in the similar template image, based on the object table, calculates a second density, which is the density of target objects included in a region of the similar template image, based on the second object number and the region table, and the object number estimation unit 111 calculates a first object number using the second density.

[0085] The object detection unit 108 detects objects from the loaded target image and counts the number of detected objects. The density estimation unit 109 determines whether the number of detected objects is equal to or greater than a predetermined number (step S901). If the number is equal to or greater than the predetermined number (YES in step S901), the density estimation unit 109 performs density estimation as described in the first embodiment (step S909). On the other hand, if the number is less than the predetermined number (NO in step S901), the density estimation unit 109 uses the depth map feature as a query to search for similar template images that are similar to the target image that is the target of object scale estimation processing (step S902).

[0086] The search process for similar template images is repeatedly performed on records included in the image table of the image database 110a (steps S903 to S908). The density estimation unit 109 extracts template images from the image table of the image database 110a whose similarity is equal to or greater than a predetermined value. Alternatively, the similarity calculated by the density estimation unit 109 and the template images in the image table can be presented to the user, allowing the user to select a similar template image. Extraction based on similarity and user selection may be combined. By employing user selection, the visual characteristics of the image itself that is the target of the object scale estimation process are taken into consideration, which is expected to result in a more realistic object scale estimation process.

[0087] The density estimation unit 109 acquires objects in the template image extracted from the image database 110a (step S905). Specifically, for a template image with a certain image ID extracted from the image table of the image database 110a, the density estimation unit 109 counts the number of objects included in the template image by referring to the object table. Note that when counting the number of objects, objects with a predetermined reliability or higher may be counted. If the number of objects is less than the predetermined number (NO in step S906), the density estimation unit 109 performs processing on the next image.

[0088] If the number of objects is equal to or greater than the predetermined number (YES in step S906), the density estimation unit 109 calculates the density and updates the average density. Specifically, the density estimation unit 109 calculates the area of ​​the region indicated in the region table, and calculates the density based on the number and area of ​​the objects. The density estimation unit 109 performs the processes of steps S903 to S907 on the template image extracted from the image table, and calculates the average value of the calculated densities.

[0089] By using the average density, multiple search results are taken into consideration, and it is expected that the estimated density will be a more accurate value.In addition to the method of using a simple average, various methods can be applied to calculate the average, such as a method of calculating an average weighted according to the similarity.

[0090] 11 is a diagram schematically illustrating an example of an output of the result of the object scale estimation process in the object scale estimation system 100. The difference from the output example in the first embodiment is that a similar image search result display section 1001 is displayed.

[0091] The similar image search result display section 1001 displays the image ID, the similarity of the depth map features (the similarity between the target depth map features and the template depth map features), the number of objects in the template image corresponding to the image ID, and the depth map and template image in the image table.

[0092] (Object Scale Estimation Processing Procedure) The density estimation processing procedure will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing the object scale estimation processing. User 1100 is a user of object scale estimation system 100a. Computer 1120 represents one aspect of object scale estimation system 100a realized as hardware, and is, for example, an electronic computer including input device 102, display device 103, and main function unit 104.

[0093] First, the user 1100 inputs a request to estimate the number of objects to the computer 1120 (step S1101). The computer 1120 sends a request for an input image, which is an image for which the number of objects is to be estimated, to the storage device 101 (step S1102), and the storage device 101 sends the input image to the computer 1120 (step S1103).

[0094] The computer 1120 performs depth estimation processing (step S1104), area extraction processing (step S1105), and object detection processing (detection of the number of objects in the vicinity) (step S1106) on the target image, which is the input image. Note that the processing shown in FIG. 11 is an example, and the order of these three processes can be reversed.

[0095] The computer 1120 registers the image recognition results, including values ​​acquired by the depth estimation process, the area estimation process, and the object detection process, in the image database 110a (S1107). The image database 110a notifies the completion of registration (step S1108).

[0096] The computer 1120 displays the image recognition result to the user 1100 (step S1109). As shown in the image recognition result display unit 704 in Fig. 10 , the image recognition result is displayed in a manner in which a rectangular portion indicating an object detected by the object detection process and a diagonally shaded portion indicating an estimated region extracted by the region extraction process are superimposed on the loaded target image.

[0097] If the number of objects is less than the predetermined number in the object detection process (step S1106), the computer 1120 performs a density estimation process (step S1110). As shown in Fig. 10 , the density estimation process (step S1110) is a process from the similar image search (step S1111) to the selection of a template image to be used for density estimation (step S1114).

[0098] The computer 1120 performs a similar image search (step S1111) and extracts search results from the image database 110a (step S1112).

[0099] The computer 1120 displays the search result display to the user 1100 (step S1113). As shown in the similar image search result display section 1001 in Fig. 11 , the result display of the similar image search (and density estimation) includes the image ID, the similarity of the depth map features (the similarity between the target depth map features and the template depth map features), the number of objects in the template image corresponding to the image ID, and the depth map and images in the image table.

[0100] The user 1100 selects a template image to be used for density estimation from the results of the similar image search (step S1114). The user's selection is input to the computer 1120.

[0101] The computer 1120 estimates the number of objects at a long distance based on the selected template image (step S1115). The computer 1120 displays the number of objects to the user 1100 (step S1116). The number of objects is displayed in the manner shown in the scale estimation result display section 705 of FIG. 11 .

[0102] (Functions and Effects) Depending on the object being imaged, the captured image may contain only a few objects at close range, making it impossible to perform density estimation through object detection. Even in such cases, density estimation can be performed based on similar images, making it possible to perform object scale estimation processing.

[0103] [Other Embodiments] In the present disclosure, a house is the target of the object-scale estimation process, but the present disclosure is not limited to this. For example, when a flood occurs in a mountainous region, driftwood may be washed into a river. Driftwood may also be used as the target of the object-scale estimation process. Furthermore, at resource mining facilities, mined resources may be left to pile up. Sediments may also be used as the target of the object-scale estimation process.

[0104] Although the present disclosure has described a case where a single image is used as the target of object scale estimation processing, the present disclosure is not limited to this. When capturing an object in multiple images, it is also possible to perform object scale estimation processing on the object formed by combining the multiple images.

[0105] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention.

[0106] The following are examples of aspects that may be included in the present invention, but the present invention is not limited to these. (Aspect 1) An object scale estimation device comprising: a depth estimation unit that generates a depth map of a target image, which is an image that is the subject of an object scale estimation process; an object detection unit that detects target objects, which are objects to be detected, in the target image; a region extraction unit that extracts an estimation region that is a region that is estimated to include the target objects; a density estimation unit that estimates the density of the target objects based on the target objects and the depth map; and an object number estimation unit that estimates the area of ​​the estimation region based on the estimation region and the depth map, and calculates a first object number, which is the number of target objects in the estimation region, based on the density and area of ​​the target objects. (Aspect 2) The object scale estimation device according to Aspect 1, wherein the depth estimation unit generates the depth map based on a depression angle and an altitude at which the target image was captured. (Aspect 3) The object scale estimation device according to Claim 1 or 2, wherein the target image is an image captured by a monocular camera, and the depth estimation unit performs monocular depth estimation to generate the depth map. (Aspect 4) The object scale estimation device according to any one of Aspects 1 to 3, further comprising: an image database including: an image table associating template images with template depth map features that are features included in a depth map of the template images; an object table including objects included in the template images and types of the objects; and a region table indicating regions extracted from the template images and types of the regions; wherein, when the density estimation unit cannot estimate the density, it extracts from the image table a similar template image that is a template image having template depth map features similar to target depth map features that are features of a depth map of the target image; calculates a second object number that is the number of target objects included in the similar template image based on the object table; and calculates a second density that is the density of the target objects included in the region of the similar template image based on the second object number and the region table; and the object number estimation unit calculates the first object number using the second density.(Aspect 5) The object scale estimation device according to any one of Aspects 1 to 4, wherein the similar template image is selected by a user. (Aspect 6) An object scale estimation method comprising: a depth estimation step of generating a depth map of a target image that is an image that is the target of an object scale estimation process; an object detection step of detecting, in the target image, a target object that is an object to be detected; a region extraction step of extracting an estimation region that is a region that is estimated to include the target object; a density estimation step of estimating a density of the target object based on the target object and the depth map; and an object number estimation step of estimating an area of ​​the estimation region based on the estimation region and the depth map, and calculating a first object number that is the number of target objects in the estimation region based on the density and area of ​​the target object. (Aspect 7) An object scale estimation system comprising: an imaging device that captures a target image that is an image that is the subject of an object scale estimation process; a depth estimation unit that generates a depth map of the imaging device; an object detection unit that detects, in the target image, a target object that is an object to be detected; a region extraction unit that extracts an estimation region that is a region that is estimated to include the target object; a density estimation unit that estimates the density of the target objects based on the target objects and the depth map; and an object number estimation unit that estimates the area of ​​the estimation region based on the estimation region and the depth map, and calculates a first object number that is the number of target objects in the estimation region based on the density and area of ​​the target objects. (Aspect 8) The object scale estimation system according to Aspect 7, wherein the imaging device is mounted on a drone. (Aspect 9) The object scale estimation system according to Aspect 7 or 8, further comprising a display device that displays an image recognition result in which a first figure representing the target object and a second figure representing the estimation region are superimposed on the target image.

[0107] 10: Captured image, 11: Image after object detection, 12: Rectangular part, 13: Image after region extraction, 100 / 100a: Object size estimation system, 101: Storage device, 102: Input device, 103: Display device, 104: Main function unit, 105: Image input unit, 106: Region extraction unit, 107: Depth estimation unit, 108: Object detection unit, 109: Density estimation unit, 110 / 110a: Image database, 111: Object number estimation unit, 112: Display unit, 201: Processor, 202: Main storage device, 203: Processing program, 301: Target image, 302: Depth estimation processing, 303: Depth map, 304: Region extraction processing, 305: Image after region extraction, 306: Object detection processing processing, 307: image after object detection, 308: density estimation processing, 309: density information, 310: area estimation processing, 311: area information, 312: object number estimation processing, 700: mouse cursor, 701: button, 702: input image display unit, 703: size estimation execution button, 704: image recognition result display unit, 705: size estimation result display unit, 801: target image, 802: object detection processing, 803: image after object detection, 804: density estimation impossible, 805: depth estimation processing, 806: depth map, 807: similar image retrieval processing, 808: similar image retrieval result, 809: density estimation processing, 1001: similar image retrieval result display unit, 1100: user, 1120: computer

Claims

1. An object scale estimation device comprising: a depth estimation unit that generates a depth map of a target image, which is an image that is the subject of an object scale estimation process; an object detection unit that detects a target object, which is an object to be detected, in the target image; a region extraction unit that extracts an estimation region that is a region that is estimated to contain the target object; a density estimation unit that estimates the density of the target object based on the target object and the depth map; and an object number estimation unit that estimates the area of ​​the estimation region based on the estimation region and the depth map, and calculates a first object number, which is the number of target objects in the estimation region, based on the density and area of ​​the target objects.

2. The object scale estimation device according to claim 1, wherein the depth estimation unit generates the depth map based on the depression angle and altitude at which the target image was captured.

3. The object scale estimation device according to claim 1, wherein the target image is an image captured by a monocular camera, and the depth estimation unit performs monocular depth estimation to generate the depth map.

4. The object scale estimation device of claim 1, further comprising an image database including an image table associating a template image with template depth map features which are features included in the depth map of the template image, an object table including objects included in the template image and types of the objects, and a region table indicating regions extracted from the template image and the types of the regions, wherein when the density estimation unit is unable to estimate density, it extracts from the image table a similar template image which is a template image having template depth map features similar to target depth map features which are features of the depth map of the target image, calculates a second object number which is the number of target objects included in the similar template image based on the object table, and calculates a second density which is the density of the target objects included in the region of the similar template image based on the second object number and the region table, and the object number estimation unit calculates the first object number using the second density.

5. The object scale estimation device according to claim 4, wherein the similar template image is selected by a user.

6. A method for estimating object scale, comprising: a depth estimation step of generating a depth map of a target image, which is an image that is the subject of an object scale estimation process; an object detection step of detecting a target object, which is an object to be detected, in the target image; a region extraction step of extracting an estimation region, which is a region estimated to contain the target object; a density estimation step of estimating the density of the target object based on the target object and the depth map; and an object number estimation step of estimating the area of ​​the estimation region based on the estimation region and the depth map, and calculating a first object number, which is the number of target objects in the estimation region, based on the density and area of ​​the target object.

7. An object scale estimation system comprising: an imaging device that captures a target image which is an image that is the subject of an object scale estimation process; a depth estimation unit that generates a depth map of the imaging device; an object detection unit that detects a target object which is the object to be detected in the target image; a region extraction unit that extracts an estimation region which is a region which is estimated to include the target object; a density estimation unit that estimates the density of the target objects based on the target objects and the depth map; and an object number estimation unit that estimates the area of ​​the estimation region based on the estimation region and the depth map, and calculates a first object number which is the number of target objects in the estimation region based on the density and area of ​​the target objects.

8. The object scale estimation system of claim 7, wherein the imaging device is mounted on a drone.

9. An object scale estimation system as described in claim 7, further comprising a display device that displays the image recognition results by superimposing a first figure indicating the target object and a second figure indicating the estimated area on the target image.

Citation Information

Patent Citations

  • Method and device for determining number of targets in group region, equipment and storage medium

    CN110287929A

  • Object number determination method and device, storage medium and electronic equipment

    CN110472599A

  • Information processing device, control method, and program

    WO2017159060A1

Cited By

  • Device and method

    WO2026150527A1