Position detection device, position detection method, and position detection program

Abstract

This position detection device recognizes the presence position of an object of interest in a three-dimensional space, and acquires a plurality of images captured from different imaging sites, the captured images including three-dimensional point group information about the space, and the area around the object in the space. An area detection unit uses the acquired plurality of images as input and determines whether an object of interest appears in the plurality of images. When the object of interest appears in each of the plurality of images, the area detection unit detects the area of the object in the images. An identification unit identifies a point group area corresponding to the object of interest on the basis of the point group information and the area of the object detected in the images. The position detection unit recognizes points corresponding to the object of interest from the point group information inside the area identified by the identification unit, and identifies the position of the object of interest in the space.

Description

[Technical field]

[0001] The present disclosure relates to a position detection device, a position detection method, and a position detection program, and more particularly to a technique for recognizing a point cloud and a target object captured in an image and detecting the position in a three-dimensional space. [Background technology]

[0002] There is a device called MMS (Mobile Mapping System) that acquires shape data of three-dimensional space by mounting various measuring devices on a vehicle. Point cloud data can be acquired by using LiDAR (Light Detection And Ranging) as a measuring device. By combining the point cloud data with position information acquired by GPS (Global Positioning System) etc., three-dimensional point cloud information of space can be acquired.

[0003] By analyzing the acquired 3D point cloud information, it is possible to acquire the exact positions of structures on the ground and construct a highly accurate 3D map. In order to acquire the positions of structures, a recognition technology is required to recognize target objects from 3D point cloud data, as shown in Non-Patent Document 1. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-095156 A [Patent Document 2] JP 2016-018444 A [Non-patent literature]

[0005] [Non-Patent Document 1] QI, Charles R., et al. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR 2017. [Non-Patent Document 2] QI, Charles R., et al. Frustum pointnets for 3d object detection from rgb-d data. In CVPR 2018. [Non-Patent Document 3] J. Redmon and A. Farhadi. YOLO9000: better, faster, stronger. In CVPR 2017. [Non-Patent Document 4] HE, Kaiming, et al. Deep residual learning for image recognition. In: CVPR 2016. Summary of the Invention [Problem to be solved by the invention]

[0006] However, since processing large-scale point cloud data requires huge computational costs, there is a technology that limits the range of point clouds to be processed based on the results of two-dimensional image recognition, as shown in Non-Patent Document 2, to quickly recognize the position of a target object.

[0007] In the technology shown in Non-Patent Document 2, the range of the point cloud in which the target object exists is narrowed down to the range of a quadrangular pyramid in three-dimensional space corresponding to the rectangle of the two-dimensional image recognition result. However, since the distance to the target object is unknown, it is necessary to target the point cloud from the shortest distance to the longest distance that can be measured, and the height of the quadrangular pyramid and the range of the point cloud that can be narrowed down become very large.

[0008] In addition, the point cloud within the narrowed-down quadrangular pyramid contains points extracted from occluding objects in front of the target object, and objects and buildings behind the target object. Therefore, it is necessary to remove these unnecessary point clouds, which can cause a decrease in accuracy.

[0009] The disclosed technology has been made in consideration of the above points, and aims to provide a position detection device, a position detection method, and a position detection program that can quickly and accurately detect the position of a target object from three-dimensional point cloud information by using the result of recognizing the target object from an image, even if other objects are included in front of or behind the target object. [Means for solving the problem]

[0010] A first aspect of the present disclosure is a position detection device that recognizes the location of a target object in a three-dimensional space, including: a point cloud acquisition unit that acquires three-dimensional point cloud information of the space; an image acquisition unit that acquires multiple images captured from different shooting points, the multiple images being images including the surroundings of an object in the space; a region detection unit that receives the multiple images acquired by the image acquisition unit as input, determines whether a target object is captured in the multiple images, and detects a region of the object in each of the multiple images if the target object is captured in each of the multiple images; an identification unit that identifies a region of the point cloud corresponding to the target object based on the point cloud information and the region of the object detected in each image; and a position detection unit that recognizes a point corresponding to the target object from the point cloud information within the region identified by the identification unit, and identifies the position of the target object in the space.

[0011] A second aspect of the present disclosure is a position detection method for recognizing the location of a target object in a three-dimensional space, comprising the steps of: acquiring three-dimensional point cloud information of the space; acquiring a plurality of images captured from different shooting points, the plurality of acquired images being input; determining whether a target object is captured in the plurality of images; if the target object is captured in each of the plurality of images, detecting an area of ​​the object in each image; identifying an area of ​​the point cloud corresponding to the target object based on the point cloud information and the area of ​​the object detected in each image; recognizing a point corresponding to the target object from the point cloud information within the identified area; and identifying the position of the target object in the space.

[0012] A third aspect of the present disclosure is a position detection program that recognizes the location of a target object in a three-dimensional space, the position detection program causing a computer to execute the following processes: acquire three-dimensional point cloud information of the space; acquire a plurality of images captured from different shooting points, the plurality of acquired images being images including the surroundings of the object in the space; determine whether the target object is captured in the plurality of images; if the target object is captured in each of the plurality of images, detect an area of ​​the object in each image; identify an area of ​​the point cloud corresponding to the target object based on the point cloud information and the area of ​​the object detected in each image; recognize a point corresponding to the target object from the point cloud information within the identified area; and identify the position of the target object in the space. Effect of the Invention

[0013] According to the disclosed technology, even if other objects are included in front of or behind the target object, the position of the target object in three-dimensional space can be narrowed down using the results of recognizing the target object from an image, and the position of the target object can be detected quickly and accurately from the three-dimensional point cloud information. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the position detection device. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the position detection device. [Diagram 3] FIG. 2 is an explanatory diagram of an object region of an image. [Figure 4] FIG. 2 is an explanatory diagram of an object region of a point cloud. [Diagram 5] FIG. 11 is an explanatory diagram of integration of object regions of point clouds. [Figure 6] 4 is a flowchart showing a flow of position detection by the position detection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0016] The configuration of this embodiment will be described below.

[0017] FIG. 1 is a block diagram showing a hardware configuration of a position detection device 100. As shown in FIG.

[0018] 1, the position detection device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0019] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, a position detection program is stored in the ROM 12 or the storage 14.

[0020] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system, and various data.

[0021] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0022] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0023] The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0024] Next, a description will be given of each functional configuration of the position detection device 100. Fig. 2 is a block diagram showing the functional configuration of the position detection device 100 of this embodiment. Each functional configuration is realized by the CPU 11 reading out a position detection program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0025] As shown in FIG. 2, the position detection device 100 includes a point cloud acquisition unit 102, an image acquisition unit 104, a region detection unit 106, a specification unit 108, and a position detection unit 110.

[0026] The point cloud acquisition unit 102 acquires three-dimensional point cloud information in a three-dimensional space (hereinafter simply referred to as space) by a LIDAR, etc. Point cloud information acquired in advance may be received as an input.

[0027] The image acquisition unit 104 acquires multiple images and shooting information by shooting with a camera from different positions. The multiple images may be shot with one moving camera, or multiple cameras may shoot a space from different positions. It is assumed that the image is shot in a space including the surroundings of an object. Images acquired in advance may be accepted as input. As the shooting information, characteristic information on the image shooting device and information for identifying the positional relationship between the image and the three-dimensional point cloud are acquired together. The characteristic information on the image shooting device may be, for example, the shooting angle of the camera used for shooting, and information for correcting lens distortion. The information for identifying the positional relationship may be, for example, information on the shooting position and shooting direction of the image in the coordinate system of the three-dimensional point cloud.

[0028] The area detection unit 106 receives the multiple images acquired by the image acquisition unit 104 as input, and applies image recognition technology to detect the area of ​​the object in each of the multiple images when the target object is captured in each of the multiple images. Any technology can be used for the image recognition technology, and for example, the technology of Non-Patent Document 3 can be used. The image recognition technology is trained in advance so that it can recognize the target object. The target object may be a category of objects such as a car or a person, or may be an instance such as a specific car model or a person. In addition, it may be possible to select the target object to be recognized from multiple target objects.

[0029] Moreover, the area detection unit 106 judges whether or not the target object is captured in the multiple images. The area detection unit 106 judges the likelihood of the target object being captured for each image based on specific information capable of identifying the approximate position of the target object including map information, characteristic information, and information for identifying the positional relationship. Images that are judged to be unlikely to include the target object are excluded from the processing target. This can reduce the cost of calculation processing. The height judgment judges the likelihood of the target object being captured based on the distance from the image capture point to the position candidate range and the proportion of the position candidate range that falls within the angle of view of the image capture device. The position candidate range of the target object is set to a range within a certain distance from the position of the target object acquired from the map information.

[0030] When a target object is recognized in the image recognition, the area detection unit 106 outputs the area of ​​the recognized target object in the image. The shape of the area is arbitrary, and as shown in Fig. 3, the area may be a rectangular detection frame or a set of pixels corresponding to the target object.

[0031] If the target object is an object such as a car, there may be multiple objects in the space. Therefore, it may be possible to determine whether the target object recognized in each image is the same object, and to carry out the subsequent processing separately for each object. For example, the determination of whether the object is the same can be based on the photographing position of the image, and if the photographing position is within a certain range, it can be considered to be the same object. If there are two or more target objects, the subsequent processing is carried out for each target object. Note that if it is known that there is only one target object in the space, the determination of whether the target object is the same object may be omitted.

[0032] Furthermore, the area detection unit 106 can use an image recognition technique such as that described in Patent Document 1 to determine whether the target object recognized in each image is the same object. For example, an image within the range of the detected object area is cut out, and the image recognition technique described in Patent Document 1 or the like is used for the pair of cut-out images to determine whether the objects have the same characteristics. It is also possible to determine whether the objects are the same by applying geometric verification based on local features to the pair of cut-out images, and to consider them to be the same object when the score of the determination result is equal to or greater than a threshold value. This makes it possible to avoid erroneous determination of the object area when there are multiple target objects.

[0033] The identification unit 108 calculates and integrates object areas for each recognition result of the shooting location for target objects determined to be the same object based on the point cloud information and the object areas detected in each image by the area detection unit 106, thereby identifying the area of ​​the point cloud.

[0034] As shown in Figure 4, the object region in the point cloud is generally a cone with a base similar to the object region in the image. In this case, points outside the measurement range of the point cloud or a certain range may be excluded from the object region of the point cloud. In addition, if the image has depth information, the object region of the point cloud may be the integration of the space corresponding to the depth information in pixel units of the object region.

[0035] The identification unit 108 calculates a plurality of object regions for each recognition result of the same target object, and then integrates the plurality of object regions to narrow down the object regions in the point cloud. For example, as shown in FIG. 5, when there are two object regions in the point cloud, the overlapping portion of the two regions may be regarded as the region after integration. Also, a score based on the reliability of image recognition may be given to each object region, and when the object region is included in a plurality of object regions, the scores of each region may be summed up, and the object region with a score equal to or greater than a threshold may be regarded as the region after integration. At this time, a plurality of image recognition technologies may be further applied to the region detected by the region detection unit 106, and the reliability of each image recognition result may be added together with a weight to be used as a score to be added to each object region. Also, as the image recognition technology, for example, the image recognition technology based on the convolutional neural network shown in Non-Patent Document 4 and the image recognition based on the local feature shown in Patent Document 2 may be applied. The reliability of each image recognition result may be added together with a weight to be used as a score to be added to each object region, and the regions with a score equal to or greater than a threshold are obtained and integrated.

[0036] The position detection unit 110 detects the exact position of the target object in space by recognizing points corresponding to the target object from the point cloud information in the integrated region identified by the identification unit 108, and outputs the result. In addition, additional information such as the posture of the target object and a list of points corresponding to the object may be calculated and output as the additional information. Any method may be used for point cloud recognition of the target object, and for example, the technology described in Non-Patent Document 1 may be used.

[0037] Next, the operation of the position detection device 100 will be described.

[0038] 6 is a flowchart showing the flow of the position detection process by the position detection device 100. The position detection process is performed by the CPU 11 reading out a position detection program from the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. The CPU 11 executes the processes as each part of the position detection device 100.

[0039] In step S100, the CPU 11 causes the point cloud acquisition unit 102 to acquire three-dimensional point cloud information, and causes the image acquisition unit 104 to acquire a plurality of images and shooting information.

[0040] Next, the CPU 11 performs the process as the area detection unit 106. The processes in steps S102 to S108 are performed for each of the multiple images (each image). Step S110 processes the recognition result.

[0041] In step S102, the CPU 11 judges, for each of the multiple images, whether the possibility that a target object exists in the image is high or low. If the possibility is high, the process proceeds to step S104, and if the possibility is low, the image is not subjected to image recognition and is excluded from subsequent processing.

[0042] In step S104, the CPU 11 applies an image recognition technique to detect the area of ​​the object in each image.

[0043] In step S106, the CPU 11 determines whether or not a target object has been recognized. If it has been recognized, the process proceeds to step S108, and if it has not been recognized, the target object is excluded from subsequent processing.

[0044] In step S108, the CPU 11 detects the area of ​​the recognized target object.

[0045] In step S110, the CPU 11 judges whether the target object appearing in two or more images is the same object based on the recognition result. The judgment method uses the method described above. If the objects are not the same object, the following processes are performed separately.

[0046] Next, the CPU 11 performs processing as the identification unit 108. The processing from step S110 onwards is performed for each object determined to be the same target object. The processing of step S110 is performed for each recognition result corresponding to the object.

[0047] In step S112, the CPU 11 calculates an object region for each recognition result.

[0048] In step S114, CPU 11 integrates a plurality of object regions for the target object using the integration method described above, thereby identifying the region of the point cloud of the target object.

[0049] In step S116, CPU 11, as position detection unit 110, detects the accurate position of the target object in space by recognizing points corresponding to the target object from the point cloud information within the integrated area, and outputs the result.

[0050] As described above, according to the position detection device 100 of this embodiment, the position of a target object can be detected from three-dimensional point cloud information at high speed and with high accuracy.

[0051] In the above embodiment, the position detection process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. The position detection process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0052] In the above embodiment, the position detection program is pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may be downloaded from an external device via a network.

[0053] The following supplementary notes are further disclosed regarding the above embodiment.

[0054] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, In position detection, the position of a target object in a three-dimensional space is recognized. Acquire three-dimensional point cloud information of the space; acquiring a plurality of images taken from different shooting positions including the surroundings of the object in the space; The acquired multiple images are input, and it is determined whether a target object is captured in the multiple images. If the target object is captured in each of the multiple images, an area of ​​the object is detected in each image. Identifying a region of the point cloud corresponding to the target object based on the point cloud information and a region of the object detected in each image; Recognizing points corresponding to a target object from the point cloud information within the identified region, and identifying a position of the target object in the space. The position detection device is configured as follows.

[0055] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to execute a position detection process, In position detection, the position of a target object in a three-dimensional space is recognized. Acquire three-dimensional point cloud information of the space; acquiring a plurality of images taken from different shooting positions including the surroundings of the object in the space; The acquired multiple images are input, and it is determined whether a target object is captured in the multiple images. If the target object is captured in each of the multiple images, an area of ​​the object is detected in each image. Identifying a region of the point cloud corresponding to the target object based on the point cloud information and a region of the object detected in each image; Recognizing points corresponding to a target object from the point cloud information within the identified region, and identifying a position of the target object in the space. Non-transitory storage medium. [Explanation of symbols]

[0056] 100 Position detection device 102 Point cloud acquisition section 104 Image acquisition unit 106 Area detection unit 108 Specific section 110 Position detection unit

Claims

1. A position detection device that recognizes the position of a target object in a three-dimensional space, A point cloud acquisition unit that acquires three-dimensional point cloud information of the space; an image acquisition unit that acquires a plurality of images captured from different shooting points, the images including the surroundings of the object in the space; an area detection unit that receives the plurality of images acquired by the image acquisition unit, determines whether a target object is captured in the plurality of images, and detects an area of ​​the object in each image when the target object is captured in each of the plurality of images; an identification unit that identifies a region of the point cloud corresponding to the target object based on the point cloud information and a region of the object detected in each image; a position detection unit that recognizes points corresponding to a target object from the point cloud information within the area identified by the identification unit and identifies a position of the target object in the space; Including, The specific information is information that can identify the approximate location of the target object, including map information; The characteristic information is information related to a photographing device of the image, The information for identifying the positional relationship is the shooting position and shooting direction of the image in the coordinate system of the three-dimensional point cloud, the area detection unit determines, for each of the images, a likelihood that the target object is included based on the identification information, the characteristic information, and the information for identifying the positional relationship, and excludes the images determined to be unlikely to include the target object from processing. Position detection device.

2. the area detection unit determines an area within a certain distance from the position of the target object acquired from the map information as a candidate area for the position of the target object, The position detection device according to claim 1 , further comprising: a position detection unit configured to detect whether the target object is captured in the image based on a distance from the image capture point to the position candidate range and a proportion of the position candidate range that is within an angle of view of an image capture device.

3. The area detection unit determines whether the target object is the same object based on the shooting position of the image or image recognition of the detected object area, 3. The position detection device according to claim 1, wherein the identification unit is configured to identify the area of ​​the point cloud by calculating and integrating an object area of ​​each of the target objects that are determined to be the same object.

4. 4. The position detection device according to claim 3, wherein the identification unit adds a score based on an image recognition result to each of a plurality of object regions corresponding to regions of the same target object detected by the region detection unit, and performs the integration for the object regions whose scores are equal to or greater than a threshold.

5. 5. The position detection device according to claim 3, wherein the identification unit applies image recognition based on a convolutional neural network and image recognition based on a local feature amount to the area detected by the area detection unit, and performs the integration by adding a weighted sum of reliability of each image recognition result to each object area as a score.

6. A position detection method for recognizing a position of a target object in a three-dimensional space, comprising: Acquire three-dimensional point cloud information of the space; acquiring a plurality of images taken from different shooting positions including the surroundings of the object in the space; The acquired multiple images are input, and it is determined whether a target object is captured in the multiple images. If the target object is captured in each of the multiple images, an area of ​​the object is detected in each image. Identifying a region of the point cloud corresponding to the target object based on the point cloud information and a region of the object detected in each image; Recognizing points corresponding to a target object from the point cloud information within the identified region, and identifying a position of the target object in the space. A position detection method for causing a computer to execute a process, comprising the steps of: fruit, The specific information is information that can identify the approximate location of the target object, including map information; The characteristic information is information related to a photographing device of the image, The information for identifying the positional relationship is the shooting position and shooting direction of the image in the coordinate system of the three-dimensional point cloud, The process of detecting the region determines the likelihood that the target object is captured for each image based on the identification information, the characteristic information, and information for identifying the positional relationship, and excludes images that are determined to be unlikely to contain the target object from being processed.

7. A position detection program for recognizing the position of a target object in a three-dimensional space, Acquire three-dimensional point cloud information of the space; acquiring a plurality of images taken from different shooting positions including the surroundings of the object in the space; The acquired multiple images are input, and it is determined whether a target object is captured in the multiple images. If the target object is captured in each of the multiple images, an area of ​​the object is detected in each image. Identifying a region of the point cloud corresponding to the target object based on the point cloud information and a region of the object detected in each image; Recognizing points corresponding to a target object from the point cloud information within the identified region, and identifying a position of the target object in the space. A position detection program for causing a computer to execute a process, fruit, The specific information is information that can identify the approximate location of the target object, including map information; The characteristic information is information related to a photographing device of the image, The information for identifying the positional relationship is the shooting position and shooting direction of the image in the coordinate system of the three-dimensional point cloud, The process of detecting the area is a position detection program that determines the likelihood that the target object is captured for each image based on the identification information, the characteristic information, and information for identifying the positional relationship, and excludes images that are determined to be unlikely to contain the target object from being processed.

Images