Three-dimensional object detection method and apparatus

Abstract

This invention provides a method and apparatus for detecting three-dimensional objects. [Solution] The three-dimensional object detection device according to the embodiment of the present disclosure includes a sensor that captures a plurality of two-dimensional images of an external object, a processor that detects the external object based on the plurality of two-dimensional images, generates a plurality of two-dimensional bounding boxes of the external object, and obtains a plurality of three-dimensional bounding boxes of the external object based on the plurality of two-dimensional bounding boxes, and a display that displays the final three-dimensional bounding box among the plurality of three-dimensional bounding boxes to the outside.

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional object detection method and apparatus using multi-view images taken using X-ray computed tomography (CT) equipment.

Background Art

[0002] Three-dimensional reconstruction technology is utilized in various fields such as medical imaging, X-ray object detection, autonomous driving and robotics, gaming & virtual reality / augmented reality (VR / AR), architecture and urban planning, film and visual effects (VFX), and so on.

[0003] In particular, X-ray-based luggage (object) detection technology has been driving important innovations, thereby greatly improving the accuracy and efficiency of security screening at airports / ports and the like.

[0004] However, when using conventional X-ray equipment for luggage detection, information about the luggage is provided as a two-dimensional image, but there are limitations in accurately grasping the shape and position of the object with only a two-dimensional image.

Summary of the Invention

Problems to be Solved by the Invention

[0005] An object of the present disclosure is to provide a three-dimensional object detection method for reconstructing a three-dimensional bounding box related to a luggage (object) through a relatively small number of two-dimensional multi-view images taken by luggage (object) detection X-ray CT equipment in order to solve the above-described problems of the prior art.

[0006] Furthermore, this disclosure aims to provide a three-dimensional object detection method that acquires a two-dimensional bounding box of an object via a two-dimensional image taken with a geometry-calibrated X-ray multi-source, and reconstructs a three-dimensional bounding box of the object using the two-dimensional bounding box in the conventional Visual Hull method. [Means for solving the problem]

[0007] A three-dimensional object detection device according to one embodiment of the present disclosure includes: a sensor that captures a plurality of two-dimensional images relating to an external object; a processor that detects the external object based on the plurality of two-dimensional images, generates a plurality of two-dimensional bounding boxes relating to the external object, and obtains a plurality of three-dimensional bounding boxes relating to the external object based on the plurality of two-dimensional bounding boxes; and a display that displays the final three-dimensional bounding box among the plurality of three-dimensional bounding boxes externally.

[0008] Furthermore, the processor may be characterized by removing false positive values ​​from the plurality of three-dimensional bounding boxes to obtain the final three-dimensional bounding box.

[0009] Furthermore, the processor may be characterized by projecting each of the plurality of three-dimensional bounding boxes onto a plurality of two-dimensional bounding boxes, comparing the projected plurality of two-dimensional bounding boxes with the plurality of pre-generated two-dimensional bounding boxes, selecting the final two-dimensional bounding box that minimizes the error with the plurality of two-dimensional bounding boxes, and generating the final three-dimensional bounding box using the selected final two-dimensional bounding box.

[0010] Furthermore, the sensor may include multiple cameras that capture the multiple two-dimensional images using an X-ray imaging method.

[0011] Furthermore, each of the multiple cameras may be characterized by capturing the multiple two-dimensional images while undergoing geometric calibration.

[0012] Furthermore, the processor may be characterized by generating the plurality of three-dimensional bounding boxes from each of the plurality of two-dimensional bounding boxes based on calibration information relating to the plurality of cameras.

[0013] Furthermore, each of the multiple cameras can be applied to all of the multiple two-dimensional images through various shooting methods such as cone or fanbeam.

[0014] Furthermore, the processor may be characterized by setting the width values ​​of the plurality of two-dimensional bounding boxes to the respective z-axis values ​​of the plurality of three-dimensional bounding boxes.

[0015] A three-dimensional object detection device according to another embodiment of the present disclosure is a method for detecting a three-dimensional object, which may include the steps of: capturing a plurality of two-dimensional images relating to an external object; detecting the external object based on the plurality of two-dimensional images; generating a plurality of two-dimensional bounding boxes relating to the external object; obtaining a plurality of three-dimensional bounding boxes relating to the external object based on the plurality of two-dimensional bounding boxes; and displaying the final three-dimensional bounding box among the plurality of three-dimensional bounding boxes externally.

[0016] The procedure may also include the step of removing false positive values ​​from the plurality of three-dimensional bounding boxes to obtain the final three-dimensional bounding box.

[0017] Furthermore, the method may include the steps of projecting each of the plurality of three-dimensional bounding boxes onto a plurality of two-dimensional bounding boxes; comparing the projected plurality of two-dimensional bounding boxes with the plurality of pre-generated two-dimensional bounding boxes; selecting the final two-dimensional bounding box that minimizes the error with the plurality of two-dimensional bounding boxes; and generating the final three-dimensional bounding box using the selected final two-dimensional bounding box.

[0018] The method may also include the step of capturing the plurality of two-dimensional images using an X-ray imaging method with the plurality of cameras included in the three-dimensional object detection device.

[0019] Furthermore, the procedure may include the step of capturing the multiple two-dimensional images with the multiple cameras after geometric calibration has been performed.

[0020] The method may also include the step of generating the plurality of three-dimensional bounding boxes from each of the plurality of two-dimensional bounding boxes based on calibration information relating to the plurality of cameras.

[0021] The method may also include the step of capturing the plurality of two-dimensional images through various imaging methods, such as a cone or a fan beam.

[0022] Furthermore, the procedure may include setting the z-axis value of each of the multiple three-dimensional bounding boxes using the width values ​​of the multiple two-dimensional bounding boxes and calibration information. [Effects of the Invention]

[0023] According to this disclosure, the three-dimensional boundary region of an object detected can be easily reconstructed from multiple images taken with multiple geometrically aligned (geometry-calibrated) X-ray equipment (sources).

[0024] Also, according to the present disclosure, it is possible to easily detect the boundary box position of a three-dimensional object using only a relatively small number of X-ray sources acquired by a stationary gantry CT imaging method.

[0025] Also, according to the present disclosure, different from the conventional rotating gantry CT method in which a camera rotates while taking a large number of images with the object stopped, by using the stationary gantry CT imaging method, it is not necessary for the subject to stop, so the imaging time can be reduced.

[0026] Also, according to the present disclosure, after voxelizing only the xy-plane voxels by the visual hull method, the z-axis value of the three-dimensional boundary box can be obtained using the width (z-axis) of the two-dimensional boundary box detected in the two-dimensional image and calibration information, so the reconstruction operation for the 3D boundary box can be performed quickly.

Brief Description of Drawings

[0027] [Figure 1] It is a flowchart showing a method for detecting a three-dimensional object according to an embodiment of the present disclosure. [Figure 2] It is a block diagram showing the configuration of an object detection device according to an embodiment of the present disclosure. [Figure 3] It is a diagram showing an example of an imaging method according to an embodiment of the present disclosure. [Figure 4] It shows the process of detecting a two-dimensional boundary box according to an embodiment of the present disclosure. [Figure 5] It is a diagram showing the process in which an object detection device according to an embodiment of the present disclosure performs three-dimensional voxelization on an object. [Figure 6] It is a diagram showing the process in which an object detection device according to an embodiment of the present disclosure performs three-dimensional voxelization on an object. [Figure 7]The three-dimensional bounding box reconstructed by embodiments of this disclosure is shown. [Figure 8] The three-dimensional bounding box reconstructed by embodiments of this disclosure is shown. [Figure 9] Embodiments of this disclosure illustrate the process by which an object detection device visualizes a three-dimensional bounding box. [Modes for carrying out the invention]

[0028] [Explanation of terms used in this specification] All embodiments described below are illustrative and provided to aid in understanding this disclosure, and can be modified and implemented in various forms different from those described herein. In describing this disclosure, if it is determined that a specific description of a relevant known function or component would unnecessarily obscure the essence of this disclosure, such description will be omitted.

[0029] The attached drawings are not drawn to actual scale to aid in understanding the disclosure, and the dimensions of some components may be exaggerated. When a reference number is provided for each component, the same reference number is used whenever possible, even if the same component is shown in other drawings.

[0030] Furthermore, in describing the components of the embodiments of this disclosure, terms such as “First,” “Second,” “A,” “B,” “(a),” and “(b)” may be used. These terms are merely for distinguishing a component from other components, and do not limit the nature, order, or sequence of the component. When it is stated that one component is “connected,” “joined,” or “connected” to another component, it should be understood that the component may be directly connected, joined, or connected to the other component, but another component may be “connected,” “joined,” or “connected” between the component and the other component.

[0031] Therefore, the embodiments described herein and the configurations shown in the drawings are merely the most preferred embodiments of the disclosure and do not represent all of the technical ideas of the disclosure; various variations of the disclosure are possible.

[0032] Furthermore, terms or words used in this specification and claims should not be limited to their ordinary or lexicographical meanings, but should be interpreted in a sense and concept consistent with the technical idea of ​​this disclosure, in accordance with the principle that inventors may appropriately define the concepts of terms in order to best describe their disclosure.

[0033] Furthermore, the singular expressions used in this application include plural expressions unless they clearly mean something different in context.

[0034] [Three-dimensional object detection method: Figure 1] Figure 1 is a flowchart showing a three-dimensional object detection method according to an embodiment of the present disclosure.

[0035] As shown in Figure 1, the three-dimensional object detection method (S100) includes steps S110, S120, S130, S140, S150, S151, and S160, and a detailed explanation is as follows.

[0036] First, the object detection device according to the embodiment of the present disclosure (described later with reference to Figure 2) captures multiple images of an external object using multiple cameras (S110). Here, the object detection device can capture images based on a fanbeam type X-ray imaging method. Here, the object detection device can capture images of an object one row at a time. Here, the object may include a hazardous article (i.e., a gun, sword, scissors, or any form of hazardous article). Here, the imaging method may include, and is not limited to, a fanbeam, but may also include, and is not limited to, a cone-shaped imaging method.

[0037] Next, the object detection device detects one object from among multiple images (S120). Here, the object detection device can detect objects from images using a deep learning algorithm.

[0038] Subsequently, the object detection device acquires a two-dimensional bounding box for the object (S130). Here, the object detection device can acquire a two-dimensional bounding box for the object using the deep learning algorithm.

[0039] Next, the object detection device performs three-dimensional voxelization on the object based on the two-dimensional bounding box acquired for the object (S140). Here, the object detection device can perform three-dimensional voxelization on the object using a known Visual Hull algorithm. For example, the object detection device can perform three-dimensional voxelization using the Visual Hull algorithm based on two-dimensional bounding boxes detected from multiple images and calibration information for each camera.

[0040] Next, the object detection device generates a three-dimensional bounding box for the object using three-dimensional voxels relating to the object (S150).

[0041] Furthermore, the object detection device can capture images a certain number of times or more and remove outliers from the three-dimensional bounding boxes acquired based on the images. Here, when the object detection device projects each two-dimensional bounding box onto a three-dimensional voxel, it can reconstruct the three-dimensional bounding box, leaving only the voxels that belong to all of the two-dimensional bounding boxes.

[0042] Furthermore, the object detection device visualizes and displays the three-dimensional bounding box on the three-dimensional voxel space of the overall scene (S160).

[0043] [Three-dimensional object detection device: Figure 2] Figure 2 is a block diagram showing the configuration of an object detection device according to an embodiment of the present disclosure.

[0044] As shown in Figure 2, the object detection device 200 according to the embodiment of the present disclosure may include a sensor 210, a processor 220, and a display 230.

[0045] The sensor 210 may include a camera 211 for capturing / acquiring a two-dimensional image of an external object 21. Here, the sensor may include multiple cameras.

[0046] The processor 220 can detect objects within a two-dimensional image using a two-dimensional image captured by a camera. The processor 220 can generate a two-dimensional bounding box for the detected object. The processor 220 can perform three-dimensional voxelization on the object using the two-dimensional bounding box for the object, and generate a three-dimensional bounding box for the object using the three-dimensional voxels. Here, the processor 220 can detect objects within a two-dimensional image and generate a two-dimensional bounding box for the detected object using a deep learning model 221 built into the processor.

[0047] The display 230 can visualize the three-dimensional bounding box by displaying it externally according to the processor's control.

[0048] Furthermore, the parameters related to the operation of each of the multiple cameras can be set as an internal parameter K, which includes DSD (detector to source distance) and a unit conversion value (s), and as an external parameter [R|T] with rotation having 3 degrees of freedom for each x, y, and z axis, using the following equations 1, 2, and 3.

[0049]

number

[0050]

number

[0051]

number

[0052] Furthermore, after setting the parameters for each of the multiple cameras, the processor can project a specific point p3d in three dimensions onto a specific point p2d in two dimensions using the following equations 4 and 5.

[0053]

number

[0054]

number

[0055] Furthermore, the processor uses only the height values ​​of the image to utilize the visual hull on the xy plane of the three-dimensionally reconstructed voxel. Since the values ​​of the xy plane on the voxel have (X,Y,0,1) values ​​in homogeneous coordinates, the processor can use the camera matrix P obtained using the intrinsic and extrinsic parameters described above to calculate the two-dimensional coordinates from the three-dimensional coordinates of a specific point using equations 6, 7, and 8 below.

[0056]

number

[0057]

number

[0058]

number

[0059] The processor can then perform a visual hull by using the two-dimensional image coordinates (u,v) obtained in this way to determine whether the v value belongs to a region detected in the two-dimensional image ((v>2d bbox_left_top_y) and (v<2d bbox_right_bottom_y)), thereby trimming the xy-plane voxels and extracting regions from multiple images that satisfy all of the conditions.

[0060] However, the embodiments of this disclosure feature a fan-beam type shooting method to aid in the derivation of simple formulas and intuitive understanding. Here, the image width value is unaffected by perspective, and the camera rotational degrees of freedom is 1 degree of freedom with respect to the Z axis. When using a fan-beam type shooting method, the processor can calculate the camera parameters using the following equations 9, 10, and 11.

[0061]

number

[0062]

number

[0063]

number

[0064] [Fan-beam type shooting method: Figure 3] Figure 3 shows an example of an image acquisition method according to an embodiment of the present disclosure.

[0065] As shown in Figure 3, according to the embodiment of the present disclosure, while the luggage 30 containing the object moves along the z-axis direction 3, the camera 310 can photograph the object and capture / acquire a two-dimensional image 31.

[0066] Here, the camera 310 can capture a two-dimensional image 31 in a fan shape. That is, the camera 310 can capture one row at a time for each piece of luggage 30.

[0067] Here, camera 310 can capture two-dimensional images using the X-ray CT imaging method.

[0068] Next, the processor (processor 220 in Figure 2) can obtain a two-dimensional bounding box 32 for the object using the two-dimensional image.

[0069] [Detection of a two-dimensional bounding box: Figure 4] Figure 4 shows the two-dimensional boundary box detection process according to an embodiment of the present disclosure.

[0070] As shown in Figures 4(A), 4(B), and 4(C), the object detection device can capture two-dimensional images 441, 442, and 443 of an object. Specifically, the object detection device can capture multiple images 441, 442, and 443 of an object at multiple angles via multiple cameras located at different locations.

[0071] The object detection device can detect specific objects 451, 452, and 453 contained in the two-dimensional images 441, 442, and 443.

[0072] Here, the processor of the object detection device can detect specific objects in a two-dimensional image using a deep learning model built into the processor.

[0073] Furthermore, the object detection device's processor can use a deep learning model built into the processor to detect and acquire two-dimensional bounding boxes 451, 452, and 453 in each image relating to a specific object.

[0074] [Voxelization process: Figures 5 and 6] Figures 5 and 6 illustrate the process by which an object detection device in an embodiment of this disclosure performs three-dimensional voxelization on an object.

[0075] As shown in Figure 5, the object detection device can perform three-dimensional voxelization of objects using the visual Hull algorithm on a two-dimensional xy voxel plane.

[0076] Here, the object detection device obtains a two-dimensional bounding box based on two-dimensional images taken from multiple camera positions 511, 512, 513, 514, and 515, and obtains a three-dimensional xy-plane voxel 560 that has been trimmed using the visual hull algorithm with the initial xy-plane voxel 550.

[0077] For example, the object detection device can use a visual hull algorithm to reconstruct the voxels containing the detected object, based on calibration information for each camera position 511, 512, 513, 514, and 515, and the two-dimensional bounding box of the detected object.

[0078] Here, in order to ensure physical meaning or scale consistency, we assume that the pixel value of a two-dimensional image has the same units as the voxel value of a three-dimensional voxel space.

[0079] As shown in Figure 6, the object detection device can obtain three-dimensional voxel values ​​670 by projecting the two-dimensional bounding boxes 651 and 652 onto three-dimensional xy-plane voxels.

[0080] Specifically, the object detection device projects each two-dimensional bounding box onto a three-dimensional xy-plane voxel, retaining only the voxels belonging to all the two-dimensional bounding boxes, thereby obtaining the three-dimensional voxels related to the object, and using the three-dimensional voxels to generate / reconstruct the three-dimensional bounding box.

[0081] In embodiments of this disclosure, when the object detection device captures an object in a fanbeam manner, the object detection device sets the width value of the two-dimensional bounding box to the same value as the z-axis value (z-axis length in Figure 3) of the reconstructed three-dimensional bounding box. Therefore, the object detection device can obtain the z-axis length of the three-dimensional bounding box using the width value of the two-dimensional bounding box.

[0082] [Reconstruction of the three-dimensional bounding box: Figures 7 and 8] Figures 7 and 8 show the three-dimensional bounding box reconstructed according to the embodiments of this disclosure.

[0083] As shown in Figures 7(A), 7(B), 7(C), and 7(D), the object detection device can display / visualize the restored / acquired three-dimensional bounding boxes 781, 782, 783, and 784 via a display.

[0084] As shown in Figures 7(A), 7(B), 7(C), and 7(D), the three-dimensional bounding box can be reconstructed as the number of cameras or the number of images captured by the cameras increases or decreases.

[0085] As shown in Figure 8, the object detection device can acquire two-dimensional bounding boxes 881 and 882 for objects 81 and 82 contained in two-dimensional images taken at various angles, and acquire three-dimensional bounding boxes 891 and 892 based on the two-dimensional bounding boxes.

[0086] Here, the object detection device can separately save a 3D voxel file (RAW file) of the overall 2D image in order to visualize the 3D bounding box, and visualize the 3D bounding box based on this file.

[0087] [Removal of false positive values] The three-dimensional bounding box generated as described above may contain incorrect information, either by simultaneously detecting various objects within the image or by false positives generated by the deep learning algorithm.

[0088] Therefore, the object detection device can remove outliers by reprojecting the reconstructed three-dimensional bounding box (three-dimensional voxel) onto the two-dimensional xy-plane using the geometric relationship between three and two dimensions, and selecting only the two-dimensional bounding box that minimizes the error with the previously acquired two-dimensional bounding box.

[0089] Subsequently, the object detection device can reconstruct / acquire the final three-dimensional bounding box using the reprojected two-dimensional bounding box.

[0090] [Visualization of a three-dimensional bounding box: Figure 9] Figure 9 shows the process by which an object detection device visualizes a three-dimensional bounding box according to an embodiment of the present disclosure.

[0091] As shown in Figure 9, the object detection device can visualize / display the final reconstructed three-dimensional bounding boxes 982, 981, and 983 within the two-dimensional image 900 captured by the camera.

[0092] [Method of Interpretation of This Specified Version] Although embodiments of the present disclosure have been described in more detail above with reference to the attached drawings, the present disclosure is not necessarily limited to these embodiments and can be modified and implemented in various ways without departing from the technical concept of the present disclosure.

[0093] Therefore, the embodiments disclosed herein are for illustrative purposes only, and not to limit the technical concept of the disclosure, and such embodiments do not limit the scope of the technical concept of the disclosure. Accordingly, the embodiments described above should be understood to be illustrative and not limiting in all respects. The scope of protection of this disclosure should be interpreted as follows, and all technical concepts within an equivalent scope should be interpreted as being included in the scope of rights of this disclosure.

Claims

1. A three-dimensional object detection device, A sensor that captures multiple two-dimensional images of an external object, A processor that detects the external object based on the plurality of two-dimensional images, generates a plurality of two-dimensional bounding boxes relating to the external object, and obtains a plurality of three-dimensional bounding boxes relating to the external object based on the plurality of two-dimensional bounding boxes, A three-dimensional object detection device, comprising: a display that shows the final three-dimensional bounding box among the plurality of three-dimensional bounding boxes to the outside;

2. The aforementioned processor, The three-dimensional object detection device according to claim 1, characterized in that false positive values ​​are removed from the plurality of three-dimensional bounding boxes to obtain the final three-dimensional bounding box.

3. The aforementioned processor, Each of the aforementioned three-dimensional bounding boxes is projected onto a plurality of two-dimensional bounding boxes, The projected plurality of two-dimensional bounding boxes are compared with the plurality of pre-generated two-dimensional bounding boxes, Select the final two-dimensional bounding box that minimizes the error with the aforementioned plurality of two-dimensional bounding boxes. The three-dimensional object detection device according to claim 2, characterized in that it generates the final three-dimensional bounding box using the selected final two-dimensional bounding box.

4. The three-dimensional object detection device according to claim 1, wherein the sensor includes a plurality of cameras that capture the plurality of two-dimensional images using an X-ray imaging method.

5. Each of the aforementioned multiple cameras is The three-dimensional object detection device according to claim 4, characterized in that it captures the plurality of two-dimensional images while geometric calibration is performed.

6. The aforementioned processor, The three-dimensional object detection device according to claim 5, characterized in that it generates the plurality of three-dimensional bounding boxes from each of the plurality of two-dimensional bounding boxes based on calibration information relating to the plurality of cameras.

7. Each of the aforementioned multiple cameras is The three-dimensional object detection device according to claim 4, characterized in that it captures the plurality of two-dimensional images via a fan beam type shooting method.

8. The aforementioned processor, The three-dimensional object detection device according to claim 7, characterized in that the width values ​​of the plurality of two-dimensional bounding boxes are set to the respective z-axis values ​​of the plurality of three-dimensional bounding boxes.

9. A method for a three-dimensional object detection device to detect a three-dimensional object, The steps include taking multiple two-dimensional images of an external object, The steps include detecting the external object based on the plurality of two-dimensional images, The steps include generating a plurality of two-dimensional bounding boxes relating to the aforementioned external object, The steps include obtaining a plurality of three-dimensional bounding boxes relating to the external object based on the plurality of two-dimensional bounding boxes, A method comprising the step of displaying the final three-dimensional bounding box among the plurality of three-dimensional bounding boxes to the outside.

10. The method according to claim 9, further comprising the step of removing false positive values ​​from the plurality of three-dimensional bounding boxes to obtain the final three-dimensional bounding box.

11. The steps include projecting each of the aforementioned plurality of three-dimensional bounding boxes onto a plurality of two-dimensional bounding boxes, The steps include comparing the projected plurality of two-dimensional bounding boxes with the plurality of pre-generated two-dimensional bounding boxes, The steps include selecting the final two-dimensional bounding box that minimizes the error with the aforementioned plurality of two-dimensional bounding boxes, The method according to claim 10, comprising the step of generating a final three-dimensional bounding box using the selected final two-dimensional bounding box.

12. The method according to claim 9, further comprising the step of capturing the plurality of two-dimensional images in an X-ray imaging manner using a plurality of cameras included in the three-dimensional object detection device.

13. The method according to claim 12, further comprising the step of capturing the plurality of two-dimensional images with the plurality of cameras while geometric calibration has been performed.

14. The method according to claim 13, further comprising the step of generating a plurality of three-dimensional bounding boxes from each of the plurality of two-dimensional bounding boxes based on calibration information relating to the plurality of cameras.

15. The method according to claim 12, further comprising the step of capturing the plurality of two-dimensional images via a fan beam type shooting method.

16. The method according to claim 15, further comprising the step of setting the width value of the plurality of two-dimensional bounding boxes to the z-axis value of each of the plurality of three-dimensional bounding boxes.

Images