Object recognition device and object recognition method

Abstract

To provide an object recognition device and an object recognition method with which it is possible to appropriately estimate the object position or the number of pieces of material.SOLUTION: An object recognition device 10 according to the present invention is constituted to comprise: a data acquisition unit 11 that acquires two-dimensional moving-image data in which an object group 20 including a plurality of substances 22 to be recognized are imaged from multiple directions and a distance image of the object group which is synchronized with the two-dimensional moving-image data; a data analysis unit 12 that obtains three-dimensional point cloud data of the object group on the basis of the distance image of the object group; and an object detection unit 13 that estimates a boundary box BB of the substances to be recognized, for each frame of the two-dimensional moving-image data, projects boundary boxes of the same identification number to a representative image via a partial point cloud corresponding to the boundary box of the three-dimensional point cloud data, determines the circumscribing rectangle of each of two-dimensional point clouds of the projected boundary boxes, and integrates the circumscribing rectangles to detect the substances to be recognized.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an object recognition device and an object recognition method. [Background technology]

[0002] In recent years, with the declining birthrate and aging population, the number of workers is decreasing, and technologies for counting the number of various objects have been proposed to replace visual confirmation work such as checking inventory levels. For example, the abstract of the following Patent Document 1 states, "...a counting processing unit that detects a bounding box for each material contained in the input image data based on the input image data, outputs for each bounding box whether it is a material of a predetermined type, and counts the number of materials of each type..." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-26697 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned technology, there is a demand for more appropriately estimating the positions of objects to be recognized, such as materials, or the number of objects. The present invention has been made in consideration of the above-mentioned circumstances, and has an object to provide an object recognition device and an object recognition method that can appropriately estimate the object position or number of materials. [Means for solving the problem]

[0005] In order to solve the above problems, the object recognition device of the present invention comprises a data acquisition unit that acquires two-dimensional video data of an object group including multiple recognition targets, captured from multiple directions, and a distance image of the object group synchronized with the two-dimensional video data; a data analysis unit that obtains three-dimensional point cloud data of the object group based on the distance image of the object group; and an object detection unit that estimates a bounding box of the object to be recognized for each frame of the two-dimensional video data, projects a bounding box with the same identification number onto a representative image via a partial point cloud corresponding to the bounding box of the 3D point cloud data, determines the circumscribing rectangle of each of the two-dimensional point clouds of the projected bounding boxes, and integrates the circumscribing rectangles to detect the object to be recognized. Effect of the Invention

[0006] According to the present invention, the positions or the number of objects to be recognized, such as materials, can be appropriately estimated. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining an overview of an object recognition method in an object recognition device. [Diagram 2] FIG. 2 is a diagram showing the relationship between the bounding box of a two-dimensional frame image and three-dimensional point cloud data. [Diagram 3] FIG. 13 is a diagram showing the relationship between a partial point group of a bounding box of three-dimensional point group data and a representative image. [Figure 4A] FIG. 13 is a diagram illustrating how the bounding box in the two-dimensional frame image of frame 1 is displayed as a circumscribing rectangle in the representative image. [Figure 4B] FIG. 13 is a diagram illustrating how the bounding box in the two-dimensional frame image of frame 2 is displayed as a circumscribing rectangle in the representative image. [Diagram 5] 13A and 13B are diagrams illustrating a method of merging circumscribing rectangles corresponding to bounding boxes with the same identification number. [Figure 6] FIG. 1 is a block diagram showing a configuration of an object recognition device. [Figure 7] FIG. 1 is a block diagram of a computer. [Figure 8]4 is a flow diagram illustrating an object detection process of an object detection unit. FIG. [Figure 9] FIG. 13 is a diagram showing a display example of a representative image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [Overview of the embodiment] Conventional counting methods that detect target objects in images of stacked materials detect and count materials in a single captured image. However, when stacked materials are hidden by each other and cannot be captured in a single image no matter what angle they are captured from, or when there is a large amount of materials and they cannot all be captured in a single image, it is not possible to count the materials accurately.

[0009] For this reason, in the object recognition device 10 of the embodiment, two-dimensional image data (each frame of the video) captured from multiple angles and three-dimensional point cloud data are acquired synchronously (more preferably simultaneously), the recognition result (bounding box) based on the two-dimensional image data is cross-referenced with the three-dimensional point cloud information, and the bounding box and point cloud are linked to recognize the number and positions of objects to be recognized.

[0010] First, an outline of an object recognition method in the object recognition device of the embodiment will be described with reference to FIG. FIG. 1 illustrates an object recognition method in which cylindrical objects (pipes) serving as scaffolding materials at a construction site are treated as recognition target objects 22, the number of recognition target objects 22 in an object group 20 consisting of a plurality of recognition target objects 22, and the position of the recognition target objects 22 are estimated.

[0011] 1 is data obtained by performing point cloud accumulation processing on a three-dimensional frame point cloud TQ(n) (n is a frame number), which is a depth image of the object group 20 acquired in synchronization with two-dimensional image data when the object group 20 is captured as a video. In other words, the three-dimensional point cloud data TP shows the surface of the object group 20 in three dimensions, and indicates the absolute coordinates of the shape and position of the recognition target object 22.

[0012] The two-dimensional frame image DG(n) in Fig. 1 is two-dimensional frame image data (n is a frame number) obtained when a moving image of an object group 20 is captured. In each frame image, an end face of a recognition target object 22 is detected, and the detected rectangular area is shown as a bounding box BB (bounding box). Each frame of the two-dimensional frame image DG(n) is an image in which the object group 20 is captured from a different angle, and even if the bounding box BB of the same recognition target object 22 is a bounding box, the position and size are different.

[0013] The object recognition device of the embodiment performs an integrated analysis of the bounding boxes BB of each frame extracted from a plurality of two-dimensional frame images DG(n) using the three-dimensional point cloud data TP, maps them onto a representative image of a 2D plane, and generates overall image data IGD that can grasp the entire object group 20. By performing an integrated analysis using the three-dimensional point cloud data TP, the object recognition device can grasp and count the positions of recognition target objects 22 that are hidden from a certain direction, simultaneously with other recognition target objects 22.

[0014] Next, a method of integrating and analyzing the bounding box BB of each frame with the three-dimensional point cloud data TP and mapping it onto a two-dimensional representative image will be described in detail with reference to FIGS. 2, 3, 4A, and 4B.

[0015] FIG. 2 is a diagram showing the relationship between a bounding box BB of a two-dimensional frame image DG(n) indicating a frame image with frame number n, and three-dimensional point cloud data TP. Since the positional relationship between the 3D point cloud data TP and the 2D frame image DG(n) can be determined from the shooting position data PS and shooting angle of the image of frame n, it is possible to obtain the 3D point cloud (hatched area (filled area) in the figure) that corresponds to the bounding box BB of the 2D frame image DG(n) in the 3D point cloud data TP. Hereinafter, the 3D point cloud that corresponds to the bounding box BB will be referred to as the partial point cloud of the bounding box.

[0016] In the two-dimensional frame image DG(n), there is a bounding box BB corresponding to each of the multiple recognition target objects 22, and in the three-dimensional point cloud data TP, there are also multiple partial point clouds of the bounding box corresponding to the bounding box BB.

[0017] Also, if an identification number BID is assigned to each recognition target object 22, there may be bounding boxes BB with the same identification number BID in different frames of the two-dimensional frame images DG(n) and DG(n+1), but even in this case, the three-dimensional point cloud data TP has a plurality of bounding box partial point clouds corresponding to the bounding box BB for each frame. In other words, the three-dimensional point cloud data TP has a plurality of bounding box partial point clouds related to the bounding box BB with the same identification number BID. Note that the positions of the frames photographed differ, so the positions of the plurality of bounding box partial point clouds in the three-dimensional point cloud data TP are also different.

[0018] Next, the relationship between the partial point group of the bounding box BB of the three-dimensional point group data TP and the representative image will be described with reference to FIG.

[0019] If the representative image is taken at the same position as one of the frames in a video recording of the object group 20, the positional relationship between the three-dimensional point cloud data TP and the representative image can be determined from the shooting position data PS and the shooting angle, and therefore a two-dimensional point cloud of the boundary box BB, which is the partial point cloud of the boundary box in the three-dimensional point cloud data TP projected onto the representative image, can be obtained. Then, a circumscribing rectangle indicating a partial region including the two-dimensional point group of the projected bounding box BB is obtained.

[0020] The 3D point cloud data TP contains multiple partial point clouds of bounding boxes with the same identification number BID, and by projecting each of these onto a representative image, a 2D point cloud of each bounding box BB can be obtained, and a circumscribing rectangle indicating the partial area containing the 2D point cloud of the projected bounding box BB can be obtained. Similarly, the 3D point cloud data TP also contains multiple partial point clouds of bounding boxes corresponding to multiple objects 22 to be recognized in the 2D frame image DG(n), and the 2D point clouds of each bounding box BB and their circumscribing rectangles can be obtained.

[0021] That is, the bounding boxes BB of each of the recognition target objects 22 in all of the 2D frame images DG(n) can be projected onto the representative image via the 3D point cloud data TP. Then, the 2D point clouds of each of the projected bounding boxes BB and their circumscribing rectangles can be obtained. Therefore, even if there is a 2D frame in which the bounding box BB of a certain recognition target object 22 does not exist because it is shaded due to the shooting angle, all of the recognition target objects 22 can be detected because the bounding boxes BB of the other captured 2D frames are projected onto the representative image.

[0022] Next, the relationship between the two-dimensional frame image DG(n), the three-dimensional point group data TP, and the representative image relating to the bounding box BB with the same identification number BID will be described with reference to FIGS. 4A and 4B.

[0023] 4A is a diagram illustrating how a bounding box BB in a two-dimensional frame image DG(1) of frame number 1 is displayed as a circumscribing rectangle in a representative image. In more detail, a partial point group of the bounding box is obtained in the three-dimensional group data TP in correspondence with the bounding box BB in the two-dimensional frame image DG(1). The partial point group of the bounding box is then projected onto the representative image as a two-dimensional point group of the bounding box BB, and its circumscribing rectangle is obtained.

[0024] Fig. 4B is a diagram illustrating that in a two-dimensional frame image DG(2) of frame number 2, a bounding box BB having the same identification number BID as the bounding box BB in Fig. 4A is displayed as a circumscribing rectangle in the representative image. In more detail, a partial point group of the bounding box is obtained in the three-dimensional group data TP in correspondence with the bounding box BB in the two-dimensional frame image DG(2). The partial point group of the bounding box is then projected onto the representative image as a two-dimensional point group of the bounding box BB, and its circumscribing rectangle is obtained.

[0025] At this time, the circumscribing rectangle of the 2D point cloud of frame 1 corresponding to the bounding box BB of frame 1 is also acquired in the representative image, and therefore the circumscribing rectangle of the 2D point cloud corresponding to each of the bounding box BB of frame 1 and the bounding box BB of frame 2 is acquired in the representative image. In other words, the bounding boxes BB of all 2D frame images DG(n) can be collected in the representative image as the circumscribing rectangle of the 2D point cloud of the bounding box BB.

[0026] 4A and 4B, the larger the angle of the shooting direction is with respect to the representative image, the more horizontally elongated the circumscribing rectangle corresponding to the bounding box BB of the frame becomes. In other words, when the edge of the object group 20 is used as the representative image, the circumscribing rectangle corresponding to the bounding box BB has the most horizontally elongated shape, so it is desirable to use the front of the object group 20 as the representative image.

[0027] Here, the representative image (entire image data IGD) will be described. The bounding boxes BB of all frames of the two-dimensional frame image DG(n) are projected onto the representative image, but from the standpoint of the counting accuracy of the recognition target objects 22, it is desirable to select as the representative image a frame having a large number of bounding boxes BB, which indicate that the recognition target objects 22 have been detected.

[0028] Furthermore, the first frame of the two-dimensional frame image DG(n) is set as a representative image in accordance with the procedure for counting the recognition target objects 22 in the object group 20. That is, in accordance with the procedure in which the worker checks the object group 20 to be counted and photographs the object group 20, the frame at the start of photographing is set as the representative image.

[0029] When the object group 20 is larger than the field of view, it is possible to photograph the object group 20 in order from the edge of the object group 20. For this reason, it is preferable to use the central frame of the two-dimensional frame image DG(n) as the representative image.

[0030] In the above, we have described a case where an image at the same position as one of the frames of a video of the object group 20, i.e., one of the two-dimensional frame images DG(n), is used as the representative image (entire image data IGD). However, this is not limited to the above, and a still image captured from the front of the entire object group 20 may be used as the representative image, and the bounding boxes BB of all the two-dimensional frame images DG(n) may be combined into a circumscribing rectangle.

[0031] In this case, when a still image to be used as a representative image is captured, a depth image of the object group 20 is simultaneously acquired to generate 3D point cloud data, which is then matched by 3D registration with the 3D point cloud data TP obtained when a video of the object group 20 is captured. This makes it possible to project a partial point cloud of the bounding box in the 3D point cloud data TP onto the still image to be used as the representative image to generate a 2D point cloud of the bounding box BB, and to obtain a circumscribing rectangle.

[0032] Next, a method for integrating circumscribing rectangles corresponding to bounding boxes BB having the same identification number BID will be described with reference to Fig. 5. By integrating circumscribing rectangles in this manner, the positions and number of objects to be recognized can be appropriately estimated. The left image in Figure 5 shows multiple circumscribing rectangles (3 in the figure) of the bounding box BB of the same BID projected onto a representative image for the recognition target object 22 at the bottom left of the object group 20, and the right image shows the integrated circumscribing rectangle.

[0033] The circumscribing rectangle indicates the result of object detection of the target object 22, but by integrating the circumscribing rectangles resulting from photographing at multiple angles, it is possible to determine the position and size of the target object 22 regardless of the photographing direction.

[0034] The circumscribing rectangles can be integrated by calculating the average, median, or maximum of the coordinates of the circumscribing rectangles. For example, the maximum coordinate of the outline rectangle can be calculated by integrating the minimum and maximum coordinates of the four corners of the outline rectangle. When multiple bounding rectangles are integrated, the resulting rectangle may be taken as a bounding rectangle again.

[0035] Alternatively, the center position of an integrated figure may be determined by taking the average or median of the center positions of a plurality of circumscribing rectangles, and the size of the integrated figure may be the size of the recognition target object 22 estimated from the distance at the time of shooting. When integrating circumscribing rectangles, if the coordinate values ​​of the obtained circumscribing rectangle are significantly different from the estimated values, the circumscribing rectangle may be excluded as an outlier from the circumscribing rectangles to be integrated.

[0036] Furthermore, the method for integrating the circumscribing rectangles may be made selectable, and the integration result may be confirmed on a representative image that displays the integrated circumscribing rectangle. Also, the integration method may be set in advance depending on the type of the recognition target object 22.

[0037] Hereinafter, the object recognition device according to the embodiment will be specifically described. FIG. 6 is a block diagram showing a configuration of an object recognition device 10 according to an embodiment. The object recognition device 10 comprises a data acquisition unit 11, a data analysis unit 12, an object detection unit 13, a counting processing unit 14, and a representative image display unit 15.

[0038] The data acquisition unit 11 includes a camera (not shown) that captures two-dimensional video data of the object group 20, a distance image sensor or a three-dimensional sensor (not shown) such as a LiDAR (Light Detection and Ranging) that acquires three-dimensional point cloud data of the object group 20, and an IMU (Inertial Measurement Unit) sensor. The data acquisition unit 11 may also acquire the three-dimensional point cloud data of the object group 20 by a stereo camera or monocular depth estimation.

[0039] As a result, when an operator counting the recognition target objects 22 in the object group 20 scans the object group 20 with the data acquisition unit 11, the data acquisition unit 11 acquires a two-dimensional frame image DG(n) obtained by dividing the two-dimensional video data into frame units, three-dimensional point cloud data TQ (distance image information), and IMU data DI which is the detection result of the IMU sensor. At this time, it is desirable to acquire the three-dimensional point cloud data TQ (distance image information) and the IMU data DI in synchronization with the two-dimensional frame image DG(n).

[0040] In detail, the IMU sensor is a sensor unit that integrates physical information sensors such as an acceleration sensor, a rotational angular acceleration sensor, a gyro sensor, a magnetic field sensor, an air pressure sensor, and a temperature sensor into one package. When acquiring two-dimensional video data and three-dimensional point cloud data, the data acquisition unit 11 judges whether the moving speed, shooting distance, and shooting angle of the data acquisition unit 11 detected by the IMU sensor are within the allowable range. This suppresses the acquisition of abnormal two-dimensional video data and three-dimensional point cloud data, and prevents a decrease in analysis accuracy.

[0041] The data analysis unit 12 outputs three-dimensional point cloud data TP, shooting position data PS, and depth data DP based on the data supplied from the data acquisition unit 101. Here, the shooting position data PS is acquired based on the IMU data DI, Visual SLAM, LiDAR SLAM, etc., and is data indicating the position of the data acquisition unit 11 (position where the data was acquired). The depth data DP is data obtained by converting the three-dimensional point cloud data TQ, TP, and is data indicating the distance from the position of the data acquisition unit 11 (position where the data was acquired) to each three-dimensional point on the surface of the object group 20.

[0042] The three-dimensional point cloud data TP is obtained by aligning the point coordinates of each frame of the three-dimensional point cloud data TQ corresponding to the frames of the two-dimensional frame image DG(n) to absolute positions in space using a technique called registration.

[0043] The object detection unit 13 is a processing unit that estimates a bounding box BB of the recognition target object 22 for each frame of the two-dimensional frame image DG(n), and as described in Figures 4A and 4B, projects the bounding box BB with the same identification number BID onto the representative image via a partial point group corresponding to the bounding box BB of the three-dimensional point cloud data TP, obtains the circumscribing rectangle of each of the two-dimensional point groups of the projected bounding box BB, and integrates the circumscribing rectangles to detect the recognition target object 22.

[0044] In detail, since the image of the end face of the cylindrical recognition target object 22 is a circle or an ellipse, the object detection unit 13 first detects the circle or ellipse in each frame of the two-dimensional frame image DG(n) and obtains a rectangular frame surrounding the detected circle or ellipse as a bounding box BB. The method for detecting the end face of the recognition target object 22 may be an object detection method using deep learning, or may be an object detection method using HoG features or the like.

[0045] In each frame of the two-dimensional frame image DG(n), the same identification number BID is assigned to the bounding box BB of the same recognition target object 22. The detection of the bounding box BB is performed for all frames of the two-dimensional frame image DG(n).

[0046] The details of the process in the object detection unit 13 of mapping the bounding box BB of the same identification number BID onto the representative image via the partial point cloud of the 3D point cloud data TP, determining the circumscribing rectangle of each of the mapped 2D point clouds, and integrating the circumscribing rectangles to detect the recognition target object 22 will be described in detail with reference to Figure 8.

[0047] The counting processing unit 14 counts the number of recognition target objects 22 detected by the object detection unit 13 by integrating and determining the circumscribing rectangles.

[0048] The representative image display unit 15 displays a rectangular figure obtained by integrating the circumscribing rectangles and superimposing it on the representative image showing the overall image of the object group 20, and also displays the number of recognition target objects 22 counted by the counting processing unit 14. This makes it possible to easily grasp the number of recognition target objects 22 in the object group 20, and to confirm the detection status of the bounding boxes, thereby making it possible to grasp the accuracy of the counting results.

[0049] In the above, the object recognition device 10 is described as having the data acquisition unit 11, but the data acquisition unit 11 may be configured separately from other components. For example, the data acquisition unit 11 may be a mobile terminal such as a smartphone equipped with a camera, LiDAR, and IMU sensor. In this case, the representative image display unit 15 may display the representative image and the counting result on the mobile terminal.

[0050] FIG. 7 is a block diagram of a computer 980 . The object recognition device 10 of the embodiment is configured by a computer 980 shown in Fig. 7. In Fig. 7, the computer 980 includes a CPU 981, a storage unit 982, a communication I / F (interface) 983, an input / output I / F 984, and a media I / F 985. Here, the storage unit 982 includes a RAM 982a, a ROM 982b, and a HDD 982c. The communication I / F 983 is connected to a communication circuit 986. The input / output I / F 984 is connected to an input / output device 987. The media I / F 985 reads and writes data from a recording medium 988.

[0051] The ROM 982b stores an IPL (Initial Program Loader) executed by the CPU, etc. The HDD 982c stores application programs, various data, etc. The CPU 981 executes application programs, etc. loaded from the HDD 982c to the RAM 982a to realize various functions. The configuration diagram of the object recognition device 10 shown in FIG. 6 shows, as blocks, functions realized by application programs, etc.

[0052] Next, the object detection process of the object detection unit 13 will be described in detail with reference to the flow diagram of FIG.

[0053] In step S81, the object detection unit 13 estimates the bounding box BB of the recognition target object 22 in all frames of the two-dimensional frame image DG(n), and assigns the same identification number BID to the bounding box BB of the same recognition target object 22. In step S82, the object detection unit 13 selects one unselected identification number BID for which no processing has been performed.

[0054] In step S83, the object detection unit 13 selects one unprocessed bounding box BB belonging to the identification number BID selected in step S82 in a different frame of the two-dimensional frame image DG(n). In step S84, the object detection unit 13 obtains, in the three-dimensional point cloud data TP, a partial point cloud of a bounding box corresponding to the bounding box BB selected in step S83.

[0055] At this time, the object detection unit 13 determines only the points with a short distance as a partial point group of the bounding box corresponding to the bounding box BB. Furthermore, the object detection unit 13 may be configured to statistically exclude outliers in the distances to the corresponding point cloud, thereby improving the accuracy of object detection.

[0056] In step S85, the object detection unit 13 determines whether all bounding boxes BB belonging to the identification number BID selected in step S82 have been processed, and if they have been processed (Yes in S85), proceeds to step S86, and if they have not been processed (No in S86), returns to step S83.

[0057] In step S86, the object detection unit 13 projects the partial point clouds of all the bounding boxes BB belonging to the identification number BID selected in step S82 in the three-dimensional point cloud data TP onto the representative image. In this case, a projection transformation such as enlargement or reduction may be performed depending on the angle difference with respect to the object group 20 between the representative image and the frame of the two-dimensional frame image DG(n) corresponding to the partial point group of the bounding box BB projected onto the representative image.

[0058] In step S87, the object detection unit 13 obtains a circumscribed rectangle of the two-dimensional point cloud of all the bounding boxes BB belonging to the identification number BID selected in step S82 and projected onto the representative image in step S86. In step S88, the object detection unit 13 merges the circumscribing rectangles in step S88 as described with reference to FIG.

[0059] In step S89, the object detection unit 13 determines whether or not the identification numbers BID of all the bounding boxes BB estimated in step S81 have been processed. If the identification numbers BID of all the bounding boxes BB estimated in step S81 have not been processed (No in S89), the object detection unit 13 returns to step S82, and if the identification numbers BID of all the bounding boxes BB estimated in step S81 have not been processed (Yes in S89), the object detection process of the object detection unit 13 is terminated.

[0060] Next, a display example of the representative image displayed by the representative image display unit 15 (see FIG. 6) will be described with reference to FIG. The representative image display unit 15 displays the number of objects 22 to be recognized counted by the counting processing unit 14, and also displays a rectangular figure formed by integrating the circumscribing rectangle of the two-dimensional point cloud of the bounding box BB of the object 22 to be recognized, superimposed on the representative image showing the overall image of the object group 20, as shown in Figure 9.

[0061] At this time, since the object recognition device 10 also detects hidden recognition target objects 22, a rectangular figure (indicated by a dashed line) obtained by integrating the circumscribing rectangle of the two-dimensional point cloud of the bounding box BB is displayed superimposed on the representative image. Therefore, as shown in Fig. 9, the display of the rectangular figure indicating the bounding box of the recognition target object 22 becomes complicated, making it difficult to judge the object detection status of each recognition target object 22.

[0062] Therefore, in the object recognition device 10, the representative image display unit 15 displays a rectangular figure showing the bounding box of the hidden target object 22 in a distinguishable manner, and allows the user to set whether or not to display the rectangular figure showing the bounding box of the hidden target object 22. This makes it easier to grasp the object detection situation of the target object 22. [Explanation of symbols]

[0063] 10 Object recognition device 11 Data Acquisition Section 12 Data Analysis Department 13 Object detection unit 14 Counting processing section 15 Representative image display section 20 Object group 22 Recognition target object

Claims

1. a data acquisition unit that acquires two-dimensional video data obtained by capturing an object group including a plurality of recognition targets from a plurality of directions and a distance image of the object group synchronized with the two-dimensional video data; a data analysis unit for obtaining three-dimensional point cloud data of the object group based on a range image of the object group; an object detection unit that estimates a bounding box of a recognition target object for each frame of the two-dimensional video data, projects the bounding box having the same identification number onto a representative image via a partial point group corresponding to the bounding box of the three-dimensional point group data, obtains a circumscribing rectangle of each of the two-dimensional point groups of the projected bounding box, and integrates the circumscribing rectangles to detect the recognition target object; Object recognition device.

2. The object recognition device according to claim 1 , the object detection unit determines a frame of the two-dimensional video data having a large number of estimated bounding boxes as a representative image. Object recognition device.

3. The object recognition device according to claim 1 , the object detection unit uses the first frame or a central frame of the two-dimensional video data as a representative image; Object recognition device.

4. The object recognition device according to claim 1 , The object detection unit is A still image of a group of objects different from the frames of the two-dimensional video data is set as the representative image; generating three-dimensional point cloud data from depth images of the object group acquired when capturing the still images; The three-dimensional point cloud data is 3D registered with the three-dimensional point cloud data of the object group created by the data analysis unit to obtain a representative image. Object recognition device.

5. The object recognition device according to claim 1 , the object detection unit calculates an average, a median, or a maximum value of coordinates of each circumscribing rectangle of the two-dimensional point cloud of the projected bounding box, and integrates the circumscribing rectangles; Object recognition device.

6. The object recognition device according to claim 1 , the object detection unit obtains a circumscribing rectangle of a figure obtained by integrating the circumscribing rectangles of the two-dimensional point cloud of the projected bounding box; Object recognition device.

7. The object recognition device according to claim 1 , the object detection unit calculates an average or a median of center positions of circumscribing rectangles of the two-dimensional point cloud of the projected bounding box as a center position of an integrated figure, and calculates a size of the integrated figure as a size of the recognition target object estimated from a distance; Object recognition device.

8. The object recognition device according to claim 1 , When projecting a bounding box onto a representative image via a partial point cloud corresponding to the bounding box of the three-dimensional point cloud data, the object detection unit performs a projection transformation of enlargement or reduction according to an angle difference to an object group between the representative image and a frame of a two-dimensional frame image corresponding to the partial point cloud of the bounding box BB to be projected onto the representative image. Object recognition device.

9. The object recognition device according to claim 1 , further comprising: a representative image display unit that displays a rectangular figure indicating a boundary box of an obscured object to be recognized in a distinguishable manner and enables a user to set whether or not to display the rectangular figure indicating the boundary box of the obscured object to be recognized; Object recognition device.

10. A data acquisition unit acquires two-dimensional video data obtained by capturing an object group including a plurality of recognition targets from a plurality of directions and a distance image of the object group synchronized with the two-dimensional video data; A data analysis unit obtains three-dimensional point cloud data of the object group based on a range image of the object group; an object detection unit estimating a bounding box of a recognition target object for each frame of the two-dimensional video data, determining partial point clouds of the three-dimensional point cloud data corresponding to each of the bounding boxes having the same identification number, projecting the partial point clouds corresponding to the bounding boxes of the three-dimensional point cloud data onto a representative image, determining circumscribing rectangles of each of the two-dimensional point clouds of the projected bounding boxes, and integrating the circumscribing rectangles to detect the recognition target object; 1. An object recognition method comprising:

Images