Information processing device, monitoring system, integration method, and integration program

Abstract

To correctly detect objects by compensating for the parts that are not detected because they are in the shadow of other objects.SOLUTION: An information processing device (1) includes a projection unit (105) that projects a point group corresponding to a first object detected from point group data onto a depth of a second object detected at a position deeper than the first object, and an integration unit (106) that integrates the projected point group corresponding to the first object into a point group corresponding to the second object to make it part of the second object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for detecting an object from point cloud data. [Background technology]

[0002] Techniques for detecting objects from point cloud data have been known for some time. For example, the following Patent Document 1 discloses a crane monitoring device that detects a load suspended from a hook and its shape using three-dimensional point cloud data generated by a point cloud data generator disposed in a position overlooking the hook of the crane. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-61911 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, LiDAR (Light Detection and Ranging) or the like is used as the point cloud data generating unit. LiDAR is a technology that measures the shape of an object and the distance to the object by irradiating the object with laser light and analyzing the reflected light. Therefore, if there is an object blocking the laser light at a position shallower than the object to be detected (a position closer to the laser light irradiating unit of the LiDAR), the detection of the object may be hindered. In particular, when the technology of Patent Document 1 is applied, the hook of the crane is located between the point cloud data generating unit and the suspended load, so that at least a part of the suspended load is shaded by the hook and cannot be detected.

[0005] In this way, when detecting an object using point cloud data, there is a problem that a part of an object at a deeper position cannot be detected because it is shadowed by an object at a shallower position. One aspect of the present invention aims to realize an information processing device or the like that can correctly detect an object by compensating for a part that is not detected because it is shadowed by another object. [Means for solving the problem]

[0006] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a projection unit that projects a point cloud corresponding to a first object detected from point cloud data including points indicating the surface positions of a plurality of objects, onto a depth of a second object detected at a position deeper than the first object, and an integration unit that integrates the projected point cloud corresponding to the first object into a point cloud corresponding to the second object to make it part of the second object.

[0007] In order to solve the above problems, a monitoring system according to one embodiment of the present invention is a crane monitoring system that includes a generating device that generates point cloud data including at least points indicating the hook of the crane, a load suspended by the hook, and the surface positions of objects surrounding the load, an information processing device that projects the point cloud corresponding to the hook detected from the point cloud data onto the depth of the load and integrates the projected point cloud corresponding to the hook into the point cloud corresponding to the load to make it part of the load, and an alarm device that issues an alarm depending on the distance between the integrated load and the objects surrounding the load.

[0008] In order to solve the above problems, an integration method according to one embodiment of the present invention is an integration method executed by one or more information processing devices, and includes a projection step of projecting a point cloud corresponding to a first object detected from point cloud data including points indicating the positions of the surfaces of a plurality of objects, to a depth of a second object detected at a position deeper than the first object, and an integration step of integrating the projected point cloud corresponding to the first object with a point cloud corresponding to the second object to make it part of the second object. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to correctly detect an object by compensating for undetected portions that are in the shadow of another object. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a configuration of a main part of an information processing device according to an embodiment of the present invention; [Diagram 2] 1 is a diagram showing a configuration example of a crane monitoring system according to an embodiment of the present invention; [Diagram 3] 1A to 1C are diagrams illustrating a method for detecting a wire according to an embodiment of the present invention. [Figure 4] FIG. 13 is a diagram showing an example in which each point clustered to a suspended load among the detected point cloud is divided according to its depth. [Diagram 5] FIG. 11 is a diagram illustrating an example of projection by a projection unit. [Figure 6] 4 is a flowchart illustrating an example of processing executed by an information processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [System Configuration] The configuration of a crane monitoring system 100 according to one embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the monitoring system 100. The monitoring system 100 is a system for safety management of a crane CR, and as shown in the figure, the monitoring system 100 includes an information processing device 1, an alarm device 2, and a generation device 3.

[0012] The crane CR is a machine that lifts and moves a heavy object (load). The crane CR includes a carriage CR1, a base CR2, a boom CR3, a cable CR4, a hook CR5, a crane room CR6, and a wire CR7.

[0013] The cart section CR1 carries a base section CR2. The base section CR2 is a structure that supports the boom CR3 and the like. The base section CR2 and the cart section CR1 are connected so that the base section CR2 can rotate on the cart section CR1. This enables the crane CR to transport a load located within 360 degrees around it. The rotation range may be less than 360 degrees. The crane CR may also be one that cannot rotate within a horizontal plane.

[0014] One end of the boom CR3 is rotatably connected to the base CR2. A pulley (not shown) is provided at the other end of the boom CR3, and a cable CR4 hangs down from the pulley.

[0015] The cable CR4 is a rope for hoisting a load, and a hook CR5 for hanging the load is attached to the end of the cable CR4. When a load is to be moved by the crane CR, the load is sling-hooked onto the hook CR5 and lifted. In the example of Fig. 2, a load A1 is suspended from the hook CR5 via a wire CR7. The wire CR7 is for suspending the load A1, and is attached to the load A1.

[0016] The crane room CR6 is a space that houses various devices for operating the crane CR, and the operator of the crane CR (hereinafter simply referred to as the operator) operates the crane CR from the crane room CR6. For example, the operator can adjust the length of the boom CR3, raise and lower the hook CR5 (load) by winding up and down the cable CR4, and rotate the structure of the base CR2.

[0017] A generating device 3 is attached near the tip of the boom CR3 of the crane CR. The generating device 3 generates point cloud data including at least points indicating the hook CR5 of the crane CR, the load suspended by the hook CR5, and the surfaces of objects around the load. For example, when the generating device 3 generates point cloud data in the state shown in FIG. 2, point cloud data is generated that indicates the positions of the surfaces of each object within the cone-shaped measurement area AR1, specifically the hook CR5, the load A1, objects B1 and B2, etc. The size of the measurement area AR1 can change depending on the height of the generating device 3 (the length and angle of the boom CR3).

[0018] The generating device 3 may be any device capable of generating the above-mentioned point cloud data. For example, various laser scanners and the like can be applied as the generating device 3. For example, LiDAR (Light Detection And Ranging) may be applied as the generating device 3. The data format of the point cloud data is not particularly limited. In this embodiment, an example will be described in which each point included in the point cloud data is expressed by coordinate values ​​(x, y, z) in an orthogonal coordinate system defined by three mutually orthogonal axes xyz. For example, as shown in FIG. 2, a coordinate system may be applied in which the vertical downward direction is the positive direction of the x-axis direction, and the y-axis and z-axis perpendicular to each other are defined on a horizontal plane perpendicular to the x-axis. In this case, the yz plane is the surface to be measured. In addition, the origin of the coordinate system, that is, a point with coordinate values ​​(0,0,0) may be the position of the generating device 3. The unit of the coordinate values ​​may be, for example, meters. In the following, a point indicating the position of the generating device 3 (for example, a point with coordinate values ​​(0,0,0)) is called a reference point.

[0019] As will be described in more detail later, the information processing device 1 has the function of projecting the point cloud corresponding to hook CR5 detected from the point cloud data generated by the generating device 3 onto the depth of the load A1, and integrating the projected point cloud corresponding to hook CR5 into the point cloud corresponding to the load A1 to make it part of the load A1.

[0020] The warning device 2 issues a warning according to the distance between the suspended load A1 after integration by the information processing device 1 and the objects B1 and B2 around the suspended load A1. The calculation of the distance may be performed by the warning device 2, but in this embodiment, an example will be described in which the information processing device 1 calculates the distance and instructs the warning device 2 to issue a warning according to the calculated distance.

[0021] For example, the information processing device 1 may instruct the alarm device 2 to issue an alarm when the distance between the load A1 and the object B1 or B2 becomes equal to or less than a predetermined threshold (e.g., 2 m). This can prevent the operator from causing the load A1 to come into contact with the object B1 or B2 without noticing that the load A1 and the object B1 or B2 are approaching each other. The alarm may be issued in a manner that can be recognized by the operator. For example, the alarm device 2 may issue an alarm by any one of sound, light, vibration, and image display, or a combination of a plurality of these. In this way, the warning device 2 issues a warning according to the calculated distance, which contributes to the safety management of the crane CR. When a worker or the like enters the measurement area AR1, the information processing device 1 calculates the distance between the suspended load A1 and the person who entered, so that the information processing device 1 also contributes to ensuring the safety of the worker or the like.

[0022] The device configuration of the monitoring system 100 can be changed as appropriate. For example, the information processing device 1 and the generating device 3 can be configured as a single device having a single housing. The same applies to the information processing device 1 and the alarm device 2. Furthermore, the monitoring system 100 may include, for example, a monitoring device that displays an image showing the suspended load and objects around it, in addition to the alarm device 2 that issues an alarm by voice.

[0023] Furthermore, each device included in the monitoring system 100 may be battery-powered. Furthermore, communication between each device included in the monitoring system 100 may be performed by wireless communication or wired communication. Furthermore, in the monitoring system 100, the location of the information processing device 1 is arbitrary and is not particularly limited. For example, the information processing device 1 may be placed in a control center or the like located away from the crane CR. Furthermore, for example, the information processing device 1 may be placed in the crane room CR6. In this case, the information processing device 1 may be provided with the function of the alarm device 2, and the alarm device 2 may be omitted.

[0024] As described above, the monitoring system 100 includes a generating device 3 that generates point cloud data including at least points indicating the hook CR5 of the crane CR, the load A1 suspended by the hook CR5, and the surface positions of the objects B1 and B2 surrounding the load A1, an information processing device 1 that projects the point cloud corresponding to the hook CR5 detected from the point cloud data onto the depth of the load A1 and integrates the projected point cloud corresponding to the hook CR5 with the point cloud corresponding to the load A1 to make it part of the load A1, and an alarm device 2 that issues an alarm depending on the distance between the integrated load A1 and the objects B1 and B2 surrounding the load A1.

[0025] The monitoring system 100 can correctly detect the load A1, including the portion shaded by the hook CR5. The monitoring system 100 issues an alarm according to the distance between the correctly detected integrated load A1 and the objects B1 and B2 around the load A1, and can therefore contribute to the safety management of the crane CR by issuing an alarm at an appropriate time.

[0026] [Device configuration] The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the main configuration of the information processing device 1. As shown in the figure, the information processing device 1 includes a control unit 10 that controls each unit of the information processing device 1, and a storage unit 11 that stores various data used by the information processing device 1. The information processing device 1 may be, for example, a personal computer, a server, or a workstation.

[0027] The information processing device 1 also includes a communication unit 12 for the information processing device 1 to communicate with other devices. The communication unit 12 functions as an interface for controlling the alarm device 2, and also functions as an interface for acquiring point cloud data from the generation device 3. Note that the communication with the alarm device 2 and the communication with the generation device 3 may be performed via separate communication units. The communication unit 12 may perform wired communication or wireless communication. For example, the communication unit 12 may be one that performs communication via a Universal Serial Bus (USB), or one that performs communication via a wireless or wired Local Area Network (LAN).

[0028] The information processing device 1 also includes an input unit 13 that accepts input of various data to the information processing device 1, and an output unit 14 that allows the information processing device 1 to output various data. The output unit 14 may be, for example, a display unit that displays an image. In this case, the information processing device 1 can display, on the output unit 14, point cloud data, objects detected from the point cloud data, calculation results of distances between objects, and the like.

[0029] The display destination of such information is not particularly limited. For example, the information may be displayed on a display device connected to the information processing device 1, or if the alarm device 2 shown in FIG. 2 has a display unit, the information may be displayed on the display unit, or the information may be displayed on a display device connected to the alarm device 2. When the information is displayed on an external display device, for example, a remote desktop function, a VNC (Virtual Network Computing) function, or an SSH (Secure Shell) X11 forward function may be used. Information may also be displayed on an external display device by a dedicated application. A normal image capturing device may be placed near the generating device 3, and an image captured by the image capturing device may be displayed together with the above-mentioned information.

[0030] The control unit 10 also includes a data acquisition unit 101, a clustering unit 102, an object detection unit (detection unit) 103, a depth estimation unit 104, a projection unit 105, an integration unit 106, a distance calculation unit 107, a notification control unit 108, and a display control unit 109.

[0031] The data acquisition unit 101 acquires point cloud data including at least points indicating the surfaces of a plurality of objects. For example, the data acquisition unit 101 may acquire the point cloud data generated by the generating device 3 shown in Fig. 2 through communication via the communication unit 12. The point cloud data may be coordinate values ​​indicating the position of each point in a three-dimensional space.

[0032] The clustering unit 102 clusters each point included in the point cloud data acquired by the data acquisition unit 101. Clustering of point cloud data can also be said to be division or classification of the point cloud data. Clustering of point cloud data may be performed based on the positional relationship of each point. For example, the clustering unit 102 may classify points located close to each other into one cluster, or may classify points located on the same plane into one cluster.

[0033] The object detection unit 103 detects a plurality of objects from point cloud data acquired by the data acquisition unit 101, the point cloud data including at least points indicating the surfaces of the plurality of objects. A specific method of object detection will be described later in the section "Object Detection". The object detection unit 103 also detects a wire (for example, wire CR7 in FIG. 2) for suspending a load from a hook of a crane. A method of wire detection will be described later in the section "Wire Detection".

[0034] The depth estimation unit 104 estimates the depth of the object detected by the object detection unit 103. Details of the depth estimation method will be described later in the section "depth estimation." Note that here, depth means the distance to the generating device 3. The farther the distance to the generating device 3 is (the larger the x-coordinate value is), the deeper the position is.

[0035] The projection unit 105 projects a point cloud corresponding to a first object detected from point cloud data including points indicating the positions of the surfaces of a plurality of objects, to the depth of a second object detected at a position deeper than the first object. For example, in the example of Fig. 2, a point cloud (which can also be called a cluster) corresponding to the hook CR5 is detected from the point cloud data generated by the generation device 3, and a point cloud corresponding to the suspended load A1 is detected at a position deeper than the hook CR5. In this case, the projection unit 105 projects the point cloud corresponding to the hook CR5 to the depth of the suspended load A1.

[0036] Here, when the point cloud data is clustered, one object may be divided into multiple clusters. Also, an object consisting of multiple surfaces with different depths, such as the suspended load A1 in Fig. 2, may be classified into different clusters for each surface. The integration unit 106 integrates into a single object an object that is actually one object but whose parts are detected as different objects because each part is classified into a different cluster.

[0037] The integration may be performed on clusters (objects) that satisfy a predetermined condition. For example, when the heights of a set of objects to be determined whether to integrate and the depths of the surfaces of the objects satisfy a predetermined condition and the outer edges of the set of objects overlap, the integration unit 106 may integrate the objects into a single object.

[0038] Furthermore, the integration unit 106 integrates the point cloud corresponding to the first object projected by the projection unit 105 with the point cloud corresponding to the second object, and makes it a part of the second object. For example, when the projection unit 105 projects the point cloud corresponding to the hook CR5 to the depth of the suspended load A1, the integration unit 106 integrates the projected point cloud corresponding to the hook CR5 with the point cloud corresponding to the suspended load A1, and makes it a part of the suspended load A1. This makes it possible to correctly detect the suspended load A1 by compensating for the part that is not detected because it is in the shadow of the hook CR5.

[0039] The distance calculation unit 107 calculates the distance between the objects detected by the object detection unit 103. For example, the distance calculation unit 107 calculates the distance between the load A1 and the surrounding objects B1 and B2. The method of calculating the distance is not particularly limited. For example, the distance calculation unit 107 may detect the contour of the load A1 from the point cloud of the load A1 obtained by integration by the integration unit 106, and may also detect the contours of the objects B1 and B2 from the point clouds of the objects B1 and B2 detected by the object detection unit 103. Then, the distance calculation unit 107 may calculate the shortest distance from the contour of the load A1 to the contour of the object B1 as the distance between the load A1 and the object B1. Similarly, the distance calculation unit 107 may calculate the shortest distance from the contour of the load A1 to the contour of the object B2 as the distance between the load A1 and the object B2.

[0040] Note that if the load and the object are located at positions apart in the depth direction, they will not come into contact with each other. Therefore, the distance calculation unit 107 may determine the presence or absence of an object that may come into contact with the load based on the respective depths of the load and the object estimated by the depth estimation unit 104, and if it is determined that such an object exists, calculate the distance between the object and the load.

[0041] The notification control unit 108 issues a notification to an operator of the crane CR or the like according to the distance calculated by the distance calculation unit 107. For example, the notification control unit 108 may cause the alarm device 2 to output an alarm when the distance calculated by the distance calculation unit 107, for example, the distance between a suspended load and an object around it, becomes equal to or less than a predetermined threshold value. Of course, the manner of notification and the content of the notification are arbitrary and are not limited to this example. For example, the notification control unit 108 may issue a notification via the output unit 14, or may issue a notification of the distance calculated by the distance calculation unit 107. Also, the notification may be issued by displaying an image, in which case the notification control unit 108 may be omitted and the display control unit 109 may issue the notification.

[0042] The display control unit 109 causes the display device to display the objects detected by the object detection unit 103. The display device may be provided in the information processing device 1 or in another device. For example, the display control unit 109 may obtain a contour line of each object detected by the object detection unit 103 and cause the display device to display the contour line. For example, the display control unit 109 may cause the display device to display the contour lines of the detected wire, hook, suspended load, and objects (which may be people) around the suspended load. Note that, for objects whose point clouds have been integrated by the integration unit 106, it is desirable for the display control unit 109 to display a contour line obtained from the integrated point clouds.

[0043] The display control unit 109 may also change the display color of the suspended load and the objects around it, and when multiple surrounding objects are detected, may change the display color of each of those objects. The position of the suspended load and the positions of the objects around the suspended load may change over time. For this reason, it is preferable that the generating device 3 generates time-series point cloud data, and the information processing device 1 detects objects at each time point from the point cloud data at that time point and calculates the distance between the objects. Then, it is preferable that the display control unit 109 updates the previous drawing content based on the latest detection result when an object is detected from the point cloud data measured at the next time point after drawing an object detected from the point cloud data measured at a certain time point.

[0044] As described above, the information processing device 1 includes a projection unit 105 that projects a point cloud corresponding to a first object detected from point cloud data including points indicating the surface positions of multiple objects, to the depth of a second object detected at a position deeper than the first object, and an integration unit 106 that integrates the projected point cloud corresponding to the first object with a point cloud corresponding to the second object to make it part of the second object.

[0045] When a first object and a second object are detected from point cloud data including points indicating the positions of the surfaces of a plurality of objects, a part of the second object detected at a position deeper than the first object may not be detected because it is in the shadow of the first object. In this respect, according to the above configuration, the point cloud corresponding to the first object is projected to the depth of the second object, and the projected point cloud corresponding to the first object is integrated with the point cloud corresponding to the second object to form a part of the second object. This makes it possible to correctly detect the second object by compensating for the part that is not detected because it is in the shadow of the first object.

[0046] As described above, the first object may be the hook CR5 of the crane CR, and the second object may be the load A1 suspended by the hook CR5. This configuration makes it possible to correctly detect the load A1 by compensating for the portion that is not detected due to being in the shadow of the hook CR5. For example, even when a narrow load A1 is divided into two objects by being in the shadow of the hook CR5, these can be integrated into a single load A1 and correctly detected.

[0047] [Object detection] Here, a specific method of object detection by the object detection unit 103 will be described with reference again to Fig. 2. When point cloud data is acquired near the load A1 of the crane CR shown in Fig. 2, the object detection unit 103 can detect the hook CR5, the load A1, objects B1 and B2, and the ground from the point cloud data. When these are the detection targets, the object detection unit 103 first detects the ground, then detects the hook CR5 and the load A1, and finally detects objects B1 and B2.

[0048] In detecting the ground, the object detection unit 103 may detect the deepest cluster among the clusters into which the clustering unit 102 has classified the point cloud as the ground. For example, the object detection unit 103 may calculate the average value of the x-coordinate values ​​of all points included in each cluster into which the clustering unit 102 has classified the point cloud, and detect the cluster with the largest calculated average value as the ground. Alternatively, the object detection unit 103 may divide each cluster into a grid, and calculate the average value of the x-coordinate values ​​of the points included in each grid. Then, the object detection unit 103 may detect the cluster with the largest calculated average value as the ground. In addition, the object detection unit 103 may detect a cluster at a depth close to the cluster detected as the ground as the ground. For example, the object detection unit 103 may detect the deepest cluster and a cluster whose distance from the deepest cluster is equal to or less than a predetermined threshold as the ground.

[0049] In detecting the hook CR5 and the load A1 after detecting the ground, the object detection unit 103 detects the clusters corresponding to the hook CR5 and the load A1 from among the clusters into which the clustering unit 102 has classified the point cloud, excluding the cluster corresponding to the ground. For example, if the generating device 3 is attached directly above the hook CR5, the point corresponding to the load A1 will be located near the center when the point cloud data is projected onto the yz plane. In this way, since the distribution range of the points corresponding to the load A1 can be specified in advance, the object detection unit 103 only needs to search for the cluster corresponding to the load A1 within the distribution range.

[0050] For example, when the object detection unit 103 projects the points of each cluster onto the yz plane, it may detect as the cluster corresponding to the suspended load A1 the cluster whose center is within the above distribution range, whose area on the yz plane is greater than the upper surface of the hook CR5, and whose depth is the shallowest.

[0051] Similarly, it is possible to specify in advance the distribution range of the point cloud corresponding to the hook CR5. Therefore, the object detection unit 103 may search for the cluster corresponding to the hook CR5 within the distribution range. The cluster corresponding to the hook CR5 and the cluster corresponding to the load A1 can be distinguished from each other by the difference in depth. In addition, when detecting either the hook CR5 or the load A1, it is preferable to set the distribution range described above broadly, taking into consideration that the hook CR5 and the load A1 swing. In addition, since the hook CR5 is smaller than the load A1 and is located above it, the swing width is also smaller than that of the load A1, so the distribution range of the hook CR5 may be narrower than that of the load A1.

[0052] Then, the object detection unit 103 classifies the point cloud into clusters by the clustering unit 102, excluding the cluster corresponding to the ground and the cluster corresponding to the hook CR5 and the load A1, and classifies the remaining clusters as clusters of the objects B1 and B2. Through the above process, the hook CR5, the load A1, and the objects B1 and B2 are detected from the point cloud data.

[0053] [Wire detection] The object detection unit 103 can also detect a wire for suspending a load from a hook of a crane (for example, the wire CR7 in FIG. 2). Specifically, when the point cloud data is divided into three-dimensional grids, the object detection unit 103 may detect points included in each grid where the sum of the number of points included in a plurality of horizontally adjacent grids is equal to or less than a predetermined threshold as points indicating the position of the wire suspending the load.

[0054] According to this configuration, it is possible to detect the wire that suspends the load. Here, since the wire is connected to the load, there is a risk that the point cloud of the wire may be regarded as part of the load. If the point cloud of the wire is regarded as part of the load, it becomes difficult to accurately estimate the depth of the top surface of the load. In this regard, by having the object detection unit 103 distinguish and detect the wire and the load, it is possible to obtain an effect that it becomes easier to accurately estimate the depth of the top surface of the load. Note that the "wire that suspends the load" may include something that is not directly connected to the load. For example, the object detection unit 103 can detect the cable CR4 (the rope connected to the hook) shown in FIG. 2 as the "wire that suspends the load."

[0055] A method for detecting a wire by the object detection unit 103 will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining the method for detecting a wire. A space Sp1 shown in FIG. 3 indicates a range in which object detection is possible based on a point cloud generated by the generating device 3. When detecting a wire, the object detection unit 103 determines a detection range for the wire in the space Sp1. The detection range is determined to have a height, width, and depth that can accommodate the entire wire. For example, the object detection unit 103 may set the space between the point cloud corresponding to the load A1 and the point cloud corresponding to the hook CR5 as the detection range for the wire. As described above, since the points corresponding to the wire may be erroneously classified into a cluster such as the load A1 and the hook CR5, it is preferable to set the detection range so as to include a part of the cluster around the wire (for example, a cluster such as the load A1 and the hook CR5). In FIG. 3, a part of the determined detection range (a part corresponding to one of the multiple wires) is shown as Sp11.

[0056] Next, the object detection unit 103 divides the point cloud data included in the determined detection range into three-dimensional grids. The size of each grid is set to be sufficiently large relative to the diameter of the wire. For example, each grid may be a cube with a height, width, and depth of 10 cm. The upper right corner of FIG. 3 shows an example in which the detection range Sp11 is divided into 6 in the width direction, 7 in the height direction, and 4 in the depth direction. In this case, the total number of grids is 6×7×4=168.

[0057] Next, object detection unit 103 detects, from among grids including at least one point, a grid in which the sum of the number of points included in multiple grids adjacent to the grid in the horizontal direction is equal to or less than a predetermined threshold.

[0058] For example, the bottom right of Fig. 3 shows a top view of the fourth grid from the top in detection range Sp11. Of the grids in this fourth row, two contain points: grids G1 and G2. Grid G1 contains one point, and grid G2 contains four points. There are eight grids horizontally adjacent to grid G1, including grid G2, and the total number of points contained in these eight grids is four. On the other hand, there are eight grids horizontally adjacent to grid G2, including grid G1, and the total number of points contained in these eight grids is one.

[0059] For example, when the threshold is set to 3, the object detection unit 103 does not detect a grid G1 in which the sum of the number of points included in the eight horizontally adjacent grids exceeds 3, but detects a grid G2 in which the sum of the number of points included in the eight horizontally adjacent grids is 3 or less. In other words, the object detection unit 103 detects four points included in the grid G2 as points indicating the position of the wire. The object detection unit 103 performs such processing on the grids of each stage, thereby being able to detect a grid group including points indicating the position of the wire. The above threshold may be set according to the density of the points. In other words, the threshold may be set to a large value when the density of the points is high, and to a small value when the density of the points is low.

[0060] Furthermore, the object detection unit 103 may detect a grid including a point indicating the position of the wire, taking into consideration the number of points included in vertically adjacent grids. For example, the object detection unit 103 may detect a grid that satisfies a condition that the sum of the number of points included in multiple grids horizontally adjacent to a grid adjacent in the vertical direction (e.g., on the lower side) is equal to or less than a threshold, in addition to the above-mentioned condition that the sum of the number of points included in multiple grids horizontally adjacent to the grid adjacent in the vertical direction (e.g., on the lower side) is equal to or less than a threshold. Since the wire is detected as a group of continuous points in the vertical direction, by taking into consideration the number of points included in vertically adjacent grids, it is possible to reduce the possibility of erroneously detecting a point that does not correspond to the wire as a point on the wire.

[0061] In addition, although the object detection unit 103 of this embodiment detects a plurality of types of objects including a wire, a detection unit may be provided for each object to be detected. For example, a detection unit for detecting the ground, a detection unit for detecting a suspended load, a detection unit for detecting a hook, a detection unit for detecting a wire, and a detection unit for detecting other objects may be provided.

[0062] [Depth Estimation] Various methods can be applied to estimate the depth of a detected object. For example, the depth estimation unit 104 may identify a point with the smallest x value among the points included in the point cloud corresponding to the object detected by the object detection unit 103, and estimate the x value of that point as the depth of the object (more precisely, the depth of the top surface). However, with this method, if a point detected at a position shallower than the top surface of the object due to the influence of noise or the like is included in the point cloud of the object, the estimation result may be shallower than the actual depth of the object. In addition, when estimating the depth of a suspended load, there is a risk that the depth of a point such as a wire suspending the load may be erroneously estimated as the depth of the suspended load.

[0063] Therefore, the depth estimation unit 104 may classify each point included in the point cloud corresponding to one object detected from the point cloud data according to its depth, and estimate the depth of the upper surface of the object based on the number of points included in each class. This makes it possible to estimate the depth of the object with high accuracy.

[0064] A method for estimating the depth of an object based on a classification according to the depth of the point cloud of the object will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example in which each point clustered to a suspended load among the detected point cloud is classified according to its depth. The example in Fig. 4 shows the distribution of each point clustered to the suspended load on the xy plane.

[0065] The depth estimation unit 104 classifies each point according to its depth, that is, its position in the x-direction (vertical direction). For example, in the example of FIG. 4, each point included in the detected point group is classified into one of the sections 401, which is the shallowest position with the reference point as the upper end, to the section 410, which is the deepest position. The width of each section may be set appropriately according to the size of the detection target object, etc. For example, the width of each section may be set to 20 cm. After classifying each point as described above, the depth estimation unit 104 counts the number of points included in each section.

[0066] Here, the depth estimation unit 104 may estimate the depth of the top surface of the object based on the depth of the points included in the shallowest section among the sections in which the number of points included is equal to or greater than a predetermined threshold. This configuration makes it possible to accurately estimate the depth of the object. This is because when the points included in the point cloud corresponding to one object are divided according to their depth, the number of points changes significantly in the sections near the top surface of the object. The above threshold may be set according to the density of the points.

[0067] For example, in the example of FIG. 4, when the threshold value is set to 15, the depth estimation unit 104 searches for a section that satisfies a condition that the number of points included is equal to or greater than the threshold value (hereinafter, referred to as the first condition), starting from the section with the shallowest position. As a result, the section 408 is detected as the section with the shallowest position that satisfies the first condition. In this case, the depth estimation unit 104 estimates the depth of the top surface of the object (the suspended load in this example) based on the depth of the points included in the section 408. For example, the depth estimation unit 104 may estimate the depth of the shallowest point among the points included in the section 408 as the depth of the top surface of the object. Also, for example, the depth estimation unit 104 may estimate the average depth of the points included in the section 408 as the depth of the top surface of the object.

[0068] The depth estimation unit 104 may estimate the depth of the top surface of the object based on the depth of the point included in the shallowest section among sections in which the ratio of the number of points to the maximum number of points included in one section is equal to or greater than a predetermined threshold. This threshold may also be set appropriately according to the density of the points, similar to the other thresholds described above.

[0069] Even in point clouds corresponding to the same object, the number of points included in the point clouds may differ depending on the model of the generating device 3 that generates the point cloud data, the positional relationship between the generating device 3 and the object, etc. In this respect, by estimating the depth of the top surface of the object based on the depth of the points included in the shallowest section among sections in which the ratio of the number of points to the maximum number of points included in one section is equal to or greater than a predetermined threshold, it is possible to take into account the difference in the number of points included in the point clouds corresponding to one object. Therefore, it becomes possible to accurately estimate the depth of the object.

[0070] In this case, the depth estimation unit 104 counts the number of points included in each section, and then identifies the maximum number of points included in one section. In the example of Fig. 4, the number of points included in section 408 is 20, which is the largest among sections 401 to 410, so 20, which is the number of points included in section 408, is identified as the maximum value. In addition, the depth estimation unit 104 calculates, for each section, the ratio of the number of points included in that section to the maximum value. In the example of Fig. 4, the ratios in sections 407 to 410 are 0.2, 1.0, 0.8, and 0.1, respectively.

[0071] Next, the depth estimation unit 104 searches for a section that satisfies a condition that the calculated ratio is equal to or greater than a predetermined threshold (hereinafter, referred to as the second condition), starting from the shallowest section. This allows the shallowest section that satisfies the second condition to be detected. The depth estimation unit 104 then estimates the depth of the top surface of the object based on the depth of the points included in the detected section. For example, when the threshold is set to 0.2, the section 407 is detected as the shallowest section that satisfies the second condition. The depth estimation unit 104 may also detect the shallowest section that satisfies at least one of the first and second conditions described above.

[0072] [About projection] The projection unit 105 projects a point cloud corresponding to a first object detected from point cloud data including points indicating the positions of the surfaces of a plurality of objects, to the depth of a second object detected at a position deeper than the first object. The first object is an object detected between the generating device 3 and the second object. For example, in the example of FIG. 2, the first object may be the hook CR5, and the second object may be the suspended load A1. For example, the depth of the surface of the second object estimated by the depth estimation unit 104 may be applied as the depth of the second object.

[0073] Furthermore, the projection unit 105 may perform a process for projecting each point included in the point cloud corresponding to the first object onto a line connecting a reference point indicating the position in the three-dimensional space of the generating device 3 that generates the point cloud data and a point included in the point cloud corresponding to the first object. This makes it possible to project the point cloud corresponding to the first object onto an appropriate position according to the positional relationship between the generating device 3, the first object, and the second object.

[0074] An example of projection by the projection unit 105 will be described below with reference to FIG. 5. FIG. 5 is a diagram showing an example of projection by the projection unit 105. 501a in FIG. 5 is a side view showing the positional relationship between the generation device 3, the hook CR5, and the load A1, and 501b is a top view showing the positional relationship between the generation device 3, the hook CR5, and the load A1. As shown in these figures, the hook CR5 is located directly below the generation device 3, and the load A1 is located directly below the hook CR5. The depth (width in the z-axis direction) of the load A1 is approximately the same as the depth of the hook CR5. For this reason, the central portion A12 of the load A1 is in the shadow of the hook CR5, and the generation device 3 cannot generate a point corresponding to the central portion A12. Therefore, when object detection is performed from the point cloud data generated by the generation device 3, the hook CR5 is detected, and the left end portion A11 and the right end portion A13 of the load A1 are each detected as independent objects.

[0075] The center of Fig. 5 shows the distribution of each point included in the point cloud data generated when the generating device 3, the hook CR5, and the load A1 are in the positional relationship shown in 501a and 501b of Fig. 5. More specifically, 502a in Fig. 5 shows the distribution of each point included in the generated point cloud data when projected onto the xy plane, and 502b in the same figure shows the distribution of each point included in the generated point cloud data when projected onto the yz plane. Note that 502a and 502b in Fig. 5 show the position of the generating device 3 as a reference point.

[0076] As shown in 502a and 502b of FIG. 5, the point cloud data generated by the generating device 3 includes a point cloud cr5 corresponding to the hook CR5, a point cloud a11 corresponding to the left end portion A11 of the load A1, and a point cloud a13 corresponding to the right end portion A13 of the load A1. As shown in 502a, the point clouds a11 and a13 are located deeper than the point cloud cr5. Also, as shown in 502b, when the point clouds a11, a13, and cr5 are projected onto the yz plane, these point clouds are approximately aligned in a straight line in the y-axis direction. However, due to the difference in depth between the hook CR5 and the load A1, a space is generated between these point clouds in the y-axis direction.

[0077] When projecting each point included in the point group cr5, the projection unit 105 obtains the equation of a straight line connecting the points included in the point group cr5 and the reference point. The equation of this straight line can be obtained from the coordinates of the reference point and the coordinates of the points included in the point group cr5. Then, the projection unit 105 obtains the coordinates of a point on this straight line that has a depth of d1. The obtained point is the coordinate of the point after projection. The projection unit 105 similarly obtains the coordinates of each point included in the point group cr5 after projection. Note that the depth d1 is the depth of an object detected at a position deeper than the point group cr5, and in the example of FIG. 5, it is the depth of the left end part A11 or the right end part A13 of the suspended load A1. Below, an example will be described in which the depth of the left end part A11 of the suspended load A1 is d1.

[0078] 503a and 503b in Fig. 5 show a point group cr5' obtained by projecting each point included in the point group cr5 as described above. As shown in 503a, the depth of each point included in the point group cr5' is d1, the same as the point group a11. Also, as shown in 503b, there is no space in the y-axis direction between the point group cr5' generated by linearly projecting as described above and the point groups a11 and a13.

[0079] Therefore, the integration unit 106 can integrate the point cloud cr5' into the point cloud a11 corresponding to the left end portion A11 of the suspended load A1, and make it a part of the left end portion A11. In addition, since the integrated point cloud and the point cloud a13 are adjacent to each other, the integration unit 106 can integrate these point clouds. As a result, the point cloud a11, the point cloud cr5', and the point cloud a13 are integrated, and these point clouds represent one suspended load A1. Note that the integration unit 106 may integrate the point cloud a11 after integrating the point cloud cr5' into the point cloud a13.

[0080] Here, the load A1 and the hook CR5 may swing like a pendulum. Even when the load A1 and the hook CR5 swing like this, the generation device 3, the hook CR5, and the load A1 are maintained in a state of being aligned in a straight line. Therefore, even when using point cloud data acquired when the load A1 and the hook CR5 are swinging, the point cloud corresponding to the hook CR5 can be projected to the depth of the load A1 by the method described above, and the point cloud corresponding to the hook CR5 and the point cloud corresponding to the load A1 can be integrated.

[0081] [Processing flow] The flow of processing executed by the information processing device 1 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of processing executed by the information processing device 1. Fig. 6 shows processing immediately after the generation device 3 completes measurement of point cloud data. In addition, the following describes processing for detecting the hook CR5, the suspended load A1, and the objects B1 and B2 in Fig. 2.

[0082] In S11, the data acquisition unit 101 acquires point cloud data measured by the generating device 3. For example, the data acquisition unit 101 may acquire the point cloud data from the generating device 3 by communication via the communication unit 12. In addition, in S12, the clustering unit 102 classifies the point cloud data acquired in S11 into clusters.

[0083] In S13, the object detection unit 103 performs object detection based on the classification result in S12. For example, the object detection unit 103 may first detect a cluster corresponding to the ground from among the clusters into which at least one point was classified in S12. Then, the object detection unit 103 detects clusters corresponding to the hook CR5 and the suspended load A1 from the remaining clusters excluding the cluster corresponding to the ground, and detects clusters including a predetermined number of points or more from the remaining clusters as clusters corresponding to the objects B1 and B2.

[0084] In S14, the object detection unit 103 detects the wire CR7. Specifically, the object detection unit 103 first determines a detection range for the wire CR7. Next, the object detection unit 103 divides the point cloud data included in the determined detection range into three-dimensional grids. Then, the object detection unit 103 detects a grid including at least one point, in which the sum of the number of points included in multiple grids horizontally adjacent to the grid is equal to or less than a predetermined threshold. Each point included in each grid detected in this way indicates the position of the wire CR7.

[0085] In S15, the depth estimation unit 104 estimates the depth of the object detected in S13. For example, the depth estimation unit 104 may classify each point included in the point cloud corresponding to one object detected in S13 according to its depth, and estimate the depth of the upper surface of the object based on the number of points included in each class. In S15, the depth estimation unit 104 estimates the depth of the upper surfaces of at least the suspended load A1 (or each part if detected as multiple parts) and the objects B1 and B2.

[0086] In S16 (projection step), the projection unit 105 projects a point cloud corresponding to a first object, which is one of the objects detected in S13, onto the depth of a second object detected in S13 at a position deeper than the first object. Specifically, the projection unit 105 projects a point cloud corresponding to the hook CR5 onto the depth of the suspended load A1 (more precisely, the depth of a portion of the suspended load).

[0087] In S17 (integration step), the integration unit 106 integrates the point cloud corresponding to the first object projected in S16 with the point cloud corresponding to the second object to make it a part of the second object. Specifically, the integration unit 106 integrates the point cloud corresponding to the hook CR5 with the point cloud corresponding to the load A1 (more precisely, a part of the load A1) to make it a part of the load A1. Furthermore, if the point cloud corresponding to the load A1 after integration is adjacent to another point cloud (more specifically, another part of the load A1), the integration unit 106 integrates those point clouds. Furthermore, in S17, if there is an object that satisfies a predetermined integration condition, the integration unit 106 also integrates the point clouds of those objects.

[0088] In S18, the distance calculation unit 107 calculates the distance between two objects to be calculated among the multiple objects detected in S13. Specifically, the distance calculation unit 107 calculates the distance between the load A1 and the object B1, and also calculates the distance between the load A1 and the object B2. Note that the point cloud integrated in S17 is used as the point cloud representing the load A1.

[0089] In S19, the notification control unit 108 judges whether an alarm is necessary based on the distance calculated in S18. For example, the notification control unit 108 may judge that an alarm is necessary (YES in S19) when the distance calculated in S18 is equal to or less than a predetermined threshold. If YES is judged in S19, the process proceeds to S20, and if NO is judged in S19, the process proceeds to S21. In S20, the notification control unit 108 issues an alarm that the distance between the suspended load and the surrounding objects is short. For example, the notification control unit 108 may instruct the alarm device 2 to issue an alarm.

[0090] In S21, the display control unit 109 causes the display device to display the objects detected in S13. Note that, among the objects detected in S13, the display control unit 109 causes those integrated in S17 to be displayed in an integrated state. Furthermore, the display control unit 109 may display various information such as the distance calculated in S18 in addition to information indicating the detected objects.

[0091] In S22, the data acquisition unit 101 judges whether or not to end the process. If it is judged as YES in S22, the process of FIG. 6 ends. On the other hand, if it is judged as NO in S22, the process returns to S11, and the data acquisition unit 101 acquires new point cloud data (measured at a later time than the previously acquired data). The end condition in S22 may be determined in advance. For example, the end condition may be that a user's end operation is detected or that a predetermined time has passed since the start of the process.

[0092] The timing of executing the process of S14, i.e., the detection of the wire CR7, is not limited to the above example. For example, the object detection unit 103 may detect the wire CR7 prior to S12. In this case, the object detection unit 103 may set a detection range for the wire CR7 directly below the generating device 3 and detect the wire CR7 within the detection range.

[0093] By detecting the wire CR7 before S12, the processes of S12 and S13 can be performed on points other than the points indicating the position of the wire CR7 among the points shown in the point cloud data acquired in S11. This makes it possible to facilitate the clustering process of S12 and the object detection process of S13 and to improve their accuracy.

[0094] The above-described process of Fig. 6 includes the integration method according to this embodiment. That is, the integration method according to this embodiment includes a projection step (S16) of projecting a point cloud corresponding to a first object detected from point cloud data including points indicating the positions of the surfaces of a plurality of objects to the depth of a second object detected at a position deeper than the first object, and an integration step (S17) of integrating the projected point cloud corresponding to the first object with the point cloud corresponding to the second object to make it a part of the second object. Therefore, it becomes possible to correctly detect an object by compensating for the part that is not detected due to the shadow of another object.

[0095] [Modifications] A part of the above-mentioned processing performed by the information processing device 1 may be configured to be performed by another computer system. For example, a part of the processing of S11 to S22 in FIG. 6 may be configured to be performed by another computer system. In this case, the information processing device 1 transmits information required for the calculation to the other computer system by communication via a network. Then, the information processing device 1 receives the calculation result transmitted from the other computer system. In this way, the integration method according to this embodiment can be executed by one or more information processing devices.

[0096] [Reference example 1] An information processing device according to a reference example of the present invention includes a data acquisition unit that acquires point cloud data including points indicating the positions of wires that suspend a load, and a detection unit that detects, when the point cloud data is divided into three-dimensional grids, points included in each grid where the sum of the number of points included in multiple horizontally adjacent grids is equal to or less than a predetermined threshold, as points indicating the positions of the wires. Note that the information processing device may or may not include a function of projecting and integrating point cloud data (i.e., a projection unit 105 and an integration unit 106).

[0097] [Reference example 2] An information processing device according to a reference example of the present invention includes a detection unit that detects an object from point cloud data including points indicating the positions of the surface of the object, and a depth estimation unit that classifies each point included in the point cloud corresponding to the detected object according to its depth and estimates the depth of the top surface of the object based on the number of points included in each class. Note that the information processing device may or may not include a function of projecting and integrating point cloud data (i.e., a projection unit 105 and an integration unit 106).

[0098] [Software implementation example] The functions of the information processing device 1 can be realized by a program for causing a computer to function as the information processing device 1, and a program (integrated program) for causing a computer to function as each control block of the information processing device 1 (particularly each part included in the control unit 10).

[0099] In this case, the information processing device 1 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the above program. The control device and storage device execute the above program to realize each function described in the above embodiment.

[0100] The above program may be recorded in one or more computer-readable recording media, not temporarily. The recording media may or may not be included in the information processing device 1. In the latter case, the above program may be supplied to the information processing device 1 via any wired or wireless transmission medium.

[0101] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0102] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0103] 1. Information processing device 103 Object detection unit (detection unit) 104 Depth Estimation Unit 105 Projection section 106 Integration Department 2 Alarm device 3 Generator 100 Surveillance System

Claims

1. a projection unit that projects a point cloud corresponding to a first object detected from point cloud data including points indicating the positions of surfaces of a plurality of objects, onto a depth of a second object detected at a position deeper than the first object; and an integration unit that integrates the projected point cloud corresponding to the first object into a point cloud corresponding to the second object to form the point cloud as a part of the second object.

2. the first object is a hook of a crane; The information processing device according to claim 1 , wherein the second object is a load suspended by the hook.

3. 3. The information processing device according to claim 1, wherein the projection unit performs a process of projecting each point included in the point cloud onto a straight line connecting a reference point indicating a position in three-dimensional space of a generating device that generates the point cloud data and a point included in the point cloud corresponding to the first object.

4. 3. The information processing device according to claim 2, further comprising a detection unit that detects, when the point cloud data is divided into three-dimensional grids, points included in each grid where a sum of the number of points included in multiple horizontally adjacent grids is equal to or less than a predetermined threshold value, as points indicating the position of a wire suspending the load.

5. 3. The information processing device according to claim 1, further comprising a depth estimation unit that divides each point included in a point cloud corresponding to an object detected from the point cloud data according to its depth, and estimates the depth of the top surface of the object based on the number of points included in each division.

6. The information processing device according to claim 5 , wherein the depth estimation unit estimates the depth of the top surface of the object based on the depth of a point included in a shallowest section among the sections in which the number of points included is equal to or greater than a predetermined threshold.

7. The information processing device according to claim 5 , wherein the depth estimation unit estimates the depth of the top surface of the object based on the depth of a point included in a shallowest section among the sections in which a ratio of the number of points to the maximum number of points included in a section is equal to or greater than a predetermined threshold.

8. A generating device that generates point cloud data including at least points indicating the positions of a hook of a crane, a load suspended by the hook, and surfaces of objects surrounding the load; an information processing device that projects a point cloud corresponding to the hook detected from the point cloud data onto a depth of the suspended load, and integrates the projected point cloud corresponding to the hook into a point cloud corresponding to the suspended load to form a part of the suspended load; A crane monitoring system including the integrated load and an alarm device that issues an alarm depending on the distance between the load and objects around the load.

9. 1. An integration method executed by one or more information processing devices, comprising: a projection step of projecting a point cloud corresponding to a first object detected from point cloud data including points indicating positions of surfaces of a plurality of objects, to a depth of a second object detected at a position deeper than the first object; and integrating the projected point cloud corresponding to the first object with the point cloud corresponding to the second object to make the point cloud part of the second object.

10. 2. An integrated program for causing a computer to function as the information processing device according to claim 1, the integrated program causing a computer to function as the projection unit and the integrated unit.

Images