Volume measurement apparatus, system, method, and program

Abstract

To provide a volume measurement apparatus (100) or the like that can contribute to measuring a volume of a raw material heap at low cost and efficiently without affecting operation of a factory. The volume measurement apparatus (100) comprises: a point group conversion part (112) converting depth information from a depth sensor (201) that shoots a raw material yard into point group data; a raw material heap detection part (113) detecting a point group related to a raw material heap from the point group data, using repose angle information related to a repose angle of the raw material heap; and a raw material heap volume calculation part (114) that calculates a volume of the raw material heap non-occlusion part that can be shot from the depth sensor (201) in the raw material heap based on the point group related to the raw material heap; estimates a volume of the raw material heap occlusion part that cannot be shot from the depth sensor (201) in the raw material heap, using at least the point group related to the raw material heap; and calculates a volume of the raw material heap, which is a sum of the calculated volume of the raw material heap non-occlusion part and the estimated volume of the raw material heap occlusion part. Figure 15

Description

Volume measuring device, system, method, and program

[0001] [Description of Related Applications] The present invention is based on the priority claim of Japanese Patent Application No. 2018-218148 (filed November 21, 2018), the entire contents of which are incorporated herein by reference. The present invention relates to a volume measuring device, system, method, and program for measuring the volume of a raw material pile in a raw material yard.

[0002] In glass factories and steel mills, silica sand, which is the raw material for glass, and iron ore, which is the raw material for steel, are stored indoors or outdoors on the ground in raw material yards (raw material storage areas) that are surrounded by walls on three sides. FIG. 17 is an image of a raw material yard 10 surrounded by walls 12 on three sides, as seen from above. Each raw material pile 11, where raw materials are piled up, is surrounded on three sides by walls 12. The remaining amount of raw material changes from moment to moment due to factory operations. Currently, the remaining amount of raw material is often managed visually by workers.

[0003] To maintain high productivity, it is important to quickly and accurately grasp the remaining amount of raw materials. In other words, a system that can automatically manage remaining amounts in real time is required. When building a raw material remaining amount management system, it is possible to use a depth sensor (depth measurement sensor) as a sensor for measuring the remaining amount. Patent Documents 1 to 3 describe technologies for managing the shape of raw material piles using depth sensors. The technologies described in Patent Documents 1 to 3 measure the shape of raw material piles by attaching depth sensors such as laser scanners, 2D (2-Dimensions) laser range finders, and stereo cameras to yard machines such as stackers and reclaimers. Systems using such depth sensors generally use multiple depth sensors to acquire three-dimensional data by photographing the object to be measured from multiple viewpoints and perform measurements.

[0004] JP 2010-286436 A JP 2011-157187 A JP 2012-193030 A JP 2016-61674 A

[0005] C. BRADFORD BARBER et al., "The Quickhull Algorithm for Convex Hulls", ACM Transactions on Mathematical Software, Vol. 22, No. 4, December 1996, Pages 469-483. (https: / / www.cise.ufl.edu / ~ungor / courses / fall06 / papers / QuickHull.pdf)

[0006] The following analysis is provided by the present inventors.

[0007] In the techniques described in Patent Documents 1 to 3, a depth sensor is attached to a yard machine, so that while the shape of the raw material pile is being measured, work such as receiving and unloading raw materials cannot be performed. Therefore, attaching a depth sensor to a yard machine for measurement may affect factory operations.

[0008] Furthermore, although depth sensors have become increasingly affordable in recent years, those that offer sufficient measurement range, accuracy, and fineness for industrial applications remain expensive. From the perspective of maintainability, using multiple depth sensors is undesirable. If a system were to be constructed using a single depth sensor, the problem of occlusion would arise, whereby an object in the foreground obscures an object behind it in three-dimensional space. For example, if a pile of raw materials is photographed from the front, the area behind the pile will be hidden, making it impossible to obtain depth information for that area.

[0009] One possible solution to this occlusion problem is to use an unmanned aerial vehicle (drone) equipped with a depth sensor to photograph the pile of raw materials while moving from above, as in the technology described in Patent Document 4. However, because building-type raw material yards have ceilings, it is difficult for the drone to fly at a sufficient distance from the pile of raw materials, and the wind from the drone's propellers may blow up the raw materials, making it impossible to measure them at all.

[0010] Another possible solution to the occlusion problem is to have workers walk around the pile of raw materials using a single depth sensor. However, if the raw material yard is large and has multiple units, not only is measurement time and laborious, but it also requires measurements to be taken while avoiding yard machinery that is currently in operation, which may make measurements inefficient.

[0011] The main object of the present invention is to provide a volume measuring device, system, method, and program that can contribute to measuring the volume of raw material piles at low cost and efficiently without affecting factory operations.

[0012] A volume measuring device according to a first aspect includes a point cloud conversion unit that converts depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; a raw material pile detection unit that detects a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile; and a raw material pile volume calculation unit that calculates the volume of a raw material pile non-occlusion portion that can be photographed by the depth sensor based on the point cloud relating to the raw material pile, estimates the volume of a raw material pile occlusion portion that cannot be photographed by the depth sensor using at least the point cloud relating to the raw material pile, and calculates the volume of the raw material pile by adding up the volume of the calculated raw material pile non-occlusion portion and the volume of the estimated raw material pile occlusion portion.

[0013] The volume measurement system according to the second viewpoint includes a depth sensor that photographs a raw material yard having a pile of raw materials, and the volume measurement device according to the first viewpoint.

[0014] A volume measurement method according to a third aspect includes the steps of converting depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile; calculating a volume of a raw material pile non-occlusion portion in the raw material pile that can be photographed by the depth sensor based on the point cloud relating to the raw material pile; estimating a volume of a raw material pile occlusion portion in the raw material pile that cannot be photographed by the depth sensor using at least the point cloud relating to the raw material pile; and calculating a volume of the raw material pile by adding together the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

[0015] A program according to a fourth aspect causes hardware resources to execute the following processes: converting depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile; calculating a volume of a raw material pile non-occlusion portion in the raw material pile that can be photographed from the depth sensor based on the point cloud relating to the raw material pile; estimating a volume of a raw material pile occlusion portion in the raw material pile that cannot be photographed from the depth sensor using at least the point cloud relating to the raw material pile; and calculating a volume of the raw material pile that is the sum of the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

[0016] The above-described program can be recorded on a computer-readable storage medium. The storage medium storing the program can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present disclosure can also be embodied as a computer program product. The program is input to a computer device from an input device or an external device via a communication interface, stored in a storage device, and drives a processor according to predetermined steps or processes. The processing results, including intermediate states as necessary, can be displayed at each stage on a display device, or the computer device can communicate with the outside via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and, if necessary, a display device, all of which are connectable to each other via a bus.

[0017] The first to fourth aspects can contribute to measuring the volume of a raw material pile efficiently at low cost without affecting the operation of the factory.

[0018] 1 is a block diagram schematically showing the configuration of a volume measurement system according to embodiment 1. FIG. 2 is an image diagram schematically showing an example of a state in which a photographing device in the volume measurement system according to embodiment 1 is placed in a raw material yard. FIG. 3 is a flowchart diagram schematically showing the operation of an information processing unit of a volume measurement device in the volume measurement system according to embodiment 1. FIG. 4 is an image diagram schematically showing an example of point cloud data relating to a raw material yard viewed from the photographing position of a photographing device in the volume measurement system according to embodiment 1. FIG. 5 is an image diagram schematically showing another example of point cloud data relating to a raw material yard viewed from the photographing position of a photographing device in the volume measurement system according to embodiment 1. FIG. 6 is an image diagram schematically showing an example of point cloud data in a state in which an obstacle is present in a raw material yard viewed from the photographing position of a photographing device in the volume measurement system according to embodiment 1. FIG. 7 is an image diagram schematically showing an example of a point cloud in a state in which the obstacle has been removed from the raw material yard viewed from the photographing position of a photographing device in the volume measurement system according to embodiment 1. FIG. 8 is an image diagram schematically showing an example of a side view of the raw material yard being photographed by a photographing device in the volume measurement system according to embodiment 1. 1 is an image diagram schematically showing an example of a state in which a convex hull created by the information processing unit of the volume measurement device in the volume measurement system according to embodiment 1 is viewed from the side. FIG. 2 is an image diagram schematically showing an example of a tetrahedron extracted from a convex hull created by the information processing unit of the volume measurement device in the volume measurement system according to embodiment 1. FIG. 3 is an image diagram schematically showing another example of a state in which a raw material yard photographed by the photographing device in the volume measurement system according to embodiment 1 is viewed from the side. FIG. 4 is an image diagram schematically showing another example of a state in which a photographing device in the volume measurement system according to embodiment 1 is placed in a raw material yard. FIG. 5 is a block diagram schematically showing the configuration of a volume measurement system according to embodiment 2. FIG. 6 is a block diagram schematically showing the configuration of a volume measurement device according to embodiment 3. FIG. 7 is a flowchart diagram schematically showing the operation of the volume measurement device according to embodiment 3. FIG. 8 is an image diagram of an example of a raw material yard viewed from above. FIG. 9 is a block diagram schematically showing the configuration of hardware resources.

[0019] In the present disclosure described below, the volume measuring device according to Mode 1 and its deformation modes can be appropriately selected and combined.

[0020] The volume measuring device according to Mode 1 may include a point cloud conversion unit that converts depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard. The volume measuring device may include a raw material pile detection unit that detects a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile. The volume measuring device may include a raw material pile volume calculation unit that calculates a volume of a raw material pile non-occlusion portion in the raw material pile that can be photographed by the depth sensor based on the point cloud relating to the raw material pile, estimates a volume of a raw material pile occlusion portion in the raw material pile that cannot be photographed by the depth sensor using at least the point cloud relating to the raw material pile, and calculates a volume of the raw material pile that is the sum of the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

[0021] In a modified mode of the volume measuring device according to Mode 1, the raw material pile detection unit, when detecting a point cloud relating to the raw material pile, can detect a point cloud relating to the raw material pile where the angle formed between a plane on the slope of the raw material pile and the ground in the raw material yard satisfies the angle of repose in the angle of repose information. The volume measuring device can further include an acquisition unit that acquires the depth information relating to the raw material yard from the depth sensor. The point cloud conversion unit, when converting into point cloud data relating to the raw material yard, can convert the depth information relating to the raw material yard acquired by the acquisition unit into point cloud data relating to the raw material yard. The volume measuring device can further include an angle of repose storage unit that stores angle of repose information for rainy weather and angle of repose information for non-rainy weather. Furthermore, when detecting the point cloud related to the raw material pile, the raw material pile detection unit can check whether a rain sensor that detects rainfall in the raw material yard is detecting rainfall, read the angle of repose information for rainy weather from the angle of repose storage unit when the rain sensor detects rainfall, and read the angle of repose information for non-rainy weather from the angle of repose storage unit when the rain sensor does not detect rainfall, and detect the point cloud related to the raw material pile from the point cloud data related to the raw material yard using the read-out angle of repose information for rainy weather or the angle of repose information for non-rainy weather. Furthermore, when detecting the point cloud related to the raw material pile, the raw material pile detection unit can measure angle of repose information using image data from a camera, and detect the point cloud related to the raw material pile from the point cloud data related to the raw material yard using the measured angle of repose information. In addition, when measuring the angle of repose information, the raw material pile detection unit creates a model that learns the surface texture of the raw material pile from the image data using deep learning, detects the coordinates of the raw material pile using the created model, and measures the angle of repose information from the depth information related to the raw material yard at the coordinates of the detected raw material pile.The raw material pile detection unit can further check whether the point cloud data related to the raw material yard includes a point cloud related to an obstacle between the raw material pile and the depth sensor, and if the point cloud related to the obstacle is included, remove the point cloud related to the obstacle from the point cloud related to the detected raw material pile and interpolate a point cloud related to a missing portion of the point cloud related to the raw material pile resulting from the removal of the point cloud related to the obstacle. When calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit can calculate the volume of the raw material pile non-occlusion portion based on the point cloud related to the raw material pile obtained by interpolating the point cloud related to the missing portion by the raw material pile detection unit. When interpolating the point cloud related to the missing portion, the raw material pile detection unit can interpolate the point cloud related to the missing portion using an image interpolation method. Furthermore, when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit can calculate the volume of the raw material pile non-occlusion portion based on the point cloud related to the raw material pile detected by the raw material pile detection unit when there is no point cloud related to the obstacle detected by the raw material pile detection unit. Furthermore, when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit can create a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the raw material pile, calculate the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and set the calculated volume of the entire convex hull as the volume of the raw material pile non-occlusion portion. Furthermore, the volume measuring device can further include a wall position storage unit that stores wall position information related to the positions of walls in the raw material yard. Furthermore, when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit uses the point cloud related to the raw material pile to check whether or not there is a wall in the raw material yard, and if there is a wall, reads out the wall position information from the wall position memory unit and estimates the volume of the raw material pile occlusion portion using the point cloud related to the raw material pile and the wall position information.Furthermore, when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit can cut out a portion corresponding to the distance between the vertex of the raw material pile and the wall from the point cloud related to the raw material pile, create a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the cut-out portion, calculate the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and set the calculated volume of the entire convex hull as the volume of the raw material pile occlusion portion. Furthermore, when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit can estimate the volume of the raw material pile occlusion portion using the point cloud related to the raw material pile when there is no wall.

[0022] In the present disclosure, a volume measurement system according to mode 2 can include a depth sensor that photographs a raw material yard having a pile of raw materials, and a volume measurement device according to mode 1.

[0023] In the present disclosure, a volume measurement method according to Mode 3 can include converting depth information relating to a raw material yard, obtained from a depth sensor that photographs the raw material yard having a raw material pile, into point cloud data relating to the raw material yard. The volume measurement method can also include detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile. The volume measurement method can also include calculating a volume of a raw material pile non-occlusion portion of the raw material pile that can be photographed by the depth sensor based on the point cloud relating to the raw material pile, estimating a volume of a raw material pile occlusion portion of the raw material pile that cannot be photographed by the depth sensor using at least the point cloud relating to the raw material pile, and calculating a volume of the raw material pile by summing the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

[0024] In the present disclosure, a program according to Mode 4 can cause a hardware resource to execute a process of converting depth information relating to a raw material yard having a raw material pile, obtained from a depth sensor that photographs the raw material yard, into point cloud data relating to the raw material yard. The program can also cause a hardware resource to execute a process of detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using angle of repose information relating to the angle of repose of the raw material pile. The program can also cause a hardware resource to execute a process of calculating a volume of a raw material pile non-occlusion portion in the raw material pile that can be photographed by the depth sensor based on the point cloud relating to the raw material pile, estimating a volume of a raw material pile occlusion portion in the raw material pile that cannot be photographed by the depth sensor using at least the point cloud relating to the raw material pile, and calculating a volume of the raw material pile that is the sum of the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

[0025] The following description of the embodiments will be made with reference to the drawings. Note that, where reference numerals are used in this application, they are intended solely to facilitate understanding and are not intended to limit the scope of the present invention to the illustrated embodiments. Furthermore, the following embodiments are merely illustrative and do not limit the scope of the present invention. Furthermore, connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown, input and output ports exist at the input and output ends of each connection line in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, and the like shown in this disclosure. The same applies to input / output interfaces. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. The computer device is configured to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless.

[0026] [Embodiment 1] A volume measurement system according to embodiment 1 will be described with reference to the drawings. Fig. 1 is a block diagram that schematically shows the configuration of the volume measurement system according to embodiment 1. Fig. 2 is an image diagram that schematically shows an example of a state in which an imaging device in the volume measurement system according to embodiment 1 is arranged in a raw material yard.

[0027] The volume measurement system 1 is a system for measuring the volume of a raw material pile (11 in FIG. 2) in a raw material yard (10 in FIG. 2) (see FIG. 1). The volume measurement system 1 includes a volume measurement device 100 and a photographing device 200.

[0028] 2 shows two raw material yards 10, but the number is not limited to this. In each raw material yard 10, a raw material pile 11 is placed on the ground 14, and three sides of the raw material pile 11 are surrounded by walls 12 that are in contact with the ground 14. The shape of the raw material pile 11 does not need to be an ideal cone, and the raw material pile 11 may have irregularities.

[0029] The volume measuring device 100 is a device that automatically measures (manages) the volume of a raw material pile (11 in FIG. 2) in a raw material yard (10 in FIG. 2) using a photographing device 200 that photographs the raw material yard (10 in FIG. 2) (see FIG. 1). The volume measuring device 100 can use hardware resources (e.g., an information processing device, a computer) including, for example, a processor, a memory, a network interface, etc. The volume measuring device 100 has an information processing unit 110, a storage unit 120, an input unit 130, an output unit 140, and a communication unit 150.

[0030] The information processing unit 110 is a functional unit that processes information (see FIG. 1 ). The information processing unit 110 is communicatively connected to the storage unit 120, the input unit 130, the output unit 140, and the communication unit 150. The information processing unit 110 includes an acquisition unit 111, a point cloud conversion unit 112, a raw material pile detection unit 113, and a raw material pile volume calculation unit 114. The information processing unit 110 may be implemented using a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processor Unit). In this case, the information processing unit 110 may virtually configure the acquisition unit 111, point cloud conversion unit 112, raw material pile detection unit 113, and raw material pile volume calculation unit 114 by executing a program while utilizing the storage unit 120 that stores the program.

[0031] The acquisition unit 111 is a processing unit that acquires (collects) depth information (depth information related to the raw material yard (10 in Figure 2); three-dimensional data) generated by the depth sensor 201 of the imaging device 200 (see Figure 1).

[0032] The point cloud conversion unit 112 is a processing unit that converts the depth information (depth information related to the raw material yard (10 in Figure 2)) acquired by the acquisition unit 111 into point cloud data (point cloud data related to the raw material yard (10 in Figure 2)) (see Figure 1).

[0033] The raw material pile detection unit 113 is a processing unit that detects (extracts) a point cloud related to the raw material pile (11 in Figure 2) from the point cloud data (point cloud data related to the raw material yard (10 in Figure 2)) converted by the point cloud conversion unit 112 (see Figure 1).

[0034] The raw material pile detection unit 113 performs a process of checking whether or not rain has been detected by the rain sensor 205 of the image capture device 200. If the raw material yard (10 in FIG. 2) is indoors and is not affected by rain, the raw material pile detection unit 113 can omit the process of checking whether rain has been detected.

[0035] The raw material pile detection unit 113 performs a process of reading out angle of repose information from the angle of repose storage unit 121. When the rain sensor 205 does not detect rainfall, the raw material pile detection unit 113 reads out angle of repose information for non-rainy weather from the angle of repose storage unit 121. When the rain sensor 205 detects rainfall, the raw material pile detection unit 113 reads out angle of repose information for rainy weather from the angle of repose storage unit 121. Note that when the raw material yard (10 in FIG. 2) is indoors and the process of checking for rainfall detection is omitted, the raw material pile detection unit 113 reads out angle of repose information for non-rainy weather from the angle of repose storage unit 121.

[0036] The raw material pile detection unit 113 uses the angle of repose information (angle of repose information for non-rainy weather or angle of repose information for rainy weather) read from the angle of repose memory unit 121 to perform processing to detect a point cloud related to the raw material pile 11 from the point cloud data converted by the point cloud conversion unit 112 (point cloud data related to the raw material yard (10 in Figure 2)).

[0037] The raw material pile detection unit 113 performs processing to check whether or not a point cloud relating to an obstacle (20 in FIG. 6; for example, a pillar) exists between the raw material pile (11 in FIG. 2) and the photographing device 200 in the point cloud data (point cloud data relating to the raw material yard (10 in FIG. 2)) converted by the point cloud conversion unit 112. Note that if there is no obstacle (20 in FIG. 6) between the raw material pile (11 in FIG. 2) and the photographing device 200, the raw material pile detection unit 113 can omit the processing to check for the existence of a point cloud relating to the obstacle (20 in FIG. 6).

[0038] If a point cloud relating to an obstacle (20 in FIG. 6) is present, the raw material pile detection unit 113 removes the point cloud relating to the obstacle (20 in FIG. 6) from the point cloud relating to the detected raw material pile 11, and interpolates the point cloud of the missing portion (17 in FIG. 7) in the point cloud relating to the raw material pile 11 created by removing the point cloud relating to the obstacle (20 in FIG. 6). If a point cloud relating to an obstacle (20 in FIG. 6) is not present, the raw material pile detection unit 113 does not perform the removal process or the interpolation process. "Interpolation" refers to calculating the value of an unmeasured portion based on the value of the portion obtained by measurement.

[0039] The raw material pile volume calculation unit 114 is a processing unit that calculates the volume (remaining amount of raw material) of the raw material pile (11 in FIG. 2) (see FIG. 1).

[0040] The raw material pile volume calculation unit 114 performs processing to calculate the volume of the raw material pile non-occlusion portion (11a in FIG. 8) in the raw material pile (11 in FIG. 2) based on the point cloud related to the raw material pile (the detected point cloud or the interpolated point cloud). Here, the raw material pile non-occlusion portion 11a is the portion of the raw material pile 11 that is not occluded (the portion that can be photographed (visible) by the depth sensor 201).

[0041] The raw material pile volume calculation unit 114 performs a process of checking whether or not a wall exists using the point cloud related to the raw material pile (the detected point cloud or the interpolated point cloud). If a wall exists, the raw material pile volume calculation unit 114 performs a process of reading out wall position information from the wall position storage unit 122. If a wall does not exist, the raw material pile volume calculation unit 114 does not read out wall position information from the wall position storage unit 122. Note that if a wall does not exist to begin with, the raw material pile volume calculation unit 114 can omit the process of checking whether a wall exists and the process of reading out wall position information.

[0042] The raw material pile volume calculation unit 114 performs processing to estimate the volume of the raw material pile occlusion portion (11b in FIG. 8) in the raw material pile (11 in FIG. 2) using at least the point cloud related to the raw material pile (the detected point cloud or the interpolated point cloud). If a wall exists, the raw material pile volume calculation unit 114 also uses the read-out wall position information to estimate the volume of the raw material pile occlusion portion (11b in FIG. 8) in the raw material pile (11 in FIG. 2).

[0043] The raw material pile volume calculation unit 114 performs processing to calculate the remaining amount (volume) of the raw material pile (11 in FIG. 2) by adding up the volume of the calculated raw material pile non-occlusion portion (11a in FIG. 8) and the volume of the estimated raw material pile occlusion portion (11b in FIG. 8).

[0044] The storage unit 120 is a functional unit that stores information such as data, programs, etc. (See FIG. 1.) The storage unit 120 includes a repose angle storage unit 121 and a wall position storage unit 122.

[0045] The angle of repose memory unit 121 stores information related to the angle of repose (angle of repose information) of the raw material pile (11 in Figure 2). The angle of repose information includes angle of repose information for non-rainy weather and angle of repose information for rainy weather. The angle of repose is the angle between the horizontal plane and the slope of the pile of particles formed when particles are dropped from a certain height and remain stable without collapsing spontaneously. The angle of repose can be obtained by measuring the angle between the slope of the pile of raw material (11 in Figure 2) piled on a flat surface and the horizontal plane. The angle of repose is determined by the size of the raw material particles, the roundness and shape of the particle corners, etc., but is also significantly affected by the moisture content. Therefore, angle of repose information for non-rainy weather and angle of repose information for rainy weather is provided. The angle of repose information can be input to the angle of repose memory unit 121 via the input unit 130. If the raw material yard (10 in FIG. 2) is indoors and is not affected by rain, the angle of repose information for rainy weather can be omitted.

[0046] The wall position storage unit 122 stores information (wall position information) relating to the position of a wall (12 in FIG. 2) in the raw material yard (10 in FIG. 2). The wall position information may be obtained from design information of the raw material yard (10 in FIG. 2), or may be obtained by measurement using a measuring device such as a laser rangefinder. The wall position information can be input to the wall position storage unit 122 from the input unit 130. Note that if a wall (12 in FIG. 2) does not originally exist in the raw material yard (10 in FIG. 2), the wall position storage unit 122 can be omitted.

[0047] The input unit 130 is a functional unit that inputs information through an operator's operation (see FIG. 1 ). The input unit 130 may be, for example, a keyboard, a mouse, a touch panel, a microphone, a button, or other input means, and may also be an information terminal or the like having an input unit that is communicably connected via a communication unit (not shown).

[0048] The output unit 140 is a functional unit that displays the measured volume of the raw material pile, etc. (see FIG. 1). The output unit 140 can be, for example, a display for displaying, a printer for printing, or other output means, and may also be an information terminal or the like having an output unit communicably connected via a communication unit (not shown). Furthermore, instead of using the output unit 140, information may be output using another information terminal or the like (not shown; for example, a terminal used by the user) connected via the communication unit and a network.

[0049] The communication unit 150 is a functional unit that is connected to enable communication (wireless communication or wired communication) with the communication unit 203 of the imaging device 200. The communication unit 150 may be connected to enable communication with the communication unit 203 of the imaging device 200 via a network (not shown).

[0050] The imaging device 200 is a device that captures an image of a subject (see FIG. 1). The imaging device 200 is installed so as to be able to overlook the entirety of at least one raw material yard (10 in FIG. 2). The imaging device 200 may be installed so as to capture an image of the raw material yard (10 in FIG. 2) from above. As shown in FIG. 2, a imaging device 200 may be installed for each raw material yard 10. The imaging device 200 includes a depth sensor 201, a sensor control unit 202, a communication unit 203, a battery 204, and a rain sensor 205.

[0051] The depth sensor 201 is a sensor that captures an image of a subject and generates depth information, which is three-dimensional data (see FIG. 1 ). The depth sensor 201 may be, for example, a laser scanner, a 2D laser rangefinder, a stereo camera, a ToF (Time of Flight) sensor, or a LiDAR (Light Detection and Ranging) sensor. It is preferable that the imaging device 200 includes one depth sensor 201, but two or more depth sensors 201 may also be used.

[0052] The sensor control unit 202 is a functional unit that controls the depth sensor 201 (see FIG. 1).

[0053] The communication unit 203 is a functional unit that is connected to the communication unit 150 of the volume measuring device 100 so as to be able to communicate (wireless communication, wired communication) with the communication unit 150 (see FIG. 1 ). The communication unit 203 may be connected to the communication unit 150 of the volume measuring device 100 so as to be able to communicate with the communication unit 150 via a network (not shown).

[0054] The battery 204 is a driving power source for the image capturing device 200 (see FIG. 1). Instead of the battery 204, an external power source (not shown) may be used.

[0055] The rain sensor 205 is a sensor that detects rainfall (see FIG. 1). Instead of the rain sensor 205, a rainfall confirmation unit (not shown) that acquires weather information via the Internet and confirms whether or not it is raining may be used. Alternatively, without using the rain sensor 205, an operator may confirm whether or not it is raining and operate the input unit 130 of the volume measuring device 100 to select angle of repose information (angle of repose information for non-rainy weather or angle of repose information for rainy weather) in the angle of repose storage unit 121.

[0056] Next, the operation of the information processing unit of the volume measuring device in the volume measurement system according to embodiment 1 will be described with reference to the drawings. FIG. 3 is a flowchart diagram schematically illustrating the operation of the information processing unit of the volume measuring device in the volume measurement system according to embodiment 1. FIG. 4 is an image diagram schematically illustrating an example of point cloud data relating to the raw material yard as viewed from the photographing position of the photographing device in the volume measurement system according to embodiment 1. FIG. 5 is an image diagram schematically illustrating another example of point cloud data relating to the raw material yard as viewed from the photographing position of the photographing device in the volume measurement system according to embodiment 1. FIG. 6 is an image diagram schematically illustrating an example of point cloud data in a state where an obstacle is present in the raw material yard as viewed from the photographing position of the photographing device in the volume measurement system according to embodiment 1. FIG. 7 is an image diagram schematically illustrating an example of a point cloud relating to a state where the obstacle has been removed from the raw material yard as viewed from the photographing position of the photographing device in the volume measurement system according to embodiment 1. FIG. 8 is an image diagram schematically illustrating an example of a state where the raw material yard photographed by the photographing device in the volume measurement system according to embodiment 1 is viewed from the side. FIG. 9 is an image diagram schematically illustrating an example of a side view of a convex hull created by the information processing unit of the volume measurement device in the volume measurement system according to embodiment 1. FIG. 10 is an image diagram schematically illustrating an example of a tetrahedron extracted from a convex hull created by the information processing unit of the volume measurement device in the volume measurement system according to embodiment 1. FIG. 11 is an image diagram schematically illustrating another example of a side view of a raw material yard photographed by a photographing device in the volume measurement system according to embodiment 1. FIG. 12 is an image diagram illustrating a cropped non-occluded portion of the raw material pile in another example of a side view of a raw material yard photographed by a photographing device in the volume measurement system according to embodiment 1. FIG. 13 is an image diagram schematically illustrating another example of a state in which a photographing device in the volume measurement system according to embodiment 1 is placed in the raw material yard. For the configuration of the volume measurement system, please refer to FIG. 1.

[0057] It is assumed that information (wall position information) relating to the position of the wall 12 in the raw material yard (10 in FIG. 2) is stored in advance in the wall position storage unit 122. It is also assumed that information (angle of repose information) relating to the angle of repose of the raw material pile 11 is stored in advance in the angle of repose storage unit 121.

[0058] First, the acquisition unit 111 of the information processing unit 110 of the volume measuring device 100 acquires depth information relating to the raw material yard (10 in FIG. 2) generated by the depth sensor 201 of the photographing device 200 (see step A1 in FIG. 3).

[0059] Next, the point cloud conversion unit 112 of the information processing unit 110 of the volume measuring device 100 converts the depth information acquired by the acquisition unit 111 into point cloud data (point cloud data relating to the raw material yard (10 in Figure 2)) (see step A2 in Figure 3).

[0060] Next, the raw material pile detection unit 113 of the information processing unit 110 of the volume measuring device 100 checks whether the rain sensor 205 detects rainfall (see step A3 in FIG. 3).

[0061] If rainfall is not detected (NO in step A3), the raw material pile detection unit 113 reads out the angle of repose information for non-rainy weather from the angle of repose storage unit 121 (see step A4 in FIG. 3).

[0062] If rainfall is detected (YES in step A3), the raw material pile detection unit 113 reads out the angle of repose information for rainy weather from the angle of repose storage unit 121 (see step A5 in FIG. 3).

[0063] After step A4 or step A5, the raw material pile detection unit 113 uses the angle of repose information (angle of repose information for non-rainy weather or angle of repose information for rainy weather) read from the angle of repose memory unit 121 to detect (extract) a point cloud related to the raw material pile (11 in Figure 4) from the point cloud data related to the raw material yard (10 in Figure 2) converted by the point cloud conversion unit 112 (see step A6 in Figure 3).

[0064] Next, the raw material pile detection unit 113 checks whether or not there is a point cloud relating to an obstacle (20 in FIG. 6; for example, a pillar) existing between the raw material pile (11 in FIG. 2) and the photographing device (200 in FIG. 2) in the point cloud data relating to the raw material yard (10 in FIG. 2) converted by the point cloud conversion unit 112 (see step A7 in FIG. 3). If there is no point cloud relating to an obstacle (20 in FIG. 6) (NO in step A7), the process proceeds to step A9.

[0065] If a point cloud relating to an obstacle (20 in FIG. 6) exists (YES in step A7), the raw material pile detection unit 113 removes the point cloud relating to the obstacle (20 in FIG. 6) from the point cloud relating to the raw material pile (11 in FIG. 6) detected (extracted) in step A6, and interpolates the point cloud relating to the missing portion (17 in FIG. 7) in the point cloud relating to the raw material pile (11 in FIG. 6) created by removing the point cloud relating to the obstacle (20 in FIG. 6) (see step A8 in FIG. 3).

[0066] If there is no point cloud relating to the obstacle (20 in FIG. 6) (NO in step A7), or after step A8, the raw material pile volume calculation unit 114 of the information processing unit 110 of the volume measuring device 100 calculates the volume of the raw material pile non-occlusion portion (11a in FIG. 8) based on the point cloud relating to the raw material pile (11 in FIG. 8) (the point cloud detected (extracted) in step A6 or the point cloud interpolated in step A8) (see step A9 in FIG. 3).

[0067] Next, the raw material pile volume calculation unit 114 checks whether or not there is a wall (12 in FIG. 2) in the raw material yard (10 in FIG. 2) using the point cloud (the point cloud detected in step A6 or the point cloud interpolated in step A8) related to the raw material pile (11 in FIG. 2) (see step A10 in FIG. 3). If there is no wall (12 in FIG. 2) (NO in step A10), the process proceeds to step A12.

[0068] If a wall (12 in FIG. 2) exists (YES in step A10), the raw material pile volume calculation unit 114 reads out wall position information from the wall position storage unit 122 (see step A11 in FIG. 3).

[0069] If a wall (12 in FIG. 2) does not exist (NO in step A10), or after step A11, the raw material pile volume calculation unit 114 estimates the volume of the raw material pile occlusion portion (11b in FIG. 8) using the point cloud (detected point cloud or interpolated point cloud) related to the raw material pile (11 in FIG. 2) (see step A12 in FIG. 3). Here, if the wall position information has been read out in step A11, the raw material pile volume calculation unit 114 also uses the wall position information to estimate the volume of the raw material pile occlusion portion (11b in FIG. 8).

[0070] Next, the raw material pile volume calculation unit 114 calculates the volume (remaining amount of raw material) of the raw material pile (11 in FIG. 2) by adding the volume of the raw material pile non-occlusion portion (11a in FIG. 8) calculated in step A9 and the volume of the raw material pile occlusion portion (11b in FIG. 8) estimated in step A12 (see step A13 in FIG. 3).

[0071] Finally, the output unit 140 outputs (displays) the volume (remaining amount of raw material) of the raw material pile (11 in FIG. 2) calculated in step A13 (see step A14 in FIG. 3), and then ends the process. After the end, when a predetermined time has elapsed, the process may return to the start and repeat steps A1 to A13.

[0072] Here, a method for detecting the point group relating to the raw material pile (11 in FIG. 4) in step A6 will be described.

[0073] Referring to FIG. 4, the equation of a plane 16 on the slope of the raw material pile 11 is expressed as [Equation 1], and a normal vector n ｓ The equation of the ground 14 in the raw material yard 10 is expressed as [Equation 3], and the normal vector n ｇ The angle θ between the plane 16 and the ground 14 is expressed as the normal vector n ｓ , n ｇ Using this, it can be expressed as in [Equation 5].

[0074] [Formula 1]

[0075] [Formula 2]

[0076] [Formula 3]

[0077] [Formula 4]

[0078] [Formula 5]

[0079] The point cloud for which the angle θ between the plane 16 and the ground 14 satisfies the angle of repose is detected as the point cloud relating to the raw material pile 11. As shown in Figure 5, the point cloud relating to the raw material pile 11 can be detected in the same way even if it has a complex shape.

[0080] Next, the removal of points related to obstacles (20 in FIG. 6) and the interpolation of points related to missing portions (17 in FIG. 7) in step A8 will be described.

[0081] If a columnar (rectangular, cylindrical, etc.) obstacle 20 as shown in FIG. 6 exists between the imaging device (200 in FIG. 2) and the pile of raw material (11 in FIG. 2), removing the point cloud related to the obstacle 20 from the point cloud related to the pile of raw material (11 in FIG. 6) will result in a point cloud related to the pile of raw material 11 having a missing portion 17, as shown in FIG. 7. The missing portion 17 is interpolated based on the point cloud related to the pile of raw material 11 that has already been acquired. Examples of interpolation methods include image interpolation methods such as linear interpolation, nearest neighbor interpolation, and concentration storage method.

[0082] Next, a method for calculating the volume of the non-occlusion portion (11a in FIG. 8) of the raw material pile (11 in FIG. 8) in step A9 will be described.

[0083] Referring to Fig. 8, the volume of the non-occlusion portion 11a of the raw material pile can be found by creating a convex hull based on the point cloud related to the raw material pile (11 in Fig. 8) detected (extracted) in step A6 or interpolated in step A8. The convex hull can be created, for example, by an algorithm using the Quickhull method described in Non-Patent Document 1. The convex hull created by the algorithm using the Quickhull method is composed of multiple tetrahedrons as shown in Fig. 9. As shown in Fig. 10, if the base area of ​​the tetrahedron is S and the height is h, the volume V of the tetrahedron is ｉ can be calculated from the formula for the volume of a triangular pyramid [Equation 6]. The volume of the entire convex hull can be calculated by calculating the sum of the volumes of the multiple tetrahedrons, and the calculated volume of the entire convex hull can be used as the volume of the raw material pile non-occlusion portion 11a.

[0084] [Formula 6]

[0085] Next, a method for estimating the volume of the raw material pile occlusion portion (11b in FIG. 8) of the raw material pile (11 in FIG. 8) in step A8 will be described.

[0086] Referring to FIG. 8, when the vertices of the raw material pile 11 viewed from the side are designated as A, B, and C, the distance d between A and C in the z-axis direction is ａｃ and the distance d between B and C in the z-axis direction ｂｃ When the relationship satisfies [Equation 7], the volume V of the raw material pile occlusion portion 11b ｂ is the volume V of the non-occlusion portion 11a of the raw material pile ａ Then, it can be expressed as in [Equation 8].

[0087] [Formula 7]

[0088] [Formula 8]

[0089] Also, as shown in FIG. ａｃ and d ｂｃ When the relationship between the two is expressed as [Equation 9], as shown in FIG. 12, the part corresponding to the occlusion part 11b of the raw material pile is cropped from the point cloud related to the raw material pile 11 (corresponding to the point cloud related to the non-occlusion part 11a of the raw material pile) (from the vertex C of the raw material pile 11 to ｂｃ The volume V of the raw material pile occlusion portion 11b is calculated by cutting out the raw material pile occlusion portion 11b by the distance to the wall (corresponding to the distance between the vertex C and the wall), creating a convex hull of the point cloud related to the cropped portion, and calculating the sum of the volumes of the tetrahedrons that make up the created convex hull. ｂ The volume calculation using the convex hull here is the same as the volume calculation method for the raw material pile non-occlusion portion 11a in step A9.

[0090] [Formula 9]

[0091] 13, when the approximate stacking area 15 is determined, the raw material pile 11 in the raw material yard 10 does not need to be surrounded by walls on three sides. In this case, the volume of the raw material pile 11 can be measured in the same manner as described above by assuming that there is a wall around the periphery of the stacking area 15.

[0092] The volume measurement system described above can be used in the management of raw materials and products in the smart factory field, management of coal fallout in mining, management of raw materials in the food manufacturing industry, management of chips in the paper industry, and management of waste in the waste disposal industry.

[0093] According to the first embodiment, by calculating the volume of the non-occluded portion 11a of the raw material pile and estimating the volume of the occluded portion 11b of the raw material pile to calculate the volume of the entire raw material pile 11, it is possible to contribute to measuring the volume of the raw material pile efficiently at low cost without affecting factory operations. Also, according to the first embodiment, even if there are some defects or irregularities in the raw material pile 11, interpolation can be performed, so there is no impact on the measurement. Furthermore, according to the first embodiment, by calculating the volume of the raw material pile 11 by selectively using the angle of repose information for non-rainy weather and the angle of repose information for rainy weather, it is possible to reduce false detections due to changes in weather.

[0094] [Embodiment 2] A volume measurement system according to embodiment 2 will be described with reference to the drawings. Fig. 14 is a block diagram showing a schematic configuration of the volume measurement system according to embodiment 2.

[0095] The second embodiment is a modified example of the first embodiment, in which, instead of selectively using pre-set angle of repose information for rainy weather and non-rainy weather using the angle of repose memory unit (121 in FIG. 1) and the rain sensor (205 in FIG. 1), the angle of repose information is measured in real time using image data from the camera 206 provided in the photographing device 200.

[0096] The photographing device 200 has a camera 206. The camera 206 photographs an object and generates image data. The camera 206 outputs the generated image data to the volume measuring device 100 via the communication unit 203. The camera 206 may be, for example, a monocular RGB camera or a stereo camera capable of generating RGB (Red Green Blue) image data. Note that if the depth sensor 201 is capable of generating image data, the camera 206 may be omitted and the image data generated by the depth sensor 201 may be used.

[0097] The raw material pile detection unit 113 of the information processing unit 110 of the volume measuring device 100 measures the angle of repose information using the image data from the camera 206 together with the depth information acquired by the acquisition unit 111, without performing rainfall detection (step A3 in FIG. 3 ) or reading out the angle of repose information (steps A4 and A5 in FIG. 3 ). For example, a method for measuring the angle of repose information may involve creating a model that learns the surface texture of the raw material pile from the image data using deep learning, detecting the coordinates of the raw material pile using the created model, and measuring the angle of repose information from the depth information related to the raw material yard at the detected coordinates of the raw material pile. Using the measured angle of repose information, the raw material pile detection unit 113 detects (extracts) a point cloud related to the raw material pile (11 in FIG. 4 ) from the point cloud data related to the raw material yard (10 in FIG. 2 ) converted by the point cloud conversion unit 112.

[0098] The other configurations and operations are the same as those in the first embodiment.

[0099] According to the second embodiment, similar to the first embodiment, it is possible to contribute to measuring the volume of a raw material pile at low cost and efficiently without affecting factory operations, and by automating the measurement of angle of repose information, it is possible to reduce false detections due to various factors.

[0100] [Embodiment 3] A volume measuring device according to embodiment 3 will be described with reference to the drawings. Fig. 15 is a block diagram schematically showing the configuration of the volume measuring device according to embodiment 3. Fig. 16 is a flow chart diagram schematically showing the operation of the volume measuring device according to embodiment 3.

[0101] The volume measuring device 100 is a device that measures the volume of a raw material pile 11. The volume measuring device 100 has a point cloud conversion unit 112, a raw material pile detection unit 113, and a raw material pile volume calculation unit 114 (see FIG. 15).

[0102] The point cloud conversion unit 112 converts depth information relating to the raw material yard 10 from the depth sensor 201 that photographs the raw material yard 10 having the raw material pile 11 into point cloud data relating to the raw material yard 10 (see step B1 in Figure 16).

[0103] The raw material pile detection unit 113 detects the point cloud relating to the raw material pile 11 from the point cloud data relating to the raw material yard 10 using the repose angle information relating to the repose angle of the raw material pile 11 (see step B2 in FIG. 16).

[0104] The raw material pile volume calculation unit 114 calculates the volume of the raw material pile non-occlusion portion 11a in the raw material pile 11 that can be photographed by the depth sensor 201, based on the point cloud related to the raw material pile 11 (see step B3 in FIG. 16 ). The raw material pile volume calculation unit 114 estimates the volume of the raw material pile occlusion portion 11b in the raw material pile 11 that cannot be photographed by the depth sensor 201, using at least the point cloud related to the raw material pile 11 (see step B4 in FIG. 16 ). The raw material pile volume calculation unit 114 calculates the volume of the raw material pile 11 by adding up the calculated volume of the raw material pile non-occlusion portion 11a and the estimated volume of the raw material pile occlusion portion 11b (see step B5 in FIG. 16 ).

[0105] According to embodiment 3, by calculating the volume of the non-occlusion portion 11a of the raw material pile and estimating the volume of the occlusion portion 11b of the raw material pile to calculate the volume of the entire raw material pile 11, it is possible to contribute to measuring the volume of the raw material pile at low cost and efficiently without affecting factory operations.

[0106] The volume measuring devices according to the first to third embodiments can be configured using so-called hardware resources (information processing devices, computers), and may use those having the configuration shown in Fig. 18. For example, the hardware resources 300 include a processor 301, a memory 302, a network interface 303, and the like, which are interconnected by an internal bus 304.

[0107] 18 is not intended to limit the hardware configuration of the hardware resource 300. The hardware resource 300 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 301 included in the hardware resource 300 is not intended to be limited to the example shown in FIG. 18, and for example, a plurality of processors 301 may be included in the device. The processor 301 may be, for example, a central processing unit (CPU), a microprocessor unit (MPU), or the like.

[0108] The memory 302 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0109] The network interface 303 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0110] The functions of the hardware resource 300 are realized by the processing modules described above. The processing modules are realized, for example, by the processor 301 executing a program stored in the memory 302. The programs can be updated by downloading them via a network or by using a storage medium that stores the programs. Furthermore, the processing modules may be realized by semiconductor chips. In other words, it is sufficient that the functions performed by the processing modules can be realized by executing software on some kind of hardware.

[0111] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0112] [Supplementary Note 1] The present invention can have a configuration of the volume measuring device according to the first aspect.

[0113] [Supplementary Note 2] When detecting the point cloud relating to the raw material pile, the raw material pile detection unit detects the point cloud relating to the raw material pile when an angle formed by a plane on the slope of the raw material pile and the ground in the raw material yard satisfies the angle of repose in the angle of repose information.

[0114] [Supplementary Note 3] The volume measuring device according to Supplementary Note 1 or 2, further comprising an acquisition unit that acquires the depth information relating to the raw material yard from the depth sensor, and wherein the point cloud conversion unit converts the depth information relating to the raw material yard acquired by the acquisition unit into point cloud data relating to the raw material yard when converting into point cloud data relating to the raw material yard.

[0115] [Appendix 4] The volume measuring device according to any one of Appendices 1 to 3, further comprising an angle of repose memory unit that stores angle of repose information for rainy weather and angle of repose information for non-rainy weather, wherein when detecting the point cloud relating to the raw material pile, the raw material pile detection unit checks whether a rain sensor that detects rainfall in the raw material yard is detecting rainfall, reads the angle of repose information for rainy weather from the angle of repose memory unit when the rain sensor detects rainfall, reads the angle of repose information for non-rainy weather from the angle of repose memory unit when the rain sensor does not detect rainfall, and detects the point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using the read angle of repose information for rainy weather or the angle of repose information for non-rainy weather.

[0116] [Appendix 5] The volume measuring device described in any one of Appendices 1 to 3, wherein when detecting the point cloud relating to the raw material pile, the raw material pile detection unit measures angle of repose information using image data from a camera, and detects the point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using the measured angle of repose information.

[0117] [Appendix 6] The volume measuring device according to Appendix 5, wherein the raw material pile detection unit, when measuring the angle of repose information, creates a model that learns the surface texture of the raw material pile from the image data using deep learning, detects the coordinates of the raw material pile using the created model, and measures the angle of repose information from the depth information related to the raw material yard at the coordinates of the detected raw material pile.

[0118] [Supplementary Note 7] The raw material pile detection unit further checks whether or not there is a point cloud relating to an obstacle between the raw material pile and the depth sensor in the point cloud data relating to the raw material yard, and if there is a point cloud relating to the obstacle, removes the point cloud relating to the obstacle from the point cloud relating to the detected raw material pile, and interpolates a point cloud relating to a missing portion in the point cloud relating to the raw material pile created by removing the point cloud relating to the obstacle; and when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit calculates the volume of the raw material pile non-occlusion portion based on the point cloud relating to the raw material pile obtained by interpolating the point cloud relating to the missing portion by the raw material pile detection unit.

[0119] [Supplementary Note 8] The volume measuring device according to Supplementary Note 7, wherein the raw material pile detection unit interpolates the point cloud relating to the missing portion using an image interpolation method when interpolating the point cloud relating to the missing portion.

[0120] [Supplementary Note 9] The volume measuring device according to Supplementary Note 7 or 8, wherein when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit calculates the volume of the raw material pile non-occlusion portion based on the point cloud related to the raw material pile detected by the raw material pile detection unit when there is no point cloud related to the obstacle detected by the raw material pile detection unit.

[0121] [Supplementary Note 10] The volume measuring device according to any one of Supplementary Notes 1 to 9, wherein when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit creates a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the raw material pile, calculates the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and sets the calculated volume of the entire convex hull as the volume of the raw material pile non-occlusion portion.

[0122] [Supplementary Note 11] The volume measuring device according to any one of Supplementary Notes 1 to 10, further comprising a wall position memory unit that stores wall position information relating to the position of a wall in the raw material yard, wherein when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit checks whether or not a wall is present in the raw material yard using the point cloud relating to the raw material pile, and if a wall is present, reads the wall position information from the wall position memory unit and estimates the volume of the raw material pile occlusion portion using the point cloud relating to the raw material pile and the wall position information.

[0123] [Supplementary Note 12] When estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit cuts out a portion corresponding to the distance between the vertex of the raw material pile and the wall from the point cloud related to the raw material pile, creates a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the cut-out portion, calculates the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and sets the calculated volume of the entire convex hull as the volume of the raw material pile occlusion portion.

[0124] [Supplementary Note 13] The volume measuring device according to Supplementary Note 11 or 12, wherein the raw material pile volume calculation unit estimates the volume of the raw material pile occlusion portion by using the point cloud related to the raw material pile when there is no wall.

[0125] [Supplementary Note 14] The present invention can be configured as a volume measurement system according to the second aspect.

[0126] [Supplementary Note 15] The present invention can be embodied as a volume measurement method according to the third aspect.

[0127] [Supplementary Note 16] The present invention can take the form of a program according to the fourth aspect.

[0128] The disclosures of the above-mentioned patent and non-patent documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical concepts thereof. Furthermore, various combinations and selections (or non-selections, if necessary) of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concepts, including the claims and drawings. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate values, lower values, and smaller ranges are deemed to be included, even if not explicitly stated. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, are also deemed to be included in the disclosures of this application, in accordance with the spirit of the present invention.

[0129] 1 Volume measurement system 10 Raw material yard 11 Raw material pile 11a Raw material pile non-occlusion part 11b Raw material pile occlusion part 12 Wall 13 Camera 14 Ground 15 Stacking range 16 Plane 17 Missing part 20 Obstacle 100 Volume measurement device 110 Information processing unit 111 Acquisition unit 112 Point cloud conversion unit 113 Raw material pile detection unit 114 Raw material pile volume calculation unit 120 Memory unit 121 Angle of repose memory unit 122 Wall position memory unit 130 Input unit 140 Output unit 150 Communication unit 200 Imaging device 201 Depth sensor 202 Sensor control unit 203 Communication unit 204 Battery 205 Rain sensor 206 Camera 300 Hardware resources 301 Processor 302 Memory 303 Network interface 304 Internal bus

Claims

1. A volume measuring device comprising: a point cloud conversion unit that converts depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; a raw material pile detection unit that detects a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using repose angle information relating to the angle of repose of the raw material pile; and a raw material pile volume calculation unit that calculates the volume of a raw material pile non-occlusion portion that can be photographed from the depth sensor based on the point cloud relating to the raw material pile, estimates the volume of a raw material pile occlusion portion that cannot be photographed from the depth sensor using at least the point cloud relating to the raw material pile, and calculates the volume of the raw material pile as the sum of the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

2. The volume measuring device according to claim 1, wherein when detecting the point cloud relating to the raw material pile, the raw material pile detection unit detects the point cloud relating to the raw material pile when the angle formed by a plane on the slope of the raw material pile and the ground in the raw material yard satisfies the angle of repose in the angle of repose information.

3. A volume measuring device as described in claim 1 or 2, further comprising an angle of repose memory unit that stores angle of repose information for rainy weather and angle of repose information for non-rainy weather, wherein when detecting the point cloud related to the raw material pile, the raw material pile detection unit checks whether a rain sensor that detects rainfall in the raw material yard is detecting rainfall, reads the angle of repose information for rainy weather from the angle of repose memory unit when the rain sensor detects rainfall, and reads the angle of repose information for non-rainy weather from the angle of repose memory unit when the rain sensor does not detect rainfall, and detects the point cloud related to the raw material pile from the point cloud data related to the raw material yard using the read angle of repose information for rainy weather or the angle of repose information for non-rainy weather.

4. A volume measuring device as described in claim 1 or 2, wherein when detecting the point cloud relating to the raw material pile, the raw material pile detection unit measures angle of repose information using image data from a camera, and detects the point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using the measured angle of repose information.

5. The raw material pile detection unit further checks whether or not there is a point cloud relating to an obstacle between the raw material pile and the depth sensor in the point cloud data relating to the raw material yard, and if there is a point cloud relating to the obstacle, removes the point cloud relating to the obstacle from the point cloud relating to the detected raw material pile and interpolates a point cloud relating to a missing portion in the point cloud relating to the raw material pile created by removing the point cloud relating to the obstacle; and when calculating the volume of the raw material pile non-occlusion portion, the raw material pile volume calculation unit calculates the volume of the raw material pile non-occlusion portion based on the point cloud relating to the raw material pile obtained by interpolating the point cloud relating to the missing portion by the raw material pile detection unit. A volume measuring device as described in any one of claims 1 to 4.

6. A volume measuring device according to any one of claims 1 to 5, wherein when calculating the volume of the non-occlusion portion of the raw material pile, the raw material pile volume calculation unit creates a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the raw material pile, calculates the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and sets the calculated volume of the entire convex hull as the volume of the non-occlusion portion of the raw material pile.

7. A volume measuring device as described in any one of claims 1 to 6, further comprising a wall position memory unit that stores wall position information relating to the position of a wall in the raw material yard, wherein when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit uses the point cloud relating to the raw material pile to check whether or not there is a wall in the raw material yard, and if there is a wall, reads the wall position information from the wall position memory unit and estimates the volume of the raw material pile occlusion portion using the point cloud relating to the raw material pile and the wall position information.

8. The volume measuring device according to claim 7, wherein, when estimating the volume of the raw material pile occlusion portion, the raw material pile volume calculation unit cuts out a portion corresponding to the distance between the vertex of the raw material pile and the wall from the point cloud related to the raw material pile, creates a convex hull composed of a plurality of tetrahedrons based on the point cloud related to the cut-out portion, calculates the volume of the entire convex hull by calculating the sum of the volumes of the plurality of tetrahedrons, and sets the calculated volume of the entire convex hull as the volume of the raw material pile occlusion portion.

9. A volume measurement system comprising: a depth sensor that photographs a raw material yard having a pile of raw materials; and a volume measurement device according to any one of claims 1 to 8.

10. A volume measurement method comprising the steps of: converting depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using repose angle information relating to the angle of repose of the raw material pile; calculating a volume of a raw material pile non-occlusion portion in the raw material pile that can be photographed from the depth sensor based on the point cloud relating to the raw material pile, estimating a volume of a raw material pile occlusion portion in the raw material pile that cannot be photographed from the depth sensor using at least the point cloud relating to the raw material pile, and calculating a volume of the raw material pile by adding up the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.

11. A program that causes hardware resources to execute the following processes: converting depth information relating to a raw material yard having a raw material pile from a depth sensor that photographs the raw material yard into point cloud data relating to the raw material yard; detecting a point cloud relating to the raw material pile from the point cloud data relating to the raw material yard using repose angle information relating to the angle of repose of the raw material pile; calculating the volume of a raw material pile non-occlusion portion that can be photographed from the depth sensor based on the point cloud relating to the raw material pile, estimating the volume of a raw material pile occlusion portion that cannot be photographed from the depth sensor using at least the point cloud relating to the raw material pile, and calculating the volume of the raw material pile as the sum of the calculated volume of the raw material pile non-occlusion portion and the estimated volume of the raw material pile occlusion portion.