Quantity calculation device and quantity calculation method

Abstract

To provide an object amount calculation device capable of shortening a processing time for calculating an amount of an object.SOLUTION: An object amount calculation device includes: an acquisition unit which acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model; a user interface which accepts an operation indicating attribute information having priority than the other attribute information among at least two pieces of attribute information respectively owned by the first three-dimensional model and the second three-dimensional model; a positioning unit 302e which positions the first three-dimensional model and the second three-dimensional model on the basis of the attribute information indicated by the operation accepted by the user interface; and a calculation unit which calculates a difference between the first three-dimensional model and the second three-dimensional model positioned by the positioning unit 302e, and outputs an amount of the difference and attribute information owned by the second three-dimensional model and corresponding to the difference.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a quantity calculation device and a quantity calculation method. [Background technology]

[0002] There is a technique for measuring a quantity such as the volume or weight of an object located in a given space (see, for example, Patent Document 1).

[0003] The technology disclosed in Patent Document 1 compares images of the collapse site before and after the collapse occurs, aligns the images, generates a three-dimensional model using the aligned images, and calculates the amount of damage at the collapse site based on the generated three-dimensional model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2002-328021 A Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable for the process of calculating the amount of objects located in a given space, such as the amount of damage, to take a short processing time.

[0006] The present disclosure provides a quantity calculation device and the like that can reduce the processing time for calculating the quantity of an object. [Means for solving the problem]

[0007] A quantity calculation device according to one embodiment of the present disclosure includes an acquisition unit that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; a user interface that accepts an operation indicating attribute information that has priority over other attribute information among at least two pieces of attribute information possessed by each of the first three-dimensional model and the second three-dimensional model; an alignment unit that aligns the first three-dimensional model and the second three-dimensional model based on the attribute information indicated by the operation accepted by the user interface; and a calculation unit that calculates a difference between the first three-dimensional model and the second three-dimensional model aligned by the alignment unit, and outputs the amount of the difference and attribute information possessed by the second three-dimensional model and corresponding to the difference.

[0008] Moreover, a quantity calculation method according to one embodiment of the present disclosure includes an acquisition step of acquiring a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; a reception step of receiving, via a user interface, an operation indicating attribute information that has priority over other attribute information among at least two pieces of attribute information possessed by each of the first three-dimensional model and the second three-dimensional model; an alignment step of aligning the first three-dimensional model and the second three-dimensional model based on the attribute information indicated by the operation accepted by the user interface; and a calculation step of calculating a difference between the first three-dimensional model and the second three-dimensional model aligned in the alignment step, and outputting the amount of the difference and attribute information possessed by the second three-dimensional model and corresponding to the difference.

[0009] The present disclosure may be realized as a program that causes a computer to execute the steps included in the above-mentioned quantity calculation method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or a signal indicating the program. The program, information, data, and signal may be distributed via a communication network, such as the Internet. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a quantity calculation device and the like that can reduce the processing time for calculating the quantity of an object. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining an overview of a sediment volume measurement system according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an overview of the process of extracting attribute information, which is executed by the sediment volume measurement system according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an outline of a process for calculating the amount of sediment, which is executed by the sediment amount measurement system according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of a sediment volume measurement system according to an embodiment. [Diagram 5] FIG. 5 is a block diagram showing a configuration of an imaging device included in the sediment volume measurement system according to the embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of a control device provided in the sediment volume measurement system according to the embodiment. [Figure 7] FIG. 7 is a block diagram showing the configuration of a sediment amount calculation device provided in the sediment amount measurement system according to the embodiment. [Figure 8] FIG. 8 is a flowchart showing a process of aligning a three-dimensional model, which is executed by the sediment volume measurement system according to the embodiment. [Figure 9]FIG. 9 is a diagram for explaining the alignment of two three-dimensional models executed by the alignment unit. [Figure 10] FIG. 10 is a sequence diagram showing the processing of the sediment volume measurement system according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing a process for calculating the amount of sediment, which is executed by the sediment amount measurement system according to the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of an image presented by a user interface included in the sediment volume measurement system according to the embodiment when aligning two three-dimensional models. [Figure 13] FIG. 13 is a diagram showing an example of difference information presented by a user interface included in the sediment volume measurement system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Summary of the Disclosure) A quantity calculation device according to one embodiment of the present disclosure includes an acquisition unit that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; an alignment unit that aligns the first three-dimensional model and the second three-dimensional model based on attribute information possessed by each of the first and second three-dimensional models; and a calculation unit that calculates an amount of difference between the first and second three-dimensional models aligned by the alignment unit, and outputs attribute information of the difference and the amount of difference.

[0013] Conventionally, there is a method for measuring (calculating) the amount of sediment that has flowed into a given space due to the effects of collapses, etc., at a landslide disaster site, for example. Calculating the amount of sediment using laser measurement can calculate the amount with high accuracy, but there is a problem that the calculation takes time and costs. Therefore, in the quantity calculation device according to the present disclosure, for example, a three-dimensional model (first three-dimensional model) showing a given space in a state where an object for which the amount of sediment, etc. is to be calculated does not exist in the given space is compared with a three-dimensional model (second three-dimensional model) showing the given space in a state where the object exists. According to this, by appropriately aligning the first three-dimensional model and the second three-dimensional model, the difference between the first three-dimensional model and the second three-dimensional model can be easily calculated. Therefore, according to the quantity calculation device according to the present disclosure, the processing time for calculating the amount of an object can be shortened.

[0014] Also, for example, the quantity calculation device according to one aspect of the present disclosure further includes a generation unit that generates at least one of the first three-dimensional model and the second three-dimensional model from a plurality of images each showing the specified space.

[0015] According to this, the amount of difference can be calculated using a simple configuration such as a camera for generating an image.

[0016] Also, for example, when the difference includes multiple pieces of attribute information that are different from each other, the calculation unit classifies and outputs the multiple pieces of attribute information by type, and displays the multiple pieces of attribute information together by type on a display device.

[0017] According to this, the types of objects included in the differences are classified and displayed on the display device, making it easier for a user checking the display device to understand the types included in the differences.

[0018] Also, for example, the calculation unit calculates an amount included in the difference for each of the types.

[0019] According to this, the amount of each type of object included in the difference is displayed on the display device, making it easier for a user checking the display device to understand the amount of each type included in the difference.

[0020] Also, for example, the amount of the difference is at least one of the volume and weight of the difference.

[0021] According to this, the quantity calculation device according to the present disclosure can calculate at least one of the difference amount and volume.

[0022] Also, for example, the alignment unit extracts partial models having attribute information common to the first three-dimensional model and the second three-dimensional model from each of the first three-dimensional model and the second three-dimensional model, aligns the partial models by moving each of the partial models so that coordinates of corresponding positions of each of the extracted partial models match, and then aligns the first three-dimensional model and the second three-dimensional model by moving the first three-dimensional model and the second three-dimensional model in accordance with the movement of each of the partial models.

[0023] This allows the quantity calculation device according to the present disclosure to align the first and second three-dimensional models without time-consuming processing such as alignment by comparing the shapes of the three-dimensional models, etc. Therefore, the processing time of the quantity calculation device according to the present disclosure can be further shortened.

[0024] Moreover, a quantity calculation method according to one embodiment of the present disclosure includes an acquisition step of acquiring a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; an alignment step of aligning the first three-dimensional model and the second three-dimensional model based on attribute information possessed by each of the first and second three-dimensional models; and a calculation step of calculating an amount of difference between the first and second three-dimensional models aligned in the alignment step, and outputting attribute information possessed by the difference and the amount of difference.

[0025] According to this, by appropriately aligning the first 3D model and the second 3D model, it is possible to easily calculate the difference between the first 3D model and the second 3D model. Therefore, according to the quantity calculation method according to the present disclosure, it is possible to shorten the processing time for calculating the amount of an object.

[0026] Also, for example, in the acquisition step, the first three-dimensional model showing the specified space before the soil and sand flow in and the second three-dimensional model showing the specified space after the soil and sand flow in are acquired, and in the calculation step, the amount of soil and sand is calculated as the amount of the difference.

[0027] According to this, by appropriately aligning the three-dimensional models showing a predetermined space before and after a disaster in which sediment flows into the predetermined space, the amount of sediment, which is the difference, can be easily calculated. Therefore, according to the quantity calculation method disclosed herein, the processing time for calculating the amount of sediment can be shortened.

[0028] Each embodiment of the present disclosure will be described in detail below with reference to the drawings. Each embodiment described below shows a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection of the components, steps, order of steps, etc. shown in each embodiment below are examples and are not intended to limit the present disclosure.

[0029] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In addition, in each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations may be omitted or simplified.

[0030] (Embodiment) [overview] FIG. 1 is a diagram for explaining an overview of a sediment volume measurement system according to an embodiment.

[0031] A sediment volume measurement system 1000 according to an embodiment (see FIG. 4, for example) includes a plurality of cameras 100 for capturing images (hereinafter also referred to as photographing) of a predetermined three-dimensional space A1 (hereinafter also simply referred to as space A1). As a specific example, space A1 is a space above a road 20 where sediment 21 has accumulated. As another specific example, space A1 is a space above a river where sediment 21 or driftwood has accumulated. The sediment volume measurement system 1000 is an example of a material quantity calculation device according to the present disclosure.

[0032] The multiple cameras 100 are disposed at different positions from one another, and capture a common three-dimensional space (e.g., space A1). Therefore, the areas captured by the multiple cameras 100 in the space A1 have at least a partially overlapping area. For example, the multiple cameras 100 are installed at different positions from one another so as to partially surround the space A1 including the soil 21 on the road 20. Furthermore, the multiple cameras 100 have different attitudes from one another. The areas captured by the multiple cameras 100 overlap at least partially. The reason that the areas captured overlap at least partially is to generate (reconstruct) a three-dimensional model of the space A1 (e.g., point cloud data) based on the video data obtained by capturing images.

[0033] The areas to be imaged may overlap among some of the multiple cameras 100, or may overlap among all of the multiple cameras 100.

[0034] Furthermore, the multiple cameras 100 may be the same cameras that capture images at different times, i.e., moving cameras. Therefore, if the cameras 100 are moving cameras, there may be one or more cameras 100, and the number is not limited.

[0035] In the process performed in three-dimensional reconstruction (the process of generating a three-dimensional model of space A1), a three-dimensional model 22 showing space A1 is calculated (generated) using a plurality of videos (hereinafter, videos are also referred to as images) thus obtained respectively from the plurality of cameras 100. The process performed in three-dimensional reconstruction, for example, associates the plurality of images obtained respectively from the plurality of cameras 100, and calculates the three-dimensional model 22 by performing geometric calculations from the obtained correspondences.

[0036] Here, the three-dimensional model refers to data that expresses the geometric shape of a predetermined three-dimensional space (space A1 in this embodiment). A model for expressing a three-dimensional shape is, for example, a point cloud consisting of a plurality of three-dimensional points. In addition, the three-dimensional model may be expressed by voxels or meshes.

[0037] It should be noted that these representations of the three-dimensional model are merely examples and do not limit the method of representing the three-dimensional model.

[0038] In addition, the data used to generate the three-dimensional model does not have to be an image and is not particularly limited. For example, the data used to generate the three-dimensional model may be data obtained from a measuring instrument such as a LiDAR.

[0039] LiDAR is a sensor for measuring the distance to an object. By scanning (moving and measuring) a given area with LiDAR, the position of an object located in the given area can be calculated.

[0040] This type of measuring instrument, such as LiDAR, generates a 3D model by emitting electromagnetic waves and acquiring the waves reflected by the measurement target. Specifically, the measuring instrument measures the time it takes for the electromagnetic waves to be emitted, reflected by the measurement target, and returned to the measuring instrument, and calculates the distance between the measuring instrument and a point on the surface of the measurement target using the measured time and the wavelength of the electromagnetic waves.

[0041] For example, the measuring instrument emits electromagnetic waves in a plurality of predetermined radial directions from a reference point of the measuring instrument. For example, the measuring instrument emits electromagnetic waves at a first angular interval around the horizontal direction and emits electromagnetic waves at a second angular interval around the vertical direction. Therefore, the measuring instrument can calculate the three-dimensional coordinates of a plurality of points on the measurement target by calculating the distance between the measuring instrument and the measurement target in each of a plurality of directions around the measuring instrument.

[0042] The measuring instrument may measure the distance to the measurement target by emitting, for example, a laser beam as an electromagnetic wave, or by emitting, for example, a millimeter wave as an electromagnetic wave.

[0043] Fig. 2 is a diagram for explaining an overview of the process of extracting attribute information executed by the sediment volume measurement system 1000 according to the embodiment. Specifically, Fig. 2(a) is a diagram showing an example of an image generated by capturing an image of space A1. Fig. 2(b) is a diagram showing a three-dimensional model showing space A1. Fig. 2(c) is a diagram showing an enlarged view of a pixel having attribute information of "tree" in the image shown in Fig. 2(a).

[0044] The sediment volume measurement system 1000 extracts attribute information using a plurality of image data captured by a plurality of cameras 100 within the space A1, or a three-dimensional model reconstructed from the space A1.

[0045] Here, the attribute information is information indicating the result of recognizing what meaning a specific region in an image or a three-dimensional model has. For example, the attribute information is information indicating an attribute such as an object included in an arbitrary region in an image being a "tree". As another example, the attribute information is information indicating an attribute such as an arbitrary partial point group in a three-dimensional model expressed by a point group being a "tree". That is, "tree", "driftwood", "sand", "house", etc. are each an example of attribute information. In this way, for example, a three-dimensional model has attribute information for each region. The region may be a predetermined range in the three-dimensional model, or may be each three-dimensional point constituting the three-dimensional model, and is not particularly limited. The region may also be each partial model described later. That is, a three-dimensional model may have attribute information for each partial model described later.

[0046] The process of extracting the attribute information is realized, for example, by performing semantic segmentation, which is a process of meaningfully labeling a pixel or a minimum structural unit of a three-dimensional model by using information surrounding the pixel or the minimum structural unit of the three-dimensional model.

[0047] Specifically, for example, semantic segmentation is a process in which a label of "tree" is attached to each pixel in an image of a "tree" existing in a specified three-dimensional space, as shown in FIG. 2(c). Each pixel can be labeled by referring to information on surrounding pixels. The information on surrounding pixels may be R (Red), G (Green), and B (Blue) values, luminance values, a distribution of pixel values, or a gradient. According to this, for example, a partial model that is a collection of points to which attribute information of "trees" is attached can be treated as a partial model having attribute information of "trees".

[0048] The attribute information is extracted, for example, by pattern recognition from data indicating a predetermined "tree" etc. The user may associate the attribute information with an object included in the image or three-dimensional model by operating a user interface 400 (see FIG. 4) described later.

[0049] Fig. 3 is a diagram for explaining an outline of a process for calculating the amount of sediment executed by the sediment amount measurement system 1000 according to the embodiment. Specifically, Fig. 3(a) is a diagram showing a three-dimensional model showing the space A1 before a disaster. Fig. 3(b) is a diagram showing a three-dimensional model showing the space A1 after a disaster. Fig. 3(c) is a diagram showing a process for calculating difference information executed by the sediment amount measurement system 1000. Fig. 3(d) is a diagram showing an example of difference information output by the sediment amount measurement system 1000.

[0050] The sediment volume measurement system 1000 extracts information (difference information) about the sediment 21 by calculating the difference between a three-dimensional model before the disaster (hereinafter also referred to as a first three-dimensional model) that reconstructs the space A1 and a three-dimensional model after the disaster (hereinafter also referred to as a second three-dimensional model).

[0051] The sediment volume measurement system 1000 extracts portions of the two three-dimensional models that show more significant changes before and after the disaster. The calculation results of the difference are presented to the user as difference information.

[0052] Here, the difference information is, for example, the volume of a three-dimensional partial area in a three-dimensional mesh model where the surfaces of the three-dimensional model before the disaster and the three-dimensional model after the disaster do not overlap or intersect. The amount of difference calculated in this way is, for example, the amount of soil and sand accumulated in a specified three-dimensional space after the disaster.

[0053] In addition, the difference information may include information on a mixture containing multiple different attribute information, such as attribute information obtained by extracting attribute information in areas of the two three-dimensional models where changes are more significant between before and after the disaster, for example, the proportion of driftwood or rubble contained in the soil and sand 21.

[0054] A mixture is, for example, a collection of multiple partial models having different attribute information from each other. In the above example, the mixture is soil and sand. The mixture (i.e., soil and sand) is a three-dimensional model composed of a partial model (first partial model) having attribute information indicating "driftwood" and a partial model (second partial model) having attribute information indicating "rubble". The difference information includes, for example, one or more pieces of attribute information contained in the mixture, includes the volume of the mixture as the amount of difference, and includes information indicating the proportion of the volume of the mixture occupied by the first partial model and the second partial model.

[0055] Also, the mixture may be classified into a plurality of categories according to the type of attribute information. For example, the mixture may be composed of a partial model (first partial model) having attribute information indicating "driftwood", a partial model (second partial model) having attribute information indicating "rubble", a partial model (third partial model) having attribute information indicating "sign", and a partial model (fourth partial model) having attribute information indicating "traffic light". In this case, the mixture may be classified into a mixture (first mixture) composed of the first partial model and the second partial model consisting of natural objects, and a mixture (second mixture) composed of the third partial model and the fourth partial model consisting of artificial objects. Also, in this case, the difference information may include information indicating the ratio of the first mixture to the volume of the mixture and the ratio of the second mixture to the volume of the mixture.

[0056] As described above, the difference information includes, for example, difference information indicating the amount of difference between two three-dimensional models and attribute information of the difference. The amount of difference is, for example, at least one of the volume and weight of the difference.

[0057] Note that the above is an example of the difference information, and the difference information is not limited to this example.

[0058] In addition, the scale of the three-dimensional model may be adjusted to a desired scale for either the pre-disaster or post-disaster three-dimensional model of the two three-dimensional models. For example, either the pre-disaster or post-disaster three-dimensional model of the two three-dimensional models may be adjusted to the scale of the space A1. In addition, the scale of either the pre-disaster or post-disaster three-dimensional model of the two three-dimensional models may be adjusted based on scale information provided from outside the sediment volume measurement system 1000. By adjusting the scale of the three-dimensional models to be the same, the difference between the pre-disaster three-dimensional model and the post-disaster three-dimensional model can be calculated at the same scale. This allows the sediment volume to be calculated at the scale of the space A1.

[0059] The sediment volume measurement system 1000 is an example of a material volume calculation device according to the present disclosure. The above-described method for calculating difference information may be used to calculate an object other than sediment, for example, an inventory amount in a warehouse, or may be used to calculate a construction material at a work site during construction work.

[0060] [composition] <Sediment volume measurement system> Next, the configuration of the sediment volume measurement system 1000 will be described in detail.

[0061] FIG. 4 is a block diagram showing a configuration of a sediment volume measurement system 1000 according to an embodiment.

[0062] The sediment volume measurement system 1000 includes a plurality of imaging devices 10a-10n (for example, n (an integer of 2 or more) from imaging device 10a to imaging device 10b, ..., imaging device 10n), a control device 200, a sediment volume calculation device 300, and a user interface 400. The imaging devices 10a-10n are communicatively connected to the control device 200. The sediment volume calculation device 300 and the user interface 400 are communicatively connected to the control device 200.

[0063] The imaging devices 10a to 10n may be connected to the control device 200 so as to be capable of wired communication or so as to be capable of wireless communication. The sediment amount calculation device 300 and the user interface 400 may be connected to the control device 200 so as to be capable of wired communication or so as to be capable of wireless communication.

[0064] The user interface 400 is a receiving device for receiving operations from a user. The user interface 400 is realized, for example, by a display device, a touch screen, a track pad, a keyboard, a mouse, other controllers, or a combination of these connected to the control device 200. The user interface 400 is also an example of a display device for displaying attribute information and difference information calculated by the quantity calculation device according to the present disclosure.

[0065] <Imaging device> Fig. 5 is a block diagram showing the configuration of the imaging device 10a included in the sediment volume measurement system 1000 according to the embodiment. Since the configurations of the imaging devices 10a to 10n shown in Fig. 4 are the same in this embodiment, when describing one of the imaging devices 10a to 10n included in the sediment volume measurement system 1000, the imaging device 10a will be described. In other words, the following description of the imaging device 10a also applies to the other imaging devices 10b to 10n.

[0066] Each of the imaging devices 10a to 10n is a device equipped with a camera for capturing an image of the space A1, which corresponds to the camera 100 shown in Fig. 1. The imaging devices 10a to 10n may have a common configuration, for example.

[0067] The space A1 here is the union of the imaging areas of the multiple cameras 100.

[0068] Each of the imaging devices 10a to 10n has a camera 100 and may be equipped with a mount 110.

[0069] The camera 100 includes a memory unit 101 , a control unit 102 , an optical system 103 , and an image sensor 104 .

[0070] The storage unit 101 stores a program that is read and executed by the control unit 102. The storage unit 101 also temporarily stores video data of an imaging area captured using the image sensor 104, meta information such as a timestamp added to the video data, camera parameters of the camera 100, and shooting settings such as the currently applied frame rate or resolution.

[0071] Such a storage unit 101 is realized by using a rewritable non-volatile semiconductor memory such as a flash memory. Also, depending on the necessity of rewriting the stored data or the required storage period, a non-rewritable read-only memory (ROM) or a volatile random access memory (RAM) may be used as the storage unit 101.

[0072] The number of imaging devices included in the sediment volume measurement system 1000 is not particularly limited. For example, the number of imaging devices included in the sediment volume measurement system 1000 may be more than one. Furthermore, the characteristics of the imaging devices 10a to 10n included in the sediment volume measurement system 1000 do not have to be common. Furthermore, the camera 100 included in each of the imaging devices 10a to 10n is not limited to a mono camera, and may be a stereo camera.

[0073] Furthermore, the number of imaging devices provided in the sediment volume measurement system 1000 may be one, as long as the imaging device can image the space A1 from a plurality of viewpoints by changing the position and orientation.

[0074] The control unit 102 is realized, for example, using a CPU (Central Processing Unit), and reads and executes programs stored in the memory unit 101 to control each component of the camera 100 to perform imaging functions and other functions.

[0075] Note that control unit 102 may be realized by a dedicated circuit that controls each component of camera 100 to perform the imaging function and other functions. In other words, control unit 102 may be realized by software or hardware.

[0076] The optical system 103 is a component that forms an image of light from an imaging region on the image sensor 104, and is realized using optical elements including a lens. The optical system 103 may be configured such that the focal length and the angle of view are changeable. The optical system 103 may use a wide-angle lens or an ultra-wide-angle lens such as a fisheye lens. For example, when an image captured by a camera included in the quantity calculation device according to the present disclosure is used in a monitoring system, a wide-angle lens may be used to obtain a wide imaging region.

[0077] The image sensor 104 is realized by a solid-state imaging element such as a CCD (Charge Coupled Devices) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a MOS (Metal Oxide Semiconductor) image sensor, which receives the light collected by the optical system 103 on a light receiving surface and converts the received light into an electrical signal representing an image.

[0078] The mount 110 is a component for fixing and supporting the camera 100 in a predetermined position while the camera 100 is capturing and generating images used to calculate the amount of soil and sand, and is realized by, for example, a tripod.

[0079] In order to adjust the fixed position of the camera 100 in preparation for this imaging, the legs of the mount 110 may be adjustable in length and angle.

[0080] The platform 110 may also be provided with a mechanism for rotating a camera head in order to pan or tilt the camera 100, an elevator mechanism for moving the camera up and down, etc. Alternatively, the platform 110 may include a mechanism for supporting and moving the camera 100, such as a dolly or a crane.

[0081] The imaging devices 10a to 10n do not necessarily have to be fixed. Therefore, the imaging devices 10a to 10n may not have the mount 110.

[0082] The number of imaging devices 10a included in the sediment volume measurement system 1000 is not particularly limited as long as the imaging devices 10a are multiple fixed cameras, one or more mobile cameras, or a combination of these. Also, the imaging devices 10a are not limited to mono cameras, and may include compound-eye cameras such as stereo cameras.

[0083] <Control device> FIG. 6 is a block diagram showing the configuration of the control device 200 provided in the sediment volume measurement system 1000 according to the embodiment.

[0084] The control device 200 includes a storage unit 201 , a control unit 202 , and a timer 203 .

[0085] The control device 200 controls the imaging devices 10a to 10n and performs input and output to the user interface 400. The control device 200 also issues a command to the sediment amount calculation device 300 to calculate the amount of sediment based on the data received from each of the imaging devices 10a to 10n.

[0086] Such a control device 200 is, for example, a computer. In this case, the storage unit 201 is a storage device of this computer, and is realized by a hard disk drive, various semiconductor memories, or a combination of these. Moreover, the control unit 202 is realized by the CPU of this computer.

[0087] The timer 203 is a timer provided in the control device 200 and is referenced by the CPU that realizes the control unit 202 .

[0088] The storage unit 201 stores a program that is read and executed by the control unit 202. The storage unit 201 also stores data received from the imaging devices 10a to 10n and that is to be processed by the control unit 202.

[0089] The control unit 202 controls the imaging devices 10a to 10n and the sediment amount calculation device 300 by reading and executing the programs stored in the storage unit 201.

[0090] The control unit 202 also executes processes in response to commands from the user regarding these controls and processes. One of these processes is the control of capturing synchronized images by the multiple cameras 100 provided in each of the imaging devices 10a to 10n. One of these processes may also include a command to calculate the amount of soil and sand.

[0091] The control unit 202 functionally includes an imaging control unit 202a, a user instruction acquisition unit 202b, and a sediment amount calculation command unit 202c.

[0092] A user instruction acquisition unit 202b included in the control unit 202 is a functional component that is realized by the control unit 202 executing a program for acquiring instructions from a user.

[0093] Further, a sediment amount calculation command unit 202c included in the control unit 202 is a functional component that is realized by the control unit 202 executing a program for issuing a command to calculate the sediment amount.

[0094] The imaging control unit 202a, the user instruction acquisition unit 202b, and the sediment amount calculation command unit 202c of the control unit 202 may be realized by dedicated circuits for performing imaging control, user instruction acquisition, sediment amount calculation command, and sediment amount calculation processing, etc. In other words, the control unit 202 may be realized by software or hardware.

[0095] The imaging control unit 202a causes each of the imaging devices 10a-10n to capture an image of the space A1, which is an imaging area, at different timings. The imaging control unit 202a causes each of the imaging devices 10a-10n to capture an image of the imaging area while each of the imaging devices 10a-10n is in a predetermined position and facing a predetermined direction.

[0096] The user instruction acquiring unit 202b sends imaging situation information provided from the imaging devices 10a to 10n to the user interface 400 and acquires input from the user. The input from the user is the result of selection of data for which sediment volume calculation is to be performed, whether or not sediment volume calculation processing is to be performed, or a combination of these. When the input from the user is whether or not sediment volume calculation processing is to be performed, the user instruction acquiring unit 202b outputs whether or not sediment volume calculation processing is to be performed, for example, to the sediment volume calculation command unit 202c.

[0097] The sediment amount calculation command unit 202c causes the sediment amount calculation device 300 to execute the sediment amount calculation process based on the availability of the sediment amount calculation process received from the user instruction acquisition unit 202b. The sediment amount calculation command unit 202c may also cause the sediment amount calculation device 300 to execute the sediment amount calculation process based on the selection result of the data for which the sediment amount is to be calculated. A specific example of the process by the sediment amount calculation command unit 202c will be described later.

[0098] <Sediment volume calculation device> FIG. 7 is a block diagram showing the configuration of a sediment volume calculation device 300 included in the sediment volume measurement system 1000 according to this embodiment.

[0099] The sediment volume calculation device 300 includes a memory unit 301 and a processing unit 302 .

[0100] The sediment amount calculation device 300 processes the data received via the control device 200. Specifically, the sediment amount calculation device 300 executes a sediment amount calculation process for the sediment 21 present in the space A1 captured by the camera 100 included in each of the imaging devices 10a to 10n.

[0101] The target for which the sediment amount calculation device 300 executes the sediment amount calculation process is, for example, a space A1 after a disaster. This makes it possible to calculate the amount (amount of sediment) of sediment 21 deposited in the space A1 by calculating the difference from the three-dimensional model of the space A1 before the disaster.

[0102] In addition, the target for which the sediment amount calculation device 300 executes the sediment amount calculation process does not have to be the space A1 immediately after the disaster, but may be, for example, the space A1 during sediment removal work. In this way, the sediment amount calculation device 300 can calculate the progress of the sediment removal work by successively calculating the sediment amount.

[0103] The target on which the sediment volume calculation device 300 executes the sediment volume calculation process is not limited to the space A1 before and after the disaster, and is not particularly limited.

[0104] Such a sediment volume calculation device 300 is, for example, a computer. In this case, the storage unit 301 is a storage device of this computer, and is realized by a hard disk drive, various semiconductor memories, or a combination of these. Also, the processing unit 302 is realized by the CPU of this computer. Note that the sediment volume calculation device 300 may be realized by the same computer as the control device 200.

[0105] The storage unit 301 stores a program that is read and executed by the processing unit 302. The storage unit 301 also stores data received from the imaging devices 10a to 10n via the control device 200 and data of a three-dimensional model acquired from an external device, which is the target of processing by the processing unit 302. That is, the storage unit 301 may store a three-dimensional reconstruction result or may store an attribute information extraction result. The three-dimensional reconstruction result may be a three-dimensional model before a disaster or a three-dimensional model after a disaster.

[0106] The processing unit 302 processes the data received from the imaging devices 10a to 10n by reading and executing the programs stored in the storage unit 301. One example of this processing is three-dimensional reconstruction of the space A1.

[0107] The processing unit 302 includes an image acquisition unit 302a, a three-dimensional reconstruction unit 302b, an attribute information extraction unit 302c, a model acquisition unit 302d, a registration unit 302e, and a difference calculation unit 302f.

[0108] The image acquisition unit 302a acquires a plurality of images captured by the plurality of imaging devices 10a to 10n, respectively. The image acquisition unit 302a further acquires, together with the plurality of images, a camera label that corresponds to each of the plurality of images and indicates the camera 100 that captured each of the plurality of images. The image acquisition unit 302a may acquire the plurality of images and the camera label, for example, by acquiring an image to which the camera label is attached. Each of the plurality of images may be a still image or a moving image.

[0109] It is preferable that the multiple images used in the calibration process (three-dimensional reconstruction process) are images obtained by capturing images by the multiple imaging devices 10a to 10n at a corresponding timing (i.e., at the same time). The multiple images captured by the image acquisition unit 302a are stored in the storage unit 301. The image acquisition unit 302a may store the images and camera labels in the storage unit 301 before receiving a sediment amount calculation command from the control device 200. This allows the sediment amount calculation device 300 to start the sediment amount calculation process when it is determined that sediment amount calculation is necessary.

[0110] The three-dimensional reconstruction unit 302b generates at least one of a first three-dimensional model and a second three-dimensional model from a plurality of images each showing the space A1. More specifically, the three-dimensional reconstruction unit 302b calculates a three-dimensional shape of the space A1 using a plurality of images captured by the plurality of imaging devices 10a to 10n for the space A1. For example, the three-dimensional reconstruction unit 302b generates a three-dimensional model of the space A1 at a first time using a group of images at the first time. Also, the three-dimensional reconstruction unit 302b generates a three-dimensional model of the space A1 using a group of images at a time different from the first time. In this way, the three-dimensional reconstruction unit 302b generates a first three-dimensional model and a second three-dimensional model that are three-dimensional models of the same area and are generated at different times.

[0111] The three-dimensional reconstruction unit 302b, for example, associates a plurality of images captured by a plurality of imaging devices 10a to 10n, and calculates the three-dimensional shape of the space A1 by geometric calculation based on the correspondence. The model for expressing the three-dimensional shape may be expressed by a point cloud, voxel, or mesh.

[0112] Note that these representations of the three-dimensional model are merely examples, and the method of representing the three-dimensional model is not limited to the above.

[0113] The attribute information extraction unit 302c extracts attribute information from a plurality of images captured by the plurality of imaging devices 10a to 10n, a three-dimensional model reconstructed by the three-dimensional reconstruction unit 302b, or a three-dimensional model acquired by the model acquisition unit 302d described below. For example, the attribute information extraction unit 302c estimates the attribute of each point of a three-dimensional point group based on information of surrounding points, and calculates the estimated attribute as attribute information. In addition, for example, the attribute information extraction unit 302c estimates attribute information of an object captured in each image based on information of surrounding pixels for image data, and links the estimated attribute information to each minimum constituent unit of the reconstructed three-dimensional model.

[0114] In addition, the attribute information extraction unit 302c may acquire a three-dimensional model reconstructed by the three-dimensional reconstruction unit 302b, or may acquire a plurality of images captured by the plurality of imaging devices 10a to 10n from the storage unit 301, or may acquire a three-dimensional model acquired from the storage unit 301, or may acquire a three-dimensional model from a model acquisition unit 302d described below.

[0115] The model acquisition unit 302d acquires a three-dimensional model from the three-dimensional reconstruction unit 302b and the storage unit 301. For example, the model acquisition unit 302d acquires a model in which the space A1 before the disaster is reconstructed from the storage unit 301, and acquires a model in which the space A1 after the disaster is reconstructed from the three-dimensional reconstruction unit 302b.

[0116] The model acquisition unit 302d may acquire two or more three-dimensional models from the storage unit 301. For example, the model acquisition unit 302d acquires a first three-dimensional model showing the space A1 and a second three-dimensional model showing the space A1 and different from the first three-dimensional model. Specifically, the model acquisition unit 302d acquires a first three-dimensional model showing the space A1 at a first time and a second three-dimensional model showing the space A1 at a second time different from the first time. In this embodiment, the model acquisition unit 302d acquires two three-dimensional models of the space A1 before and after a disaster (that is, a first three-dimensional model showing the space A1 before the disaster (first time) and a second three-dimensional model showing the space A1 after the disaster (second time)) from the storage unit 301. This makes it possible to calculate the difference between the models without performing a three-dimensional reconstruction process in the three-dimensional reconstruction unit 302b.

[0117] The three-dimensional model acquired by the model acquisition unit 302d may be a model calculated from image data outside the sediment volume measurement system 1000, or may be a model calculated from laser measurement outside the sediment volume measurement system 1000.

[0118] The representation method of the acquired three-dimensional model may be voxel data, mesh data, or point cloud data.

[0119] The positioning unit 302e performs a process of unifying the coordinate systems between the three-dimensional models using two or more three-dimensional models acquired by the model acquisition unit 302d and the attribute information calculated by the attribute information extraction unit 302c, that is, a process of aligning the three-dimensional models. That is, the positioning unit 302e aligns the first three-dimensional model and the second three-dimensional model based on the attribute information of each of the first three-dimensional model and the second three-dimensional model. Specifically, the positioning unit 302e can efficiently compare the models by comparing partial regions that are part of the space A1 including three-dimensional models having the same attribute information based on the attribute information.

[0120] The position matching unit 302e associates partial models with each other while changing the combination within partial models having the same attribute information.

[0121] Next, the position matching unit 302e compares the minimum structural units between partial models having at least one piece of the same attribute information, for example, three-dimensional points in the case of a three-dimensional model expressed as a point cloud.

[0122] Here, a partial model is a set of minimum structural units that represents a partial or entire area of ​​a three-dimensional model.

[0123] The registration unit 302e searches for and associates the nearest points between the partial models, and adjusts the position and orientation of the coordinate system so that the difference in the positions of the points is small. The registration unit 302e calculates a coordinate system that minimizes the difference by repeatedly changing the correspondence of the partial models and the correspondence of the points, and calculates a three-dimensional model that has been coordinate-converted into the calculated coordinate system.

[0124] As described above, the alignment unit 302e performs alignment (coordinate conversion and size adjustment) based on the attribute information. For example, the alignment unit 302e extracts three-dimensional models (partial models) having the same attribute information from each of the multiple three-dimensional models. The alignment unit 302e compares the extracted three-dimensional models to perform alignment. Furthermore, the alignment unit 302e changes the entire three-dimensional model in accordance with the partial model changed by the alignment. That is, the alignment unit 302e extracts partial models having attribute information common to the first and second three-dimensional models from each of the first and second three-dimensional models, and aligns the first and second three-dimensional models by moving each partial model so that the coordinates of the corresponding positions of each extracted partial model match, and then aligns the first and second three-dimensional models by moving the first and second three-dimensional models in accordance with the movement of each partial model.

[0125] This allows the registration unit 302e to align the three-dimensional model more efficiently than aligning the entire three-dimensional model all at once.

[0126] The converted model may be stored in the storage unit 301.

[0127] The correspondence between the partial models and the minimum constituent units may be random or may be selected based on the feature amount. The correspondence between the partial models and the minimum constituent units is not limited to the above method. In addition, when aligning colored models, information on color differences may be used for alignment.

[0128] The scale of the three-dimensional model may be adjusted to a desired scale for either the pre-disaster or post-disaster three-dimensional model. For example, it may be adjusted to the scale of space A1. The scale of the three-dimensional model may be adjusted based on scale information provided from outside the sediment volume measurement system 1000. By adjusting the scale of the three-dimensional model, the difference can be calculated at the same scale as the pre-disaster three-dimensional model and the post-disaster three-dimensional model. This allows the sediment volume to be calculated at the scale of space A1.

[0129] The difference calculation unit 302f calculates the amount of difference between the first and second three-dimensional models, and outputs attribute information of the difference and difference information indicating the amount of difference. Specifically, the difference calculation unit 302f calculates difference information between two or more three-dimensional models whose coordinate systems have been unified by the alignment unit 302e. The difference calculation unit 302f calculates, for example, the volume of a three-dimensional area covered by surfaces that do not overlap or intersect in mesh models of the same coordinate system.

[0130] The difference information calculated by the difference calculation unit 302f may include attribute information in an area where the difference is significant, for example, information on a mixture contained in the soil. Note that the above difference information is an example and is not intended to limit the present disclosure.

[0131] The difference calculation unit 302f, for example, outputs the calculated difference information to the user interface 400. This allows the user to operate the user interface 400 to check the difference information (that is, the amount and attributes of the soil 21, which is the difference amount).

[0132] Furthermore, for example, when the difference includes a plurality of pieces of attribute information that are different from each other, the difference calculation unit 302f classifies and outputs the plurality of pieces of attribute information by type, thereby displaying the plurality of pieces of attribute information by type together on a display device (for example, the user interface 400). In this case, for example, the difference calculation unit 302f calculates the amount included in the difference for each type.

[0133] [Processing Procedure] <Alignment process> Fig. 8 is a flowchart showing the process of aligning three-dimensional models executed by the sediment volume measurement system 1000 according to the embodiment. Fig. 9 is a diagram for explaining the alignment of two three-dimensional models executed by the alignment unit 302e.

[0134] The position matching unit 302e included in the sediment volume calculation device 300 in this embodiment first matches partial models having the same attribute information, for example, partial models labeled as "house", with each other for at least two or more three-dimensional models (S31). That is, in this case, for example, both of the two three-dimensional models include partial models having attribute information indicating "house".

[0135] For example, as shown in (a) of FIG. 9, the difference between the first three-dimensional model 500 and the second three-dimensional model 510 is calculated. In this case, the positioning unit 302e first extracts a partial model having attribute information included in both the first three-dimensional model 500 and the second three-dimensional model 510. Here, the positioning unit 302e extracts a first partial model 501 included in the first three-dimensional model 500 and having attribute information of "house", and a second partial model 511 included in the second three-dimensional model 510 and having attribute information of "house". Next, as shown in (b) of FIG. 9, the positioning unit 302e associates the first partial model 501 with the second partial model 511. Specifically, the positioning unit 302e extracts one or more feature points included in the first partial model 501 and one or more feature points included in the second partial model 511, and associates the extracted feature points of the first partial model 501 with the extracted feature points of the second partial model 511. Any feature points may be extracted here. For example, the feature points are corners in the partial model. The positioning unit 302e associates the same feature points of the first partial model 501 with the extracted feature points of the second partial model 511. For example, the positioning unit 302e associates the feature points 520 of the first partial model 501 with the extracted feature points 521 of the second partial model 511.

[0136] Referring again to FIG. 8, after step S31, the alignment unit 302e compares the smallest constituent units between partial models having at least one or more pieces of the same attribute information, for example, three-dimensional points in the case of a three-dimensional model represented by a point cloud, and searches for and matches the points between the partial models that are closest to each other (S32).

[0137] Next, the positioning unit 302e adjusts the position and orientation of the coordinate system of the partial model so that the difference between the positions of the points associated in step S32 becomes small (S33). For example, as shown in (c) of FIG. 9, the positioning unit 302e changes at least one of the first partial model 501 and the second partial model 511 so that the associated feature points are located at the same coordinates. For example, the positioning unit 302e moves at least one of the first partial model 501 and the second partial model 511 so that the associated feature points, that is, the feature points 520 and 521, are located at the same coordinates. In addition, the positioning unit 302e changes the entire three-dimensional model in accordance with the change of the partial model. As a result, the positioning unit 302e changes the first three-dimensional model 500 and the second three-dimensional model 510 so that models showing the same object are located at the same position and orientation.

[0138] The positioning unit 302e may adjust the scale of the three-dimensional model when adjusting the position and orientation. The scale may be adjusted to match one of the two three-dimensional models, or the scale of both of the two three-dimensional models may be adjusted based on scale information obtained from an external source.

[0139] 8 again, after step S33, the alignment unit 302e determines whether the change in position and orientation of the partial model (for example, at least one of the first partial model 501 and the second partial model 511 shown in FIG. 9) executed in step S33 is small (S34). If the change in position and orientation during adjustment is large, proceed to step S32, and if the change is small, proceed to step S35.

[0140] The amount of change used as a reference here may be arbitrarily determined in advance. For example, when the amount of change before and after the adjustment of the position and orientation of the partial model is equal to or greater than a predetermined reference amount of change, the alignment unit 302e next executes step S32. On the other hand, when the amount of change before and after the adjustment of the position and orientation of the partial model is less than the above-mentioned reference amount of change, the alignment unit 302e next executes step S35. The amount of change is, for example, the amount of movement of each point. When the entire partial model is rotated relative to a certain reference point, the amount of change may be the rotation angle caused by the rotation.

[0141] Next, the registration unit 302e compares the entire three-dimensional model based on the position and the adjustment result, and judges whether the overall difference is large (S35). If the overall difference is large, the process proceeds to step S36, and if the difference is small, the process proceeds to step S37.

[0142] The overall difference used as a reference here may be arbitrarily determined in advance. For example, when the difference between before and after the adjustment of the position and orientation of the entire three-dimensional model is equal to or greater than a predetermined reference difference, the alignment unit 302e next executes step S36. On the other hand, when the difference between before and after the adjustment of the position and orientation of the entire three-dimensional model is less than the above-mentioned reference difference, the alignment unit 302e next executes step S37. The difference here is, for example, the amount of points whose coordinates do not match between the three-dimensional points in the two three-dimensional models.

[0143] Next, the registration unit 302e judges whether all possible combinations have been compared in steps S31 and S32 (S36). If all combinations have been compared, the process proceeds to step S37, and if not, the process proceeds to step S31. Here, for example, when the process proceeds to step S31, the registration unit 302e extracts partial models that have not yet been compared between the first 3D model 500 and the second 3D model 510, and performs correspondence between the partial models.

[0144] 8 are merely examples and are not particularly limited. For example, the process may proceed to step S36 after step S33 is completed. Alternatively, the process may not include steps S34 and S35.

[0145] Finally, the registration unit 302e outputs the coordinate system that minimizes the overall difference and the three-dimensional model after coordinate transformation (S37).

[0146] <Sequence diagram> FIG. 10 is a sequence diagram showing the processing of the sediment volume measurement system 1000 according to the embodiment.

[0147] The sediment volume measurement system 1000 starts the sediment volume calculation process when a processing instruction is received from the user.

[0148] The control device 200 sends imaging instructions to the imaging devices 10a to 10n (only the imaging device 10a is shown in FIG. 10), sends the captured images received from the imaging devices 10a to 10n as imaging information to the user interface 400, and presents the imaging information to the user (S41). The imaging information is, for example, an image of the space A1.

[0149] The user operates the user interface 400 to check the imaging information, judge whether the imaging devices 10a-10n can properly image the desired space, and whether the imaging information is sufficient to start processing, and give instructions for processing (S42). Specifically, the user operates the user interface 400 to cause the user interface 400 to transmit information indicating the above-mentioned processing instructions to the control device 200.

[0150] The user processing instruction transmitted via the user interface 400 is information or a signal indicating whether to start the sediment amount calculation process or to have the imaging devices 10a-10n continue capturing images. The user processing instruction may include information specifying which imaging information is to be used in the sediment amount calculation process or information specifying the results to be output.

[0151] Next, the control device 200 determines whether to execute the sediment amount calculation process or to have the imaging devices 10a-10n continue capturing images (S43) based on the information indicating the user's processing instruction received from the user interface 400. When the control device 200 determines to start the sediment amount calculation process (Yes in S43), it transmits a sediment amount calculation command and the captured image to the sediment amount calculation device 300.

[0152] On the other hand, when the control device 200 determines that the imaging devices 10a to 10n should continue imaging (No in S43), it transmits information indicating an instruction to continue imaging to the imaging devices 10a to 10n, causing the imaging devices 10a to 10n to continue imaging.

[0153] Next, when the sediment amount calculation device 300 receives information indicating a sediment amount calculation command from the control device 200, the device executes a sediment amount calculation process using the captured video based on the information (S44). Step S44 is, for example, the process of steps S31 to S37 shown in Fig. 9. After completing the sediment amount calculation process, the sediment amount calculation device 300 transmits information indicating the process result to the control device 200.

[0154] Next, the control device 200 organizes the processing results received from the sediment volume calculation device 300 based on the user processing instructions, and outputs (transmits) difference information to the user interface 400 (S45).

[0155] The user interface 400 presents the difference information received from the control device 200 to the user (S46). A specific example of the difference information presented to the user will be described later.

[0156] <Summary of processing steps> FIG. 11 is a flowchart showing a process for calculating the amount of sediment executed by the sediment amount measurement system 1000 according to the embodiment.

[0157] In the process of calculating the amount of sediment in the sediment amount calculation device 300 according to this embodiment, first, the image acquisition unit 302a acquires a plurality of images captured by the plurality of imaging devices 10a to 10n, respectively (S51).

[0158] Next, the three-dimensional reconstruction unit 302b executes a three-dimensional reconstruction process using the multiple images (S52). For example, the three-dimensional reconstruction unit 302b generates a first three-dimensional model that represents the space A1 and has attribute information for each region, and a second three-dimensional model that represents the space A1 and has attribute information for each region, and is different from the first three-dimensional model (for example, at a different time).

[0159] Next, the model acquisition unit 302d acquires two or more three-dimensional models from the three-dimensional reconstruction unit 302b or the storage unit 301 (S53). Note that one or more models may be acquired from the three-dimensional reconstruction unit 302b, one or more models may be acquired from the storage unit 301, or two or more models may be acquired from the storage unit 301.

[0160] Next, the attribute information extraction unit 302c extracts attribute information from the multiple images captured by the imaging devices 10a to 10n or the three-dimensional model (S54). Note that step S54 may be performed before step S53, after step S53, or both.

[0161] Next, the alignment unit 302e performs a process of unifying coordinates based on two or more three-dimensional models and attribute information (S55). Specifically, the alignment unit 302e aligns the first three-dimensional model and the second three-dimensional model based on the attribute information of each of the first three-dimensional model and the second three-dimensional model. For example, the alignment unit 302e detects corresponding points that are the same points in the first three-dimensional model and the second three-dimensional model based on the attribute information of each of the first three-dimensional model and the second three-dimensional model, and aligns the first three-dimensional model and the second three-dimensional model using the detected corresponding points. The alignment unit 302e can efficiently and accurately align the first three-dimensional model and the second three-dimensional model by comparing areas having the same attribute information in the three-dimensional model (i.e., partial models having the same attribute information).

[0162] Next, the difference calculation unit 302f calculates difference information between two or more models whose coordinate systems have been unified by the alignment unit 302e (i.e., aligned by the alignment unit 302e) (S56). The difference information includes, for example, difference information indicating the amount of difference and attribute information of the difference. Furthermore, the difference calculation unit 302f outputs, for example, the calculated difference information to the user interface 400 (more specifically, to the user interface 400 via the control device 200). The user interface 400 presents (more specifically, displays) the acquired difference information to the user.

[0163] [Example output] FIG. 12 is a diagram showing an example of an image 600 presented by the user interface 400 included in the sediment volume measurement system 1000 according to the embodiment when aligning two three-dimensional models.

[0164] The user interface 400 is realized, for example, by using a mobile terminal such as a smartphone or tablet terminal having a display device such as a touch panel display and an input device.

[0165] Note that the user may arbitrarily select attributes to be used for alignment using a tablet-type mobile terminal as shown in FIG.

[0166] For example, an image 600 includes a three-dimensional image 601 , an object selection portion 602 , and a determination portion 603 .

[0167] A three-dimensional image 601 is an image showing a three-dimensional model after a disaster.

[0168] The target selection unit 602 is an image for accepting the selection of a partial model having attribute information that the positioning unit 302e preferentially associates. For example, the user touches the attribute information or the partial model having the attribute information that the user wants to preferentially associate with the positioning unit 302e in the three-dimensional image 601 or the target selection unit 602. The user interface 400 transmits, for example, information indicating the attribute information touched by the user or the partial model having the attribute information to the control device 200 together with information indicating a user processing instruction after step S42 shown in FIG. 10. The control device 200 transmits, for example, information indicating the attribute information touched by the user or the partial model having the attribute information to the sediment amount calculation device 300 together with a sediment amount calculation command and imaging information. For example, in step S31, the positioning unit 302e first selects and associates a partial model based on the attribute information touched by the user or the information indicating the partial model having the attribute information.

[0169] FIG. 13 is a diagram showing an example of difference information presented by the user interface 400 included in the sediment volume measurement system 1000 according to the embodiment.

[0170] The user interface 400 displays the result of the amount of sediment, which is an example of the calculated difference amount, using a tablet-type mobile terminal such as that shown in Fig. 13. The user interface 400 may display the estimated weight of the amount of sediment as a calculation result in addition to the volume of the amount of sediment.

[0171] The user interface 400 may also display the mixture of the soil and sand using the attribute information obtained by the attribute information extraction unit 302c. This allows the information required for removing the soil and sand and the mixture to be obtained efficiently. The method of displaying the mixture is not limited to text, and may be achieved by coloring the corresponding portion on the three-dimensional model or image.

[0172] For example, the user interface 400 presents difference information to the user by displaying an image 610. For example, the image 610 includes a three-dimensional image 611, a highlight line 612, a difference amount information image 613, and an attribute information image 614.

[0173] The three-dimensional image 611 is an image showing a three-dimensional model after a disaster.

[0174] The highlight line 612 is a line for indicating the position, area, etc. of the difference between the 3D model before the disaster and the 3D image 611. In Fig. 13, the highlight line 612 is shown as a dashed circle for the sake of explanation, but the highlight line 612 may be any shape, such as a solid line, a dashed line, a thick line, a colored line that is colored differently from the other parts, a rectangle, a polygon, etc.

[0175] The difference amount information image 613 is an image for presenting the difference amount to the user. The difference amount information image 613 includes information indicating, for example, the volume of the difference and the estimated weight of the soil.

[0176] The attribute information image 614 is an image for presenting to the user attribute information of the partial model included in the difference. For example, the attribute information image 614 includes information indicating "soil and sand," "driftwood," "sign," "traffic light," etc. as attribute information of the partial model included in the difference. In this example, the attribute information image 614 also includes the ratio of the volume or weight of the part indicated in the attribute information to the entire difference. In this way, the amount of difference may be displayed for each attribute.

[0177] In this example, the attribute information image 614 includes a first type image 615 and a second type image 616 .

[0178] The first type image 615 is an image for collectively presenting attributes that satisfy a first condition among a plurality of attributes included in the difference.

[0179] The second type image 616 is an image for collectively presenting attributes that satisfy a second condition, among a plurality of attributes included in the difference.

[0180] In this way, the user interface may display attribute information grouped by attribute type and the amount of partial models having the attribute information.

[0181] In this example, the first condition is a natural object, and the second condition is an artificial object. These conditions may be arbitrarily determined in advance, and are not particularly limited.

[0182] [Effects, etc.] As described above, a quantity calculation device according to one embodiment of the present disclosure (e.g., sediment volume measurement system 1000, more specifically, sediment volume calculation device 300) includes an acquisition unit (e.g., model acquisition unit 302d) that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which represents the same predetermined space (e.g., space A1) and has attribute information for each region (e.g., for each partial model or for each three-dimensional point that constitutes a three-dimensional model), an alignment unit 302e that aligns the first three-dimensional model and the second three-dimensional model based on attribute information held by each of the first and second three-dimensional models aligned by the alignment unit, and a calculation unit (e.g., difference calculation unit 302f) that calculates the amount of difference between the first and second three-dimensional models and outputs the attribute information held by the difference and the amount of the difference.

[0183] Conventionally, there is a method for calculating the amount of sediment that has flowed into a given space due to the effects of collapses, etc., at a landslide disaster site, for example. Calculating the amount of sediment using laser measurement can calculate the amount with high accuracy, but there is a problem that the calculation takes time and costs. Therefore, in the quantity calculation device according to the present disclosure, for example, a three-dimensional model (first three-dimensional model) showing a given space in a state where an object for which the amount of sediment, etc. is to be calculated does not exist in the given space is compared with a three-dimensional model (second three-dimensional model) showing the given space in a state where the object exists. According to this, by appropriately aligning the first three-dimensional model and the second three-dimensional model, the difference between the first three-dimensional model and the second three-dimensional model can be easily calculated. Therefore, according to the quantity calculation device according to the present disclosure, the processing time for calculating the amount of an object can be shortened.

[0184] Also, for example, the quantity calculation device according to the present disclosure further includes a generation unit (e.g., a three-dimensional reconstruction unit 302b) that generates at least one of a first three-dimensional model and a second three-dimensional model from a plurality of images each showing a specified space.

[0185] According to this, the amount of difference can be calculated using a simple configuration such as a camera for generating an image.

[0186] Furthermore, for example, when the calculation unit includes multiple pieces of attribute information with mutually different differences, the calculation unit classifies the multiple pieces of attribute information by type and outputs the multiple pieces of attribute information, thereby displaying the multiple pieces of attribute information together by type on a display device (e.g., user interface 400).

[0187] According to this, the types of objects included in the differences are classified and displayed on the display device, making it easier for a user checking the display device to understand the types included in the differences.

[0188] Also, for example, the calculation unit calculates the amount included in the difference for each type.

[0189] According to this, the amount of each type of object included in the difference is displayed on the display device, making it easier for a user checking the display device to understand the amount of each type included in the difference.

[0190] Also, for example, the amount of difference is at least one of the volume and weight of the difference.

[0191] According to this, the quantity calculation device according to the present disclosure can calculate at least one of the difference amount and volume.

[0192] Also, for example, the alignment unit extracts partial models having attribute information common to the first three-dimensional model and the second three-dimensional model from each of the first three-dimensional model and the second three-dimensional model, aligns the partial models by moving each of the extracted partial models so that the coordinates of corresponding positions of each of the extracted partial models match, and then aligns the first three-dimensional model and the second three-dimensional model by moving the first three-dimensional model and the second three-dimensional model in accordance with the movement of each of the partial models.

[0193] This allows the quantity calculation device according to the present disclosure to align the first and second three-dimensional models without time-consuming processing such as alignment by comparing the shapes of the three-dimensional models, etc. Therefore, the processing time of the quantity calculation device according to the present disclosure can be further shortened.

[0194] Furthermore, a quantity calculation method according to one embodiment of the present disclosure includes an acquisition step (e.g., step S53) of acquiring a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region, an alignment step (e.g., step S55) of aligning the first three-dimensional model and the second three-dimensional model based on the attribute information possessed by each of the first and second three-dimensional models, and a calculation step (e.g., step S56) of calculating the amount of difference between the first and second three-dimensional models and outputting attribute information possessed by the difference and difference information indicating the amount of the difference.

[0195] According to this, by appropriately aligning the first 3D model and the second 3D model, it is possible to easily calculate the difference between the first 3D model and the second 3D model. Therefore, according to the quantity calculation method according to the present disclosure, it is possible to shorten the processing time for calculating the amount of an object.

[0196] Also, for example, in the acquisition step, a first three-dimensional model showing a specified space before the sediment is flowed in and a second three-dimensional model showing the specified space after the sediment is flowed in are acquired, and in the calculation step, the amount of sediment is calculated as the amount of the difference.

[0197] According to this, by appropriately aligning the three-dimensional models showing a predetermined space before and after a disaster in which sediment flows into the predetermined space, the amount of sediment, which is the difference, can be easily calculated. Therefore, according to the quantity calculation method disclosed herein, the processing time for calculating the amount of sediment can be shortened.

[0198] (Other embodiments) Although the quantity calculation device and the like according to the present disclosure have been described based on the above-described embodiment, the present disclosure is not limited to the above-described embodiment.

[0199] For example, in the above embodiment, the processing unit of the quantity calculation device or the like is described as being realized by a CPU and a control program. For example, each of the components of the processing unit may be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an IC (Integrated Circuit), or an LSI (Large Scale Integration). The IC or LSI may be integrated into one chip or into multiple chips. Here, the IC or LSI is referred to as an IC or LSI, but the name may change depending on the degree of integration, and may be called a system LSI, a VLSI (Very Large Scale Integration), or an ULSI (Ultra Large Scale Integration). In addition, a field programmable gate array (FPGA) that is programmed after the LSI is manufactured can also be used for the same purpose.

[0200] In addition, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the present disclosure may be realized as a computer-readable non-transitory recording medium such as an optical disk, a hard disk drive (HDD), or a semiconductor memory in which the computer program is stored. In addition, the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0201] In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to another functional block. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0202] In addition, the order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and the steps may be performed in an order other than the above. In addition, some of the steps may be performed simultaneously (in parallel) with other steps.

[0203] In addition, the present disclosure also includes forms obtained by applying various modifications to the above-described embodiments that would come to mind by a person skilled in the art, and forms realized by arbitrarily combining the components and functions of the embodiments without departing from the spirit of the present disclosure. [Industrial Applicability]

[0204] The present disclosure can be applied to a quantity calculation device that calculates a difference from a three-dimensional model, and can be applied to, for example, a device that calculates the amount of an object. [Explanation of symbols]

[0205] 10a, 10b, 10n Imaging device 20 road 21 Soil 22 Three-dimensional model 100 Cameras 101, 201, 301 Storage section 102, 202 Control unit 103 Optical system 104 Image Sensor 110 Mount 200 Control device 202a Imaging control unit 202b User instruction acquisition unit 202c Sediment volume calculation command unit 203 Timer 300 Sediment volume calculation device 302 Processing section 302a Image acquisition unit 302b 3D reconstruction unit 302c Attribute information extraction part 302d Model Acquisition Department 302e Alignment section 302f Difference calculation part 400 User Interface 500 1st 3D model 501 1st part model 510 2nd 3D Model 511 2nd part model 520, 521 feature points 600, 610 images 601, 611 3D images 602 Target Selection Section 603 Decision Section 612 Emphasis line 613 Differential Information Image 614 Attribute Information Image 615 First type image 616 Second type image 1000 Sediment volume measurement system A1 Three-dimensional space (space)

Claims

1. an acquisition unit that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; a user interface that accepts an operation of indicating attribute information that has priority over other attribute information among at least two pieces of attribute information that each of the first three-dimensional model and the second three-dimensional model has; a registration unit that performs registration between the first 3D model and the second 3D model based on attribute information indicated by an operation accepted by the user interface; a calculation unit that calculates a difference between the first 3D model and the second 3D model aligned by the alignment unit, and outputs an amount of the difference and attribute information that is included in the second 3D model and corresponds to the difference. Quantity calculation device.

2. The attribute information corresponding to the difference indicates that the difference is a mixture. The quantity calculation device according to claim 1 .

3. Attribute information corresponding to the difference indicates that the difference is a natural object or an artificial object. The quantity calculation device according to claim 1 .

4. The apparatus further includes a generating unit that generates at least one of the first three-dimensional model and the second three-dimensional model from a plurality of images showing the predetermined space. The quantity calculation device according to any one of claims 1 to 3.

5. When the difference includes a plurality of pieces of attribute information, the calculation unit classifies the plurality of pieces of attribute information by type and outputs the classified information, thereby causing the display device to display the plurality of pieces of attribute information collectively by type. The quantity calculation device according to any one of claims 1 to 4.

6. The calculation unit calculates the amount of the difference for each of the types. The quantity calculation device according to claim 5.

7. The amount of the difference is at least one of the volume and weight of the difference. The quantity calculation device according to any one of claims 1 to 6.

8. The alignment unit extracts partial models having attribute information common to the first and second three-dimensional models from each of the first and second three-dimensional models, aligns the positions by moving each of the partial models so that coordinates of corresponding positions of each of the extracted partial models match, and then aligns the positions of the first and second three-dimensional models by moving the first and second three-dimensional models in accordance with the movements of each of the partial models. The quantity calculation device according to any one of claims 1 to 7.

9. The position alignment unit aligns the first three-dimensional model and the second three-dimensional model in the partial region, which is a part of the predetermined space, based on first partial attribute information of the first three-dimensional model in the partial region and second partial attribute information of the second three-dimensional model in the partial region, the second partial attribute information being the same as the first partial attribute information. The quantity calculation device according to any one of claims 1 to 7.

10. The alignment unit moves at least one of the first three-dimensional model and the second three-dimensional model in the predetermined space other than the partial region in accordance with movement of at least one of the first three-dimensional model and the second three-dimensional model due to the alignment of the first three-dimensional model and the second three-dimensional model in the partial region. The quantity calculation device according to claim 9.

11. an attribute information extraction unit that extracts attribute information of the first three-dimensional model and attribute information of the second three-dimensional model in a state in which the first three-dimensional model and the second three-dimensional model have not been aligned by the alignment unit; The alignment unit aligns the first and second three-dimensional models using attribute information of the first and second three-dimensional models extracted by the attribute information extraction unit. The quantity calculation device according to any one of claims 1 to 10.

12. a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of which indicates the same predetermined space and has attribute information for each region; and an acquisition step of acquiring a receiving step of receiving, through a user interface, an operation indicating attribute information that has priority over other attribute information, among at least two pieces of attribute information that each of the first 3D model and the second 3D model has; a registration step of performing registration between the first 3D model and the second 3D model based on attribute information indicated by an operation accepted by the user interface; a calculation step of calculating a difference between the first 3D model and the second 3D model aligned in the alignment step, and outputting an amount of the difference and attribute information of the second 3D model corresponding to the difference. Quantity calculation method.

13. In the acquisition step, the first three-dimensional model showing the predetermined space before soil and sand are flowed in and the second three-dimensional model showing the predetermined space after soil and sand are flowed in are acquired, In the calculation step, the amount of the soil is calculated as the amount of the difference. The method for calculating a quantity of material according to claim 12.

14. an attribute information extraction step of extracting attribute information of the first three-dimensional model and attribute information of the second three-dimensional model in a state in which the first three-dimensional model and the second three-dimensional model have not been aligned in the alignment step, In the positioning step, the first three-dimensional model and the second three-dimensional model are aligned using the attribute information of the first three-dimensional model extracted in the attribute information extraction step and the attribute information of the second three-dimensional model. The method for calculating a quantity of material according to claim 12 or 13.

15. A method for causing a computer to execute the method for calculating the quantity of material according to any one of claims 12 to 14. program.

Images