Information processing apparatus

Abstract

Provided is an information processing apparatus including: a storage; and a processor connected to the storage, which acquires, using the storage, a first three-dimensional model representing a prescribed space in an actual space and a second three-dimensional model including a space common to the prescribed space, generates difference data representing a difference between the first three-dimensional model and the second three-dimensional model, and the difference data includes attribute information representing an attribute of a region included in the difference data, which is generated through recognition processing.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202080018282.1 (International Application No. PCT / JP2020 / 006390), filed on February 19, 2020, entitled "Material Quantity Calculation Device and Material Quantity Calculation Method". Technical Field

[0002] This disclosure relates to a quantity calculation device and a quantity calculation method. Background Technology

[0003] There is a technique for measuring quantities such as volume or weight related to an object located in a specified space (for example, see Patent Document 1).

[0004] In the technology disclosed in Patent Document 1, images of the collapse site before and after the collapse are compared to align the images, and a three-dimensional model is generated using the aligned images. The extent of damage at the collapse site is then calculated based on the generated three-dimensional model.

[0005] (Existing technical literature) (Patent Documents) Patent Document 1: Japanese Patent Application Publication No. 2002-328021 Regarding the calculation of the quantity of objects located in the specified space, such as the aforementioned disaster-affected quantity, it is hoped that the processing time can be shortened. Summary of the Invention

[0006] This disclosure provides a quantity calculation device, etc., that can shorten the processing time for calculating the quantity of an object.

[0007] One aspect of the information processing apparatus disclosed herein includes: a memory; and a processor connected to the memory, the processor using the memory to obtain a first three-dimensional model representing a predetermined space in actual space and a second three-dimensional model including a space common to the predetermined space, and generating difference data representing the difference between the first three-dimensional model and the second three-dimensional model, the difference data including attribute information representing the attributes of the regions included in the difference data, generated by identification processing.

[0008] One aspect of this disclosure relates to a quantity calculation apparatus comprising: an acquisition unit that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are each three-dimensional models representing the same defined space and having attribute information for each region; an alignment unit that aligns the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation unit that calculates the amount of difference between the first three-dimensional model and the second three-dimensional model after alignment by the alignment unit, and outputs the attribute information possessed by the difference and the amount of the difference.

[0009] Furthermore, one embodiment of the present disclosure involves a quantity calculation method comprising: an obtaining step, obtaining a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are three-dimensional models representing the same defined space and having attribute information for each region; an alignment step, aligning the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation step, calculating the quantity of the difference between the first three-dimensional model and the second three-dimensional model after alignment in the alignment step, and outputting the attribute information possessed by the difference and the quantity of the difference.

[0010] Furthermore, this disclosure can also be implemented as a program that enables a computer to perform the steps included in the above-described quantity calculation method. Moreover, this disclosure can also be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer storing the program. Furthermore, this disclosure can also be implemented as information, data, or signals representing the program. Moreover, those programs, information, data, and signals can be distributed via communication networks such as the Internet.

[0011] This disclosure provides a quantity calculation device and the like that can shorten the processing time for calculating the quantity of an object. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the general outline of the sediment measurement system involved in the embodiments.

[0013] Figure 2 This is a diagram illustrating the outline of the process performed by the sediment measurement system involved in the embodiments to extract attribute information.

[0014] Figure 3 This is a diagram illustrating the general process of sediment quantity calculation performed by the sediment quantity measurement system involved in the embodiments.

[0015] Figure 4This is a block diagram illustrating the structure of the sediment measurement system involved in the embodiment.

[0016] Figure 5 This is a block diagram illustrating the structure of the camera device included in the sediment measurement system described in the embodiment.

[0017] Figure 6 This is a block diagram illustrating the structure of the control device included in the sediment measurement system of the embodiment.

[0018] Figure 7 This is a block diagram illustrating the structure of the sediment quantity calculation device included in the sediment quantity measurement system of the embodiment.

[0019] Figure 8 This is a flowchart illustrating the alignment process of the three-dimensional model performed by the sediment measurement system involved in the embodiment.

[0020] Figure 9 It is a diagram used to illustrate the alignment of two three-dimensional models performed by the alignment section.

[0021] Figure 10 This is a sequence diagram illustrating the processing of the sediment measurement system involved in the embodiment.

[0022] Figure 11 This is a flowchart illustrating the sediment quantity calculation process performed by the sediment quantity measurement system involved in the embodiment.

[0023] Figure 12 This is an example of an image showing a prompt from the user interface of the sediment measurement system described in the embodiment when aligning two three-dimensional models.

[0024] Figure 13 This is a diagram illustrating an example of the difference information displayed by the user interface of the sediment measurement system involved in the embodiment. Detailed Implementation

[0025] (Summary of this disclosure) One aspect of this disclosure relates to a quantity calculation apparatus comprising: an acquisition unit that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are each three-dimensional models representing the same defined space and having attribute information for each region; an alignment unit that aligns the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation unit that calculates the amount of difference between the first three-dimensional model and the second three-dimensional model after alignment by the alignment unit, and outputs the attribute information possessed by the difference and the amount of the difference.

[0026] Conventionally, methods have been used to measure (calculate) the amount of sediment flowing into a specified space due to the effects of landslides or other geological disasters. While laser-based sediment quantity calculations can achieve high accuracy, they are time-consuming and costly. Therefore, in the quantity calculation apparatus of this disclosure, for example, a three-dimensional model (first three-dimensional model) of the specified space representing the absence of an object whose quantity of sediment is to be calculated is compared with a three-dimensional model (second three-dimensional model) representing the presence of that object. By appropriately aligning the first and second three-dimensional models, the difference between them can be easily calculated. Therefore, the quantity calculation apparatus of this disclosure reduces the processing time for calculating the quantity of an object.

[0027] Furthermore, for example, one embodiment of the mass calculation device disclosed herein also includes a generation unit that generates at least one of the first three-dimensional model and the second three-dimensional model based on a plurality of images representing the defined space.

[0028] Therefore, the amount of difference can be calculated using a simple structure such as a camera used to generate images.

[0029] Furthermore, for example, the calculation unit, when the difference includes multiple attribute information that are different from each other, categorizes and outputs the multiple attribute information by each category, thereby displaying the multiple attribute information together on the display device according to each category.

[0030] Accordingly, the categories of objects included in the difference are categorized and displayed on the display device, thus enabling users who wish to examine the display device to easily understand the categories included in the difference.

[0031] Furthermore, for example, the calculation unit calculates the amount included in the difference for each of the categories.

[0032] Accordingly, the quantity of each category of objects included in the difference is displayed on the display device, thus enabling the user who wants to verify the display device to easily understand the quantity of each category included in the difference.

[0033] Furthermore, for example, the amount of the difference is at least one of the volume and weight of the difference.

[0034] Accordingly, the quantity calculation device disclosed herein is capable of calculating at least one of the difference in quantity and volume.

[0035] Furthermore, for example, the alignment section extracts partial models with common attribute information from the first three-dimensional model and the second three-dimensional model respectively. After aligning the partial models by moving them so that the coordinates of the corresponding positions of the extracted partial models are consistent, the first three-dimensional model and the second three-dimensional model are moved according to the movement of the partial models, thereby aligning the first three-dimensional model and the second three-dimensional model.

[0036] Accordingly, the mass calculation device of this disclosure can align the first three-dimensional model and the second three-dimensional model even without performing time-consuming processing such as alignment by comparing the shapes of the three-dimensional models. Therefore, the processing time of the mass calculation device of this disclosure can be further shortened.

[0037] Furthermore, one embodiment of the present disclosure involves a quantity calculation method comprising: an obtaining step, obtaining a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are three-dimensional models representing the same defined space and having attribute information for each region; an alignment step, aligning the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation step, calculating the quantity of the difference between the first three-dimensional model and the second three-dimensional model after alignment in the alignment step, and outputting the attribute information possessed by the difference and the quantity of the difference.

[0038] Accordingly, by properly aligning the first three-dimensional model and the second three-dimensional model, the difference between the first three-dimensional model and the second three-dimensional model can be easily calculated. Therefore, the quantity calculation method disclosed herein can shorten the processing time for calculating the quantity of an object.

[0039] Furthermore, for example, in the obtaining step, a first three-dimensional model representing the specified space before the sediment inflow and a second three-dimensional model representing the specified space after the sediment inflow are obtained; in the calculation step, the amount of sediment is calculated as the amount of the difference.

[0040] Accordingly, by properly aligning a three-dimensional model of the specified space before and after a disaster, representing the inflow of sediment into that space, the amount of sediment, which becomes differential, can be easily calculated. Therefore, the material quantity calculation method disclosed herein can shorten the processing time for calculating the amount of sediment.

[0041] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, each embodiment described below is merely a specific example illustrating the present disclosure. Therefore, the numerical values, shapes, materials, constituent elements, arrangement of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the scope of the present disclosure.

[0042] Furthermore, each figure is a schematic diagram and not a rigorous illustration. Also, in each figure, substantially identical components are assigned the same number, and sometimes repeated descriptions are omitted or simplified.

[0043] (Implementation Method) [summary] Figure 1 This is a diagram illustrating the general outline of the sediment measurement system involved in the embodiments.

[0044] The sediment measurement system 1000 involved in the implementation method (for example, refer to...) Figure 4 The system is equipped with multiple cameras 100 for capturing images (hereinafter also referred to as photographs) within a defined three-dimensional space A1 (hereinafter also referred to as space A1). As a specific example, space A1 is the space on a road 20 where silt 21 is deposited. As another specific example, space A1 is the space on a river where silt 21 or driftwood is deposited. The silt quantity measurement system 1000 is an example of the quantity calculation apparatus involved in this disclosure.

[0045] Each of the multiple cameras 100 is positioned at a different location and captures images of a common three-dimensional space (e.g., space A1). Therefore, the areas captured by the multiple cameras 100 within space A1 have at least a portion of overlapping regions. For example, the multiple cameras 100 are positioned at different locations to surround a portion of space A1 containing mud and sand 21 on the road 20. Furthermore, the multiple cameras 100 are each in a different orientation. Moreover, at least a portion of the areas captured by the multiple cameras 100 overlaps. This overlap is due to the generation (reconstruction) of a three-dimensional model (e.g., point group data) of space A1 based on the image data obtained through the cameras.

[0046] In addition, the area that becomes the subject of the camera can be repeated by a portion of the multiple cameras 100, or it can be repeated by all of the multiple cameras 100.

[0047] Furthermore, multiple cameras 100 can also be cameras that record video at different times using the same camera, i.e., mobile cameras. Therefore, if camera 100 is a mobile camera, then there can be more than one camera 100, and there is no limit to the number.

[0048] In the 3D reconstruction process (the process of generating a 3D model of space A1), a 3D model 22 representing space A1 is calculated (generated) using multiple images (hereinafter also referred to as images) obtained from multiple cameras 100 respectively. In the 3D reconstruction process, for example, a correspondence is established between the multiple images obtained from multiple cameras 100 respectively, and geometric calculations are performed based on the obtained correspondence to calculate the 3D model 22.

[0049] Here, a 3D model is a type of data that represents the geometry of a defined 3D space (space A1 in this embodiment). The model used to represent the 3D shape is, for example, a group of points consisting of multiple 3D points. Furthermore, the 3D model can be represented using voxels or meshes.

[0050] Furthermore, these representations of 3D models are just examples and do not limit the methods of representing 3D models.

[0051] Furthermore, the data used to generate the 3D model does not necessarily have to be an image; there are no particular restrictions. For example, the data used to generate the 3D model can also be data obtained from measuring instruments such as LiDAR.

[0052] LiDAR is a sensor used to measure the distance to objects. By scanning (moving and measuring) a defined area, the position of an object located within that area can be calculated.

[0053] Such measuring instruments as LiDAR generate three-dimensional models by emitting electromagnetic waves and obtaining the reflected waves from the object being measured. Specifically, the measuring instrument measures the time required from the emission of the electromagnetic wave to its reflection from the object and back to the measuring instrument. Using the measured time and the wavelength of the electromagnetic wave, it calculates the distance between the measuring instrument and a point on the surface of the object being measured.

[0054] For example, a measuring instrument emits electromagnetic waves from its reference point in multiple predetermined radial directions. For instance, the measuring instrument emits electromagnetic waves at first angular intervals around the horizontal direction and at second angular intervals around the vertical direction. Therefore, by calculating the distances between the measuring instrument and the object being measured in multiple directions around the measuring instrument, the measuring instrument can calculate the three-dimensional coordinates of multiple points on the object being measured.

[0055] Measuring instruments, for example, irradiate with laser light as an electromagnetic wave. Alternatively, measuring instruments, for example, measure the distance between themselves and the object being measured by emitting millimeter waves as electromagnetic waves.

[0056] Figure 2 This is a diagram illustrating the outline of the process for extracting attribute information performed by the sediment measurement system 1000 according to the embodiments. Specifically, Figure 2 (a) is a diagram representing an example of an image generated by photographing space A1. Figure 2 (b) is a diagram showing the three-dimensional model of space A1. Figure 2 (c) is an enlarged representation of having Figure 2 The image shown in (a) is a graph of the pixel attributes of the "tree".

[0057] The sediment measurement system 1000 uses multiple image data from multiple cameras 100 that capture images of space A1 or reconstructs a three-dimensional model of space A1 to extract attribute information.

[0058] Here, attribute information refers to information indicating the meaning of a specific region in an image or 3D model. Attribute information is, for example, information indicating that an object included in any region of an image is a "tree." Furthermore, as another example, attribute information is information indicating that any partial group of points in a 3D model represented as a group of points is a "tree." That is, "tree," "driftwood," "mud," and "house" are examples of attribute information. Thus, for example, a 3D model has attribute information for each region. This region can be a predetermined area within the 3D model, or it can be each 3D point constituting the 3D model; there is no particular limitation. Furthermore, this region can also be each partial model described later. That is, a 3D model can also have attribute information for each partial model described later.

[0059] Extracting attribute information is achieved, for example, through semantic segmentation. Semantic segmentation is a process that assigns meaningful labels to the surrounding information of a pixel or the smallest constituent unit of a 3D model.

[0060] Specifically, for example, semantic segmentation is, such as Figure 2 As shown in (c), the process of assigning a "tree" label to each pixel of a "tree" existing in a defined three-dimensional space is described. Labels can be assigned to each pixel by referring to information from surrounding pixels. This information can be R(Red), G(Green), B(Blue) values, brightness values, pixel value distributions, or gradients. Accordingly, for example, a set of points assigned "tree" attribute information, i.e., a partial model, can be considered as a partial model possessing "tree" attribute information.

[0061] The extraction of attribute information is performed, for example, through pattern recognition based on data representing a pre-determined "tree," etc. The user interface 400 (see below) is used... Figure 4 This allows for the manipulation of attribute information to establish a correspondence between the image or the objects included in the 3D model.

[0062] Figure 3 This is a diagram illustrating a summary of the sediment quantity calculation process performed by the sediment quantity measurement system 1000 according to the embodiment. Specifically, Figure 3 (a) is a diagram representing a three-dimensional model of space A1 before the disaster. Furthermore, Figure 3 (b) is a diagram representing a three-dimensional model of space A1 after the disaster. Furthermore, Figure 3 (c) is a diagram representing the processing of the calculated difference information performed by the sediment measurement system 1000. Figure 3 (d) is a graph representing an example of the difference information output by the sediment measurement system 1000.

[0063] The sediment measurement system 1000 extracts information (difference information) about sediment 21 by calculating the difference between a pre-disaster three-dimensional model (hereinafter also referred to as the first three-dimensional model) and a post-disaster three-dimensional model (hereinafter also referred to as the second three-dimensional model) of reconstructed space A1.

[0064] The sediment measurement system 1000 extracts the most significant changes between pre- and post-disaster conditions from two 3D models. The calculated differences are then presented to the user as discrepancy information.

[0065] Here, the difference information is, for example, the volume of a 3D region in a 3D mesh model where the faces of the pre-disaster 3D model and the post-disaster 3D model do not overlap or intersect. The amount of difference thus calculated is, for example, the amount of sediment deposited in a given 3D space after the disaster.

[0066] Furthermore, the difference information can also include information about the mixture, including multiple different attribute information obtained by extracting attribute information from the parts of the two 3D models where the changes before and after the disaster are more obvious, such as the proportion of driftwood or rubble included in sediment 21.

[0067] A mixture is, for example, a collection of multiple partial models having distinct attribute information. In the example above, the mixture is silt. Furthermore, the mixture (i.e., silt) is a three-dimensional model composed of partial models (part 1 model) having attribute information representing "driftwood" and partial models (part 2 model) having attribute information representing "rubble." The difference information, for example, includes information on one or more attributes included in the mixture, includes the volume of the mixture as a measure of difference, and includes information indicating the proportion of the volume of the mixture occupied by the part 1 model and the proportion occupied by the part 2 model.

[0068] Furthermore, the mixture can be classified into multiple categories based on each type of attribute information. For example, suppose the mixture consists of a partial model (part 1 model) with attribute information representing "driftwood," a partial model (part 2 model) with attribute information representing "rubble," a partial model (part 3 model) with attribute information representing "road sign," and a partial model (part 4 model) with attribute information representing "traffic light." In this case, the mixture can also be classified as a mixture composed of part 1 and part 2 models made of natural materials (mixture 1) and a mixture composed of part 3 and part 4 models made of man-made materials (mixture 2). Furthermore, in this case, the difference information can also include information indicating the proportion of mixture 1 and mixture 2 in the volume of the mixture.

[0069] As described above, the difference information includes, for example, difference information representing the quantity of the difference between the two 3D models and attribute information of the difference. The quantity of the difference is, for example, at least one of the volume and weight of the difference.

[0070] Furthermore, the above is an example of discrepancy information and is not limited to discrepancy information.

[0071] Furthermore, regarding the scale of the 3D model, either the pre-disaster or post-disaster 3D model can be adjusted to the desired scale. For example, either the pre-disaster or post-disaster 3D model can be adjusted to the scale of space A1. Moreover, the scale of either the pre-disaster or post-disaster 3D model can be adjusted based on scale information provided externally from the sediment measurement system 1000. By adjusting the scale in a manner that makes the 3D models identical, it is possible to perform differential calculations on the pre-disaster and post-disaster 3D models at the same scale. Accordingly, sediment volume can be calculated at the scale of space A1.

[0072] Furthermore, the sediment measurement system 1000 is an example of the quantity calculation device disclosed herein. The method for calculating the aforementioned discrepancy information can be used to calculate the inventory of objects other than sediment, such as warehouse stock, or to calculate the quantity of construction components at a work site during construction work.

[0073] [structure] <Sediment Measurement System> Next, the structure of the sediment measurement system 1000 will be described in detail.

[0074] Figure 4 This is a block diagram illustrating the structure of the sediment measurement system 1000 involved in the embodiment.

[0075] The sediment measurement system 1000 includes multiple camera devices 10a to 10n (e.g., n (two or more integers) of camera devices 10a, 10b to 10n), a control device 200, a sediment measurement calculation device 300, and a user interface 400. The camera devices 10a to 10n are connected to the control device 200 and are capable of communication. Furthermore, the sediment measurement calculation device 300 and the user interface 400 are connected to the control device 200 and are capable of communication.

[0076] Furthermore, the camera devices 10a to 10n and the control device 200 can be connected to enable wired communication or wireless communication. Similarly, the sediment volume calculation device 300 and the user interface 400 can be connected to the control device 200 to enable wired communication or wireless communication.

[0077] User interface 400 is a receiving device for accepting operations from a user. User interface 400 may be implemented, for example, by a display device, touchscreen, touchpad, keyboard, mouse, other controller class, or a combination thereof connected to control device 200. Furthermore, user interface 400 is an example of a display device for displaying attribute information and differential information calculated by the quantity calculation device according to this disclosure.

[0078] <Camera Device> Figure 5 This is a block diagram illustrating the structure of the camera device included in the sediment measurement system 1000 according to the embodiment. Furthermore, later... Figure 4 The structures of the camera devices 10a to 10n shown are identical in this embodiment. Therefore, when describing one of the multiple camera devices 10a to 10n included in the sediment measurement system 1000, camera device 10a will be described. That is, the part describing camera device 10a thereafter is the same as that describing the other camera devices 10b to 10n.

[0079] The multiple camera devices 10a to 10n are respectively equipped with functions equivalent to camera capturing space A1. Figure 1 The camera device of the camera 100 shown. The structures that can be adopted by the multiple camera devices 10a to 10n are, for example, common to each other.

[0080] In addition, here, space A1 is the union of the respective camera areas of multiple cameras 100.

[0081] The camera devices 10a to 10n may each have a camera 100 and a bracket 110.

[0082] The camera 100 includes a storage unit 101, a control unit 102, an optical system 103, and an image sensor 104.

[0083] The storage unit 101 stores the program read and executed by the control unit 102. Furthermore, the storage unit 101 temporarily stores image data of the camera area captured by the image sensor 104, metadata such as the timestamp assigned to the image data, camera parameters of the camera 100, and applicable shooting settings such as frame rate or resolution.

[0084] Such a storage unit 101 is implemented, for example, using a rewritable non-volatile semiconductor memory such as flash memory. Furthermore, depending on whether the stored data needs to be rewritten or the necessary storage period, a non-rewritable ROM (Read-Only Memory) or a volatile RaM (Random Access Memory) can be used in the storage unit 101.

[0085] Furthermore, there is no particular limitation on the number of camera devices included in the sediment measurement system 1000. For example, the sediment measurement system 1000 may include multiple camera devices. Also, the characteristics of the various camera devices 10a to 10n included in the sediment measurement system 1000 may not be common. Furthermore, the camera 100 included in each camera device 10a to 10n is not limited to a non-stereoscopic camera, but may also be a stereoscopic camera.

[0086] Furthermore, regarding the number of camera devices in the sediment measurement system 1000, if the camera device can change its position and posture to capture images of space A1 from multiple viewpoints, it can also be one.

[0087] For example, the control unit 102 is implemented using a CPU (Central Processing Unit). The control unit 102 reads and executes the program stored in the storage unit 101, thereby controlling the various components of the camera 100 to perform camera functions and other functions.

[0088] Alternatively, the control unit 102 may be implemented by a dedicated circuit that controls the various components of the camera 100 to perform camera functions and other functions. In other words, the control unit 102 may be implemented by software or by hardware.

[0089] The optical system 103 is a component that images light from the imaging area onto the image sensor 104, and is implemented using optical elements including lenses. Furthermore, the optical system 103 can also be a structure capable of changing the focal distance and viewing angle. Moreover, ultra-wide-angle lenses such as wide-angle lenses or fisheye lenses can be used for the optical system 103. For example, in the case where the image captured by the camera in the mass calculation device according to this disclosure is used in a monitoring system, a wide-angle lens is sometimes used to obtain a wide imaging area.

[0090] The image sensor 104 is implemented by a solid-state imaging element such as a CCD (Charge Coupled Devices) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a MOS (Metal Oxide Semiconductor) image sensor, which receives light collected by the optical system 103 on the light-receiving surface and converts the received light into an electrical signal representing an image.

[0091] The bracket 110 is a component used to fix and support the camera 100, which is generating an image for calculating sediment volume by taking pictures, in a specified position. The bracket 110 is implemented by, for example, a tripod.

[0092] In addition, in order to adjust the fixed position of the camera 100 in preparation for the camera to record, the length and angle of the foot part of the bracket 110 can be adjusted.

[0093] Furthermore, the bracket 110 may also include a mechanism for rotating the gimbal to pan or tilt the camera 100, as well as a lifting mechanism for moving it up and down. Alternatively, the bracket 110 may include a mechanism such as a trolley or crane to support and move the camera 100.

[0094] Furthermore, it is not necessarily necessary to fix the camera devices 10a to 10n. Therefore, the camera devices 10a to 10n may also be structures without the bracket 110.

[0095] Furthermore, the number of cameras included in the sediment measurement system 1000 is not particularly limited, whether they are multiple fixed cameras, one or more mobile cameras, or a combination thereof. Moreover, it is not limited to non-stereo cameras, but may also include compound-eye cameras such as stereo cameras.

[0096] <Control Device> Figure 6 This is a block diagram illustrating the structure of the control device 200 included in the sediment measurement system 1000 according to the embodiment.

[0097] The control device 200 includes a storage unit 201, a control unit 202, and a timer 203.

[0098] The control device 200 controls the camera devices 10a to 10n and performs input / output to the user interface 400. Furthermore, the control device 200 sends sediment quantity calculation instructions to the sediment quantity calculation device 300 corresponding to the data received from each of the camera devices 10a to 10n.

[0099] These control devices 200 are, for example, computers. In this case, the storage unit 201 is the computer's storage device, implemented by a hard disk drive or various semiconductor memories, or a combination thereof. Furthermore, the control unit 202 is implemented by the computer's CPU.

[0100] The timer 203 is a timer provided by the control device 200 and referenced by the CPU that implements the control unit 202.

[0101] The storage unit 201 stores the program read and executed by the control unit 202. Furthermore, the storage unit 201 stores data received from the imaging devices 10a to 10n, that is, data that is the object of processing by the control unit 202.

[0102] The control unit 202 controls the aforementioned camera devices 10a to 10n and sediment quantity calculation device 300 by reading and executing the program stored in the storage unit 201.

[0103] Furthermore, the control unit 202 executes processing of user commands related to these controls and processes. One example of such processing is the control of simultaneous image capture by multiple cameras 100 on each of the camera devices 10a to 10n. Also, one example of such processing may be commands for calculating sediment load.

[0104] The control unit 202 includes, in terms of function, a camera control unit 202a, a user instruction acquisition unit 202b, and a sediment quantity calculation instruction unit 202c.

[0105] The user instruction acquisition unit 202b included in the control unit 202 is a functional component implemented by the control unit 202 executing a program for acquiring instructions from the user.

[0106] Furthermore, the sediment quantity calculation instruction unit 202c included in the control unit 202 is a functional component that is implemented by the control unit 202 executing a program for sediment quantity calculation instructions.

[0107] Furthermore, the camera control unit 202a, user instruction acquisition unit 202b, and sediment quantity calculation instruction unit 202c of the control unit 202 can also be implemented by dedicated circuits that perform camera control, user instruction acquisition, sediment quantity calculation instruction, and sediment quantity calculation processing. In other words, the control unit 202 can be implemented in software or hardware.

[0108] The camera control unit 202a causes each of the plurality of camera devices 10a to 10n to capture images of the camera area, i.e., space A1, at different multiple timings. The camera control unit 202a causes the plurality of camera devices 10a to 10n to capture images of the camera area while each of the plurality of camera devices 10a to 10n is in a predetermined position and in a posture facing a predetermined direction.

[0109] User instruction acquisition unit 202b sends camera status information provided by camera devices 10a to 10n to user interface 400 to obtain input from the user. The user input includes the selection result of the data for the object to be calculated, the feasibility of the sediment quantity calculation process, or a combination thereof. If the user input is whether the sediment quantity calculation process is feasible, user instruction acquisition unit 202b outputs the feasibility of the sediment quantity calculation process to, for example, sediment quantity calculation instruction unit 202c.

[0110] The sediment quantity calculation instruction unit 202c, based on the feasibility of sediment quantity calculation processing obtained from the user instruction acquisition unit 202b, causes the sediment quantity calculation device 300 to perform sediment quantity calculation processing. Furthermore, the sediment quantity calculation instruction unit 202c can also cause the sediment quantity calculation device 300 to perform sediment quantity calculation processing based on the selection result of the data of the object to be calculated. Specific examples of the processing of the sediment quantity calculation instruction unit 202c will be described later.

[0111] <Sediment Calculation Device> Figure 7 This is a block diagram illustrating the structure of the sediment quantity calculation device 300 included in the sediment quantity measurement system 1000 according to the embodiment.

[0112] The sediment quantity calculation device 300 includes a storage unit 301 and a processing unit 302.

[0113] The sediment quantity calculation device 300 processes the data received via the control device 200. Specifically, the sediment quantity calculation device 300 performs sediment quantity calculation processing on the sediment 21 present in the space A1 captured by the cameras 100 of each of the camera devices 10a to 10n.

[0114] The sediment quantity calculation device 300 performs sediment quantity calculation processing on, for example, space A1 after a disaster. Based on this, by calculating the difference between the three-dimensional model of space A1 before the disaster, the amount of sediment 21 deposited in space A1 (sediment quantity) can be determined.

[0115] Furthermore, the object on which the sediment volume calculation device 300 performs sediment volume calculation processing can be not the space A1 immediately following the disaster, but rather, for example, the space A1 during sediment removal operations. Accordingly, the sediment volume calculation device 300 calculates the sediment volume sequentially, thereby enabling it to calculate the progress of the sediment removal operation.

[0116] In addition, the object on which the sediment quantity calculation device 300 performs sediment quantity calculation processing may not be the space A1 before and after the disaster, and there is no particular limitation.

[0117] Such a sediment load calculation device 300 is, for example, a computer. In this case, the storage unit 301 is the computer's storage device, implemented by a hard disk drive or various semiconductor memories, or a combination thereof. Furthermore, the processing unit 302 is implemented by the computer's CPU. Alternatively, the sediment load calculation device 300 may also be implemented by the same computer as the control device 200.

[0118] The storage unit 301 stores the program read and executed by the processing unit 302. Furthermore, the storage unit 301 stores data received from the camera devices 10a to 10n via the control device 200, as well as data of the 3D model obtained from external devices—that is, data that is the object of processing by the processing unit 302. In other words, the storage unit 301 can store both 3D reconstruction results and attribute information extraction results. Moreover, as 3D reconstruction results, it can store both the 3D model before the disaster and the 3D model after the disaster.

[0119] The processing unit 302 reads and executes the program stored in the storage unit 301, thereby processing the data received from the imaging devices 10a to 10n. One example of this processing is the three-dimensional reconstruction of space A1.

[0120] The processing unit 302 includes an image acquisition unit 302a, a three-dimensional reconstruction unit 302b, an attribute information extraction unit 302c, a model acquisition unit 302d, a alignment unit 302e, and a difference calculation unit 302f.

[0121] The image acquisition unit 302a acquires multiple images captured by the multiple imaging devices 10a to 10n. Further, while acquiring the multiple images, the image acquisition unit 302a acquires camera tags corresponding to each of the multiple images and indicating which camera 100 captured each of the multiple images. The image acquisition unit 302a may also, for example, acquire the multiple images and camera tags by acquiring the images to which the camera tags are assigned. Each of the multiple images may be a still image or a moving image.

[0122] Furthermore, preferably, the multiple images used for correction processing (3D reconstruction processing) are images obtained by multiple imaging devices 10a to 10n capturing images at a corresponding time interval (i.e., at that time). The multiple images acquired by the image acquisition unit 302a are stored in the storage unit 301. Moreover, the image acquisition unit 302a may also pre-store the images and camera tags in the storage unit 301 before receiving the sediment quantity calculation command from the control device 200. Accordingly, the sediment quantity calculation device 300 can start sediment quantity calculation processing when it is determined that sediment quantity calculation is required.

[0123] The 3D reconstruction unit 302b generates at least one of a first 3D model and a second 3D model based on multiple images representing space A1. More specifically, the 3D reconstruction unit 302b calculates the 3D shape of space A1 using multiple images captured by multiple imaging devices 10a to 10n. For example, the 3D reconstruction unit 302b uses a set of images at a first moment to generate a 3D model of space A1 at that first moment. Furthermore, the 3D reconstruction unit 302b uses a set of images at moments different from the first moment to generate a 3D model of space A1. Thus, the 3D reconstruction unit 302b generates a first 3D model and a second 3D model, which are 3D models of the same region but at different times.

[0124] The 3D reconstruction unit 302b, for example, establishes correspondences between multiple images captured by multiple camera devices 10a to 10n, performs geometric calculations based on the correspondences, and thereby calculates the 3D shape of space A1. The model used to represent the 3D shape can be represented by point groups, voxels, or meshes.

[0125] Furthermore, these representations of 3D models are just one example. The methods of representing 3D models are not limited to those described above.

[0126] The attribute information extraction unit 302c extracts attribute information from multiple images captured by multiple imaging devices 10a to 10n, a 3D model reconstructed by the 3D reconstruction unit 302b, or a 3D model obtained by the model acquisition unit 302d (described later). For example, for a group of 3D points, the attribute information extraction unit 302c estimates the attributes of each point based on information from surrounding points and calculates the estimated attributes as attribute information. Furthermore, for image data, for example, the attribute information extraction unit 302c estimates the attribute information of the object captured in each image based on information from surrounding pixels and associates the estimated attribute information with each smallest constituent unit of the reconstructed 3D model.

[0127] Furthermore, the attribute information extraction unit 302c can obtain a three-dimensional model reconstructed by the three-dimensional reconstruction unit 302b, or obtain multiple images captured by multiple camera devices 10a to 10n respectively from the storage unit 301, or obtain a three-dimensional model obtained from the storage unit 301, or obtain a three-dimensional model from the model acquisition unit 302d described later.

[0128] The model acquisition unit 302d acquires a three-dimensional model from the three-dimensional reconstruction unit 302b and the storage unit 301. For example, the model acquisition unit 302d acquires a model of the space A1 before the disaster from the storage unit 301, and acquires a model of the space A1 after the disaster from the three-dimensional reconstruction unit 302b.

[0129] Alternatively, the model acquisition unit 302d may acquire two or more three-dimensional models from the storage unit 301. For example, the model acquisition unit 302d acquires a first three-dimensional model representing space A1 and a second three-dimensional model representing space A1 that is different from the first three-dimensional model. Specifically, the model acquisition unit 302d acquires a first three-dimensional model representing space A1 at a first moment and a second three-dimensional model representing space A1 at a second moment that is different from the first moment. In this embodiment, the model acquisition unit 302d acquires two three-dimensional models of space A1 before and after the disaster (i.e., a first three-dimensional model representing space A1 before the disaster (at the first moment) and a second three-dimensional model representing space A1 after the disaster (at the second moment)) from the storage unit 301. Accordingly, even if three-dimensional reconstruction processing is not performed in the three-dimensional reconstruction unit 302b, the differences between the models can be calculated.

[0130] The three-dimensional model obtained by the model acquisition unit 302d can be a model calculated based on image data outside the sediment measurement system 1000, or a model calculated by laser measurement outside the sediment measurement system 1000.

[0131] In addition, the obtained 3D model can be represented by voxel data, mesh data, or point group data.

[0132] The alignment unit 302e performs the following processing: using two or more three-dimensional models obtained by the model acquisition unit 302d and attribute information calculated by the attribute information extraction unit 302c, it unifies the coordinate system between the three-dimensional models, that is, aligns the three-dimensional models with each other. In other words, the alignment unit 302e aligns the first three-dimensional model and the second three-dimensional model based on their respective attribute information. Specifically, the alignment unit 302e effectively compares models by comparing a portion of space A1, i.e., a partial region, that includes three-dimensional models with the same attribute information, based on the attribute information.

[0133] Alignment part 302e establishes correspondences between partial models while changing the combination within partial models that have the same attribute information.

[0134] Next, the alignment section 302e compares the smallest constituent units among the partial models that have at least one or more identical attribute information. For example, if it is a three-dimensional model represented by a group of points, it compares the three-dimensional points.

[0135] Here, a partial model is a set of the smallest constituent units that represent a part or all of a region of a 3D model.

[0136] Alignment unit 302e searches for the nearest neighbor points among the partial models and establishes correspondences, adjusting the position and pose of the coordinate system by minimizing the differences in the positions of each point. Alignment unit 302e calculates the coordinate system with the smallest difference by repeatedly changing the correspondences established for the partial models and the correspondences established for the points, and calculates the 3D model that is converted from the coordinate system to the calculated coordinate system.

[0137] As described above, the alignment unit 302e performs alignment (coordinate transformation and size adjustment) based on attribute information. For example, in multiple 3D models, the alignment unit 302e extracts 3D models (partial models) with the same attribute information from the multiple 3D models respectively. The alignment unit 302e compares and aligns the extracted 3D models. Furthermore, the alignment unit 302e changes the 3D model as a whole according to the partial models that have changed through alignment. That is, the alignment unit 302e extracts partial models with common attribute information of each of the first and second 3D models from each of the first and second 3D models, and aligns the first and second 3D models by moving each partial model to align them in a manner that makes the coordinates of the corresponding positions of the extracted partial models consistent. Then, according to the movement of each partial model, the first and second 3D models are moved, thereby aligning the first and second 3D models.

[0138] Therefore, the alignment part 302e can perform the alignment of the three-dimensional model more effectively compared to the case where the entire three-dimensional model is aligned at the same time.

[0139] The transformed model can also be stored in storage unit 301.

[0140] The establishment of correspondences for partial models and the smallest constituent units can be random or based on feature selection methods. Furthermore, the establishment of correspondences for partial models and the smallest constituent units is not limited to the methods described above. Additionally, in the alignment of color models, color difference information can also be used for alignment.

[0141] Furthermore, regarding the scale of the 3D model, it is possible to adjust the 3D model from either the pre-disaster or post-disaster perspective to the desired scale. For example, it can be adjusted to the scale of space A1. Moreover, the scale of the 3D model can be adjusted based on scale information obtained externally from the sediment measurement system 1000. By adjusting the scale of the 3D model, it is possible to perform differential calculations on the pre-disaster and post-disaster 3D models at the same scale. Accordingly, sediment volume can be calculated at the scale of space A1.

[0142] The difference calculation unit 302f calculates the amount of difference between the first 3D model and the second 3D model, and outputs attribute information of the difference and difference information representing the amount of the difference. Specifically, the difference calculation unit 302f calculates the difference information between two or more 3D models whose coordinate systems have been unified by the alignment unit 302e. For example, the difference calculation unit 302f calculates the volume of a 3D region covered by surfaces in a mesh model with non-overlapping or non-intersecting surfaces in the same coordinate system.

[0143] The difference information calculated by the difference calculation unit 302f may also include attribute information of areas with significant differences, such as information on the mixtures included in the sediment. Furthermore, the difference information described above is an example and is not intended to limit the scope of this disclosure.

[0144] The difference calculation unit 302f, for example, outputs the calculated difference information to the user interface 400. Accordingly, the user operates the user interface 400 to confirm the difference information (i.e., the amount of difference, i.e., the amount and properties of sediment 21).

[0145] Furthermore, for example, when the difference calculation unit 302f includes multiple attribute information that are not mutually exclusive, it categorizes and outputs the multiple attribute information by each category, thereby displaying the multiple attribute information together on the display device (e.g., user interface 400) by each category. In this case, for example, the difference calculation unit 302f calculates the amount included in the difference by each category.

[0146] [Processing Order] <Alignment Processing> Figure 8 This is a flowchart illustrating the alignment process of a three-dimensional model performed by the sediment measurement system 1000 involved in the embodiment. Figure 9 This is a diagram used to illustrate the alignment of the two three-dimensional models performed by the alignment section 302e.

[0147] The alignment unit 302e of the sediment quantity calculation device 300 of this embodiment first establishes a correspondence between partial models with the same attribute information for at least two or more three-dimensional models. For example, partial models labeled "house" are established to correspond to each other (S31). That is, in this case, for example, both three-dimensional models include partial models with attribute information representing "house".

[0148] For example, such as Figure 9 As shown in (a), the difference between the first three-dimensional model 500 and the second three-dimensional model 510 is calculated. In this case, the alignment unit 302e first extracts the partial models with attribute information included in both the first three-dimensional model 500 and the second three-dimensional model 510. Here, it is assumed that the alignment unit 302e extracts the first partial model 501 with the attribute information of "house" included in the first three-dimensional model 500, and the second partial model 511 with the attribute information of "house" included in the second three-dimensional model 510. Next, as Figure 9 As shown in (b), the alignment unit 302e establishes a correspondence between the first part model 501 and the second part model 511. Specifically, the alignment unit 302e extracts one or more feature points included in the first part model 501 and one or more feature points included in the second part model 511, and establishes a correspondence between the extracted feature points of the first part model 501 and the extracted feature points of the second part model 511. Here, the extracted feature points can be arbitrary. For example, the feature points are the corners of the part models. The alignment unit 302e establishes a correspondence between the same features of the feature points of the first part model 501 and the extracted feature points of the second part model 511. For example, the alignment unit 302e establishes a correspondence between feature point 520 of the first part model 501 and feature point 521 of the extracted second part model 511.

[0149] Again, refer to Figure 8 After step S31, the positioning unit 302e compares the smallest constituent units among the partial models that have at least one or more identical attribute information. For example, if it is a three-dimensional model represented by a group of points, it compares the three-dimensional points, searches for the nearest neighbor points among the points in the partial models and establishes a correspondence (S32).

[0150] Next, in alignment section 302e, the position and pose of the coordinate system of a portion of the model are adjusted by reducing the difference in position between the corresponding points established in step S32 (S33). For example, as Figure 9 As shown in (c), the alignment unit 302e changes at least one of the first part model 501 and the second part model 511 so that the corresponding feature points are located at the same coordinates. For example, the alignment unit 302e moves at least one of the first part model 501 and the second part model 511 so that the corresponding feature points, namely feature points 520 and 521, are located at the same coordinates. Furthermore, the alignment unit 302e changes the entire three-dimensional model according to the change of the part model. Accordingly, the alignment unit 302e changes the models representing the same object in the first three-dimensional model 500 and the second three-dimensional model 510 so that they exist in the same position and pose.

[0151] Additionally, the alignment unit 302e can also adjust the scale of the 3D model during position and pose adjustments. Furthermore, regarding scale, it can be adjusted in a manner consistent with either of the two 3D models, or the scale of both 3D models can be adjusted based on scale information obtained from external sources.

[0152] Again, refer to Figure 8 After step S33, the alignment unit 302e determines the portion of the model executed in step S33 (e.g., Figure 9 The system checks whether the position and orientation of at least one of the first part model 501 and the second part model 511 changes little (S34). If the changes in position and orientation are large, proceed to step S32; if the changes are small, proceed to step S35.

[0153] Here, the amount of change used as a reference can be arbitrarily determined in advance. For example, in alignment unit 302e, if the amount of change before and after adjusting the position and posture of a partial model is greater than or equal to the arbitrarily determined reference amount, step S32 is then executed. On the other hand, for example, in alignment unit 302e, if the amount of change before and after adjusting the position and posture of a partial model is less than the aforementioned reference amount, step S35 is then executed. The amount of change is, for example, the amount of movement of each point. When the entire partial model is rotated relative to a reference point, the amount of change can also be based on the rotation angle of that rotation.

[0154] Next, the alignment unit 302e compares the overall 3D model based on the results of the position and pose adjustments to determine whether the overall difference is large (S35). If the overall difference is large, proceed to step S36; if the difference is small, proceed to step S37.

[0155] Here, the overall difference used as a reference can be arbitrarily determined in advance. For example, in alignment unit 302e, if the difference before and after adjusting the position and pose of the 3D model is greater than or equal to the arbitrarily determined reference difference, step S36 is then executed. On the other hand, for example, in alignment unit 302e, if the difference in the change before and after adjusting the position and pose of the 3D model is less than the aforementioned reference difference, step S37 is then executed. Here, the difference is, for example, the amount of points where the coordinates of the various 3D points in the two 3D models are inconsistent.

[0156] Next, the alignment unit 302e determines whether to compare all possible combinations from steps S31 and S32 (S36). If all combinations are completed, the process proceeds to step S37; otherwise, it proceeds to step S31. Here, for example, if the process proceeds to step S31, the alignment unit 302e extracts and establishes correspondences between the parts of the first 3D model 500 and the second 3D model 510 that have not yet been compared.

[0157] in addition, Figure 8 The steps and their order shown are merely examples and are not particularly limited. For instance, the structure could proceed from step S33 to step S36. Furthermore, the structure could also exclude steps S34 and S35.

[0158] Finally, the alignment unit 302e outputs the coordinate system with the smallest overall difference, as well as the three-dimensional model after coordinate transformation (S37).

[0159] <Sequence diagram> Figure 10 This is a sequence diagram illustrating the processing of the sediment measurement system 1000 involved in the embodiment.

[0160] The sediment measurement system 1000 begins sediment quantity calculation and processing upon receiving a user's processing instruction.

[0161] Control device 200 directs signals to camera devices 10a to 10n (in...) Figure 10 The text only indicates that the camera device 10a) sends a camera instruction to send the camera images received from the camera devices 10a to 10n as camera information to the user interface 400, and then prompts the user with the camera information (S41). The camera information is, for example, an image of space A1.

[0162] The user operates the user interface 400 to confirm the camera information and determine whether the camera devices 10a to 10n have properly captured the desired space and whether there is sufficient camera information to start processing, and then instructs the user to perform processing (S42). Specifically, the user operates the user interface 400 so that the user interface 400 sends information indicating the above-mentioned processing instructions to the control device 200.

[0163] The user processing instruction sent via the user interface 400 is information or a signal indicating whether to start sediment quantity calculation processing or whether the camera devices 10a to 10n should continue recording. Additionally, the user processing instruction may also include information determining which camera information to use for sediment quantity calculation processing, or specifying the desired output result.

[0164] Next, the control device 200 determines, based on the information indicating the user's processing instruction received from the user interface 400, whether to perform sediment quantity calculation processing or whether the camera devices 10a to 10n should continue recording (S43). If the control device 200 determines that sediment quantity calculation processing should begin ("Yes" in S43), it sends the sediment quantity calculation command and the camera image to the sediment quantity calculation device 300.

[0165] On the other hand, when the control device 200 determines that the camera devices 10a to 10n should continue to record (S43 "No"), it sends an instruction to the camera devices 10a to 10n to continue recording, so that the camera devices 10a to 10n continue to record.

[0166] Next, upon receiving information indicating a sediment quantity calculation instruction received from the control device 200, the sediment quantity calculation device 300 performs sediment quantity calculation processing using the camera image based on that information (S44). Step S44 is, for example, Figure 9 The processing steps S31 to S37 are shown. After the sediment quantity calculation is completed, the sediment quantity calculation device 300 sends information indicating the processing result to the control device 200.

[0167] Next, the control device 200 organizes the processing results received from the sediment quantity calculation device 300 according to the user's processing instructions, and outputs (sends) the difference information to the user interface 400 (S45).

[0168] The user interface 400 will prompt the user with the difference information received from the control device 200 (S46). Specific examples of the difference information prompted to the user will be described later.

[0169] Summary of Processing Order Figure 11This is a flowchart illustrating the sediment quantity calculation process performed by the sediment quantity measurement system 1000 involved in the embodiment.

[0170] In the sediment quantity calculation process of the sediment quantity calculation device 300 of this embodiment, firstly, the image acquisition unit 302a acquires multiple images captured by multiple imaging devices 10a to 10n respectively (S51).

[0171] Next, the 3D reconstruction unit 302b performs 3D reconstruction processing using multiple images (S52). For example, the 3D reconstruction unit 302b generates a first 3D model representing space A1 and having attribute information for each region, and a second 3D model representing space A1 and having attribute information for each region. The second 3D model is different from the first 3D model (e.g., at different times).

[0172] Next, the model acquisition unit 302d acquires two or more three-dimensional models from the three-dimensional reconstruction unit 302b or the storage unit 301 (S53). Alternatively, one or more models can be acquired from the three-dimensional reconstruction unit 302b and one or more models can be acquired from the storage unit 301, or two or more models can be acquired from the storage unit 301.

[0173] Next, the attribute information extraction unit 302c extracts attribute information from multiple images or three-dimensional models captured by the imaging devices 10a to 10n respectively (S54). Alternatively, step S54 may be performed before step S53, after step S53, or both.

[0174] Next, the alignment unit 302e performs unified coordinate processing based on two or more 3D models and attribute information (S55). Specifically, the alignment unit 302e aligns the first 3D model and the second 3D model based on the attribute information of the first 3D model and the second 3D model, respectively. For example, the alignment unit 302e detects corresponding points that are the same points in the first 3D model and the second 3D model based on the attribute information of the first 3D model and the second 3D model, and uses the detected corresponding points to align the first 3D model and the second 3D model. By comparing the regions in the 3D model that have the same attribute information (i.e., the parts of the model that have the same attribute information), the alignment unit 302e can perform alignment effectively and with high precision.

[0175] Next, the difference calculation unit 302f calculates the difference information between two or more models whose coordinate systems have been unified by the alignment unit 302e (i.e., aligned by the alignment unit 302e) (S56). The difference information includes, for example, difference information representing the quantity of difference and attribute information possessed by the difference. Furthermore, the difference calculation unit 302f outputs the calculated difference information to the user interface 400 (more specifically, to the user interface 400 via the control device 200). The user interface 400 displays (more specifically, prompts) the user with the obtained difference information.

[0176] [Output Example] Figure 12 This is a diagram illustrating an example of an image 600 displayed when the user interface 400 of the sediment measurement system 1000 involved in the embodiment is aligning two three-dimensional models.

[0177] The user interface 400 is implemented, for example, using a mobile terminal such as a smartphone or tablet terminal with a display device and an input device, such as a touch screen display.

[0178] Alternatively, users can also use Figure 12 The tablet-type mobile terminal shown allows users to arbitrarily select attributes for alignment.

[0179] For example, image 600 includes a three-dimensional image 601, an object selection unit 602, and a decision unit 603.

[0180] 3D image 601 is an image representing a 3D model after the disaster.

[0181] The object selection unit 602 is an image for accepting selections of partial models having attribute information preferentially established by the alignment unit 302e. A user, for example, touches the 3D image 601 or the object selection unit 602, selecting the attribute information or partial model having that attribute information that the alignment unit 302e preferentially establishes. The user interface 400, for example, displays information representing the attribute information touched by the user or the partial model having that attribute information, for example in... Figure 10 After step S42, the information indicating user processing instructions is sent to the control device 200. The control device 200, for example, sends information indicating attribute information touched by the user or information about a partial model having that attribute information, along with sediment quantity calculation instructions and camera information, to the sediment quantity calculation device 300. The alignment unit 302e, for example, in step S31, first selects and establishes a correspondence for a partial model based on the attribute information indicating user touch or information about a partial model having that attribute information.

[0182] Figure 13This is a diagram illustrating an example of the difference information displayed by the user interface 400 of the sediment measurement system 1000 involved in the embodiment.

[0183] User interface 400, using Figure 13 The tablet-type mobile terminal shown displays the result of sediment volume as an example of the calculated difference. The user interface 400 may also display the estimated weight as a calculation result, in addition to the sediment volume.

[0184] Furthermore, the user interface 400 can also display the mixture that matches the sediment portion using the attribute information obtained by the attribute information extraction unit 302c. Accordingly, the information needed for the removal of sediment and the mixture can be obtained effectively. Moreover, the method of displaying the mixture is not limited to text; it can also be achieved by coloring that portion on a 3D model or image.

[0185] For example, user interface 400 displays image 610 to prompt the user with difference information. For example, image 610 includes a three-dimensional image 611, an emphasis line 612, a difference information image 613, and an attribute information image 614.

[0186] 3D image 611 is an image representing a 3D model after the disaster.

[0187] Emphasis line 612 is a line used in the three-dimensional image 611 to represent the location, region, etc., of the difference between the three-dimensional image and the pre-disaster three-dimensional model. Figure 13 In the illustration, the emphasis line 612 is represented by a dashed circle. However, the emphasis line 612 can also be any solid line, dashed line, thick line, colored line of a different color from the rest, rectangle, polygon, etc.

[0188] The differential information image 613 is an image used to present the differential components to the user. The differential information image 613 includes information such as the volume of the differential and the estimated weight of the sediment.

[0189] The attribute information image 614 is an image used to indicate to the user the attribute information of the partial model included in the difference. For example, the attribute information image 614 includes attribute information representing "mud," "driftwood," "road sign," "traffic light," etc., which are part of the model included in the difference. Furthermore, in this example, the attribute information image 614 includes the proportion of the volume or weight of the portion shown in the attribute information as a whole of the difference. Thus, the quantity of the difference can also be displayed for each attribute.

[0190] Furthermore, in this example, the attribute information image 614 includes a first category image 615 and a second category image 616.

[0191] Image 615 of category 1 is used to highlight the attributes that satisfy the first condition among the multiple attributes included in the difference.

[0192] Image 616 of category 2 is used to highlight the attributes that satisfy the second condition among the multiple attributes included in the difference.

[0193] As mentioned above, the user interface can also display attribute information and the quantity of the partial model containing that attribute information together for each category of the attribute.

[0194] Furthermore, in this example, the first condition is a natural object, and the second condition is a man-made object. These conditions can be arbitrarily determined in advance, without any particular restrictions.

[0195] [Effects, etc.] As explained above, one embodiment of the present disclosure includes a quantity calculation device (e.g., a sediment quantity measurement system 1000, specifically a sediment quantity calculation device 300) comprising: an acquisition unit (e.g., a model acquisition unit 302d) that acquires a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are respectively three-dimensional models representing the same defined space (e.g., space A1) and having attribute information for each region (e.g., each part model or each three-dimensional point constituting the three-dimensional model structure); an alignment unit 302e that aligns the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation unit (e.g., a difference calculation unit 302f) that calculates the amount of difference between the first three-dimensional model and the second three-dimensional model aligned by the alignment unit 302e, and outputs the attribute information possessed by the difference and the amount of the difference.

[0196] Conventionally, methods have been used to measure (calculate) the amount of sediment flowing into a specified space due to the effects of landslides or other geological disasters. While laser-based sediment quantity calculations can achieve high accuracy, they are time-consuming and costly. Therefore, in the quantity calculation apparatus of this disclosure, for example, a three-dimensional model (first three-dimensional model) of the specified space representing the absence of an object whose quantity of sediment is to be calculated is compared with a three-dimensional model (second three-dimensional model) representing the presence of that object. By appropriately aligning the first and second three-dimensional models, the difference between them can be easily calculated. Therefore, the quantity calculation apparatus of this disclosure reduces the processing time for calculating the quantity of an object.

[0197] Furthermore, for example, the mass calculation apparatus disclosed herein also includes a generation unit (e.g., a three-dimensional reconstruction unit 302b) that generates at least one of a first three-dimensional model and a second three-dimensional model based on a plurality of images representing a defined space.

[0198] Therefore, the amount of difference can be calculated using a simple structure such as a camera used to generate images.

[0199] Furthermore, for example, the calculation unit (e.g., user interface 400) can display multiple attribute information together on the display device (e.g., user interface 400) by classifying and outputting the multiple attribute information according to each category when the difference includes multiple attribute information that are different from each other.

[0200] Accordingly, the categories of objects included in the difference are categorized and displayed on the display device, thus enabling users who wish to examine the display device to easily understand the categories included in the difference.

[0201] Furthermore, for example, the calculation unit calculates the quantities included in the difference for each category.

[0202] Accordingly, the quantity of each category of objects included in the difference is displayed on the display device, thus enabling the user who wants to verify the display device to easily understand the quantity of each category included in the difference.

[0203] Furthermore, for example, the quantity of the difference is at least one of the volume and weight of the difference.

[0204] Accordingly, the quantity calculation device disclosed herein is capable of calculating at least one of the difference in quantity and volume.

[0205] Furthermore, for example, in the alignment section, partial models with common attribute information of the first and second three-dimensional models are extracted from the first and second three-dimensional models respectively. The partial models are aligned by moving them so that the coordinates of the corresponding positions of the extracted partial models are consistent. Then, the first and second three-dimensional models are moved according to the movement of the partial models, thereby aligning the first and second three-dimensional models.

[0206] Accordingly, the mass calculation device of this disclosure can align the first three-dimensional model and the second three-dimensional model even without performing time-consuming processing such as alignment by comparing the shapes of the three-dimensional models. Therefore, the processing time of the mass calculation device of this disclosure can be further shortened.

[0207] Furthermore, one embodiment of the present disclosure relates to a quantity calculation method comprising: an obtaining step (e.g., step S53), obtaining a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, wherein the first three-dimensional model and the second three-dimensional model are three-dimensional models representing the same defined space and having attribute information for each region; an alignment step (e.g., step S55), aligning the first three-dimensional model and the second three-dimensional model according to the attribute information respectively possessed by the first three-dimensional model and the second three-dimensional model; and a calculation step (e.g., step S56), calculating the quantity of the difference between the first three-dimensional model and the second three-dimensional model, and outputting the attribute information possessed by the difference and the difference information representing the quantity of the difference.

[0208] Accordingly, by properly aligning the first three-dimensional model and the second three-dimensional model, the difference between the first three-dimensional model and the second three-dimensional model can be easily calculated. Therefore, the quantity calculation method disclosed herein can shorten the processing time for calculating the quantity of an object.

[0209] Furthermore, for example, in the above-described obtaining step, a first three-dimensional model representing a predetermined space before the sediment inflow and a second three-dimensional model representing a predetermined space after the sediment inflow are obtained, and in the above-described calculation step, the amount of the sediment is calculated as the amount of the difference.

[0210] Accordingly, by properly aligning a three-dimensional model of the specified space before and after a disaster, representing the inflow of sediment into that space, the amount of sediment, which becomes differential, can be easily calculated. Therefore, the material quantity calculation method disclosed herein can shorten the processing time for calculating the amount of sediment.

[0211] (Other implementation methods) The above description addresses the quantity calculation device and other related devices involved in this disclosure, based on the aforementioned embodiments. However, this disclosure is not limited to the aforementioned embodiments.

[0212] For example, as described in the above embodiments, the processing unit of the quantity calculation device, etc., is implemented by a CPU and a control program. For example, the components of this processing unit may each be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may also include, for example, semiconductor devices, integrated circuits (ICs), or large-scale integration (LSIs). ICs or LSIs may be integrated on a single chip or on multiple chips. Here, although referred to as ICs or LSIs, depending on the degree of integration, they may be called system LSIs, very large-scale integration (VLSIs), or ultra-large-scale integration (ULSIs). Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after manufacturing LSIs can also be used for the same purpose.

[0213] Furthermore, the general or specific form of this disclosure can also be implemented by a system, apparatus, method, integrated circuit, or computer program. Alternatively, it can be implemented by a computer-readable non-transitory recording medium such as an optical disc, HDD (Hard Disk Drive), or semiconductor memory storing the computer program. Furthermore, it can be implemented by any combination of system, apparatus, method, integrated circuit, computer program, and recording medium.

[0214] Furthermore, taking the division of functional blocks in the block diagram as an example, multiple functional blocks can be implemented as a single functional block, and a single functional block can also be divided into multiple functional blocks. A portion of the functionality can also be moved to other functional blocks. Moreover, the functions of multiple functional blocks with similar capabilities can be processed in parallel or time-division processed by a single piece of hardware or software.

[0215] Furthermore, the execution order of the steps in the flowchart is an example given to illustrate this disclosure, and the order may differ from the above. Also, some of the steps described above may be executed simultaneously (in parallel) with other steps.

[0216] Furthermore, any form obtained by performing various modifications to the above embodiments that can be conceived by those skilled in the art, or any form achieved by arbitrarily combining the constituent elements and functions of the embodiments without departing from the spirit of this disclosure, is included in this disclosure.

[0217] This disclosure is applicable to a quantity calculation device that calculates differences based on a three-dimensional model, and is applicable to, for example, a device for calculating the quantity of an object.

[0218] Symbol Explanation 10a, 10b, 10n camera devices 20 Roads 21. Sediment 22 Three-dimensional mode 100 cameras Storage Departments 101, 201, and 301 Control Departments 102 and 202 103 Optical System 104 Image Sensor 110 bracket 200 control device 202a Camera Control Department 202b User Instruction Acquisition Department 202c Sediment Calculation Command Section 203 Timer 300 Sediment Calculation Device 302 Processing Department 302a Image Acquisition Unit 302b 3D Reconstruction Department 302c Attribute Information Extraction Department 302d model acquisition department 302e Alignment section 302f Difference Calculation Department 400 User Interface 500 First three-dimensional model 501 Part 1 Model 510 Second 3D Model 511 Part 2 Model Feature points 520 and 521 Images 600 and 610 601, 611 3D images 602 Object Selection Section 603 Decision Department 612 Emphasis Line 613 Differential Component Information Image 614 Attribute Information Image 615 Category 1 Images 616 Category 2 Images 1000 Sediment Measurement System A1 Three-dimensional space (space)

Claims

1. An information processing device, comprising: Memory; and The processor is connected to the memory. The processor uses the memory. Obtain a first three-dimensional model representing a defined space in actual space, and a second three-dimensional model containing spaces common to the defined space. Generate difference data representing the difference between the first three-dimensional model and the second three-dimensional model, wherein the difference data includes attribute information representing the attributes of the regions contained in the difference data, generated through identification processing.

2. The information processing apparatus according to claim 1, wherein, The attribute information includes information indicating that it is a mixture of multiple attributes.

3. The information processing apparatus according to claim 2, wherein, The attribute information includes: information representing the volume of the region corresponding to the mixture, information representing each of the at least two attributes constituting the mixture, and proportion information representing the proportion of the at least two attributes in the mixture.

4. The information processing apparatus according to claim 1, wherein, The identification process includes assigning labels to the smallest constituent units of a pixel or 3D model using surrounding information.

5. The information processing apparatus according to claim 1, wherein, The attribute information is assigned to each three-dimensional point or each part of the model that constitutes the three-dimensional model.

6. The information processing apparatus according to claim 1, wherein, The attribute information corresponds to a category, and the category includes natural objects and man-made objects.

7. The information processing apparatus according to claim 6, wherein, The attribute information includes information representing the quantity of each category.

8. The information processing apparatus according to claim 1, wherein, The processor generates the differential data based on the first three-dimensional model and the second three-dimensional model, which have a unified coordinate system.

Images