Imaging and analysis device, program, and imaging and analysis method

Abstract

This enables support for understanding on-site conditions using three-dimensional models. [Solution] The imaging analysis device 100 includes a three-dimensional model generation unit 112 that generates a generated three-dimensional model 122, which is a three-dimensional model of the object to be imaged, based on imaging information, and a comparison unit 113 that compares a reference three-dimensional model, which is a three-dimensional model of an object to be compared with the generated three-dimensional model 122, with the generated three-dimensional model and calculates the difference. The generated three-dimensional model 122 is a textured three-dimensional mesh to which a texture has been applied, and the comparison unit 113 may detect the type of the object to be imaged based on the image or texture, and compare the reference three-dimensional model and the generated three-dimensional model 122 corresponding to the type of the detection result to calculate the difference in at least one of the size and shape.

Description

Technical Field

[0001] The present invention relates to an imaging analysis device, a program, and an imaging analysis method for imaging a work site and assisting in grasping the situation of the work site.

Background Art

[0002] Heavy materials such as steel sheet piles and H-shaped steel used in construction work are loaded and transported in a mixed state on a large truck. When inspecting such heavy materials, workers board the truck bed and visually check the quantity and damage status. Therefore, man-hours and safety are issues. Also, in progress confirmation at a large-scale construction site, the cost of moving to the site is high, and efficiency improvement is required through remote monitoring of images.

[0003] As a method for solving these problems, there is a comparison between a three-dimensional model of materials without defects such as damage and a building constructed as planned, and a three-dimensional model showing the actual materials and the building under construction. If the difference can be grasped by comparison, efficient inspection and progress grasping can be achieved.

[0004] As another method for solving problems, there is a three-dimensional mesh (three-dimensional model, digital twin) to which a texture showing the loading status of the truck load and the construction site is applied. If the reproduction accuracy of such a three-dimensional model is high, visual confirmation from a free viewpoint, counting, and dimension measurement are possible.

[0005] As a comparison technique between three-dimensional models, there is a quantity calculation device described in Patent Document 1. This quantity calculation device includes an acquisition unit that acquires a first three-dimensional model showing a predetermined space and a second three-dimensional model showing the predetermined space and different from the first three-dimensional model. The quantity calculation device also includes an alignment unit that aligns the first three-dimensional model and the second three-dimensional model based on the attribute information possessed by each of the first three-dimensional model and the second three-dimensional model. Further, the quantity calculation device includes a calculation unit that calculates the amount of difference between the first three-dimensional model and the second three-dimensional model and outputs the attribute information possessed by the difference and the difference information indicating the amount of the difference. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2020 / 179438 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] The material quantity calculation device described in Patent Document 1 has the problem of high processing costs because it recognizes attribute information and aligns the positions of two three-dimensional models. There is a need for support in understanding on-site conditions, including inspection of goods and confirmation of construction progress, by reproducing objects at the work site as a three-dimensional model (digital twin) and comparing it with a reference three-dimensional model, or by improving the accuracy of the reproduction. This invention was made in view of the above background, and aims to provide an imaging analysis device, program, and imaging analysis method that enable support for understanding on-site conditions using a three-dimensional model. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the imaging analysis apparatus according to the present invention includes a three-dimensional model generation unit that generates a generated three-dimensional model, which is a three-dimensional model of the object to be imaged, based on imaging information; a comparison unit that compares the generated three-dimensional model with a reference three-dimensional model, which is a three-dimensional model of an object that is also a three-dimensional model of an object to be compared with the object to be imaged, and calculates the difference; and a unit that determines the difference in the object to be imaged. minutes It includes an additional imaging support unit that outputs a suitable position and orientation for imaging a certain additional imaging location. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an imaging analysis device, program, and imaging analysis method that enable support for understanding on-site conditions using a three-dimensional model. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0012] [Figure 1] This is a functional block diagram of the imaging and analysis device according to the first embodiment. [Figure 2] This is a screen configuration diagram of the imaging analysis results screen according to the first embodiment. [Figure 3] This is a screen configuration diagram of the imaging analysis results screen, which displays a pointer indicating the position and orientation of additional imaging according to the first embodiment. [Figure 4] This is a screen configuration diagram of the imaging analysis results screen after additional imaging according to the first embodiment. [Figure 5] This is a flowchart of the imaging analysis process according to the first embodiment. [Figure 6] This is a schematic diagram showing the video analysis device according to the second embodiment. [Figure 7] This is an example of video data of a bridge pier. [Figure 8] This is a flowchart of the measurement visualization process according to the second embodiment. [Figure 9] This is a data flow diagram of the video analysis device according to the second embodiment. [Figure 10] This is a hardware configuration diagram of the video analysis device according to the second embodiment. [Figure 11] This is a schematic diagram showing the video analysis device according to the third embodiment. [Figure 12] This is a data flow diagram of the video analysis device according to the third embodiment. [Figure 13] This is a schematic diagram showing the video analysis device according to the fourth embodiment. [Figure 14] This is a flowchart of the measurement visualization process according to the fourth embodiment. [Figure 15]It is a hardware configuration diagram showing an example of a computer that realizes the functions of the imaging analysis device and the video analysis device according to the above-described embodiment.

Embodiment for Carrying Out the Invention

[0013] The imaging analysis device in the following embodiment (embodiment) for carrying out the present invention will be described. The imaging analysis device generates a three-dimensional model based on three-dimensional point cloud information and images (videos) obtained by imaging a work site with a plurality of LiDAR (Light Detection And Ranging) and cameras. This three-dimensional model is, for example, a three-dimensional mesh with texture (image) of heavy materials such as steel sheet piles and H-shaped steel loaded on a truck. Next, the imaging analysis device compares the generated three-dimensional model (generated three-dimensional model) with a reference three-dimensional model (reference three-dimensional model) and outputs the difference. The difference is, for example, the difference in length in the longitudinal direction of the steel sheet pile, and is the length of the portion cut from the reference steel sheet pile. In the inspection of heavy materials such as steel sheet piles and H-shaped steel, in addition to grasping the damage situation, obtaining the length and quantity of the heavy materials is included in the inspection.

[0014] Conventionally, workers have measured the length of heavy materials while riding on the truck bed for inspection. By using the imaging analysis device, not only the efficiency of the inspection work of workers can be improved, but also the safety of workers can be improved.

[0015] The imaging analysis device can also be used to grasp and monitor the progress of construction. By comparing the three-dimensional model of the building (reference three-dimensional model) that has been constructed according to the construction plan as the reference three-dimensional model with the generated three-dimensional model, the administrator can grasp the difference / progress status from the construction plan.

[0016] In this way, the imaging analysis device can assist in confirming the situation at the work site. Hereinafter, the imaging analysis device will be described by taking the inspection of heavy materials such as steel sheet piles and H-shaped steel as an example.

[0017] ≪Configuration of Imaging Analysis Device≫ Figure 1 is a functional block diagram of the imaging and analysis device 100 according to the first embodiment. Multiple LiDARs 411 and cameras 412 are connected to the imaging and analysis device 100. The pairs of LiDARs 411 and cameras 412 are assumed to be installed close to each other. The LiDAR 411 transmits the captured three-dimensional point cloud information to the imaging and analysis device 100. The camera 412 transmits the captured image (video) to the imaging and analysis device 100. Hereinafter, the LiDARs 411 and cameras 412 will be collectively referred to as the imaging device. The three-dimensional point cloud information and images will also be collectively referred to as imaging information. The LiDARs 411 and cameras 412 may be an integrated imaging device. Furthermore, since the LiDARs 411 and cameras 412 have different imaging positions and fields of view, their correspondence may be calculated using the alignment method for both described in the second embodiment and subsequent embodiments.

[0018] The imaging and analysis device 100 is a computer and comprises a control unit 110, a storage unit 120, and an input / output unit 180. User interface devices such as a display, keyboard, and mouse are connected to the input / output unit 180. The input / output unit 180 is equipped with a communication device and can transmit and receive data with the LiDAR 411 and camera 412. A media drive may also be connected to the input / output unit 180, enabling data exchange using a recording medium.

[0019] ≪Imaging and Analysis Device: Memory Unit≫ The memory unit 120 is composed of memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores an imaging information database 130, an imaging device information database 140, an object information database 150, a reference three-dimensional model database 160, a recognition model 121, a generated three-dimensional model 122, and a program 128. The various contents of the memory unit 120 may be stored in an external storage device such as a cloud server and read as needed.

[0020] ≪Memory Unit: Imaging Information Database / Imaging Device Information Database≫ The imaging information database 130 stores three-dimensional point cloud information and images captured by the imaging device, associated with the imaging device's identification information and the time of acquisition. The imaging device information database 140 stores information such as identification information, type, specifications, installation location, and installation orientation related to the imaging devices, including the LiDAR 411 and camera 412.

[0021] ≪Memory Unit: Object Information Database / Reference 3D Model Database≫ The object information database 150 stores information about the object to be imaged (the object to be imaged). In the first embodiment, the object to be imaged is heavy material, and information such as the name, type, size, imaged image, and identification information of the three-dimensional model (reference three-dimensional model) of the reference heavy material is associated with it and stored in the object information database 150. "Reference" means that the object (heavy material) is in the same condition as a new product before cutting or before any damage or deformation occurs, and serves as a standard for comparison with the heavy material subject to inspection.

[0022] The reference three-dimensional model database 160 stores reference three-dimensional models, which are three-dimensional models of standard heavy materials, associated with identification information. These three-dimensional models are not necessarily generated based on three-dimensional point cloud information; for example, they may be three-dimensional models (three-dimensional CAD models) used as design information. Furthermore, when the imaging analysis device 100 is used to check the progress of construction, the reference three-dimensional model database 160 may also store design three-dimensional data such as BIM (Building Information Modeling) or CIM (Construction Information Modeling).

[0023] ≪Memory Unit: Cognitive Model≫ The recognition model 121 is a machine learning model that recognizes / identifies and detects the type of object being imaged based on the image. The image is not limited to an image captured by the camera 412, but may also be an image that is a texture of a three-dimensional model representing the object being imaged.

[0024] Examples of types include steel sheet piles and H-beams, but other heavy materials or even more specific types may also be used. Furthermore, the recognition model 121 can detect individual objects based on images showing the cross-section (vertical plane in the long direction) of the objects (heavy materials).

[0025] ≪Memory Unit: Generating Three-Dimensional Models and Programs≫ The generated 3D model 122 is a textured 3D mesh of the imaged object (heavy material) generated based on the imaging information. The method for generating the generated 3D model 122 will be described later. Program 128 includes a description of the processing to be performed by the functional unit of the control unit 110, which will be described later.

[0026] <<Imaging and Analysis System: Control Unit>> The control unit 110 includes a CPU (Central Processing Unit) and comprises an image acquisition unit 111, a three-dimensional model generation unit 112, a comparison unit 113, a display control unit 114, and an additional image acquisition support unit 115. The control unit 110 may also include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0027] ≪Control Unit: Image Acquisition Unit≫ The imaging unit 111 stores the three-dimensional point cloud information and images captured and transmitted by the LiDAR 411 and camera 412 in the imaging information database 130. The imaging unit 111 also stores the identification information of the imaging device and the time of acquisition in the imaging information database 130.

[0028] ≪Control Unit: Three-Dimensional Model Generation Unit≫ The 3D model generation unit 112 generates a 3D model, which is a textured 3D mesh, based on 3D point cloud information and images in the imaging information database 130. This 3D model is a generated 3D model that represents objects (heavy materials) present at the site. More specifically, the 3D model generation unit 112 refers to the position and orientation of the LiDAR 411 and generates a 3D mesh based on multiple 3D point cloud information. Methods for generating the 3D mesh include the Alpha Shape method and the Poisson method. The 3D mesh is composed of polygons and positions that represent the surface of the object being imaged.

[0029] One method for combining multiple three-dimensional point cloud data is ICP (Iterative Closest Point), but the three-dimensional model generation unit 112 may generate a three-dimensional mesh using a method different from ICP. For example, the three-dimensional model generation unit 112 may extract feature quantities contained in each three-dimensional point cloud data and generate a three-dimensional mesh by combining the three-dimensional point cloud data by referring to the corresponding feature quantities. Examples of feature quantities include edges and sharp points.

[0030] Next, the 3D model generation unit 112 generates a UV map and texture by referring to the 3D mesh, image, and the position and orientation of the camera 412. The texture is an image that is applied to the 3D mesh. The UV map shows the correspondence between the polygons (pixels) that make up the 3D mesh and the texture (pixels). In this case, the correspondence of the UV map may be calculated based on the image taken at the time of imaging. In this case, the output texture image is an image that has not been converted from the image taken at the time of imaging.

[0031] The three-dimensional model generation unit 112 calculates a texture based on the image captured by the camera 412 if there is only one camera 412 that has captured an image of the surface of the object to be imaged, which corresponds to a polygon. More specifically, the three-dimensional model generation unit 112 calculates the pixels of the texture corresponding to the pixels of the polygon based on the image (pixels) captured by the camera 412.

[0032] The following describes the case where multiple cameras 412 capture images of the surface of an object corresponding to a polygon. The three-dimensional model generation unit 112 calculates the pixels of the polygon (the texture pixels corresponding to those pixels) according to the image captured by the camera 412 corresponding to the pixel, and the distance between the position of the pixel (the position of the polygon) and the camera 412. For example, the three-dimensional model generation unit 112 uses the image from the camera 412 at the shortest distance. The three-dimensional model generation unit 112 may use the average value of pixels included in multiple images, or it may use a weighted average value where closer distances are given more weight.

[0033] The 3D model generation unit 112 may calculate the texture based on the imaging angle rather than the distance between the polygon and the camera 412. The 3D model generation unit 112 uses the image from the camera 412 that is closest to the camera, where the angle between the line connecting the polygon and the camera 412 and the perpendicular line of the polygon is minimized. The 3D model generation unit 112 may also use an average value for the pixels, weighting more heavily on the angle between the line connecting the polygon and the camera 412 and the perpendicular line of the polygon.

[0034] The texture includes images captured by camera 412 and images calculated based on distance or angle. Alternatively, the texture may include images corresponding to a three-dimensional mesh (polygon) represented by a UV map.

[0035] As described above, the imaging analysis device 100 includes a three-dimensional model generation unit 112 that generates a generated three-dimensional model 122, which is a three-dimensional model of the object to be imaged, based on the imaging information. The imaging information consists of three-dimensional point cloud information and images of the object being imaged. The generated 3D model 122 is a textured 3D mesh, which is a 3D mesh with a texture applied to it.

[0036] The imaging information consists of multiple three-dimensional point cloud data and images of the target object, captured by multiple imaging devices. The three-dimensional model generation unit 112 generates a three-dimensional mesh and texture by referring to the positions and orientations of multiple imaging devices (see imaging device information database 140), and also generates a UV map which represents the correspondence between the three-dimensional mesh and the texture.

[0037] The three-dimensional model generation unit 112 calculates the pixels included in the texture based on images captured by multiple imaging devices, according to the distance between the point on the object to be imaged corresponding to the pixel and the imaging device. The three-dimensional model generation unit 112 calculates pixels based on the image captured by the imaging device with the smallest distance. The three-dimensional model generation unit 112 calculates pixels based on images captured by an imaging device, which are weighted according to distance. The three-dimensional model generation unit 112 takes the pixels included in the texture and uses the average of images captured by multiple imaging devices corresponding to points on the object to be imaged that correspond to those pixels.

[0038] The texture includes an image captured by the imaging device and an image composed of pixels calculated by the three-dimensional model generation unit 112. The texture includes an image corresponding to the UV map.

[0039] ≪Control Unit: Comparison Unit≫ The comparison unit 113 compares the generated three-dimensional model 122, which is a three-dimensional model generated by the three-dimensional model generation unit 112, with the reference three-dimensional model, which is a three-dimensional model in the reference three-dimensional model database 160, and calculates the difference. More specifically, the comparison unit 113 first detects the type of object to be imaged using the recognition model 121 based on the texture of the generated three-dimensional model 122 or the image from the camera 412. The comparison unit 113 may also detect the type of object to be imaged by comparing the size and shape of the reference three-dimensional model and the generated three-dimensional model 122. In this case, the length in the longitudinal direction of the object to be imaged (e.g., steel sheet pile) is compared taking into account the possibility that it may have been cut and shortened, or that it may have been damaged or deformed.

[0040] The reference 3D model does not have to be one of the 3D models stored in the reference 3D model database 160. Alternatively, the end faces and sides of materials may be detected from the video, and their size and shape may be measured using the generated 3D model based on that information. Furthermore, if multiple types of materials are mixed, certain clusters (similar in shape or distance) may be detected based on the video and 3D information, and the above comparison may be performed for each unit. This cluster detection may be determined by using a threshold or similar method based on distance similarity using video recognition or 3D information.

[0041] Next, the comparison unit 113 uses the recognition model 121 to detect and count individual objects based on the image from the camera 412 showing the cross-section (vertical plane in the longitudinal direction) of the object to be imaged, or the texture of the generated three-dimensional model 122. The comparison unit 113 may also count the number of objects to be imaged based on the volume of the generated three-dimensional model 122 and the volume of the reference three-dimensional model. Alternatively, the comparison unit 113 may obtain the portion occupied by each object in the generated three-dimensional model 122 based on the count result.

[0042] The comparison unit 113 also calculates the size of the object to be imaged, particularly its length in the longitudinal direction, based on the generated three-dimensional model 122. Furthermore, the comparison unit 113 compares the shape of the generated three-dimensional model 122 with that of a reference three-dimensional model to detect damage or deformation as a difference. The comparison unit 113 obtains identification information of the reference three-dimensional model corresponding to the type of object to be imaged detected by referring to the object information database 150. Subsequently, the comparison unit 113 may obtain the reference three-dimensional model corresponding to the identification information from the reference three-dimensional model database 160 and compare it with the generated three-dimensional model 122 to detect the difference in size (length). It is desirable to perform these processes on the parts of the generated three-dimensional model 122 that correspond to each target object.

[0043] As described above, the imaging analysis device 100 includes a comparison unit 113 that compares the generated three-dimensional model 122 with a reference three-dimensional model, which is a three-dimensional model of an object, and the generated three-dimensional model 122, and calculates the difference. The comparison unit 113 counts the number of objects to be imaged. The comparison unit 113 detects the type of object to be imaged based on the image or texture, compares the reference three-dimensional model and the generated three-dimensional model 122 corresponding to the detected type, and calculates the difference in at least one of the size and shape.

[0044] ≪Control Unit: Display Control Unit≫ The display control unit 114 outputs the imaging analysis result screen 510 (see Figure 2 below) to a display connected to the input / output unit 180. Figure 2 is a screen configuration diagram of the imaging analysis result screen 510 according to the first embodiment. In the area 511 where the generated three-dimensional model 122 for which a type has been detected is displayed, comments 512 including the detected type, number, size (length), and damage / deformation status are added. In the area 513 where the generated three-dimensional model 122 for which no type has been detected is displayed, comments 514 indicating that it was not detected are added. Areas with dark hatching are areas that are blind spots of the imaging device. Furthermore, when multiple materials or multiple types of materials are mixed, the detected materials and material units that could not be detected (units with similar shapes or close distances measured in three dimensions) may be displayed side by side as shown in Figure 2.

[0045] Note that the undetected region 513 is not necessarily an undetected region of any type. For imaging target objects whose size (length) is not expected to change, if the comparison unit 113 determines that there is a difference in size between the generated three-dimensional model 122 and the reference three-dimensional model, the display control unit 114 displays the differing portion as an undetected region.

[0046] By pressing the "Edit" button for comments 512 and 514, an editing screen for the comment content (not shown) is displayed, allowing the operator, who is a user of the imaging analysis device 100, to modify comments 512 and 514. By pressing the "Add Comment" button 515, the operator can add a region or a comment about that region. By pressing the "Add Image" button 516, the operator can acquire a suitable position and orientation for imaging the undetected region 513.

[0047] Figure 3 is a screen configuration diagram of the imaging analysis results screen 520, which displays a pointer 526 indicating the position and orientation of additional imaging according to the first embodiment. The worker takes an image using LiDAR or a camera at the position and orientation indicated by the pointer 526 and sends the imaging information to the imaging analysis device 100. The imaging analysis device 100 regenerates the generated three-dimensional model 122, re-executes type detection and counting, and redisplays the imaging analysis results screen 530 (see Figure 4 below).

[0048] Figure 4 is a screen configuration diagram of the imaging analysis result screen 530 after additional imaging according to the first embodiment. The undetected area 513 (see Figure 2) is gone, and the steel sheet pile area 531 is correctly detected and displayed. The correct length is displayed in comment 532. By performing additional imaging in this way, the imaging analysis device 100 can reduce the undetected portion and display more accurate information.

[0049] ≪Control Unit: Additional Imaging Support Unit≫ Returning to Figure 1, let's continue the explanation of the control unit 110. The additional imaging support unit 115 calculates a suitable position and orientation (see pointer 526 in Figure 3) for imaging the undetected region 513 (see Figure 2). The additional imaging support unit 115 calculates a position and orientation that allows the additional imaging location, which is the undetected region 513, to be imaged from the front, and displays it as pointer 526.

[0050] The additional imaging support unit 115 may also transmit its position and orientation to the imaging device to guide the worker who is performing the imaging. The imaging device should preferably be a device equipped with LiDAR, a camera, a satellite positioning system antenna, an angular velocity sensor, etc., such as a smartphone with LiDAR. Such an imaging device may measure its own position and orientation and guide the worker so that imaging can be performed at the received position and orientation.

[0051] As described above, the imaging analysis device 100 includes an additional imaging support unit 115 that outputs a position and orientation suitable for imaging additional imaging locations, which are parts of the object to be imaged where differences have occurred.

[0052] <<Image Analysis Processing>> Figure 5 is a flowchart of the imaging analysis process according to the first embodiment. At the start of the imaging analysis process, it is assumed that imaging information has already been stored in the imaging information database 130. In step S11, the three-dimensional model generation unit 112 generates a three-dimensional mesh representing the object to be imaged based on multiple three-dimensional point cloud information.

[0053] In step S12, the 3D model generation unit 112 generates a UV map and texture by referencing the 3D mesh, image, and the position and orientation of the camera 412. The generation of the 3D mesh, UV map, and texture constitutes the generation of a 3D model (generated 3D model). In step S13, the comparison unit 113 uses the recognition model 121 to detect the type of the object to be imaged (generated three-dimensional model).

[0054] In step S14, the comparison unit 113 uses the recognition model 121 to detect individual objects to be imaged and counts the number of objects to be imaged. In step S15, the comparison unit 113 calculates the size of the object to be imaged based on the generated three-dimensional model 122.

[0055] In step S16, the comparison unit 113 compares the shapes of the generated three-dimensional model 122 and the reference three-dimensional model to detect damage or deformation. In step S17, the display control unit 114 outputs the imaging analysis result screen 510 (see Figure 2) to the display.

[0056] In step S18, the display control unit 114 accepts an operation from the user, the operator. If the operation is an instruction to finish (step S18 → finish), the imaging analysis process is terminated. If the operation is the pressing of the "correct" button (step S18 → correct), the display control unit 114 proceeds to step S19. If the operation is the pressing of the "add comment" button 515 (step S18 → add comment), the display control unit 114 proceeds to step S20. If the operation is the pressing of the "add image" button 516 (step S18 → additional image), the display control unit 114 proceeds to step S21.

[0057] In step S19, the display control unit 114 modifies comments 512 and 514 according to the operator's instructions and returns to step S17. In step S20, the display control unit 114 adds an area and a comment for that area according to the operator's instructions and returns to step S17.

[0058] In step S21, the additional imaging support unit 115 calculates a suitable position and orientation for imaging the undetected region 513, displays it as a pointer 526 (see Figure 3), and returns to step S11. In steps S11 and S12, the three-dimensional model generation unit 112 generates a new three-dimensional model including the additionally acquired imaging information.

[0059] As described above, the three-dimensional model generation unit 112 generates a new three-dimensional model 122 based on the additional imaging information obtained from capturing additional imaging locations and the imaging information (see steps S21, S11, and S12).

[0060] Features of the imaging and analysis device The imaging and analysis device 100 generates a three-dimensional model based on multiple three-dimensional point cloud data and images of the object to be imaged, and calculates the type, size, and number of the object to be imaged. With such an imaging and analysis device 100, it is possible to remotely identify the type of object to be imaged, count the number of objects, and check the damage status of heavy materials such as steel sheet piles, which can streamline inspection work, for example. Alternatively, it can reduce on-site work, thereby improving worker safety.

[0061] The imaging and analysis device 100 can also be used to check the progress of construction work. By comparing the three-dimensional model of the building under construction as a reference three-dimensional model with the generated three-dimensional model 122, the work manager can understand the differences from the construction plan and check the progress.

[0062] The imaging analysis device 100, in addition to measuring the size using a three-dimensional model generated based on the imaging information, calculates the difference in size / size and shape by comparing it with a reference three-dimensional model. Furthermore, the imaging analysis device 100 designates this difference as an undetected region. When additional imaging is instructed, the imaging analysis device 100 calculates an imaging position and orientation suitable for additional imaging of the region in question, and supports the additional imaging.

[0063] ≪Modification: Three-dimensional point cloud information≫ In the first embodiment described above, the three-dimensional point cloud information was obtained from the LiDAR411. However, the three-dimensional point cloud information may be generated / obtained using technologies such as artificial intelligence or stereo vision based on image information.

[0064] <<Modification: 3D Mesh Generation>> While methods such as the Alpha Shape method and the Poisson method have been described for generating a three-dimensional mesh, other methods may also be used. Here, we assume that depth cameras are installed so that they have the same viewpoint as the LiDAR411. A three-dimensional mesh can also be generated by creating depth images calculated from the three-dimensional point cloud for each LiDAR411 installed, and then integrating these depth images using a TSDF (Truncated Signed Distance Function) or similar method.

[0065] <<Modification: Integration of three-dimensional information>> While ICP was described as a method for combining multiple 3D point cloud data, other methods may also be used. Alternatively, a 3D mesh may be generated from the 3D point cloud data acquired by multiple LiDAR411s, and then these meshes may be combined to create a single 3D mesh. Alternatively, a single 3D mesh may be generated directly by combining the 3D point cloud data acquired by each LiDAR411.

[0066] As another method, the imaging and analysis device 100 generates a textured three-dimensional mesh based on the three-dimensional point cloud information acquired by each LiDAR 411. The imaging and analysis device 100 may also display multiple textured three-dimensional meshes generated simultaneously when viewing the textured three-dimensional model, making them appear as if they were a single three-dimensional model.

[0067] ≪Modification: Additional imaging≫ In the embodiment described above, the additional imaging support unit 115 calculates an imaging position and orientation suitable for additional imaging and displays it as a pointer 526 (see Figure 3). The additional imaging support unit 115 may also instruct the robot to perform additional imaging. The additional imaging support unit 115 may transmit the calculated imaging position and orientation to a self-propelled robot or drone equipped with an imaging device and instruct it to perform additional imaging. The robot transmits the additionally captured imaging information to the imaging analysis device 100.

[0068] <<Modified Version: Three-Dimensional Model Generator>> The imaging analysis device may be a three-dimensional model generation device that does not include a comparison unit 113. In such an imaging analysis device, the three-dimensional model generation unit 112 generates a generated three-dimensional model 122 based on three-dimensional point cloud information and images captured from various directions. The three-dimensional model generation unit 112 generates textures based on multiple images, and has high color reproduction accuracy of the object being imaged. By viewing this generated three-dimensional model 122 from various viewpoints, the operator can grasp the condition of the object being imaged, including any damage.

[0069] ≪Second Embodiment≫ The video analysis device according to the second embodiment measures water levels in real time solely by analyzing video from a surveillance camera. This eliminates the need to install meters for measuring water levels, eliminates the need for maintenance of water level measuring instruments and equipment, and allows for the cancellation of age-related errors in water level measurement through auto-calibration. Furthermore, it offers advantages in terms of cost and aesthetics, as well as the ability to measure water levels in real time.

[0070] Figure 6 is a schematic diagram showing the video analysis device 610 according to the second embodiment. The video analysis device 610 is comprised of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a spatial alignment unit 617, a measurement unit 618, and a display control unit 600.

[0071] The three-dimensional data 614 is obtained by pre-measuring an object using a three-dimensional measurement device such as LiDAR. The video data 611 is video data of the same object as the one captured by LiDAR, and was captured with a two-dimensional camera. The video data 611 is, for example, a video stream in which multiple video frames are arranged in chronological order, but it may also be a still image.

[0072] The three-dimensional data feature conversion unit 615 calculates three-dimensional features by converting the features by assuming two-dimensional projected data through perspective projection of the input three-dimensional data 614 from multiple viewpoints. In other words, the three-dimensional data feature conversion unit 615 extracts features from three-dimensional data obtained by measuring an object with a three-dimensional measuring device by assuming two dimensions through perspective projection from multiple viewpoints. The three-dimensional data feature conversion unit 615 extracts boundary information of surfaces detected from normals included in the three-dimensional data obtained by measuring the object as the aforementioned features. Here, the object is, for example, a bridge pier, and boundary information of surfaces detected from normals included in the three-dimensional data of the bridge pier is extracted as features and used for water level measurement. The boundary information of surfaces detected from normals included in the three-dimensional data of a structure is edge information of the shape of the structure.

[0073] The three-dimensional data features calculated by the three-dimensional data feature conversion unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 only needs to be measured once initially, and it is not necessary to continuously measure the object. Alternatively, if three-dimensional information already exists, that information may be used.

[0074] The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. In other words, the video feature conversion unit 612 extracts the features of video data in which an object is captured by a camera. The video feature conversion unit 612 extracts edge information from the video data 611 in which the object is captured as features. Here, the object is, for example, a bridge pier. In water level measurement, the video feature conversion unit 612 extracts edge information from the video in which the bridge pier is captured as features. Furthermore, the video feature conversion unit 612 may use three-dimensional data 614 when extracting features from the video data 611 in which the object is captured. This allows the video feature conversion unit 612 to suitably extract the features of the bridge pier. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613. Furthermore, the video feature conversion unit 612 may extract features using multiple video frames from different time series. This allows the video feature conversion unit 612 to extract features regardless of the noise in individual frames. Additionally, the video feature conversion unit 612 may extract features based on the position information of the three-dimensional measuring device and the position information of the camera. This allows the video feature conversion unit 612 to extract features appropriately while considering how surfaces that can be suitably measured by the three-dimensional measuring device are captured by the camera.

[0075] The spatial alignment unit 617 compares the data in the video feature storage unit 613 and the data in the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data that is closest to the video data 611. In other words, the spatial alignment unit 617 compares the features of the three-dimensional data stored in the three-dimensional data feature storage unit 616 with the features of the video data stored in the video feature storage unit to determine the two-dimensional projection data that has the closest field of view to the video data. For this comparison, since the data in the video feature retention unit 613 may detect not only the edges of structures but also edges that appear on the image due to shadows, etc., it is best to compare only the feature portion of the three-dimensional data feature retention unit 616. The spatial alignment unit 617 should perform a comparison at a predetermined height or higher based on the three-dimensional data 614. The predetermined height is, for example, the water surface. Furthermore, for this comparison, it is best to first compare two-dimensional projection data at a general viewpoint or field of view to narrow down the possible viewpoints and fields of view, and then calculate the closest field of view hierarchically by making finer changes to the viewpoint and field of view.

[0076] The measurement unit 618 measures the size of an object in the video by comparing the feature quantities of the two-dimensional projection data determined by the spatial alignment unit 617 with the feature quantities of the video data stored by the video feature quantity storage unit. The display control unit 600 displays the aligned video data and three-dimensional data, and superimposes the water levels indicating caution and danger onto the three-dimensional data. This allows the user to see whether the current water level is something to be cautious about or whether it is dangerous.

[0077] As a specific measurement method, the measurement unit 618 converts the waterline information in the video into height information in the three-dimensional data by comparing the waterline feature quantities in the video data with the feature quantities of the three-dimensional data stored in the three-dimensional data feature quantity holding unit 616.

[0078] Furthermore, the measurement unit 618 uses the feature quantities of the video data at a given time as a reference and compares the feature quantities of the video data at different times to measure the water level based on the difference. This is because, for example, when the water level of a river rises due to flooding, occlusion regions occur where structures such as bridges are obscured by water and are no longer detected as feature quantities in the video. If this is represented as a histogram of feature quantities for each water level, a large difference will occur in the histogram between the reference feature quantity and the feature quantity when the water level increases. The measurement unit 618 measures the increase in water level from the part where this difference occurs and converts it into a water level by adding the height to the reference value. In this case, when determining the water level, the measurement unit 618 may compare the histogram with the reference starting from the highest water level and select a point where the value has changed more than a certain threshold, or a point where the trend is significantly different from the surrounding heights. This makes it possible to calculate river water level information in real time based on video data captured by surveillance cameras. Furthermore, the measurement unit 618 performs measurements in short time units to measure the water level, but it is not necessary to perform spatial alignment processing each time.

[0079] Figure 7 shows an example of video data of a bridge pier. Bridge piers are installed in rivers, and the shape of the structure and the waterline are visible on the surface of the piers. By calculating the height information of the three-dimensional data of this shape and waterline, and comparing it with the features of reference video data, river water level information can be obtained. To obtain the shape information of this structure, it is preferable to detect boundary line (edge) information between different surfaces from the three-dimensional data based on the condition of the normal vectors, which are vectors that represent the surface of the surface, and use this as a feature.

[0080] Figure 8 is a flowchart of the measurement visualization process according to the second embodiment. First, the three-dimensional data feature transformation unit 615 acquires high-precision three-dimensional data 614 (step S40). Then, the three-dimensional data feature transformation unit 615 extracts boundary information of surfaces detected from the normals contained in the three-dimensional data 614 (step S41). Next, the spatial alignment unit 617 associates the image features with the boundary information of the surfaces included in the three-dimensional data (step S42). The measurement unit 618 measures the difference in occlusion among the associated image features, where feature points appearing in the two-dimensional projection data are hidden by the video data, and creates a height-specific match amount histogram (step S43). By comparing this with the histogram at the reference time, the water level is determined (step S44), and the process shown in Figure 8 is completed.

[0081] Figure 9 is a data flow diagram of the video analysis device 610 according to the second embodiment. During data collection, the LiDAR 710 performs three-dimensional measurement of the target object and generates a point cloud 721. The video analysis device 610 then generates a mesh 722 and hands it over to the Raycast unit 730. The Raycast unit 730 corresponds to, for example, the Raycast function of the game engine Unity (registered trademark), and performs a process that emits a transparent ray from a specific object and obtains the coordinates of another object that the ray hits.

[0082] When the camera is installed, the Raycast unit 730 determines the two-dimensional projection data that is closest to the video data when the camera captures the target object, based on camera parameters 731 that indicate the position and direction of the camera that captures the image, from the objects represented by the mesh 722. This two-dimensional projection data is stored in the virtual normal image feature database 740. The virtual normal image feature database 740 further stores the LiDAR / camera transformation matrix 741 and Raycast resolution data 742.

[0083] During operation, the surveillance camera 750 captures the target object and generates an RGB image 751. The video analysis device 610 generates feature data 752 from the RGB image 751 and passes it to the feature matching unit 760 and the feature comparison unit 770. The feature matching unit 760 compares the feature points of the two-dimensional projection data stored in the virtual normal image feature database 740 with the feature data 752. The feature matching unit 760 then outputs the best-matching normal image feature 761. The feature comparison unit 770 compares the feature data 752, the best-matching normal image feature 761, and the feature quantities at the reference time to generate a height-specific match amount histogram 771. The water level estimation unit 780 estimates the current water level by determining this height-specific match amount histogram 771.

[0084] Figure 10 is a diagram showing the specific hardware configuration of the video analysis device 610 according to the second embodiment. The video analysis device 610 is, for example, a computer and incorporates a feature processing plugin 830 and a water level measurement plugin 840. The video analysis device 610 also stores an image capture information database 881, three-dimensional data 882, an image feature database 883, feature matching processing results 884, and analysis results 885. The video analysis device 610 is connected to a surveillance camera 810, a video management device 820, a three-dimensional measurement device 850 such as LiDAR, and an initial data generation device 860.

[0085] The surveillance camera 810 captures video and hands it over to the video management device 820. The video management device 820 stores the video captured by the surveillance camera 810 and outputs image data to the feature processing plugin 830 and the water level measurement plugin 840.

[0086] The three-dimensional measuring device 850 is, for example, a LiDAR, which measures an object in three dimensions, writes the shooting location ID to the shooting information database 881, and writes the mesh data and shooting location ID to the three-dimensional data 882. The initial data generation device 860 comprises a virtual normal image generation unit 861 and a feature extraction unit 862. The virtual normal image generation unit 861 generates a normal image from the shooting location ID and mesh data. The feature extraction unit 862 generates a feature image and a depth image from the normal image. The initial data generation device 860 stores the normal image, feature image, and depth image in the image feature database 883.

[0087] The feature processing plugin 830 comprises a feature extraction unit 831 and a feature matching unit 832. The feature extraction unit 831 extracts features from image data and outputs them to the feature matching unit 832. The feature matching unit 832 receives the features of the image data as input, along with the shooting location ID and camera parameters from the shooting information database 881.

[0088] The feature matching unit 832 passes the shooting location ID to the image feature database 883 and reads the feature image corresponding to the shooting location ID. The feature matching unit 832 matches the features of the image data with the feature image and selects the normal image ID. The feature matching unit 832 stores the normal image ID and the shooting location ID in the feature matching processing result 884.

[0089] The water level measurement plugin 840 includes a feature extraction unit 841 and a water level measurement unit 842. Image data is input to the feature extraction unit 841 from the video management device 820. The feature extraction unit 841 extracts feature points from the normal image ID and image data and outputs them to the water level measurement unit 842. The water level measurement unit 842 measures the water level from the feature points, the normal image ID, the feature image, and the depth image. The water level measurement unit 842 stores the shooting location ID, shooting time, image data, and water level in the analysis results 885. The analysis result display device 870 acquires and displays the water level and other information from the analysis results 885. This makes it possible to measure the water level from the video feed of the surveillance camera 810. In this embodiment, three-dimensional data was acquired in advance using LiDAR, but methods that utilize already acquired three-dimensional information may also be used.

[0090] ≪Third Embodiment≫ The video analysis device according to the third embodiment creates an overhead view of a construction site or similar location from a free viewpoint without distortion. This makes it easy to grasp the progress of construction work.

[0091] Figure 11 is a schematic diagram showing the video analysis device 610A according to the third embodiment. The video analysis device 610A is comprised of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a spatial alignment unit 617, and a three-dimensional data generation unit 619.

[0092] The three-dimensional data 614 is obtained by measuring objects, such as those at a construction site, using a three-dimensional measuring device such as LiDAR each time. The video data 611 is video of the same objects as those captured by LiDAR, etc., but captured with a two-dimensional camera.

[0093] The three-dimensional data feature transformation unit 615 calculates three-dimensional data features by performing perspective projections of the input three-dimensional data 614 from multiple viewpoints, assuming two-dimensional projected data. The three-dimensional data features calculated by the three-dimensional data feature transformation unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 is acquired each time the construction progresses.

[0094] The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613.

[0095] The spatial alignment unit 617 compares the data from the video feature storage unit 613 and the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data that is closest to the video data 611. The three-dimensional data generation unit 619 generates texture-mapped three-dimensional data by mapping the texture of the video data onto the three-dimensional data based on the two-dimensional projection data determined by the spatial alignment unit 617. In other words, the three-dimensional data generation unit 619 associates video data with mesh information generated from the three-dimensional data corresponding to the two-dimensional projection data determined by the spatial alignment unit 617. By rendering this three-dimensional data with a game engine such as Unity®, it becomes possible to view the object from a free viewpoint.

[0096] Figure 12 is a data flow diagram of a modified image analysis device 620. The video analysis device 620 includes a texture extraction unit 622, a mesh data conversion unit 625, and a viewpoint transformation rendering unit 626, and is connected to multiple cameras 621a, 621b, a long-range LiDAR 623, and a portable LiDAR 624.

[0097] Multiple cameras 621a and 621b are positioned at different viewpoints, capturing images of the object from multiple perspectives. The long-range LiDAR 623 measures the three-dimensional data of the object from a point at a long distance. The portable LiDAR 624 measures the three-dimensional data of the object from a point in close proximity.

[0098] The mesh data generation unit 625 integrates the three-dimensional data of the object and converts it into mesh data. The mesh data generated by the mesh data generation unit 625 is output to the texture extraction unit 622.

[0099] The texture extraction unit 622 receives video input from multiple cameras 621a and 621b. The texture extraction unit 622 then converts the three-dimensional mesh data into two-dimensional projection data as viewed from each viewpoint of cameras 621a and 621b, associates each two-dimensional projection data with each video, and extracts textures from the video to be applied to the mesh data. In this way, the texture extraction unit 622 creates mesh data with textures applied.

[0100] The viewpoint transformation rendering unit 626 is, for example, the game engine Unity®, and renders textured mesh data from an arbitrary viewpoint. This makes it possible to observe an object, such as a construction site, from a desired viewpoint. In this embodiment, three-dimensional information is captured sequentially using LiDAR, but methods that utilize already acquired three-dimensional information are also acceptable. Examples of already acquired three-dimensional information include BIM (Building Information Modeling) and CIM (Construction Information Modeling).

[0101] ≪Fourth Embodiment≫ The video analysis device according to the fourth embodiment measures the length and volume of heavy temporary construction materials on the spot simply by taking a picture with a mobile device such as a smartphone, based on three-dimensional data acquired by LiDAR. The video analysis device grasps the condition of the material bundle from the video captured by the mobile device and performs matching processing with the length analysis information of the cross section. As a result, for example, it is possible to measure a material with a length of 20m from a distance of 20m and classify the length of the material in 10cm increments in real time.

[0102] The video analysis device according to the fourth embodiment enables the measurement of the length and volume of cargo on a truck all at once. This makes it possible to perform work that was previously done manually, one item at a time, by climbing onto the truck bed, safely and efficiently.

[0103] Figure 13 is a schematic diagram showing the video analysis device 610B according to the fourth embodiment. The video analysis device 610B is comprised of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a video recognition unit 601, and a measurement unit 602.

[0104] The three-dimensional data 614 is obtained by pre-measuring an object using a three-dimensional measurement device such as LiDAR. The video data 611 is a video of the same object as the one captured by LiDAR, but it was captured with a two-dimensional camera.

[0105] The three-dimensional data feature transformation unit 615 calculates three-dimensional data features by performing perspective projections of the input three-dimensional data 614 from multiple viewpoints, assuming two-dimensional projected data. The three-dimensional data features calculated by the three-dimensional data feature transformation unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 only needs to be measured once initially, and it is not necessary to continuously measure the object.

[0106] The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613.

[0107] The image recognition unit 601 recognizes an object from the image data 611, determines whether it is a bundle of materials, and if the objects are overlapping, it recognizes the number of individual pieces or items. The image recognition unit 601 then compares the feature data from the image feature storage unit 613 with the feature data from the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data that is closest to the image data 611, and recognizes the three-dimensional data corresponding to the recognized object.

[0108] The measurement unit 602 measures the size of an object based on the range of the three-dimensional data of the object recognized by the image recognition unit 601.

[0109] Specifically, this size measurement involves measuring the length, width, and height based on the three-dimensional data of the recognized object (for example, a bundle of materials), calculating the volume from that data, and detecting the number and shape of materials contained in the bundle from the video, and then measuring the length of each individual material.

[0110] Figure 14 is a flowchart of the measurement visualization process according to the fourth embodiment. The user first measures three-dimensional data from the side of the material using LiDAR (step S50). Next, the user takes pictures of the cross-section and overview of the material with a camera (step S51). The measured three-dimensional data and the captured video data are input to the video analysis device 610B. The video recognition unit 601 of the video analysis device 610B recognizes the bundle of materials from the video data 611 (step S52).

[0111] The video recognition unit 601 of the video analysis device 610B performs a matching process between the video data 611 and the three-dimensional data 614 to understand the status of the material bundle (step S53). This makes it possible to recognize the portion of the three-dimensional data 614 that represents the material bundle. Furthermore, the video recognition unit 601 of the video analysis device 610B analyzes the number of target materials from their cross-sectional shape (step S54). After that, the measurement unit 602 of the video analysis device 610B analyzes the length of the materials in the longitudinal direction (step S55), and the process shown in Figure 14 is completed.

[0112] <<Other variations>> Although several embodiments and modifications of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein and elsewhere, as well as in the scope of the invention and its equivalents as described in the claims.

[0113] Hardware Configuration The imaging analysis device 100 and video analysis devices 610, 610A, 610B, and 620 according to the above-described embodiment are implemented by a computer 900 having a configuration such as that shown in Figure 15. Figure 15 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the imaging analysis device 100 and video analysis devices 610, 610A, 610B, and 620 according to the above-described embodiment. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, input / output interface 905 (labeled as input / output I / F (Interface) in Figure 15), communication interface 906 (labeled as communication I / F in Figure 15), and media interface 907 (labeled as media I / F in Figure 15). The computer 900 may be equipped with an HDD (Hard Disk Drive) instead of the SSD 904, or it may be equipped with an HDD in addition to the SSD 904.

[0114] The CPU 901 operates based on programs stored in the ROM 902 or SSD 904 and is controlled by the control unit 110 in Figure 1. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

[0115] The CPU 901 controls input devices 910, such as a mouse or keyboard, and output devices 911, such as a display or printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905.

[0116] SSD904 stores programs executed by CPU901 and data used by those programs. Communication interface906 receives data from other devices (not shown) via the communication network (e.g., LiDAR411,710 and cameras412,612a,612b) and outputs it to CPU901, and also transmits data generated by CPU901 to other devices via the communication network.

[0117] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

[0118] For example, when the computer 900 functions as an imaging analysis device 100 or video analysis devices 610, 610A, 610B, or 620 according to the above-described embodiment, the CPU 901 of the computer 900 realizes the functions of the imaging analysis device 100 or video analysis devices 610, 610A, 610B, or 620 by executing a program 128 (see Figure 1) loaded onto the RAM 903. The CPU 901 reads the program from the recording medium 912 and executes it. Alternatively, the CPU 901 may read the program from another device via a communication network, or it may install the program 128 from the recording medium 912 onto the SSD 904 and execute it. [Explanation of Symbols]

[0119] 100 Imaging and Analysis Device (Image Analysis Device) 111 Image acquisition unit 112 Three-dimensional model generation unit 113 Comparison Section 114 Display Control Unit 115 Additional Imaging Support Unit 121 Recognition Models 122 Generative 3D Models 128 Programs 130 Imaging Information Database 140 Imaging device information database 150 Object Information Database 160 Reference 3D Model Database (Reference 3D Models) 510, 520, 530 Imaging analysis results screen 610, 610A, 610B Image Analysis System (Imaging Analysis System) 611 Video Data 612 Video Feature Conversion Unit 613 Video Feature Storage Unit 614 Three-dimensional data 615 Three-dimensional data feature transformation unit 616 Three-dimensional data feature retention unit 617 Spatial alignment section 618 Measurement Unit 620 Video Analysis Device 622 Texture Extraction Unit 625 Mesh Data Generation Unit 626 Viewpoint Transformation Rendering Unit 621a Camera 621b Camera 623 Long Range LiDAR 624 Portable LiDAR 710 LiDAR 619 Three-dimensional data generation unit 721 point cloud 722 mesh 730 Raycast section 731 Camera Parameters 740 Virtual Normal Image Feature Database 741 LiDAR / Camera Transformation Matrix 742 Raycast resolution data 750 surveillance cameras 751 RGB image 752 Feature Data 760 Feature Matching Section 770 Feature Comparison Section 761 Normal Image Features 771 Match amount histogram by height 780 Water level estimation section 830 Feature Processing Plugin 840 Water Level Measurement Plugin 881 Shooting Information Database 882 3D data 883 Image Feature Database 810 Surveillance Cameras 820 Video Management Device 850 Three-dimensional measuring device 860 Initial Data Generation Device 861 Virtual Normal Image Generation Unit 862 Feature Extraction Unit 831 Feature Extraction Unit 832 Feature Matching Section 841 Feature Extraction Unit 842 Water level measurement unit 885 Analysis results 870 Analysis result display device

Claims

1. A three-dimensional model generation unit generates a three-dimensional model of the object being imaged based on imaging information, A comparison unit compares a reference three-dimensional model, which is a three-dimensional model of an object to be compared with the generated three-dimensional model and is also a three-dimensional model of an object to be compared with the object to be imaged, with the generated three-dimensional model and calculates the difference. The system includes an additional imaging support unit that outputs a suitable position and orientation for imaging the additional imaging area, which is the part of the object to be imaged where the difference occurs. Imaging and analysis device.

2. The three-dimensional model generation unit, Based on the additional imaging information obtained from capturing the aforementioned additional imaging locations and the aforementioned imaging information, a new three-dimensional model is generated. The imaging and analysis apparatus according to claim 1.

3. The aforementioned imaging information is, Multiple three-dimensional point cloud information and images of the object to be imaged, captured by multiple imaging devices, The generated three-dimensional model is, A textured 3D mesh is a 3D mesh with a texture applied to it. The three-dimensional model generation unit, Referencing the positions and orientations of the multiple imaging devices, The three-dimensional mesh and the texture are generated, and a UV map representing the correspondence between the three-dimensional mesh and the texture is generated. The imaging and analysis apparatus according to claim 1.

4. The three-dimensional model generation unit, The pixels included in the aforementioned texture Based on the images captured by the multiple imaging devices, the distance between a point on the object to be imaged, corresponding to the pixel, and the imaging device is calculated. The imaging and analysis apparatus according to claim 3.

5. The three-dimensional model generation unit, The aforementioned pixels are calculated based on the image captured by the imaging device with the minimum distance. The imaging and analysis apparatus according to claim 4.

6. The three-dimensional model generation unit, The aforementioned pixels are calculated based on the image captured by the imaging device, weighted according to the distance. The imaging and analysis apparatus according to claim 4.

7. The three-dimensional model generation unit, The pixels included in the aforementioned texture The average of images captured by multiple imaging devices corresponding to a point on the object to be imaged that corresponds to the pixel in question. The imaging and analysis apparatus according to claim 3.

8. The aforementioned texture is The image includes an image captured by the imaging device and an image composed of pixels calculated by the three-dimensional model generation unit. The imaging and analysis apparatus according to claim 4 or 7.

9. The aforementioned texture is Includes an image corresponding to the aforementioned UV map The imaging and analysis apparatus according to claim 3.

10. The aforementioned imaging information is, The three-dimensional point cloud information and image of the object to be imaged are as follows: The generated three-dimensional model is, A textured 3D mesh is a 3D mesh with a texture applied to it. The comparison unit is, The type of the object to be imaged is detected based on the image or texture. The reference 3D model and the generated 3D model corresponding to the type of detection result are compared to calculate the difference in at least one of the size and shape. The imaging and analysis apparatus according to claim 1.

11. The comparison unit is, The number of objects to be imaged is counted. The imaging and analysis apparatus according to claim 10.

12. Computers, A three-dimensional model generation unit generates a three-dimensional model of the object being imaged based on imaging information, A comparison unit compares a reference three-dimensional model, which is a three-dimensional model of an object to be compared with the generated three-dimensional model and is also a three-dimensional model of an object to be compared with the object to be imaged, with the generated three-dimensional model and calculates the difference. The system includes an additional imaging support unit that outputs a suitable position and orientation for imaging the additional imaging area, which is the part of the object to be imaged where the difference occurs. A program for functioning as an imaging and analysis device.

13. The imaging and analysis device, The steps include generating a three-dimensional model, which is a three-dimensional model of the object being imaged, based on the imaging information, and A step of comparing the generated three-dimensional model with a reference three-dimensional model, which is a three-dimensional model of an object that is to be compared with the object to be imaged, and the generated three-dimensional model, and calculating the difference. The steps include: outputting a suitable position and orientation for imaging the additional imaging area, which is the part of the object to be imaged where the difference occurs; and executing the above steps. Imaging and analysis methods.

Images