Image generation processing device, 3D shape restoration system, image generation processing method and program

Abstract

This image generation processing device comprises: an intersection point set generation unit that acquires an intersection point set from the connection relationship among intersection points of laser lines detected in each frame of a video which is a group of successive frames imaged by an imaging means which is obtained by integrating and movably fixing a camera for imaging a target area during a specific period and a plurality of plane-crossing laser oscillation units for projecting plane-crossing laser beams onto a substance in the target area and from tracking results of the intersection points of the laser lines detected in the successive frames; a simultaneous equation generation unit that, on the basis of the fact that the intersection points in the intersection point set are on two laser planes formed by the plane-crossing laser beams, generates simultaneous equations by serially obtaining a plurality of constraint equations and satisfying the constraint equations simultaneously; a three-dimensional position estimation unit that reconstructs, in a projection space, three-dimensional coordinates of a laser plane by solving the simultaneous equations; and a three-dimension reconstruction unit that reconstructs, in the projection space, three-dimensional coordinates of laser line reflection positions by an optical cutting method using the estimated three-dimensional coordinates of the laser plane and the laser lines detected in each frame of the video.

Description

Technical Field

[0001] The present invention relates to an image generation processing apparatus, a three-dimensional shape restoration system, an image generation processing method, and a program.

Background Art

[0002] Obtaining a detailed (high-density) and accurate three-dimensional scene for various environments is important for applications to various environments that are difficult for humans to access, such as internal human body scans by endoscopes, creation of three-dimensional maps of the seabed, and acquisition of three-dimensional shapes of planetary images such as Mars and satellite images. Regarding the restoration of the three-dimensional shape of the target area (scene; subject), various methods are introduced in the background art section of Patent Document 1, including passive methods that use only images and active methods.

[0003] And, Patent Document 1 describes an image processing apparatus that restores a three-dimensional shape using the coplanarity included in a two-dimensional image of a scene and the geometric conditions of the scene. Specifically, Patent Document 1 describes an image processing apparatus that restores a three-dimensional shape from an input two-dimensional image, and based on the coplanarity extracted from the two-dimensional image, a first calculation unit that calculates a first solution that is a solution of a first intersection point that is an intersection of a first plane corresponding to the coplanarity and the first planes, and a second calculation unit that eliminates the degree of freedom of the first solution using the geometric conditions included in the scene in the two-dimensional image and calculates a second solution that is a solution of the first intersection point and the first plane.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, the camera was fixed and did not move. Furthermore, Patent Document 1 only connected the intersections of the same frame of the (moving) image. Therefore, the scene had to be fixed, and it was impossible to measure not only the movement of the camera but also the movement of objects within the scene. From the perspective of performing precise and accurate 3D reconstruction in various environments, such as measuring outdoor scenes from a car or measuring moving people, further improvement was needed.

[0006] In response to such demands, a solution for camera movement involves estimating the 3D scene and egomotion (the camera's own speed and movement) without calibration, such as Kinect. Geometry-based methods such as Fusion and vision-based methods such as Visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) are known. Geometry-based methods assume that three-dimensional shapes are dense, but in reality, they are generally not dense. On the other hand, methods based on vision, such as Visual SLAM or SfM, are considered promising due to their simplicity, requiring only a single camera and being self-calibrating. However, Visual SLAM or SfM are passive methods, based on feature point detection and matching, and can only reconstruct sparse 3D shapes when the scene has little texture. Furthermore, in extreme environments with few feature points in the scene, feature points are often lost while tracking each frame.

[0007] Until now, no image generation processing device had been known that could adapt to various environments, self-calibrate (without requiring external calibration such as synchronization between the camera and laser), and perform precise and accurate 3D reconstruction.

[0008] The problem that this invention aims to solve is to provide an image generation processing device that can adapt to various environments, is self-calibrating, and can perform precise and accurate 3D reconstruction. [Means for solving the problem]

[0009] According to a first aspect of the present invention, by using a motion image captured by a shooting means equipped with multiple plane-intersecting lasers and a camera, a set of intersection points is obtained from the laser lines detected in each frame of the motion image, a group of constraint equations based on coplanarity is obtained in a chain, and the three-dimensional coordinates of the laser plane are reconstructed in projective space by solving a system of simultaneous equations, and the three-dimensional coordinates of the reflection positions of the laser lines are reconstructed in projective space by the light section method, thereby enabling self-calibration. Since synchronization between the camera and the plane-intersecting lasers and geometric constraints are not required, it is suitable for extreme environments such as the deep sea or outer space, where it is difficult to adjust the device afterward or for human intervention to perform calibration, and even under such conditions, high-density and accurate three-dimensional reconstruction is possible. According to a second aspect of the present invention, using a shooting means equipped with multiple plane-intersecting lasers and a camera, the three-dimensional coordinates of the reflection positions of laser rays obtained by the light section method are reconstructed in projected space. At the same time, arbitrary feature points are detected by Visual SLAM or SfM to obtain the camera position and orientation in a Euclidean coordinate system and the three-dimensional reconstruction result. The projected reconstruction result of the three-dimensional coordinates of the reflection positions of laser rays is then Euclidean upgraded, and the three-dimensional reconstruction result is integrated using the camera position and orientation to reconstruct a wide range of three-dimensional shapes, thereby achieving self-calibration. This eliminates the need for geometric constraints on synchronization between the camera and the plane-intersecting lasers, and thus enables precise (high-density) and accurate three-dimensional reconstruction even in extreme environments. In other words, the inventors have found that according to the first and second embodiments, it is possible to provide an image generation processing apparatus that can adapt to various environments, does not require synchronization between the camera and the laser, and can perform detailed (high-density) and accurate 3D reconstruction, thereby solving the above-mentioned problems. The configuration of the present invention, which is a specific means for solving the above problems, and a preferred configuration of the present invention are described below.

[0010] [1] An image generation processing device for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation unit obtains an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and from the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on the two laser planes formed by intersecting plane lasers, we can obtain a chain of constraint equations. Multiple constraints A system of linear equations generation unit that generates a system of linear equations by solving them simultaneously, A 3D position estimation unit for a plane, which reconstructs the 3D coordinates of the laser plane in projective space by solving simultaneous equations, Reconstructed by the 3D position estimation unit An image generation processing apparatus including a 3D reconstruction unit that uses the 3D coordinates of the laser plane and the laser beam detected in each frame of a moving image to reconstruct the 3D coordinates of the reflection position of the laser beam in projective space using the light section method. [2] The image generation processing apparatus according to [1], wherein the intersection set generation unit creates an intersection set graph. [3] An image generation apparatus according to [1] or [2], comprising a first calculation unit that takes known relative three-dimensional positions of intersecting plane lasers and three-dimensional coordinates of the laser plane estimated in projection space as input, and upgrades the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the reflection positions of the laser rays to Euclidean coordinates. [4] A 3D point calculation unit that takes a group of consecutive frames of a moving image as input and performs Euclidean 3D reconstruction using a self-calibration method, Each frame of the video is equipped with a corresponding point detection unit that detects, as a corresponding point, the 3D points obtained by the 3D point calculation unit that are located on the laser line. A second calculation unit uses the detected corresponding points to upgrade the 3D coordinates of the laser plane and the 3D coordinates of the laser beam reflection position to Euclidean coordinates. An image generation processing apparatus according to any one of items [1] to [3], including the following. [5] The 3D point calculation unit includes a SLAM analysis unit that detects arbitrary feature points using Visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result. It includes an integration unit that reconstructs the 3D shape by integrating the 3D reconstruction results using the 3D coordinate reconstruction results of the laser beam reflection position and the camera position and orientation. The image generation processing apparatus described in [4], which realizes a laser plane self-calibration method using three-dimensional reconstruction results obtained by Visual SLAM or SfM. [6] An epipolar line calculation unit obtains an epipolar line by calculating the straight line that passes through the same intersection point in the image, based on the results of tracking the intersection points of the laser beams, A correspondence point search unit based on epipolar constraints searches for corresponding points in any frame within a moving image along the epipolar line, The system includes a correspondence point detection unit that detects, as a corresponding point, 3D points obtained by a 3D point calculation unit using Visual SLAM or SfM, that are located on a laser line connected to a searched corresponding point. A third calculation unit uses the detected corresponding points to upgrade the 3D coordinates of the laser plane and the 3D coordinates of the laser beam reflection position to Euclidean coordinates. The image generation apparatus described in [5], including the image generation apparatus. [7] From the results of tracking the intersections of the laser beams, the straight line that passes through the same intersection on the image is calculated for each intersection. Then obtain epipolar rays , Epipolar line calculation unit, A correspondence point search unit based on epipolar constraints searches for corresponding points in any frame within a moving image along the epipolar line, A 3D position estimation unit for a plane reconstructs the 3D coordinates of the laser plane in projective space using the searched corresponding points, An image generation processing apparatus according to any one of items [1] to [5], including the following. [8] An image generation processing device for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. A 3D reconstruction unit that uses the 3D coordinates of the laser plane formed by plane-intersecting lasers estimated by any method, and the laser rays detected in each frame of the moving image, to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method, A SLAM analysis unit that detects arbitrary feature points using Visual SLAM or SfM and obtains the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration unit that reconstructs the 3D shape by integrating the 3D reconstruction results using the 3D coordinate reconstruction results of the laser beam reflection position and the camera position and orientation, An image generation processing apparatus comprising: [9] An image generation processing apparatus according to any one of [1] to [8], comprising a laser beam estimation unit that calculates the estimated laser beam position in consecutive frames using a trained model of a convolutional neural network (CNN).

[10] A mask creation unit is provided which creates a mask by morphological transformation from the estimated position of the laser beam, The image generation processing apparatus described in [9] obtains three-dimensional points by applying Visual SLAM or SfM to each frame of a moving image, ignoring the brightness at the mask position.

[11] An image generation apparatus according to any one of [1] to

[10] , comprising a fourth calculation unit that re-estimates the three-dimensional coordinates of the laser plane to minimize the discrepancy between the two three-dimensional coordinates when, in a group of frames, the three-dimensional coordinates re-estimated in frame n and the three-dimensional coordinates re-estimated in frame k are at the same location within the target region.

[12] The steps include dividing the video into blocks of m consecutive frames, and integrating the three-dimensional coordinates restored by the three-dimensional restoration unit for each block to obtain a second set of three-dimensional coordinates, Image generation processing apparatus according to any one of [1] to

[11] , comprising a fifth calculation unit that re-estimates the three-dimensional coordinates of the laser plane to minimize the discrepancy between two three-dimensional coordinates when the two second three-dimensional coordinates are at the same location within the target region.

[13] An image generation processing apparatus described in any one of the items [1] to

[12] , A three-dimensional shape restoration system comprising imaging means including a single camera that images a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material within the target area, and a fixing unit that integrates and fixes these so that they can be moved.

[14] The camera and the planar intersecting laser emitter are housed inside the housing, A three-dimensional shape reconstruction system as described in

[13] , wherein the target area is underwater.

[15] The shooting means further comprises a recording unit and a moving unit, A three-dimensional shape reconstruction system according to

[13] or

[14] , wherein the shooting means moves while capturing a moving image of the target area and records it in a recording unit.

[16] An image generation processing method for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation process is performed to obtain an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on the two laser planes formed by intersecting plane lasers, we can obtain a chain of constraint equations. Multiple constraints A system of equations generation process, which involves setting up a system of equations together, The process involves reconstructing the 3D coordinates of the laser plane in projective space by solving a system of simultaneous equations, and estimating the 3D position of the plane. Reconstructed in the 3D position estimation processA 3D reconstruction process is performed using the 3D coordinates of the laser plane and the laser beam detected in each frame of the moving image to reconstruct the 3D coordinates of the reflection position of the laser beam in projective space using the light section method. An image generation processing method that includes [specific details].

[17] A SLAM analysis step to detect arbitrary feature points using Visual SLAM or SfM and obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration process to reconstruct the 3D shape by integrating the 3D reconstruction results using the 3D coordinate reconstruction results of the laser beam reflection position and the camera position and orientation, The image generation processing method according to

[16] , comprising:

[18] An image generation process for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. A 3D reconstruction unit that uses the 3D coordinates of the laser plane formed by plane-intersecting lasers estimated by any method, and the laser rays detected in each frame of the moving image, to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method, A SLAM analysis process that detects arbitrary feature points using Visual SLAM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration process to reconstruct the 3D shape by integrating the original reconstruction result of the 3D coordinates of the laser beam reflection position and the 3D reconstruction result using the camera position and orientation, An image generation processing method comprising:

[19] A program to be executed by an image generation processing device that reconstructs a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation function that obtains an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and from the tracking results of laser beam intersections detected in consecutive frames. Each intersection point in the set of intersections is one of the two plane-intersecting lasers laser Since it is on a plane, multiple constraint equations can be obtained in a chain, Multiple constraints A simultaneous equation generation function that generates simultaneous equations by solving them simultaneously, This system uses simultaneous equations to reconstruct the 3D coordinates of the laser plane in projective space, providing a 3D position estimation function for the plane. A 3D reconstruction function that uses the estimated 3D coordinates of the laser plane and the laser beam detected in each frame of the moving image to reconstruct the 3D coordinates of the reflection position of the laser beam in projective space using the light section method, A program that executes something. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image generation processing device that can adapt to various environments, is self-calibrated, and can perform detailed (high-density) and accurate 3D reconstruction. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a schematic diagram of Embodiment 1A, which is an image generation processing apparatus according to the first embodiment. [Figure 2] Figure 2(A) is a schematic diagram of a shooting means for obtaining moving images used in the image generation processing apparatus of the present invention. Figures 2(B), (C), and (D) are schematic diagrams of an example of a method for obtaining moving images used in the image generation processing apparatus of the present invention. Figure 2(E) is a schematic diagram of an example of a three-dimensional shape restored by the image generation processing apparatus of the present invention. [Figure 3] Figure 3 is a flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1A. [Figure 4] Figure 4 is another flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1A. [Figure 5] Figure 5 is a schematic diagram of Embodiment 1B, which is an image generation processing apparatus according to the first embodiment. [Figure 6] Figure 6 is a flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1B. [Figure 7] Figure 7 is another flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1B. [Figure 8] Figures 8(a) to 8(d) show flowcharts illustrating the process of creating an intersection set graph using the spatial and temporal connections between intersection points. [Figure 9] Figure 9 shows the geometry (geometric arrangement) of the Euclidean upgrade and the symbols used in calculating the energy function. [Figure 10] Figure 10 is a schematic diagram of Embodiment 2A, which is an image generation processing apparatus according to the second embodiment, as shown in Figure 1. [Figure 11] Figure 11(a) is a photograph of the setup of the photographic means used in Example 1. Figure 11(b) shows the measurement of the actual angle of the restored column. Figure 11(c) is a photograph showing an example of an image taken for the measurement method using a calibration device. Figure 11(d) shows an example of an image taken in the present invention. [Figure 12] Figure 12 is a graph showing the relationship between the number of frames used and the RMSE (Root Mean Square Error, angle) in the evaluation of the 3D shape reconstruction accuracy of Example 1. [Figure 13] Figure 13(A) shows the plane fitting error between the two planes in the evaluation of the 3D shape reconstruction accuracy of Example 1. Figure 13(B) shows the angular error in the evaluation of the 3D shape reconstruction accuracy of Example 1. [Figure 14] Figure 14(A) shows the 3D shape reconstruction results for Kinect V1. Figure 14(B) shows the 3D shape reconstruction results for Kinect Azure. Figure 14(C) shows the 3D shape reconstruction results for the hard calibration method. Figure 14(D) shows the 3D shape reconstruction results for the method of the present invention. [Figure 15]Figure 15(a) is a photograph of the setup of the imaging device used in Example 2. Figure 15(b) is a photograph corresponding to the top view of the target area (scene) used in Example 2. Figure 15(c) is a photograph of an example of the process of reconstructing the three-dimensional coordinates of the laser plane in projected space. Figure 15(d) is an example of an image of the reconstructed mannequin being measured. Figure 15(e) is the reconstructed mannequin. [Figure 16] Figure 16(A) shows the error between MAE [mm] and RMSE [mm] in the evaluation of the 3D shape reconstruction accuracy of Example 2. Figure 16(B) shows the number of reconstructed 3D points in the evaluation of the 3D shape reconstruction accuracy of Example 2. [Figure 17] Figure 17(a) shows the 3D shape reconstruction results for GT. Figures 17(b) and 17(B) show the 3D shape reconstruction results for DSO. Figures 17(c) and 17(C) show the 3D shape reconstruction results for Colmap. Figures 17(d) and 17(D) show the 3D shape reconstruction results for the hard calibration method. Figures 17(e) and 17(E) show the 3D shape reconstruction results for the method of the present invention. [Figure 18] Figure 18(A) shows the result of three-dimensional shape reconstruction using the method of the present invention. [Figure 18-1] Figures 18(a1-A), 18(a1-B), and 18(a1-C) each represent an example of the captured image required for the present invention. Figures 18(a1-1) and 18(a1-2) each show the 3D shape reconstruction result of the frame on the left side of Figure 18(A) from a different angle. [Figure 18-2] Figure 18(b1) shows the result of reconstructing the three-dimensional shape within the frame on the right side of Figure 18(A), viewed from a different angle. Figures 18(b1-A) and 18(b1-B) each represent an example of the captured image required for this invention. [Figure 18-3] Figure 18(B) shows the 3D shape reconstruction results of Colmap. Figure 18(b1) shows the 3D shape reconstruction results of the left-hand frame in Figure 18(B) viewed from a different angle. Figure 18(b2) shows the 3D shape reconstruction results of the right-hand frame in Figure 18(B) viewed from a different angle. [Figure 18-4]Figure 18(C) shows the result of 3D shape reconstruction using Meshroom. Figure 18(c1) shows the result of 3D shape reconstruction of the frame on the left side of Figure 18(C) from a different angle. Figure 18(c2) shows the result of 3D shape reconstruction of the frame on the right side of Figure 18(C) from a different angle. [Figure 19] Figure 19 shows the results of the laser beam estimation. [Figure 20] Figure 20(A) shows the traced intersections. Figure 20(B) shows the connected intersections. Figure 20(C) shows the graph of the resulting set of intersections. [Figure 21] Figure 21 shows the result of mask creation. [Figure 22] Figure 22 shows the results of the correspondence point detection. [Figure 23] Figure 23 shows the 3D shape reconstruction results before optimization. [Figure 24] Figure 24 is a diagram illustrating the algorithm of the optimization method. [Figure 25] Figure 25 illustrates the process of generating an average shape from a dual shape using a mesh. [Figure 26] Figure 26 illustrates the process of determining the loop section and the average camera position and orientation. [Figure 27] Figure 27 illustrates the process of bundle adjustment at the block level. [Figure 28] Figure 28 shows the cost changes during bundle adjustment. [Figure 29] Figure 29 illustrates the process of bundle adjustment on a frame-by-frame basis. [Figure 30] Figure 30 shows the 3D shape reconstruction result after optimization. [Figure 31] Figure 31 shows the 3D shape reconstruction results after optimization in water. [Modes for carrying out the invention]

[0013] The present invention will be described in detail below. The following descriptions of constituent elements may be based on representative embodiments or specific examples, but the present invention is not limited to such embodiments. In this specification, numerical ranges represented by "~" mean a range that includes the numbers written before and after "~" as the lower and upper limits.

[0014] [Image generation processing device] A first aspect of the image generation processing apparatus of the present invention is an image generation processing apparatus that reconstructs a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation unit obtains an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and from the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on the two laser planes formed by intersecting plane lasers, a chain of constraint equations is obtained, and a system of simultaneous equations is generated by solving the set of constraint equations simultaneously. A 3D position estimation unit for a plane, which reconstructs the 3D coordinates of the laser plane in projective space by solving simultaneous equations, It includes a 3D reconstruction unit that uses the estimated 3D coordinates of the laser plane and the laser rays detected in each frame of the moving image to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method.

[0015] A second aspect of the image generation processing apparatus of the present invention is an image generation processing apparatus that reconstructs a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. A 3D reconstruction unit that uses the 3D coordinates of the laser plane formed by plane-intersecting lasers estimated by any method, and the laser rays detected in each frame of the moving image, to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method, A SLAM analysis unit that detects arbitrary feature points using Visual SLAM or SfM and obtains the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, It includes an integration unit that reconstructs the 3D shape by integrating the 3D reconstruction results using the reconstruction results of the 3D coordinates of the reflection position of the laser beam and the camera position and orientation.

[0016] With these configurations, the image generation processing apparatus of the first or second embodiment can adapt to various environments, self-calibrate, and perform precise and accurate 3D reconstruction. Preferred embodiments of the present invention will be described below. In the following description, the first embodiment will be mainly described, but the preferred embodiments of the first embodiment are the same as the preferred embodiments of the second embodiment.

[0017] <First aspect: Embodiment 1A> A preferred embodiment of the image generation apparatus of the first embodiment will be described with reference to the drawings. Figure 1 is a schematic diagram of Embodiment 1A, which is an image generation processing apparatus according to the first embodiment. The image generation processing device 1 shown in Figure 1 comprises an intersection set generation unit, a simultaneous equation generation unit, a three-dimensional plane position estimation unit, and a three-dimensional reconstruction unit. These parts, such as the intersection set generation unit, the simultaneous equation generation unit, the three-dimensional plane position estimation unit, and the three-dimensional reconstruction unit, which are directly related to image generation processing, are also called the image generation processing unit. As shown in Figure 1, it is preferable that the image generation processing unit further comprises a laser beam estimation unit. The image generation processing unit may be integrated so that it can be executed by an application (program) that performs a predetermined function using a CPU or the like. The image generation processing device 1 shown in Figure 1 may optionally include a control unit, an input unit, a storage unit, a display unit, an operation unit, etc., and may also include an output unit, etc. The overall general function of the image generation processing device 1 is to perform image generation processing from moving images input from the shooting means 100, restore a three-dimensional shape, and output it. Furthermore, the implemented image generation processing device 1 may be a computer such as a personal computer on which an application (program) that performs a predetermined function is installed, or it may be configured as a device dedicated to image generation processing configured to perform a predetermined function. Moreover, each part constituting the image generation processing device 1 may be electrically connected to each other via a bus, or they may be interconnected via a network as a client-server system or a cloud system.

[0018] (Video) The moving image used in this invention is a series of consecutive frames captured by an imaging means that includes a single camera that captures a target region over a specific period of time, multiple plane-intersecting laser emitters that project plane-intersecting lasers onto a substance in the target region, and a fixed part that integrates these and fixes them so that they can move. Even if the fixed part moves during measurement, the following process is still possible because it is self-calibrated by the method of this invention. Specifically, if a displacement of the fixed part is detected, self-calibration should be performed again. As a method for detecting such displacement, for example, whether or not the laser intersection lies on the epipolar line can be used as a criterion. Alternatively, inconsistencies in the reconstructed shape, for example, the reconstructed results of two laser planes that intersect in the same frame do not intersect in three-dimensional space, or the three-dimensional position of the reconstructed result from the same laser line differs significantly between different frames, may be used. Figure 2(A) is a schematic diagram of an imaging means 100 for obtaining moving images used in the image generation processing apparatus of the present invention. The imaging means 100 shown in Figure 2(A) includes a single camera 101 for imaging a target area over a specific period of time, a plurality of plane-crossing laser emitters 102 for projecting plane-crossing lasers 111 onto material in the target area, and a fixing unit 103 for integrating and fixing these together so that they can be moved. Preferably, the imaging means 100 shown in Figure 2(A) further includes a housing 121, a recording unit 131, and a moving unit 141. Each intersecting laser 111 emitted from the intersecting laser emitter 102 consists of two line lasers, and the two laser planes are fixed almost perpendicularly and then precisely self-calibrated. It is assumed that the camera's intrinsic parameters are calibrated in advance. However, since the camera's intrinsic parameters can be calibrated using self-calibration methods such as SfM and SLAM, which will be described later, the system can be operated even without calibration. On the other hand, the position of the imaging means (the relative positional relationship between the camera and each laser plane) does not presuppose prior calibration and therefore requires calibration according to the present invention. During the scanning process in which the target area is photographed over a specific period, multiple images (groups of frames) constituting the moving image are captured at different camera positions and orientations by moving the entire imaging means. The position of the imaging means is initially determined by a self-calibration method such as SfM or SLAM, and then self-calibrated by the method of the present invention. Figures 2(B), (C), and (D) are schematic diagrams of an example of a method for obtaining a moving image used in the image generation processing apparatus of the present invention. As shown in Figures 2(B), (C), and (D), the entire imaging means is moved to photograph the target area (subject) by including an imaging means that includes a fixed part that integrates one camera and multiple plane-crossing laser emitters so that they can be moved. The image generation processing apparatus of the present invention receives the captured video image as input and reconstructs a three-dimensional shape. Figure 2(E) is a schematic diagram of an example of a three-dimensional shape reconstructed by the image generation processing apparatus of the present invention. Details of the imaging method will be described later in the explanation of the 3D shape reconstruction system.

[0019] (Laser beam estimation unit) The laser beam estimation unit detects laser beams in each frame of the video. The laser beam estimation unit can detect pixels by selecting those with a brightness threshold above a certain level. However, if the laser output is not large enough for the target area (scene), noise increases, requiring noise reduction. Therefore, as an algorithm to avoid these problems, it is preferable that the laser beam estimation unit uses a trained model of a convolutional neural network (CNN) to calculate the estimated laser beam position in consecutive frames. The estimated (detected) laser beams are input to the intersection point generation unit.

[0020] (Intersection set generation part) The intersection point generation unit obtains an intersection point set from the connection relationships between laser line intersections detected in each frame of the video, and from the tracking results of laser line intersections detected in consecutive frames. The resulting set of intersection points is input into the system of linear equations generator. Furthermore, in this invention, it is preferable that the intersection set generation unit creates an intersection set graph. The intersection set graph will be described later.

[0021] (System of equations generation unit) The simultaneous equation generation unit obtains a series of constraint equations from the fact that each intersection point in the set of intersection points lies on the two laser planes formed by intersecting plane lasers, and generates a system of simultaneous equations by solving the set of constraint equations simultaneously. The generated system of equations is input into the 3D position estimation unit for the plane.

[0022] (3D position estimation unit in a plane) The 3D position estimation unit for the plane reconstructs the 3D coordinates of the laser plane in projective space by solving a system of simultaneous equations. The estimated (reconstructed in projective space) 3D coordinates of the laser plane are input into the 3D reconstruction unit.

[0023] (3D reconstruction section) The 3D reconstruction unit uses the estimated 3D coordinates of the laser plane and the laser beam detected in each frame of the moving image to reconstruct the 3D coordinates of the reflection position of the laser beam in projective space using the light section method. On the other hand, the 3D reconstruction unit may use the laser rays detected in each frame of the moving image to reconstruct points on the laser rays in 3D, based on the Euclidean solution. The 3D reconstruction unit can reconstruct the intersection lines between the laser plane, which corresponds to the coplanarity used in the 3D position estimation unit for the plane, and the target region (scene). Furthermore, it can also extract and reconstruct the intersection lines between planes other than the above-mentioned plane and the target region (scene). In this way, dense shapes can be reconstructed. The restored 3D information may be stored in a memory unit, displayed on a display unit, or printed out on paper using an output unit such as a printer (not shown).

[0024] (Control Unit) The control unit is the part that controls the operation of the entire image generation processing device 1.

[0025] (Input section) The input section is the part where information is input to the image generation processing device 1 from an external source. In this embodiment, a two-dimensional moving image is input.

[0026] (Storage part) The memory unit includes fixed storage disks such as HDDs (Hard Disk Drives), removable storage disks such as CDs (Compact Discs) and DVDs (Digital Versatile Disks), and fixed or removable semiconductor memory. In this embodiment, the memory unit stores the 2D video image before processing, the 3D shape reconstructed from the 2D video image, and intermediate information obtained by processing the 2D video image. Here, the intermediate information includes, for example, information on the intersection set, information on the intersection set graph, information on the 3D coordinates of the laser plane in projective space (projection solution), information on the 3D coordinates of the reflection position of the laser ray in projective space, and information on the Euclidean solution calculated by the Euclidean upgrade calculation described later.

[0027] Furthermore, it is preferable that the memory unit stores a program for executing the image generation processing method described below. This program is invoked by the user operating the control unit and causes the functions of each of the above-mentioned parts to be executed. Specifically, the program operates each part to reconstruct 3D shape data from the input 2D moving image data.

[0028] (Display) The display unit can be, for example, an LCD, a CRT (Cathode Ray Tube), or a video projector, and it displays input 2D moving images or 3D shapes reconstructed based on these 2D moving images.

[0029] (Operation unit) The control unit is, for example, a keyboard or mouse, and by operating this control unit, the image generation processing device reconstructs a 3D shape from a 2D moving image.

[0030] (Image generation processing method of Embodiment 1A) Figure 3 is a flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1A. First, the imaging device moves around the target area (scene; object to be photographed) and projects multiple planar lasers. Then, the target area (scene; subject to be photographed) is captured, and a video image consisting of multiple images as a series of consecutive frames is obtained. Thus, in this invention, a plane-intersecting laser is used to acquire coplanarity information. Coplanarity information can be obtained by irradiating a target area with a plane-intersecting laser while taking images, and extracting the trajectory of the point irradiated by the line laser from the captured video image. The plane in the three-dimensional space through which the plane-intersecting laser passes is sometimes called the laser plane.

[0031] Next, for each frame of the video, multiple laser lines projected onto the target area are detected. Preferably, each laser line is detected independently. Detection can be achieved by selecting pixels with a brightness threshold above a certain level, but if the laser output is not large enough for the target area (scene), noise increases, requiring noise reduction. Furthermore, the process of separating and making the detected multiple laser lines independent is not obvious. Therefore, it is efficient to detect them independently using a deep neural network (DNN). In this case, it is even more efficient to not only detect the laser lines but also to assign an ID to each line, indicating which planar intersecting laser emitter emitted the line. Furthermore, it is more preferable to use a pre-trained convolutional neural network (CNN) model to calculate the estimated laser beam position in consecutive frames. Known methods can be used to create DNNs and CNNs, as well as their pre-trained models. For example, the method by Furukawa et al. (Non-Patent Literature A) can be used. (Non-patent document A) Ryo Furukawa, Genki Nagamatsu, Shiro Oka, Takahiro Kotachi, Yuki Okamoto, Shinji Tanaka, Hiroshi Kawasaki, "Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan", MICCAI workshops AE-CAI, CARE (MIAR), Vol. 6, Iss. 6, pp.249-254, 10.2019

[0032] Next, since each laser line is detected independently, the intersection points between two laser lines are calculated. It is preferable to determine the connection relationships between these intersection points by connecting them under the constraint of coplanarity. The condition that each intersection point lies on the two laser planes is a coplanarity condition. Here, a point cloud is said to be coplanar when it lies on the same plane. For example, if the surface of an object is a plane, all points on that plane are coplanar. Even if there is no pattern on the plane and the points on the plane cannot be observed as a pattern in the image, the points on the plane are still coplanar. In this way, there is a lot of coplanarity in a target region (scene) composed of planar structures. The coplanarity of points on a physically existing surface that are actually observed in an image is called explicit coplanarity (second coplanarity). Furthermore, a set of points that have such coplanarity will henceforth be described as an explicitly coplanar set of points. On the other hand, there are countless coplanar features in space that are not normally visible but become observable under certain specific conditions. For example, the trajectory of a line laser beam illuminating an object is a set of coplanar points. Such coplanar features are usually invisible and can only be observed when light strikes them. In this embodiment, this is called implicit coplanar feature (first coplanar feature). Furthermore, the group of coplanar points detected in this way is called an implicit coplanar feature, and the curve obtained by observing an implicit coplanar point group with a camera is called an implicit coplanar curve. While explicit coplanar features are generally observed only on the planar parts of an object, implicit coplanar features can be observed on any part of the object's surface, including free curved surfaces.

[0033] Next, we track the intersections of the laser beams detected in consecutive frames. If the shooting process is not terminated at this stage, the process will return to the process of photographing the target object. The image generation processing apparatus of the present invention can perform image generation processing automatically by program, enabling high-speed, real-time 3D reconstruction. Therefore, if it is expected that a sufficient set of intersection points cannot be obtained by tracking the intersection points of the laser beams, or if a sufficient set of intersection points has actually not been obtained, the process can return to the process of photographing the target object in real time.

[0034] On the other hand, if the imaging is to be stopped at this stage, the next step is to create a system of equations from the intersection points. If it is expected that a sufficient set of intersection points can be obtained by tracking the intersection points of the laser beams, or if a sufficient set of intersection points is actually obtained, the imaging may be stopped. Then, we solve the simultaneous equations and reconstruct the three-dimensional coordinates of the laser plane in projective space. Finally, the shape is reconstructed using the light sectioning method. Furthermore, since the solution obtained solely from the coplanarity condition has at least four degrees of freedom, it is necessary to eliminate the remaining degrees of freedom in order to obtain a Euclidean shape. In this specification, this is called a Euclidean upgrade (Euclidean restoration). The solution obtained by a Euclidean upgrade is called a Euclidean solution. In order to perform a Euclidean upgrade, it is necessary to use conditions other than coplanarity. However, in Embodiment 1A, a Euclidean upgrade is not required.

[0035] Figure 4 is another flowchart illustrating an image generation method using the image generation apparatus of Embodiment 1A. In Figure 4, a step of creating an intersection set graph is included between the step of tracking intersections and the step of creating simultaneous equations from the intersections. To efficiently track intersections in a group of frames, it is preferable to use a graph representation to describe the relationships between intersections and the trajectories of the intersections as an intersection set graph. The other steps in Figure 4 are the same as in Figure 3. The accuracy of the relationships between intersections and the trajectories of the intersections is important for the stability of self-calibration, and it is preferable to create an intersection set graph that can guarantee high-precision stability in a group of frames with a larger number of frames to capture than using only two adjacent frames, and that can also streamline the program. Based on the evaluation of the accuracy of coplanarity and epipolar constraints in Embodiment 1, it is preferable to have about 30 frames or more, and more preferably 40 frames or more.

[0036] For other methods, see Japanese Patent Publication No. 2009-32123.

[0037] The method described in ~

[0080] can be adapted, and the contents of that publication are incorporated herein by reference.

[0037] <First aspect: Embodiment 1B> Furthermore, a more preferred embodiment of the image generation apparatus of the present invention will be described. Figure 5 is a schematic diagram of Embodiment 1B, which is an image generation processing apparatus according to the first embodiment. The image generation processing apparatus 1 of Embodiment 1B shown in Figure 5, similar to Embodiment 1A shown in Figure 1, comprises a laser beam estimation unit, an intersection point set generation unit, a simultaneous equation generation unit, a 3D position estimation unit for a plane, and a 3D reconstruction unit. Furthermore, it comprises a mask creation unit, a 3D point calculation unit, an epipolar line calculation unit, a corresponding point search unit, a corresponding point detection unit, a Euclidean upgrade calculation unit (including the first calculation unit, the second calculation unit, and the third calculation unit), and an integration unit. The parts directly related to these image generation processes are also called the image generation processing unit. The image generation processing unit may be integrated so that it can be executed by an application (program) that performs a predetermined function, such as by a CPU. Furthermore, Embodiment 1B may include a fourth and / or fifth calculation unit for optimizing the camera position and orientation throughout the measurement and for more accurate reestimation of the three-dimensional coordinates. In the following, preferred embodiments of Embodiment 1B will be described, primarily focusing on the differences from Embodiment 1A.

[0038] (First calculation unit) Embodiment 1B preferably includes a first calculation unit that takes known relative three-dimensional positions of intersecting plane lasers and the three-dimensional coordinates of the laser plane estimated in projection space as input, and upgrades the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the reflection positions of the laser rays to Euclidean coordinates.

[0039] The first calculation unit has the function of eliminating the degrees of freedom of the 3D coordinates of the laser plane estimated in projective space and the 3D coordinates of the reflection position of the laser ray in projective space (projective solution), thereby upgrading the projective solution to a Euclidean solution.

[0040] (Epipolar line calculation unit and corresponding point search unit) In Embodiment 1B, it is preferable to include an epipolar line calculation unit that calculates the straight line that passes through the same intersection point in the image for each intersection point based on the tracking results of the laser beam intersections, thereby obtaining an epipolar line. Embodiment 1B preferably includes an epipolar constraint-based correspondence point search unit that searches for corresponding points in any frame within a moving image on the epipolar lines obtained by the epipolar line calculation unit. Preferably, the searched corresponding points are input to the corresponding point detection unit. Furthermore, it is preferable that the searched corresponding points are input to a 3D position estimation unit for the plane, which then reconstructs the 3D coordinates of the laser plane in projective space.

[0041] (Mask creation department) Embodiment 1B preferably includes a mask creation unit that creates a mask from the estimated laser beam position using morphological transformation. A known method can be used for the morphological transformation. The mask is input to the SLAM analysis unit of the 3D point calculation unit, and when detecting arbitrary feature points by Visual SLAM or SfM, it is preferable from the viewpoint of robust feature point detection to mask the laser beam region from each frame of the captured video image, and exclude the masked region from feature point detection.

[0042] (3D point calculation part) Embodiment 1B is preferable from the viewpoint of enabling precise (high-density) and accurate 3D reconstruction even in extreme environments, as it includes a 3D point calculation unit that takes a group of consecutive frames of a moving image as input and performs Euclidean 3D reconstruction using Visual SLAM or SfM. This enables self-calibration, which eliminates the need for geometric constraints on synchronization between the camera and the plane-crossing laser. Preferably, the 3D point calculation unit includes a SLAM analysis unit that detects arbitrary feature points using Visual SLAM or SfM and obtains the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result. In the 3D point calculation unit, it is preferable to apply Visual SLAM or SfM to each frame of the video image, ignoring the brightness at the mask position, to obtain 3D points (3D reconstruction results). The obtained 3D reconstruction result (3D points) of the Euclidean coordinate system is input to the corresponding point detection unit, which will be described later. The obtained camera position and orientation in the Euclidean coordinate system are input to the integration unit described later, which enables a self-calibration method for the three-dimensional position of the laser plane and the imaging device.

[0043] (Corresponding point detection unit) In Embodiment 1B, it is preferable to include a corresponding point detection unit in each frame of the moving image that detects corresponding points that lie on the laser line among the three-dimensional points obtained by the three-dimensional point calculation unit, from the viewpoint that it is easy to reconstruct the shape by optical sectioning using the coplanarity that all of these corresponding points lie on the laser line (laser plane).

[0044] Furthermore, it is preferable for the corresponding point detection unit to detect as corresponding points those 3D points obtained by the 3D point calculation unit using a self-calibration method that are located on the laser line connected to the corresponding points searched by the corresponding point search unit based on epipolar constraints, as this facilitates shape reconstruction by optical sectioning using the coplanarity that all of these corresponding points exist on the laser line (laser plane).

[0045] (Second calculation unit and third calculation unit) Embodiment 1B preferably includes a second and a third calculation unit that use the detected corresponding points to upgrade the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the laser beam reflection position to Euclidean coordinates. For the sake of explanation, when the correspondence point detection unit uses a correspondence point detected by the 3D point calculation unit that lies on the laser line among the 3D points obtained by the 3D point calculation unit, it is referred to as the second calculation unit. Furthermore, when the correspondence point detection unit uses a correspondence point detected by the 3D point calculation unit that lies on the laser line connected to a correspondence point search unit based on epipolar constraints, it is referred to as the third calculation unit.

[0046] (Fourth and fifth calculation units) Embodiment 1B may include a fourth and / or fifth calculation unit for optimizing the camera position and orientation throughout the measurement and for more accurate reestimation of the three-dimensional coordinates. The fourth calculation unit reestimates the three-dimensional coordinates of the laser plane to minimize the discrepancy between the two three-dimensional coordinates when the three-dimensional coordinates reconstructed in frame n and the three-dimensional coordinates reconstructed in frame k are at the same location within the target area in a group of consecutive frames captured by the imaging means. Here, frame n and frame k are arbitrary frames, and it is indicated that frame n and frame k are different frames. The fourth calculation unit may operate in combination with a fifth calculation unit, which will be described later.

[0047] Furthermore, Embodiment 1B may include a fifth calculation unit that includes the steps of dividing the video into blocks of m consecutive frames, and integrating the 3D coordinates reconstructed by the 3D reconstruction unit for each block to obtain a second 3D coordinate, and reestimating the 3D coordinate of the laser plane to minimize the discrepancy between the two 3D coordinates when the two second 3D coordinates are at the same position within the target area. In the step of dividing into blocks, it is preferable that the video be divided into blocks of 3 to 100 frames each, and more preferably into blocks of 5 to 50 frames each.

[0048]

number

[0049] The block is generated by integrating point clouds from several dozen frames. The camera position and orientation of the first frame is used as the orientation of the entire block, and the point clouds of the other frames are projected onto the local coordinates of the first frame using their relative orientations to the first frame. In equation (1), T i start P is the relative orientation of the i-frame to the first frame. i This represents the point cloud of the laser irradiation area in the i-frame.

[0050] In the fifth calculation unit, for example, 3D coordinates are integrated every few dozen frames to generate blocks, and at the same time, an average shape is generated in the loop section using the mesh. In this specification, a loop section refers to, for example, the section in which the captured areas overlap when the target area is photographed one and a half times around (see Figure 26). Then, the correspondence between each block and each point of the average shape is obtained using ICP (Iterative Closest Point), and the camera position and orientation relative to the block is optimized by bundle adjustment at the block level (see Figure 24). The series of steps from generating the average shape to bundle adjustment at the block level may be repeated several times. By performing bundle adjustment at the block level, the accurate camera position and orientation can be estimated even with the sparse point cloud of each frame.

[0051] The average shape can be generated by a mesh. In this case, if the pose error is large and the distance between overlapping shapes is small, a mesh will be generated in between them (Figure 25 (left side)). (Figure)) When the distance is large, the shape of the mesh becomes distorted (Figure 25 (right side)). Alternatively, before generating the mesh, the distance between shapes may be forcibly reduced by reintegrating the points of each frame using the average camera position and orientation in the overlapping section of the path.

number

number

[0052] The average camera position and orientation is calculated by manually specifying the start and end frames of the loop section (Figure 26 (left side)) and then taking the weighted average of the translation components of the corresponding camera position and orientation (Figure 26 (right side)). In equations (2) and (3), t i t′ is the translation component of the i-frame's orientation. i is the average of the translation components of the pose in the i-th frame, w is the weight, and is linear with respect to i.

[0053] Finally, using the determined camera position and orientation, the point cloud for each frame is projected.

number

[0054] Next, the correspondence between each point in the block and the average shape is determined using ICP. Since both the block and overall reconstruction results are obtained by integrating the lasers of each frame at the camera position and orientation, the correspondence between the points of each block and the average shape can be determined from the correspondence between the points of the overall shape and the average shape. Bundle adjustment is then performed using the determined correspondence.

[0055] Furthermore, in block-level bundle adjustment, the relative orientation within a block is fixed, which can cause abrupt changes in camera position and orientation between the final frame of a block and the first frame of the next block. For this reason, it is preferable to perform frame-level bundle adjustment in the fourth calculation unit described above to optimize the camera position and orientation so that it changes smoothly. In this case, since block-level bundle adjustment provides a position and orientation close to the correct one at the block level, problems such as lines being too close together in the average shape are resolved, and the minimization of the 3D point distance at the frame level works correctly.

[0056] (Integration Department) Embodiment 1B preferably includes an integration unit that integrates the 3D reconstruction results using the reconstruction results of the 3D coordinates of the laser beam reflection position, the camera position and orientation obtained by the 3D point calculation unit, and the re-estimation results obtained by the fourth and fifth calculation units to reconstruct the 3D shape. By integrating the 3D reconstruction results of each frame of the moving image using the light section method, it is possible to adapt to various environments, self-calibrate, and achieve precise and accurate 3D reconstruction. In particular, integrating multiple 3D reconstruction results allows for the reconstruction of a wide area.

[0057] (Image generation processing method of Embodiment 1B) Figure 6 is a flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1B. In Figure 6, similar to the flowchart illustrating the image generation processing method using the image generation processing apparatus of Embodiment 1A shown in Figure 3 or Figure 4, the following steps are performed from the moving image: (1) laser beam detection, (2) creation of an intersection set graph, and (3) reconstruction of the laser plane. In Figure 6, when performing (3) laser plane reconstruction, it is preferable to construct a system of equations from epipolar constraints in addition to coplanarity. In this case, a search for corresponding points based on epipolar constraints is performed using epipolar lines obtained by tracking intersections. Then, laser intersections on the obtained epipolar lines are detected to obtain corresponding points, and a system of equations including the corresponding points is constructed. By solving the system of equations using linear methods such as singular value decomposition (SVD), a solution with four degrees of freedom can be obtained.

[0058] Subsequently, in Figure 6, before shape reconstruction using the light section method (integrated in Figure 7 using (7) SLAM, etc.), feature points are detected using (5) SLAM, etc., and then (6) nonlinear optimized Euclidean upgrade is performed.

[0059] In Figure 6, (5) Feature point detection using SLAM, etc., arbitrary feature points are detected by Visual SLAM or SfM to obtain the camera position and orientation in a Euclidean coordinate system and the 3D reconstruction result. Since the laser rays on each frame of the video are sparse, Visual SLAM or SfM can robustly detect feature points by excluding the laser ray region from each frame of the captured video. It is preferable to create a mask by applying a morphological filter (extended) to the detected laser rays, and more specifically, it is preferable to create a mask using morphological transformation. Furthermore, it is preferable to directly apply Visual SLAM or SfM to the video image to detect arbitrary feature points and estimate the camera position and orientation (camera ego-motion) in Euclidean coordinate system and the 3D reconstruction result (3D point cloud, which is the initial depth of the target region). By utilizing the camera position and orientation, the obtained 3D point cloud is back-projected onto each frame, points that overlap with the laser beam, i.e., 3D points on the laser beam, are detected as corresponding points, and the 3D coordinates in Euclidean coordinate system for these corresponding points are stored as the initial depth.

[0060] In Figure 6, (6) the nonlinear optimized Euclidean upgrade takes known relative 3D positions of intersecting plane lasers and the 3D coordinates of the laser plane estimated in projective space as input, and upgrades the 3D coordinates of the laser plane and the 3D coordinates of the reflection positions of the laser rays to Euclidean coordinates. In this case, a scale graph may be created and estimated scale parameters may be calculated. By multiplying by the estimated scale parameters, the camera motion parameters can be modified and the Euclidean upgrade can be performed (first calculation unit). Among the known relative three-dimensional positions of intersecting plane lasers, a cross laser configuration can be considered, where the two laser planes are rotated 90 degrees from each other. Alternatively, a parallel laser configuration, where the two lasers are placed parallel to each other, is also acceptable. In these combinations, even when enclosed in an underwater housing, both laser planes can be orthogonal to the interface, thus eliminating the refraction effect, making them preferable. Note that intersecting plane lasers may also be lasers where the two laser planes are rotated at an angle other than 90 degrees from each other; for example, lasers rotated 60 degrees or 45 degrees from each other are possible. Furthermore, intersecting plane lasers may also be lasers where three or more laser planes are rotated at known angles such as 60 or 45 degrees from each other. When capturing moving images on land or in the air, increasing the density of the laser beams is more effective than eliminating the refraction effect in reconstructing a high-density and accurate three-dimensional shape. For example, using a Diffractive Optical Element (DOE) is preferable because it allows for a significant reduction in self-calibration parameters due to central projection. Furthermore, (5) it is preferable to use the initial depth of the 3D points on the laser beam obtained by feature point detection using SLAM, etc., and the camera position and orientation to perform a Euclidean upgrade on the 3D coordinates of the laser plane and the 3D coordinates of the laser reflection position (second calculation unit). Similarly, it is preferable to Euclidean upgrade the 3D coordinates of the laser plane and the 3D coordinates of the laser reflection position using the initial depth of the 3D points on the epipolar line and the camera position and orientation. In this case, it is preferable to use the epipolar line obtained by tracking the intersection points to perform a corresponding point search based on the epipolar constraint, detect 3D points on the epipolar line to obtain corresponding points, and store the 3D coordinates in the Euclidean coordinate system of the corresponding points as the initial depth (third calculation unit). By using the initial depth and camera position / orientation, the 4-degree-of-freedom solution can be upgraded to a Euclidean coordinate system using a bundle adjustment algorithm. In this process, by adding parameters other than the 4-degree-of-freedom parameters to the estimated parameters, it becomes possible to self-calibrate arbitrary parameters. In the case of conventional line reconstruction, the correspondence between multiple frames is limited to the intersections of lines, so a sufficient number cannot be obtained, making convergence practically difficult. However, in this invention, a highly accurate initial solution is obtained through coplanarity reconstruction, making it possible to achieve this.

[0061] In Figure 6, these results are finally integrated using (7) SLAM, etc. In (7) SLAM integration, the 3D shape is reconstructed by integrating the reconstructed 3D coordinates of the laser beam reflection positions and the 3D reconstruction results using the camera position and orientation. By using the estimated parameters, the optical section method can be applied to each frame, and the reconstruction lines can be integrated to reconstruct a precise and accurate shape.

[0062] Figure 7 is another flowchart illustrating an image generation process using the image generation apparatus of Embodiment 1B. The flowchart in Figure 7 provides a detailed explanation of a preferred embodiment of the flowchart in Figure 6. The details of the image generation processing method using the image generation processing apparatus of Embodiment 1B will be described below with reference to Figure 7.

[0063] -The process of tracking intersections, the process of creating a graph of intersection sets- First, we will explain the process shown in Figure 7: the process of detecting laser beams, the process of calculating the intersection points of the detected laser beams, the process of tracking the intersection points, and the process of creating a graph of the intersection points. To estimate the laser plane parameters and obtain a projected solution, a correspondence between intersections and laser lines is necessary. In this invention, a camera moves around a target region and uses moving images captured by emitting multiple intersecting plane lasers. Therefore, it is preferable to obtain the correspondence between intersections between frames using a graph-based approach to achieve robust intersection tracking as described below. Figures 8(a) to 8(d) show flowcharts illustrating the process of creating an intersection set graph using the spatial and temporal connections between intersection points. First, nodes are created by calculating the intersections of laser curves on the image detected by a DNN or CNN. These are spatially connected within the frame to generate the first graph. Intersections in the next frame are temporally connected to the current frame using a nearest neighbor approach, and the temporally connected nodes are grouped to form a single node (Figures 8(a) to 8(b)). The intersection graph obtained at this stage contains errors and noise in both spatial and temporal connections. Therefore, spatial errors are first suppressed by the following approach. First, a new graph is constructed with groups as nodes. In our setup, we do not assume special situations such as triple intersections, so each node in the new graph has only two spatial connections, such as in the vertical and horizontal directions. Therefore, false connections can be eliminated by maintaining only one directed edge in each direction in which each node has the most connections to each other. As a result, each group has at most two directed edges to each other (from Figure 8(b) to Figure 8(c)). Regarding errors in temporal connectivity, disconnections due to occlusion occur frequently. To resolve this, since there are only two spatial connections for each node as described above, if there are multiple directed edges connected to a temporal connectivity setting, temporal connectivity settings in the same direction are merged to leave only one directed edge (Figures 8(c) to (d)). As a result, the spatial and temporal connectivity between intersections is greatly improved, making it possible to track up to 50 to 100 frames. This is a sufficient number of tracking frames to achieve robust self-calibration. Specifically, as described in Example 1 below, the number of tracking frames is preferably 25 or more, more preferably 30 or more, and particularly preferably 40 or more, for stable calibration.

[0064] -The process of creating simultaneous equations from intersection points- The process of creating a system of simultaneous equations from the intersection points in FIG. 7 and the process of solving the system of simultaneous equations to restore the three-dimensional coordinates of the laser plane in the projective space will be described.

[0065] First, by obtaining the projective solution of the plane parameters due to the coplanarity constraint, the restoration of the three-dimensional coordinates of the laser plane in the projective space is performed. After graph generation, p, which is the intersection point between plane i and plane j in frame t [i,j,t] =(u [i,j,y ] , v [i,j,t] ) is obtained. The laser plane π i is represented by the following Equation 1

Equation

Equation

Equation

Equation

Equation

[0066] Next, in Figure 7, when performing laser plane reconstruction, a system of equations is constructed from the epipolar constraint in addition to the coplanarity. Based on the tracking results of the laser beam intersections, we calculate the straight lines (epipolar lines) that pass through the same intersections in the image. Because the relative positions of the camera and laser are fixed, the epipolar lines on the image remain static even if the imaging device moves around during scanning. This means that all points belonging to the same intersection lie on the same epipolar line, and therefore, using more than three frames in the calculation does not increase the essential information. Thus, the maximum number of constraints on points on the same epipolar line is two. In practice, the accuracy of the epipolar lines is important for the stability of self-calibration, and therefore, if the distance between two corresponding points is too small, the accuracy will inevitably be low, so using only two adjacent frames may lead to unstable solutions. In this invention, it is preferable to create a graph that includes the results of tracking intersections of a long series, ensuring long-range, high-precision, and projected solutions for corresponding points. Considering that increasing the number of frames to capture does not increase the constraints on the system of equations, this can be considered a kind of degeneracy condition. Therefore, we will explain which conditions can be solved by the method of the present invention. If M is the number of laser planes, L is the number of observation lines, and N is the number of images taken, then the number of unknown parameters is 3M-4. Here, 4 represents the 3 degrees of freedom (DOF) of the offset and the 1 degree of freedom of the scale parameter. The number of constraints is 2L. Therefore, a solution can be obtained if L ≥ (3M-4) / 2 is satisfied. For example, if we consider the case where a imaging device with four intersecting plane laser emitters is used, M=8 and the minimum value of L is 10.

[0067] - From the process of detecting arbitrary feature points in each image using SLAM and SfM, to Euclidean upgraving Do- Next, we will explain the process from detecting arbitrary feature points in each image using SLAM and SfM to the Euclidean upgrade process. In Figure 7, arbitrary feature points in each image are detected using SLAM and SfM, ignoring the brightness of the mask area, and 1. 3D reconstruction is performed in the Euclidean real coordinate system, and 2. Camera position and orientation estimation is performed. Subsequently, the 3D points are back-projected onto each image, and the 3D points on the laser line are detected to obtain corresponding points. Then, using the corresponding points, the 3D coordinates of the laser plane and the 3D coordinates of the laser reflection position are Euclidean upgraded. Since the projected solution due to coplanarity and epipolar constraints has a maximum of 4 degrees of freedom, it is necessary to upgrade the laser plane parameters to obtain a Euclidean solution. In the first embodiment of this, if the angle between laser planes is known (for example, a cross laser set at a high precision of 90 degrees), it can be used for Euclidean upgrade. In this case, since Euclidean upgrade is possible immediately after projected reconstruction, there are many advantages such as a significant reduction in computational load, faster processing, and reconstruction that is independent of the target region (scene). However, on the other hand, this has the problem of significantly reducing the flexibility of the system configuration, as it is necessary to create a cross laser with high precision in advance or measure its angle. Therefore, in order to avoid such limitations, in the second embodiment of the present invention, it is preferable to use 3D points generated by Visual SLAM or SfM using texture information of the target region (scene) for Euclidean upgrade. The method using Visual SLAM will be described below, but the same can be done with SfM. L is a three-dimensional line between two planes. [i,j] Let's assume that. L [i,j] The corresponding 2D line is l [i,j] Let's assume that the 2D lines detected from the captured image are s [i,t] Let's assume that. [i,t] and s [j,t] The intersection point between m[i,j,t] Let i and j be the laser plane π. i and π j This means that t represents the ID (identification number) of the captured image. [i,t] When it is detected by the line detection algorithm, s [i,t] and π i No response is provided between them. However, L [i,j] and m [i,j,t] The correspondence between them is given by the closest distance l [i,j] and m [i,j,t] It can be obtained by m [i,j,t] is s [i,t] and s [j,t] Since it is an element of m [i,j,t] is s [i,t] From the accumulation of belonging to, [i,t] and L [i] The correspondence can be estimated. Figure 9 shows a schematic diagram of the geometry (geometric arrangement) of the Euclidean upgrade and the symbols used in the calculation of the energy function below.

[0068] After obtaining 3D points using Visual SLAM, the obtained 3D points are placed in P [i] Let i be the plane π. i This means that the 3D points obtained by Visual SLAM are reprojected onto each frame of the captured video image, and the points that overlap with the detected laser beams (corresponding points) are P n [i] Let n be the identification number of the point. Furthermore, let Q be the intersection point. m [i,j] This is expressed as follows, where i and j are the identification numbers of the laser planes, and m is the identification number of the intersection point. To achieve the Euclidean upgrade, we set the energy function to be expressed in equation 5 below.

number

number

[0069] - From the process of detecting 3D points on the epipolar line and obtaining corresponding points to a Euclidean upgrade - The process involves detecting three-dimensional points on the epipolar line and obtaining corresponding points, as well as using these corresponding points to perform a Euclidean upgrade of the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the laser reflection position. This can be done in a similar manner to the process of obtaining 3D points using Visual SLAM and then performing a Euclidean upgrade. This utilizes the property that epipolar lines on a 2D image are the intersection lines of two laser planes in 3D space. As a Euclidean upgrade using known angles between laser planes, it is preferable in this invention to use a Euclidean upgrade that involves detecting 3D points on the epipolar line and obtaining corresponding points, from the viewpoint of increasing the flexibility of the system configuration, such as eliminating the need to create a high-precision cross laser in advance. In this case, since a 3D point cloud lying on the plane is obtained directly, the solution can be obtained directly by plane fitting, etc., which contributes to reducing the amount of computation, speeding up processing, and improving the stability of the solution. For estimating the plane parameters, for example, SVD (Singular Value Decomposition) can be used. Furthermore, combining Euclidean upgrades—one performed by detecting arbitrary feature points in each image using SLAM and SfM, and another performed by detecting 3D points on the epipolar line and obtaining corresponding points—is preferable from the standpoint of synergistically reconstructing a highly accurate 3D shape that cannot be reconstructed by using only one of these methods.

[0070] -Shape restoration process using light sectioning and Euclidean upgrade- As shown in Figure 7, there is no specific order for the Euclidean upgrade in the second and / or third calculation unit and the shape reconstruction process using the optical section method in the three-dimensional reconstruction unit. As described above, the second and third calculation units can upgrade both the three-dimensional coordinates of the laser plane in the projected space estimated by the three-dimensional position estimation unit of the plane, and the three-dimensional coordinates of the reflection position of the laser beam in the projected space restored by the three-dimensional restoration unit, to Euclidean coordinates. However, it is also possible to upgrade only one of the three-dimensional coordinates of the laser plane or the three-dimensional coordinates of the reflection position of the laser beam to Euclidean coordinates in advance. For example, only the three-dimensional coordinates of the laser plane in the projected space estimated by the three-dimensional position estimation unit of the plane may be upgraded to Euclidean coordinates by the second and / or third calculation units, and then the three-dimensional coordinates of the laser reflection position may be obtained by shape restoration using the light section method in the three-dimensional restoration unit. Furthermore, it is also possible to upgrade the three-dimensional coordinates of the reflection position of the laser beam in the projected space restored by the three-dimensional restoration unit to Euclidean coordinates again by the second and / or third calculation units. Note that Figure 7 shows a flow diagram of three different Euclidean upgrades performed in the first, second, and third calculation units, but it is also possible to use only some of these Euclidean upgrades. For example, it is possible to perform two Euclidean upgrades in the second and third calculation units and omit the Euclidean upgrade in the first calculation unit.

[0071] <Second aspect: Embodiment 2A> The second embodiment of the image generation processing apparatus is an image generation processing apparatus that reconstructs a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. A 3D reconstruction unit that uses the 3D coordinates of the laser plane formed by plane-intersecting lasers estimated by any method, and the laser rays detected in each frame of the moving image, to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method, A SLAM analysis unit that detects arbitrary feature points using Visual SLAM or SfM and obtains the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, It includes an integration unit that reconstructs the 3D shape by integrating the 3D reconstruction results using the reconstruction results of the 3D coordinates of the reflection position of the laser beam and the camera position and orientation. The following describes the image generation apparatus of the second embodiment, with Embodiment 2A being representative.

[0072] Figure 10 is a flowchart illustrating an image generation method using the image generation apparatus of Embodiment 2A. In Figure 10, unlike another flowchart illustrating an image generation method using the image generation apparatus of Embodiment 1B shown in Figure 7, the steps of creating an intersection set graph, calculating the intersections of detected laser lines, tracking the intersections, creating an intersection set graph, and creating simultaneous equations from the intersections are not required, and laser plane reconstruction is performed by any method. An example of such an arbitrary method is to determine the variables corresponding to the degrees of freedom remaining in the solution from the equations obtained from the geometric conditions included in the target region (scene), and to realize Euclidean reconstruction. Details can be adapted from the methods described in

[0038] to

[0047] of Japanese Patent Application Publication No. 2009-32123, and the contents of this publication are incorporated herein by reference. Although the embodiment 2A shown in Figure 10 does not show the steps for obtaining the epipolar line, the corresponding point search step based on the epipolar constraint, and the step for detecting 3D points on the epipolar line and obtaining corresponding points, these steps may also be performed. Details of the other processes are the same as those described in Figure 7.

[0073] [3D Shape Restoration System] The three-dimensional shape restoration system of the present invention comprises an image generation processing device of the present invention, and an imaging means including a camera that photographs a target area over a specific period of time, a plurality of plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target area, and a fixing unit that integrates these and fixes them so that they can be moved. The 3D shape reconstruction system of the present invention can adapt to various environments, is self-calibrating, and can perform precise and accurate 3D reconstruction. The environments it can adapt to, i.e., the target areas, include various environments that are difficult for humans to access, such as internal human body scanning using endoscopes, creation of 3D maps of the ocean floor, and acquisition of 3D shapes from planetary images such as Mars and satellite images. Among these, the 3D shape reconstruction system of the present invention is particularly capable of detailed and accurate 3D reconstruction compared to conventional techniques, especially when reconstructing 3D shapes in extreme environments with few feature points within the target area (scene). Examples of extreme environments with few feature points within the target area include environments photographed from underwater using ROVs (Remotely Operated Vehicles) or underwater drones (inspection and maintenance of underwater, seabed, harbor areas, riverbanks, lakeshores, and other underwater structures, and photography of underwater objects and organisms), environments photographed from the air using drones (photogrammetry, inspection and maintenance of land structures, etc.), environments photographed from or toward space (photography of planetary and satellite images, etc.), and environments photographing the inside of the human body using an endoscope. Furthermore, when photographing a wide area of ​​target space, some areas may be included with few feature points, and even in environments photographed indoors, for example, the present invention can perform detailed and accurate 3D reconstruction of floors, walls, etc. Similarly, because the 3D shape reconstruction system of the present invention is self-calibrated, it can perform precise and accurate 3D reconstruction even in target areas where external calibration is difficult due to reasons such as the difficulty of observing or accurately moving calibration instruments, such as in turbid water.

[0074] In the three-dimensional shape reconstruction system of the present invention, a camera and a plane-crossing laser emitter are provided inside a housing, and it is preferable that the target area is underwater. Even when the camera and plane-crossing laser emitter are provided inside the housing, according to the present invention, the effect of refraction can be suppressed by positioning the laser plane perpendicular to the housing interface, enabling precise and accurate three-dimensional reconstruction.

[0075] In the three-dimensional shape restoration system of the present invention, it is preferable that the imaging means further comprises a recording unit and a moving unit, and that the imaging means captures a moving image of the target area and records it in the recording unit. The imaging device may move autonomously, or it may be moved by external control, such as wirelessly. It is preferable for the imaging device to include a mobile unit capable of autonomous movement, from the viewpoint of application to automated map measurement and other applications.

[0076] In the 3D shape restoration system of the present invention, the camera and the plane-intersecting laser emitter are integrated and fixed in a movable manner. Therefore, the 3D shape restoration system of the present invention can be easily manufactured simply by fixing the plane-intersecting laser emitter to an existing camera-equipped drone or camera-equipped ROV. Furthermore, the 3D shape restoration system of the present invention can be easily manufactured simply by fixing the plane-intersecting laser emitter to an autonomous mobile device equipped with a camera. Alternatively, the 3D shape restoration system of the present invention can be easily manufactured by integrating and fixing a camera and a plane-intersecting laser emitter to an autonomous mobile device that can move autonomously using means other than vision, which does not have a camera.

[0077] [Image generation processing method] A first aspect of the image generation processing method of the present invention is an image generation processing method for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation process is performed to obtain an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and from the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on the two laser planes formed by intersecting plane lasers, multiple constraint equations are obtained in a chain, and a system of simultaneous equations is generated by solving the set of constraint equations simultaneously. The process involves reconstructing the 3D coordinates of the laser plane in projective space by solving a system of simultaneous equations, and estimating the 3D position of the plane. This includes a 3D reconstruction step in which the 3D coordinates of the reflection position of the laser beam are reconstructed in projective space using the light section method, based on the estimated 3D coordinates of the laser plane and the laser beam detected in each frame of the moving image.

[0078] A first aspect of the image generation processing method of the present invention preferably comprises a SLAM analysis step of detecting arbitrary feature points by Visual SLAM or SfM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, and an integration step of integrating the 3D reconstruction result using the reconstruction result of the 3D coordinates of the reflection position of the laser beam and the camera position and orientation to reconstruct a 3D shape. Other preferred embodiments of the first embodiment of the image generation processing method of the present invention are the same as described in the description of the preferred embodiments of the first embodiment of the image generation processing apparatus of the present invention.

[0079] A second aspect of the image generation processing method of the present invention is an image generation processing method for reconstructing a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. A 3D reconstruction unit that uses the 3D coordinates of the laser plane formed by plane-intersecting lasers estimated by any method, and the laser rays detected in each frame of the moving image, to reconstruct the 3D coordinates of the reflection position of the laser rays in projective space using the light section method, A SLAM analysis process that detects arbitrary feature points using Visual SLAM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, The system includes an integration step that combines the original reconstruction result of the 3D coordinates of the laser beam reflection position with the 3D reconstruction result using the camera position and orientation to reconstruct the 3D shape. A preferred embodiment of the second embodiment of the image generation processing method of the present invention is the same as the description of a preferred embodiment of the second embodiment of the image generation processing apparatus of the present invention.

[0080] The image generation processing method of the present invention can be executed sequentially by a program stored in a storage means such as an HDD.

[0081] [program] The present invention is a program to be executed by an image generation processing device that reconstructs a three-dimensional shape from an input video image, The moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target area over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto material in the target area, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation function that obtains an intersection set from the connection relationships between laser beam intersections detected in each frame of the video, and from the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on two planes formed by intersecting plane lasers, multiple constraint equations are obtained in a chain, and a system of simultaneous equations is generated by solving the set of constraint equations simultaneously. This system uses simultaneous equations to reconstruct the 3D coordinates of the laser plane in projective space, providing a 3D position estimation function for the plane. This system performs a 3D reconstruction function that uses the estimated 3D coordinates of the laser plane and the laser beam detected in each frame of the video image to reconstruct the 3D coordinates of the laser beam's reflection position in projective space using the light section method. Preferred embodiments of the program of the present invention are the same as described in the description of preferred embodiments of the image generation processing apparatus and the image generation processing method of the present invention. [Examples]

[0082] The present invention will be described in more detail below with reference to examples and comparative examples. The materials, amounts used, proportions, processing content, and processing procedures shown in the following examples can be modified as appropriate without departing from the spirit of the present invention. Therefore, the scope of the present invention should not be interpreted as being limited by the following specific examples.

[0083] [Example 1: Evaluation for self-calibration in air] To evaluate the effectiveness of self-calibration in the image generation processing method using the image generation processing apparatus of Embodiment 1B shown in Figure 7, experiments were conducted in air. In Examples 1 to 3, Euclidean upgrade using the angle between laser planes was not used; instead, two Euclidean upgrades were used: one using 3D points generated by Visual SLAM or SfM, and another using 3D points refined using epipolar lines. That is, the step in the lower right of Figure 7, "Euclidean upgrade of the 3D coordinates of the laser plane and the 3D coordinates of the laser reflection position using the relative 3D positions of the lasers of the imaging means," was not performed, and the other steps were carried out. Figure 11(a) is a photograph of the setup of the imaging device used in Example 1. Four green planar intersecting laser emitters were attached to a GoPro HERO8 camera and fixed in place, creating an imaging device that could be moved as a whole. By moving the entire imaging device, the target area (scene) of the calibration board and column in the room was scanned, and the self-calibration method was evaluated. Figure 11(b) shows the measurement of the actual angle of the restored column. Figure 11(c) is a photograph showing an example of an image taken for the measurement method using the calibration device. Figure 11(d) shows an example of an image taken that is necessary for the present invention.

[0084] (Evaluation of the accuracy of coplanarity and epipolar constraints) First, to verify the effectiveness of the intersection tracking algorithm based on the creation of an intersection set graph, the accuracy of the estimated epipolar line direction was evaluated by changing the number of frames used from the video. The results are shown in Figure 12. Figure 12 is a graph showing the relationship between the number of frames used and the RMSE in the evaluation of the 3D shape reconstruction accuracy of Example 1. From Figure 12, it can be seen that the RMSE error gradually decreases and almost converges at 31 frames. This indicates that if tracking of corresponding points can be achieved for 31 frames or more, there is a high possibility that high-precision reconstruction can be achieved.

[0085] (Evaluation of 3D shape reconstruction accuracy) Next, the 3D shape reconstruction accuracy of the present invention is compared to a hard calibration method using a grid planer for laser calibration, and to a previous technique (Mathieu Labbe and Francois Michaud. Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation: LabbeE and michaud. Journal of Field Robotics, 36, 10 2018. doi: 10.1002 / rob.21831, described in Kinect). We compared it with V1 and RTAB MAP using Kinect Azure. The results are shown in Figure 13. The bar graphs in Figures 13(A) and 13(B) represent, from left to right, the sequence of RTAB MAPs using Kinect V1, RTAB MAPs using Kinect Azure, the hard calibration method, and the method of the present invention, respectively. Figure 13(A) shows the plane fitting error between the two planes in the evaluation of the 3D shape reconstruction accuracy of Example 1. From Figure 13(A), it was confirmed that the RMSE and MAE obtained by the method of the present invention are equivalent to or better than those of Kinect Azure. Figure 13(B) shows the angular error in the evaluation of the 3D shape reconstruction accuracy of Example 1. As can be seen from Figure 13(B), the angular error was quite small for all methods. Kinect V1 performed best, which may be because the data size acquired from Kinect V1 was much larger than that of the other methods, making it statistically more stable. The 3D shapes obtained by all methods are shown in Figure 14. Figure 14(A) shows the 3D shape reconstruction results of Kinect V1 from Journal of Field Robotics, 36, 10 2018. Figure 14(B) shows the 3D shape reconstruction results of Kinect Azure from Journal of Field Robotics, 36, 10 2018. Figure 14(C) shows the 3D shape reconstruction results of the hard calibration method. Figure 14(D) shows the 3D shape reconstruction results of the method of the present invention. From Figure 14, it is clearly shown that the method of the present invention is almost the same as the hard calibration method, particularly for cross-sections in the xy plane, and is superior to existing 3D sensors.

[0086] [Example 2: Evaluation of self-calibration in an underwater environment] Next, to demonstrate the ability of the image generation processing method using the image generation processing apparatus of Embodiment 1B shown in Figure 7 to withstand extreme conditions, self-calibration in an underwater environment was evaluated. Figure 15(a) is a photograph of the setup of the imaging device used in Example 2. A scanning device consisting of a GoPro HERO8 camera with four green crosslinked line lasers mounted inside a waterproof housing was constructed and attached to an underwater ROV (BlueROV2) as shown in Figure 15(a). Several objects, such as tables and mannequins, were submerged in a swimming pool as target objects, and an underwater ROV was operated to scan the target area (scene). Figure 15(b) is a photograph corresponding to the top view of the target area (scene) used in Example 2. Figure 15(c) is a photograph of an example of the process of reconstructing the 3D coordinates of the laser plane in projected space. Figure 15(d) is an example of an image of the reconstructed mannequin being measured. Figure 15(e) is the reconstructed mannequin. The accuracy of 3D shape reconstruction was compared with previous techniques such as hard calibration, Colmap, and Kinect fusion. The results are shown in Figure 16. The bar graphs in Figures 16(A) and 16(B) represent, from left to right, Direct Sparuse Odometry (DSO), Colmap, hard calibration, and the method of the present invention, respectively. Figure 16(A) shows the error between MAE [mm] and RMSE [mm] in the evaluation of the 3D shape reconstruction accuracy of Example 2. Figure 16(B) shows the number of reconstructed 3D points in the evaluation of the 3D shape reconstruction accuracy of Example 2. Figures 16(A) and 16(B) show the previous Visual SLAM (J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. In arXiv:1607.02565, July 2016) It can be confirmed that DSO (Distributed Spectroscopy) cannot correctly restore the shape because it has almost no texture on the object. Hard calibration is best, but the self-calibration algorithm of the present invention achieves nearly the same accuracy, demonstrating the effectiveness of the method of the present invention. The three-dimensional shapes obtained by all methods are shown in Figure 17. Figure 17(a) is by Shahram Izadi. Figures 17(b) and 17(B) show the 3D shape reconstruction results of GT from David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. Kinectfusion: Real-time 3D reconstruction and interaction using a moving depth camera. In UIST '11 Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559-568. ACM, October 2011. ISBN 978-1-4503-0716-1. Figures 17(b) and 17(B) show the 3D shape reconstruction results of DSO from J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. In arXiv:1607.02565, July 2016. Figures 17(c) and 17(C) are from Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In Colma at Conference on Computer Vision and Pattern Recognition (CVPR), 2016 Figure 17 shows the 3D shape reconstruction results for p. Figures 17(d) and 17(D) show the 3D shape reconstruction results for the hard calibration method. Figures 17(e) and 17(E) show the 3D shape reconstruction results for the method of the present invention. Figure 17 clearly shows that the method of the present invention is almost the same as the hard calibration method and is superior to existing 3D sensors.

[0087] [Example 3: Evaluation of self-calibration over a wide area] Next, to demonstrate the ability of the image generation processing method using the image generation processing apparatus of Embodiment 1B shown in Figure 7 to handle large areas, a large area was reconstructed by moving the imaging device indoors. This was compared with the well-known SfM techniques Colmap and Meshroom. The results are shown in Figures 18, 18-1 to 18-3. Note that all methods are self-calibrated, and large areas are reconstructed. Figure 18(A) shows the result of 3D shape reconstruction using the method of the present invention. Figures 18(a1-A), 18(a1-B), and 18(a1-C) each show an example of the captured image required for the present invention. Figures 18(a1-1) and 18(a1-2) show the result of 3D shape reconstruction within the frame on the left side of Figure 18(A) from a different angle. Figure 18(b1) shows the result of 3D shape reconstruction within the frame on the right side of Figure 18(A) from a different angle. Figures 18(b1-A) and 18(b1-B) each show an example of the captured image required for the present invention. From Figure 18(A), it can be seen that the method of the present invention can reconstruct the 3D shape with high density. Figure 18(B) shows the 3D shape reconstruction results of Colmap from Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. Figure 18(b1) is shown in Figure 18(B) Figure 18(b2) shows the 3D shape reconstruction results from the left-hand frame of Figure 18(B), viewed from a different angle. From Figure 18(B), it can be seen that Colmap hardly reconstructs the 3D shape. Figure 18(C) shows the 3D shape reconstruction results from Meshroom. Figure 18(c1) shows the 3D shape reconstruction results for the left frame of Figure 18(C) from a different angle. Figure 18(c2) shows the 3D shape reconstruction results for the right frame of Figure 18(C) from a different angle. From Figure 18(C), it can be seen that Meshroom has low accuracy in 3D shape reconstruction and produces large holes. These figures (Figure 18) show that while areas with little texture, such as floor surfaces, cannot be restored using well-known SfM techniques, these areas with little texture can be restored to high density using the method of the present invention.

[0088] [Example 4: Confirmation of other processes] In Example 1, laser ray estimation, intersection point graph creation, mask creation, and corresponding point detection were verified in the target area (scene) including the calibration board. Figure 19 shows the results of laser ray estimation. From Figure 19, it can be seen that laser rays in an image can be estimated using a CNN. Figure 20(A) shows the traced intersections. Figure 20(B) shows the connected intersections. Figure 20(C) shows the graph of the created set of intersections. From Figure 20(C), it can be seen that by tracing the intersections, the set of intersections can be graphed and a unique correspondence can be made. Figure 21 shows the results of mask creation. From Figure 21, it can be seen that a mask can be created from the estimated laser rays using morphological transformation, and by excluding the mask position from feature point extraction by Visual SLAM, corrected feature points that ignore brightness gradients can be obtained. Figure 22 shows the results of the correspondence point detection. From Figure 22, it can be seen that there are a sufficient number of 3D points estimated by Visual SLAM along the laser line and that they can be detected as correspondence points.

[0089] [Example 5: Evaluation 2 for self-calibration in air] The experiment was conducted using the same method as in Example 1, with photographic data captured by moving the camera around the mannequin approximately one and a half times in mid-air. The reconstruction results showed that the floor area was curved due to the accumulation of errors in camera position and orientation (Figure 23(c)). Therefore, all frames were divided into blocks of several tens of frames each, and an average shape was generated for each block (loop section). Then, an average shape was generated by integrating the reconstruction results of all frames, and this was used as a provisional target shape. The process of bringing the individual block shapes closer to this target shape was repeated (Figure 24).

[0090]

number

[0091] The block was generated by integrating point clouds from several dozen frames. The camera position and orientation of the first frame was used as the orientation of the entire block, and the point clouds of the other frames were projected onto the local coordinates of the first frame using their relative orientations to the first frame. In equation (1), T i start P is the relative orientation of the i-frame to the first frame. i This represents the point cloud of the laser irradiation area in the i-frame.

[0092] The average shape was generated by meshing, as shown in Figure 25. In this case, if the orientation error is large and the distance between overlapping shapes is small, a mesh is generated in between them (Figure 2 5 (left diagram), the mesh shape becomes distorted at long distances (Figure 25 (right diagram)). Therefore, in Example 5, before generating the mesh, the distance between shapes was forcibly reduced by reintegrating the points of each frame using the average camera position and orientation in the overlapping section of the path.

number

number

[0093] The average camera position and orientation was calculated by manually specifying the start and end frames of the loop section (Figure 26 (left panel)) and then using the weighted average of the translation components of the corresponding camera position and orientation (Figure 26 (right panel)). In equations (2) and (3), t i t′ is the translation component of the i-frame's orientation. i is the average of the translation components of the pose in the i-th frame, w is the weight, and is linear with respect to i.

[0094] Finally, using the determined camera position and orientation, we projected the point cloud for each frame.

number

[0095] Next, as shown in Figure 27, the correspondence between each point in the block and the average shape was determined using ICP. Since both the block and overall reconstruction results are obtained by integrating the lasers of each frame at the camera position and orientation, the correspondence between the points of each block and the average shape can be determined from the correspondence between the points of the overall shape and the average shape. Bundle adjustment was performed using the determined correspondence. The cost function was calculated from the following formula.

number

[0096] Here T Bi is the parameter optimized for the orientation of the i-th block. j p' is included in the block. j is p j These are points on the average shape that correspond to them. Also, ￣T i B i+1 is the relative orientation of frame i to the camera position and orientation of frame i+1 in the previous iteration, and w is the weight. The second term was added to prevent the overall shape from deforming too much by incorporating the change in relative orientation between blocks into the cost.

[0097] Figure 28 shows the cost progression during camera position and orientation optimization according to Example 5. The cost fluctuates because the average shape is regenerated each time, and the large initial value in frame-by-frame optimization is due to the addition of the cost of relative orientation constraints across all frames. The cost converges in block-by-block optimization, indicating that the optimization was performed successfully using this method which employs blocks and average shapes.

[0098] Furthermore, in block-level bundle adjustment, the relative orientation within a block is fixed, causing abrupt changes in camera position and orientation between the final frame of a block and the first frame of the next block. Therefore, as shown in Figure 29, a frame-level bundle adjustment was performed last to optimize the camera position and orientation T for each frame so that the camera position and orientation change smoothly. As a result of the aforementioned bundle adjustment, a position and orientation close to the correct one was obtained at the block level, resolving issues such as lines being too close together in the average shape, and thus the minimization of the 3D point distance at the frame level worked correctly.

number

[0099] Here ￣T i i+1 This is the relative pose calculated from the camera position and pose during reconstruction, not the camera position and pose before frame-by-frame bundle adjustment. This is because short-range changes in camera position and pose can be estimated with high accuracy during reconstruction, and the goal is for the final optimized camera position and pose to change smoothly, reflecting this advantage.

[0100] In Example 5, after frame-level optimization, the value converged to a significantly smaller value than before optimization, demonstrating that the optimization proceeded correctly even with sparse point clouds in each frame, thanks to the good initial values ​​provided by block-level optimization. Looking at the final camera position and orientation (Figure 30(a)), it can be seen that the camera position and orientation (Figure 23(a)) did not change abruptly, and the floor shape also became flat (Figure 30(c)). In addition, the RMSE when the reconstructed shape of the same scene using Colmap[3] was set as Ground Truth was also reduced by the optimization (Table 1).

[0101] [Table 1]

[0102] [Example 6: Evaluation of self-calibration in an underwater environment 2] Next, the same experiment as in Example 2 was conducted underwater, and optimization was performed using the same method as in Example 5. Because the texture fluctuates underwater, Colmap does not function correctly, and ground truth cannot be obtained, so the flatness of the seabed was evaluated visually. By optimizing using the same method as in Example 5, the bottom shape approached a flat surface (Figure 31), confirming the effectiveness of the mesh-based optimization method.

[0103] From the above embodiments, it has been shown that the image generation processing apparatus of the present invention is suitable for image capture and 3D shape reconstruction under extreme conditions. Furthermore, the image generation processing apparatus of the present invention employs a self-calibration technique for the light section method. Since the light section method requires only a few intersecting plane lasers with cameras attached, it has significant advantages in size and energy consumption compared to existing 3D sensors. Therefore, in this respect as well, the image generation processing apparatus of the present invention is suitable for image capture and 3D shape reconstruction under extreme conditions. Conventional methods for detecting intersecting plane lasers require known geometric constraints to achieve a Euclidean upgrade, making it difficult to construct high-precision systems in practice. The image generation processing apparatus of the present invention efficiently overcomes this limitation by using Visual SLAM results and bundle adjustments specifically designed for plane parameter estimation. While the accuracy of each 3D point in Visual SLAM is not necessarily high, this can be efficiently resolved by solving only the uncertainty of the 4-degree-of-freedom solution. This represents a significant advantage when considering practical industrial applicability.

[0104] Furthermore, as shown in Examples 5 and 6, the effectiveness of optimization methods in 3D shape reconstruction was confirmed. Specifically, the effectiveness of a method that uses a mesh to create an average shape as a provisional target shape and performs optimization in block units by integrating point clouds from multiple frames was confirmed. [Explanation of symbols]

[0105] 1 Image generation processing device 100 Shooting methods 101 Camera 102 Planar intersecting laser emitter 103 Fixed part 111 Plane Crossing Laser 121 Housing 131 Records Section 141 Mobile Unit

Claims

1. An image generation processing device for restoring a three-dimensional shape from an input moving image, wherein the moving image is a group of consecutive frames photographed by photographing means including one camera that photographs a target area for a specific period, a plurality of planar intersection laser transmitting units that project a planar intersection laser onto a substance in the target area, and a fixing unit that integrally fixes these and enables movement, an intersection set generation unit that obtains an intersection set from the connection relationship between the intersections of the laser lines detected in each frame of the moving image and the tracking result of the intersections of the laser lines detected in consecutive frames; a simultaneous equation generation unit that successively obtains a plurality of constraint equations from the fact that each intersection of the intersection set lies on two laser planes formed by the planar intersection laser, and generates a simultaneous equation by联立 these plurality of constraint equations; a three-dimensional position estimation unit of a plane that restores the three-dimensional coordinates of the laser plane in the projective space by solving the simultaneous equation; a three-dimensional restoration unit that restores the three-dimensional coordinates of the reflection position of the laser line in the projective space by the optical cutting method using the three-dimensional coordinates of the laser plane restored by the three-dimensional position estimation unit and the laser lines detected in each frame of the moving image; An image generation processing device comprising the above.

2. The image generation processing device according to claim 1, wherein the intersection set generation unit creates an intersection set graph.

3. The image generation processing device according to claim 1, comprising a first calculation unit that inputs the known relative three-dimensional positions between the planar intersection lasers and the three-dimensional coordinates of the laser plane estimated in the projective space, and upgrades the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the reflection position of the laser line to Euclidean coordinates.

4. A three-dimensional point calculation unit that performs Euclidean three-dimensional restoration by a self-calibration method with a group of consecutive frames of the moving image as an input; in each frame of the moving image, a corresponding point detection unit that detects, as corresponding points, those of the three-dimensional points obtained by the three-dimensional point calculation unit that exist on the laser line; a second calculation unit that upgrades the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the reflection position of the laser line to Euclidean coordinates using the detected corresponding points; The image generation processing device according to claim 1, comprising the above.

5. The 3D point calculation unit includes a SLAM analysis unit that detects arbitrary feature points using Visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result. The system includes an integration unit that integrates the three-dimensional reconstruction results using the reconstruction results of the three-dimensional coordinates of the reflection position of the laser beam and the camera position and orientation to reconstruct the three-dimensional shape. The image generation apparatus according to claim 4, wherein the self-calibration method of the laser plane is realized using the three-dimensional reconstruction result obtained by the Visual SLAM or SfM.

6. An epipolar line calculation unit obtains an epipolar line by calculating the straight line that passes through the same intersection point in the image, based on the results of tracking the intersection points of the aforementioned laser beams, for each intersection point. A correspondence point search unit based on epipolar constraint searches for a corresponding point in any frame within the video image on the epipolar line, The system includes a corresponding point detection unit that detects, as corresponding points, three-dimensional points obtained by the three-dimensional point calculation unit using Visual SLAM or SfM, that are located on a laser line connected to the searched corresponding point. A third calculation unit that uses the detected corresponding points to upgrade the three-dimensional coordinates of the laser plane and the three-dimensional coordinates of the reflection position of the laser beam to Euclidean coordinates, The image generation apparatus according to claim 5, including the following:

7. An epipolar line calculation unit obtains an epipolar line by calculating the straight line that passes through the same intersection point in the image for each intersection point, based on the results of tracking the intersection points of the aforementioned laser beams. A correspondence point search unit based on epipolar constraint searches for a corresponding point in any frame within the video image on the epipolar line, A three-dimensional position estimation unit for a plane reconstructs the three-dimensional coordinates of the laser plane in projective space using the corresponding points that were searched, The image generation apparatus according to claim 1, including the following:

8. An image generation processing device that reconstructs a three-dimensional shape from an input video image, The aforementioned moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target region over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target region, and a fixing unit that integrates these and fixes them so that they can move. A three-dimensional reconstruction unit that uses the three-dimensional coordinates of the laser plane formed by the intersecting plane lasers estimated by any method and the laser rays detected in each frame of the moving image to reconstruct the three-dimensional coordinates of the reflection position of the laser rays in projective space by the light section method, A SLAM analysis unit that detects arbitrary feature points using Visual SLAM or SfM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration unit that integrates the three-dimensional reconstruction results using the reconstruction results of the three-dimensional coordinates of the reflection position of the laser beam and the camera position and orientation to reconstruct the three-dimensional shape, An image generation processing apparatus comprising:

9. The image generation processing apparatus according to claim 1, comprising a laser beam estimation unit that calculates the position of the estimated laser beam in consecutive frames using a trained model of a convolutional neural network (CNN).

10. The system includes a mask creation unit that creates a mask using morphological transformation from the position of the estimated laser beam, The image generation apparatus according to claim 9, wherein Visual SLAM or SfM is applied to each frame of the video image, ignoring the brightness at the mask position, to obtain a three-dimensional point.

11. The image generation apparatus according to claim 1, further comprising a fourth calculation unit that, in the group of frames, if the three-dimensional coordinates restored in frame n and the three-dimensional coordinates restored in frame k are at the same location within the target region, re-estimates the three-dimensional coordinates of the laser plane to minimize the discrepancy between the two three-dimensional coordinates.

12. The process includes the steps of dividing the video into blocks of m consecutive frames, and integrating the three-dimensional coordinates restored by the three-dimensional restoration unit for each block to obtain a second set of three-dimensional coordinates. The image generation apparatus according to claim 1, further comprising a fifth calculation unit that re-estimates the three-dimensional coordinates of the laser plane to minimize the discrepancy between two three-dimensional coordinates when the two second three-dimensional coordinates are at the same location within the target region.

13. An image generation apparatus according to any one of claims 1 to 12, A three-dimensional shape restoration system comprising a camera that photographs a target area over a specific period of time, a plurality of plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target area, and a fixing unit that integrates and fixes these so that they can be moved.

14. The camera and the plane-crossing laser emitter are located inside the housing. The three-dimensional shape restoration system according to claim 13, wherein the target area is underwater.

15. The aforementioned shooting means further comprises a recording unit and a moving unit, The three-dimensional shape restoration system according to claim 13, wherein the shooting means moves while capturing a moving image of the target area and records it in the recording unit.

16. An image generation processing method for reconstructing a three-dimensional shape from an input video image, The aforementioned moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target region over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target region, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation step is performed to obtain an intersection set from the connection relationships between laser beam intersections detected in each frame of the aforementioned video, and the tracking results of laser beam intersections detected in consecutive frames. A system of linear equations generation step is performed by obtaining a series of constraint equations from the fact that each intersection point of the set of intersection points lies on the two laser planes formed by the intersecting plane lasers, and then solving the series of constraint equations simultaneously to generate a system of linear equations, A three-dimensional position estimation step of a plane, in which the three-dimensional coordinates of the laser plane are reconstructed in projected space by solving the aforementioned system of equations, A three-dimensional reconstruction step is performed using the three-dimensional coordinates of the laser plane reconstructed in the three-dimensional position estimation step and the laser beam detected in each frame of the moving image to reconstruct the three-dimensional coordinates of the reflection position of the laser beam in projective space using the light section method. An image generation processing method that includes [a specific feature].

17. A SLAM analysis process that detects arbitrary feature points using Visual SLAM or SfM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration step of restoring a three-dimensional shape by integrating the three-dimensional restoration results using the restoration results of the three-dimensional coordinates of the reflection position of the laser beam and the camera position and orientation, The image generation processing method according to claim 16, comprising:

18. An image generation processing method for reconstructing a three-dimensional shape from an input video image, The aforementioned moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target region over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target region, and a fixing unit that integrates these and fixes them so that they can move. A three-dimensional reconstruction unit that uses the three-dimensional coordinates of the laser plane formed by the intersecting plane lasers estimated by any method and the laser rays detected in each frame of the moving image to reconstruct the three-dimensional coordinates of the reflection position of the laser rays in projective space by the light section method, A SLAM analysis process that detects arbitrary feature points using Visual SLAM to obtain the camera position and orientation in a Euclidean coordinate system and a 3D reconstruction result, An integration step of restoring a three-dimensional shape by integrating the original restoration result of the three-dimensional coordinates of the reflection position of the laser beam and the three-dimensional restoration result using the camera position and orientation, An image generation processing method comprising:

19. A program to be executed by an image generation processing device that reconstructs a three-dimensional shape from an input video image, The aforementioned moving image is a series of consecutive frames captured by a shooting means that includes one camera that photographs a target region over a specific period of time, multiple plane-crossing laser emitters that project plane-crossing lasers onto a substance in the target region, and a fixing unit that integrates these and fixes them so that they can move. An intersection set generation function obtains an intersection set from the connection relationships between laser beam intersections detected in each frame of the aforementioned video, and the tracking results of laser beam intersections detected in consecutive frames. Since each intersection point in the set of intersection points lies on the two laser planes formed by the intersecting plane lasers, a series of constraint equations are obtained, and a system of simultaneous equations is generated by solving these constraint equations simultaneously. A three-dimensional position estimation function for a plane, which reconstructs the three-dimensional coordinates of the laser plane in projective space by solving the aforementioned system of equations, A three-dimensional reconstruction function that uses the three-dimensional coordinates of the laser plane reconstructed in the three-dimensional position estimation function and the laser beam detected in each frame of the moving image to reconstruct the three-dimensional coordinates of the reflection position of the laser beam in projective space using the light section method, A program that executes something.

Images