Three dimensional measurement method, three dimensional measurement device, and three dimensional measurement system

Abstract

To perform highly accurate three dimensional measurement by enabling effective extraction of feature points even for an object having few characteristic patterns and shapes.MEANS FOR SOLVING THE PROBLEM: A three-dimensional measurement system includes a camera that captures images of an object while moving, a projector that projects a projection pattern onto the object, and an information processing device that receives images from the camera and issues projection commands to the projector. The information processing device includes: a feature point extraction unit that extracts feature points in two images including the projection pattern captured before and after the camera moves; a feature point matching unit that searches for pairs of feature points indicating the same point and calculates a feature point matching set; a movement amount estimation unit that estimates a movement amount of the camera from the feature point matching set; and a model generation unit that generates a three-dimensional model of the object from the movement amount of the camera.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to three-dimensional measurement technology.

Background Art

[0002] In plant construction, building construction, vehicle manufacturing, etc., in order to reflect construction drawings on the construction site, it is important to perform highly accurate alignment on site. For example, in plant construction, construction is carried out to install pipes that penetrate walls. In a general construction procedure, first, a hole is made in the wall to install the through pipe, and then the straight pipe at the front is installed. In order to shorten this construction period, it is required to realize a construction procedure in which the straight pipes on both sides of the wall are installed in advance before making a hole in the wall, and finally a hole is made in the wall to install the through pipe. For this purpose, highly accurate alignment with the opposite side of the wall is required.

[0003] Therefore, three-dimensional measurement that scans an object while moving a visual sensor such as an optical camera, an infrared camera, a ToF sensor, or LiDAR (hereinafter, may also be referred to as a camera), sequentially estimates the movement amount of the camera, and finally generates a three-dimensional model of the three-dimensional object is effective. Before making a hole in the wall, scan while moving inside the plant to the opposite side of the wall, and by three-dimensionally modeling the entire movement path, alignment with the opposite side of the wall can be achieved.

[0004] However, when there are no characteristic patterns or shapes in the shooting field of view during scanning, characteristic points (hereinafter, may also be referred to as feature points or landmarks) cannot be extracted, so the movement amount of the camera cannot be estimated, and the scan may fail. In particular, during plant construction, the walls and floor may be covered with a uniform curing sheet, etc., and there are often no features on the walls and floor. As a result, the possibility of scan failure increases.

[0005] Regarding this point, in Patent Document 1, in the navigation of a mobile system that processes Simultaneous Localization and Mapping (SLAM), an algorithm is disclosed that ranks landmarks based on visual similarity and selects the best landmark candidate from the information obtained by a visual sensor.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] In the prior art, when there are few features on walls or floors, effective feature points cannot be extracted. For example, in Patent Document 1, even if the best landmark is selected by ranking, effective feature points cannot be obtained. When effective feature points cannot be extracted, the error increases in the estimated calculation of the camera movement amount, making high-precision three-dimensional measurement difficult.

[0008] From the above, an object of the present invention is to provide a three-dimensional measurement system that enables high-precision three-dimensional measurement by enabling the extraction of effective feature points even for an object with few characteristic patterns or shapes.

Means for Solving the Problems

[0009] One aspect of the present invention is a three-dimensional measurement method that executes a projection step of projecting a projection pattern onto an object, a photographing step of photographing at least two images of the object with a moving camera, an input step of inputting the two images into an information processing device, a feature point extraction step of extracting feature points from the two images by the information processing device, a feature point matching step of searching for pairs of the feature points indicating the same point and calculating a feature point matching pair by the information processing device, a moving amount estimation unit step of estimating the moving amount of the camera from the feature point matching pair by the information processing device, and a model generation step of generating a three-dimensional model of the object from the moving amount of the camera by the information processing device.

[0010] Another aspect of the present invention is a three-dimensional measurement device that takes at least two images of an object onto which a projection pattern is projected, with a moving camera as input, and includes a feature point extraction unit, a feature point matching unit, a camera moving amount estimation unit, and a model generation unit. The feature point extraction unit extracts feature points from the two images, the feature point matching unit searches for pairs of the feature points indicating the same point and calculates a feature point matching pair, the camera moving amount estimation unit estimates the moving amount of the camera from the feature point matching pair, and the model generation unit generates a three-dimensional model of the object from the moving amount of the camera.

[0011] Another aspect of the present invention includes a camera that photographs an object while moving, a projector that projects a projection pattern onto the object, and an information processing device that receives an image from the camera and issues a projection command to the projector. The information processing device includes a feature point extraction unit that extracts feature points in the image from two images including the projection pattern photographed before and after the camera moves, a feature point matching unit that searches for pairs of the feature points indicating the same point and calculates a feature point matching pair, a moving amount estimation unit that estimates the moving amount of the camera from the feature point matching pair, and a model generation unit that generates a three-dimensional model of the object from the moving amount of the camera.

Advantages of the Invention

[0012] According to the present invention, even for an object with few characteristic patterns or shapes, high-precision three-dimensional measurement can be achieved by enabling effective extraction of characteristic points. In addition, problems, configurations, and effects other than those described above will be clarified by the description of the embodiments for carrying out the following invention.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 23

Figure 24A

Figure 24B

Figure 24C

Figure 25

Figure 26

Figure 27

Figure 28

Mode for Carrying Out the Invention

[0014] Hereinafter, examples of the present invention will be described with reference to the drawings. The examples are illustrative for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural.

[0015] In the drawings, the positions, sizes, shapes, ranges, etc. of the respective components shown may not represent the actual positions, sizes, shapes, ranges, etc. for the purpose of facilitating the understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0016] As examples of various types of information, descriptions may be made using expressions such as "table", "list", "queue", etc., but the various types of information may be represented by data structures other than these. For example, various types of information such as "XX table", "XX list", "XX queue" may be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", "number" are used, but these are mutually replaceable.

[0017] When there are a plurality of components having the same or similar functions, they may be described by attaching different subscripts to the same reference numeral. Also, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.

[0018] In the embodiments, the processing performed by executing a program may be described. Here, the computer executes the program by a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)), and performs the processing defined by the program while using storage resources (for example, a memory) and interface devices (for example, a communication port), etc. Therefore, the subject of the processing performed by executing the program may be the processor.

[0019] Similarly, the entity that performs the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity that performs the processing by executing the program may be an arithmetic unit and may include a dedicated circuit that performs a specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Device), or the like.

[0020] The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor and a storage resource that stores the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0021] Note that since the present invention encompasses a wide range of contents, if an overview of each embodiment is shown in advance, it is as follows. First, in Example 1, a projection pattern is projected onto an object without features to increase feature points, and then while the photographer moves with a camera, the object is three-dimensionally measured. In Example 2, the necessity of the projection pattern is determined to control the projection. In Example 3, a plurality of types of projection patterns are selected. In Example 4, the interval between feature points of the projected projection pattern is considered. In Example 5, a specific example of a projection pattern expressed by alphanumeric symbols is presented. In Example 6, the size of the projected alphanumeric symbol pattern is discussed. In Example 7, in the construction of a through pipe, before drilling a hole in the wall, the position where a hole will open on the opposite side of the wall (virtual hole penetration position) is estimated by three-dimensional measurement. In Example 8, the virtual hole penetration position is displayed on AR (Augmented Reality) glasses. In Example 9, a walking guide is incorporated into the projection pattern. Each will be described respectively.

Example

[0022] Example 1 will be described with reference to FIGS. 1 to 4 and FIGS. 5A to 5E. In Example 1, a projection pattern is projected onto an object without features to increase feature points, and then a three-dimensional measurement method of photographing the object while the photographer moves with a camera will be described. According to Example 1, even for an object with few characteristic patterns or shapes, effective extraction of feature points enables high-precision three-dimensional measurement.

[0023] FIG. 1 is a diagram showing the operation of a photographer using the three-dimensional measurement system according to Example 1 of the present invention. In Example 1, the object 101 is three-dimensionally measured to generate a three-dimensional model of the object 101. The photographer 103 scans the object 101 while carrying the camera 102 and moving. The photographer 103 is illustrated as a human in FIG. 1, but is not limited to a human as long as it can move and be equipped with the camera 102. For example, it may be a humanoid robot, a dog-type robot, a vacuum cleaner-type robot, an AGV (Automated Guided Vehicle), a vehicle, a trolley, a cart, a wagon, a drone, or the like.

[0024] The camera 102 is a device equipped with a function capable of photographing the object 101 to generate a two-dimensional image or a three-dimensional image (hereinafter, the two-dimensional image and the three-dimensional image may also be collectively referred to as an image without distinction). Examples of the camera include an optical camera, an infrared camera, a stereo camera, a ToF (Time Of Flight) sensor, LiDAR (Light Detection And Ranging), and the like. The shooting range 106 indicates the range that can be shot at the angle of view per image shot by the camera 102. Since the object 101 is generally larger than the shooting range 106, the photographer 103 moves around the object 101 to take pictures and scans the shooting range 106. A three-dimensional model of the object 101 is generated by combining a plurality of images obtained during the scanning.

[0025] Here, consider the case where the object 101 has few characteristic patterns or shapes. In the generation of the three-dimensional model, as will be described later with reference to FIG. 3, the amount of camera movement is estimated by matching the feature points included in the image, and the three-dimensional model is generated. Therefore, when the object 101 has few characteristic patterns or shapes, feature points cannot be extracted, the estimation of the camera movement amount fails, or the estimation accuracy of the camera movement amount deteriorates. Therefore, by projecting the projection pattern 105 onto the object 101 by the projector 104, the number of feature points on the object can be increased, the estimation accuracy of the camera movement amount can be improved, and as a result, high-precision three-dimensional measurement becomes possible.

[0026] Examples of the projector 104 include an LCD (Liquid Crystal Display) projector, a DLP (Digital Light Processing) projector, an LCOS (Liquid Crystal On Silicon) projector, laser projection mapping, a flexible display, an OHP (Overhead Projector), and the like.

[0027] Figure 2 is a block diagram showing an example of the overall configuration of the three-dimensional measurement system according to Embodiment 1 of the present invention. The projector 104 projects a projection pattern 105 onto the object 101, and the camera 102 captures the object 101 together with the projection pattern. The information processing device 1000 issues a projection command to the projector 104 and receives an image from the camera 102.

[0028] The information processing device 1000, the camera 102, and the projector 104 may be connected by a wired method or a wireless method. As the wired method, a wired connection method according to a wired transmission standard such as USB (Universal Serial Bus), Ether, HDMI (High-Definition Multimedia Interface), VGA (Video Graphics Array), DVI (Digital Visual Interface), etc. can be considered. As the wireless method, a wireless connection method according to a wireless communication standard such as Wi-Fi (trademark), Bluetooth (trademark), Zigbee (trademark), etc. can be considered. Further, the information processing device 1000, the camera 102, and the projector 104 may be connected via a network not shown in FIG. 2.

[0029] The information processing device 1000 stores the projection pattern 105 projected by the projector 104. Note that the projection pattern 105 may be stored in a storage area built into the projector 104. As long as the projection pattern can be transmitted to the projector 104, the location of the data is arbitrary.

[0030] The internal processing of the information processing apparatus 1000 will be described. The feature point extraction unit 1001 analyzes the image obtained from the camera 102, and extracts feature points that have a characteristic pattern in the image and can be landmarks that are easily distinguishable from other points. As algorithms for extracting feature points, Harris corner detection, Shi-Tomasi corner detection, SIFT (Scale-Invariant Feature Transformation), SURF (Speeded-Up Robust Features), FAST (Features from Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF), NN (Neural Network), SuperPoint, etc. can be considered.

[0031] The feature point matching unit 1002 compares the feature points extracted from two images, and outputs a feature point matching set by matching the feature points indicating physically the same location. As algorithms for matching, nearest neighbor search, kd-tree, exhaustive matching, comparison of the Euclidean distance of feature quantities calculated for each feature point, SuperGlue, LightGlue, etc. can be considered.

[0032] The camera movement amount estimation unit 1003 calculates the movement amount of the camera when two images are taken using the feature point matching set. Calculating the movement amount corresponds to obtaining a 3×3 rotation matrix and a 3×1 translation vector that represent the rotation and translation of the camera, or obtaining a 4×4 homogeneous transformation matrix that combines rotation and translation. As algorithms for calculating the movement amount, ICP (Iterative Closest Point), RANSAC (Random Sample Consensus), etc. can be considered.

[0033] The model generation unit 1004 generates a three-dimensional model of the object 101 by overlapping images using the camera movement amount. The technical field that performs all of the feature point extraction unit 1001, the feature point matching unit 1002, the camera movement amount estimation unit 1003, and the model generation unit 1004 together is sometimes called three-dimensional restoration or photogrammetry. As algorithms for three-dimensional restoration, SALM (Simultaneous Localization and Mapping) and SfM-MVS (Structure from Motion / Multi View Stereo) can be considered.

[0034] Although the names of various algorithms are listed, each algorithm is calculated on the premise that feature points are extracted from the image. When there are few characteristic patterns or shapes on the object 101, the accuracy of each algorithm is considered to deteriorate.

[0035] FIG. 3 is a flowchart showing an overall processing example of the three-dimensional measurement system according to the first embodiment of the present invention. In step 2001, the projector 104 starts projecting the projection pattern 105 onto the object 101. Assume that the relative positions of the projector 104 and the object 101 are fixed.

[0036] In step 2002, the camera 102 moves and shooting is performed in step 2003. Assume that the Nth photo is obtained here. For the obtained Nth image, in step 2004, the feature point extraction unit 1001 extracts feature points.

[0037] In step 2005, the feature point matching unit 1002 matches the feature points extracted from the (N - 1)th image and the Nth image to generate a feature point matching set. When N = 1, steps 2005, 2006, and 2007 are skipped. In order to ensure the matching of feature points in two adjacent images, it is desirable that the same pattern in the projection pattern be included in the (N - 1)th image and the Nth image.

[0038] In step 2006, the camera movement amount estimation unit 1003 estimates the camera movement amount from the camera position at the time of shooting the (N - 1)-th image to the camera position at the time of shooting the N-th image, starting from the camera position at the time of shooting the (N - 1)-th image.

[0039] In step 2007, the model generation unit 1004 creates a model of the object 101 based on the images from the 1st to the N-th and the camera positions at the times when each image was taken. In step 2008, if the measurement is to be continued, the process returns to step 2002, and if the measurement is to be terminated, the process ends.

[0040] FIG. 4 is a diagram showing a feature point and a feature point matching pair according to Embodiment 1 of the present invention. Image 201 is an image obtained from the camera 102. Image 201A is an image of an object (wall and floor) with few characteristic shapes and patterns, and image 201B is an image taken by moving the camera that took image 201A horizontally by 10 cm. The feature points 202 extracted from each image are shown as white minute points. Since the feature point 202A and the feature point 202B are matched as feature points at physically the same location, the points are connected by a white thin line. Such a combination in which two feature points 202 are connected by a line is a feature point matching pair. In images 201A and 201B, since there are few characteristic shapes and patterns on the object, the number of obtained feature points and the number of feature point matching pairs are relatively small.

[0041] On the other hand, in images 201C and 201D, relatively many feature points and feature point matching pairs are obtained. Image 201C is an image taken at the same location as image 201A after projecting the projection pattern 105. Image 201D is an image taken at the same location as image 201B after projecting the projection pattern 105. By projecting the projection pattern 105, it can be seen that the feature points 202C and 202D are extracted at locations that could not be obtained in images 201A and 201B. These two feature points are matched. Thus, by projecting the projection pattern 105, it is possible to obtain many feature points and feature point matching pairs even for an object with few characteristic patterns and shapes.

[0042] A specific example of the projection pattern will be described with reference to FIGS. 5A to 5E. FIG. 5A is a pattern in which grayscale rectangles are randomly arranged. The rectangles are randomly arranged so that no repeating pattern occurs to eliminate randomness.

[0043] FIG. 5B is a pattern in which black-and-white binary rectangles are randomly arranged. Compared with FIG. 12A, since the pattern stands out by using black-and-white binary values, it is expected that feature points can be easily extracted in step 2004. On the other hand, compared with FIG. 12A, since the number of vertices and intersection points of the rectangles is small, the total number of feature points to be extracted may be small.

[0044] FIG. 5C is a pattern in which black-and-white binary straight lines are arranged at equal intervals. Since this pattern still has a strong randomness, it may be difficult to match feature points.

[0045] FIG. 5D is a pattern in which black-and-white binary vertical lines and diagonal lines are arranged at equal intervals. Since the number of vertices of the pattern is larger than that of FIG. 5C, the total number of feature points to be extracted is expected to be large. On the other hand, since it still has a strong randomness as in FIG. 5C, it may be difficult to match feature points.

[0046] FIG. 5E is a pattern in which black-and-white binary circles are randomly arranged. Since many algorithms for extracting feature points in step 2004 are based on vertices and edges in the image, the number of feature points extracted in the circular pattern may be small.

Example

[0047] Example 2 of the present invention will be described with reference to FIGS. 6 to 10B. In Example 2, in addition to the three-dimensional measurement method shown in Example 1, the necessity of the projection pattern 105 is determined to control the projector 104. According to Example 2, when there is a concern that the accuracy may deteriorate due to the execution of projection, by controlling the projector 104 to stop the projection, the possibility of obtaining an effective set of feature points and feature point matching can be increased.

[0048] FIG. 6 is a block diagram showing an overall configuration example of a three-dimensional measurement system for determining necessity of projection according to Embodiment 2 of the present invention. The difference from the overall configuration example of FIG. 2 is that it includes a projection necessity determination unit 1005 and a projection control unit 1006.

[0049] The projection necessity determination unit 1005 takes as input the scores extracted by the feature point extraction unit 1001 and the feature point matching pairs obtained by the feature point matching unit 1002, and determines the necessity of projection. The method for determining necessity will be described later with reference to FIG. 7. The projection control unit 1006 controls the projector 104 based on the determination result of the projection necessity determination unit 1005, and switches between execution and stop of projection.

[0050] FIG. 7 is a flowchart showing an overall processing example of a three-dimensional measurement system for determining necessity of projection according to Embodiment 2 of the present invention. The difference from the overall processing example of FIG. 3 is that step 2001 is absent and steps 2009, 2010, and 2011 are included.

[0051] In step 2009, the projection necessity determination unit 1005 compares the number of feature points obtained in step 2004 with a predetermined threshold value to determine whether the number of feature points is sufficient. As the threshold value in step 2009, a fixed value, the number per unit pixel of the image, the degree of variation in the positions of the feature points within the field of view of the image, etc. can be considered. If it is determined that the number of feature points is insufficient, the projection control unit 1006 switches between execution and stop of projection.

[0052] In step 2010, the projection necessity determination unit 1005 compares the number of feature point matching pairs obtained in step 2005 with a predetermined threshold value to determine whether the number of feature point matching pairs is sufficient. As the threshold value in step 2010, a fixed value, the number per unit pixel of the image, the ratio of the number of feature points to the number of feature point matching pairs, the degree of variation in the positions of the feature points matched within the field of view of the image, etc. can be considered. If it is determined that the number of feature point matching pairs is insufficient, the projection control unit 1006 switches between execution and stop of projection.

[0053] In step 2011, based on the determination result of the projection necessity determination unit 1005, the projection control unit 1006 switches between executing and stopping the projection. If the projection is already being executed, the projection is stopped, and if the projection is stopped, the projection is executed. Although not shown in FIG. 7, if the thresholds of step 2009 or step 2010 are still not satisfied even after trying both cases of projection startup and stop, either one of the pre-specified projection execution or projection stop may be selected to proceed with the flow.

[0054] FIG. 8 is a diagram showing feature points extracted by the feature point extraction unit according to Embodiment 2 of the present invention. Image 201E and image 201F are images taken at the same location. In image 201E, the projection pattern 105 is not projected, and in image 201F, the projection pattern 105 is projected. An object 203 (headphones) having a characteristic shape exists within the image field of view, and many feature points are extracted by this object. Focusing on the region 204 surrounded by the ellipse in FIG. 201E, it can be seen that many feature points (indicated by fine black and white dots) such as feature point 202E are extracted. On the other hand, focusing on the same location 205 in image 201F, it can be seen that the number of extracted feature points has decreased.

[0055] From FIG. 8, it can be seen that projecting the projection pattern 105 may reduce the number of feature points that can be extracted. The reduction of feature points leads to a deterioration in the accuracy of three-dimensional measurement. From this, it can be understood that when an object 203 having a characteristic pattern or shape exists within the image field of view, projecting the projection pattern 105 does not necessarily contribute to improving the accuracy. Therefore, it is considered effective to sequentially determine the necessity of projection from the number of feature points extracted during scanning and perform three-dimensional measurement under conditions where the number of feature points increases.

[0056] FIG. 9 is a diagram showing feature points and feature point matching pairs according to Embodiment 2 of the present invention. Image 201G and image 201H are images taken with the camera moved 10 cm horizontally. FIGS. 201I and 201J are images taken at the same locations as images 201G and 201J, respectively, with the projection pattern 105 projected.

[0057] Looking at regions 206A and 206B, since there is an object 203 having a characteristic shape, it can be seen that many feature point matching pairs (indicated by fine white dots and white thin lines) were obtained. On the other hand, in regions 207A and 207B, since there is no object 203 having a characteristic shape, only a small number of feature point matching pairs were obtained. Thus, there is a possibility that the feature point matching pairs are obtained in a biased manner in local regions (such as 206A and 206B) within the image field of view.

[0058] On the other hand, looking at regions 208A and 208B, it can be seen that, despite photographing the same location as regions 207A and 207B, as a result of feature points being extractable by the projection pattern 105, new feature point matching pairs were obtained. As a result, feature point matching pairs were obtained over a wide area within the image field of view.

[0059] Due to the nature of the algorithm calculated by the camera movement amount estimation unit 1003, the accuracy of camera movement amount estimation is likely to improve when feature point matching pairs are obtained widely over the entire image field of view. From this, even when there is an object 203 having a characteristic shape within the image field of view, it is expected that projecting the projection pattern 105 can contribute to improving the accuracy of three-dimensional measurement.

[0060] FIG. 10A is a diagram showing the experimental results of the number of feature points according to Example 2 of the present invention. In FIG. 10A, at the same location, photographs were taken while changing the conditions of the presence or absence of an object 203 having a characteristic shape and the presence or absence of projection of the projection pattern 105, and the number of feature points extracted from the images was plotted. First, condition 901 corresponds to image 201A, and it can be seen that the number of extracted feature points is relatively small.

[0061] It can be seen that in condition 902 (corresponding to image 201E) where there is an object 203 having a characteristic shape and in condition 903 (corresponding to image 201C) where the projection pattern 105 was projected, the number of feature points increased by more than twice compared to condition 901.

[0062] Furthermore, in the condition 904 (corresponding to the image 201F) where the presence of the object 203 having a characteristic shape is compatible with the projection of the projection pattern 105, it can be seen that the number of feature points further increases. In this case, since it is expected that projecting the projection pattern 105 will improve the accuracy of three-dimensional measurement, the projection necessity determination unit 1005 may determine that projection is necessary.

[0063] FIG. 10B is a diagram showing the experimental results of the number of feature point matching sets according to the second embodiment of the present invention. The number of feature point matching sets when the camera is moved horizontally by 10 cm is plotted. Comparing condition 905 to condition 908, similar to conditions 901 to 904 (FIG. 10A), the number of feature point matching sets increases when the projection pattern 105 is projected. In this case, since it is described that projecting the projection pattern 105 improves the three-dimensional accuracy, the projection necessity determination unit 1005 may determine that projection is necessary.

[0064] On the other hand, when the number of feature points in condition 902 (about 470) exceeds the predetermined threshold in step 2010, or when the number of feature point matching sets in condition 906 (about 170) exceeds the predetermined threshold in step 2010, the projection necessity determination unit 1005 may determine that projection is not necessary. In this case, since the scan can continue without controlling the projector, the labor of moving the projector during the scan and the labor of installing the projector in advance on the movement path are reduced, and it is considered that the scan can be completed quickly.

[0065] The predetermined threshold may be determined from the trade-off relationship between the labor required for the scan and the accuracy of three-dimensional measurement. If higher accuracy is required, the presence or absence of sequential projection may be switched during the scan (that is, the projection may be blinked), and the one with a larger number of feature points or feature point matching sets may be sequentially selected.

Embodiment

[0066] Example 3 of the present invention will be described with reference to FIGS. 11 and 12. In Example 3, in addition to the three-dimensional measurement method shown in Example 1, the necessity of the projection pattern 105 is determined, and a plurality of projection patterns 105 are selected to control the projector 104. According to Example 3, by projecting an appropriate projection pattern 105 according to the situation within the image field of view, the possibility of obtaining an effective feature point and a feature point matching pair can be increased.

[0067] FIG. 11 is a block diagram showing an overall configuration example of a three-dimensional measurement system that selects a projection pattern 105 according to Example 3 of the present invention. The difference from the overall configuration example of FIG. 2 is that it includes a projection necessity determination unit 1005 and a projection pattern selection unit 1007.

[0068] The projection pattern 105 has at least two or more types of patterns. The projection pattern selection unit 1007 selects an appropriate projection pattern 105 based on the determination result output by the projection necessity determination unit 1005.

[0069] FIG. 12 is a flowchart showing an overall processing example of a three-dimensional measurement system that selects a projection pattern according to Example 3 of the present invention. The difference from the overall processing example of FIG. 3 is that it includes steps 2009, 2010, and 2012.

[0070] In step 2012, based on the determination of the projection necessity determination unit 1005, the projection pattern selection unit 1007 switches a plurality of projection patterns 105. As a result, in steps 2004 and 2005, conditions for obtaining more feature points and feature point matching pairs are explored. As a search method, for example, the patterns may be switched in order or randomly, and the pattern that obtains the most feature points and feature point matching pairs may be selected.

[0071] As another switching method, basically, the projection pattern 105 is switched so as to be visually prominent. For example, it is conceivable to select a pattern of a color that is in a complementary color relationship with the color of the object 101. Alternatively, it is conceivable to select a pattern such that the projection pattern 105 is intensively projected onto a region in the image field where the number of feature points is locally small. By projecting the light amount of the projector 104 onto a limited area in this way, the luminance of the pattern may be improved. At this time, the projector 104 may be controlled to adjust the aperture or focus of the lens of the projector 104.

[0072] Alternatively, it is conceivable to select a pattern that is not similar to the pattern of the object 101. For example, when the object 101 has a straight stripe pattern, projecting a pattern parallel to the stripe pattern may make it difficult to increase the number of feature points, so it is effective to rotate the pattern. In the case of a pattern in which straight stripe patterns are regularly arranged, the same image is taken even if the camera moves. This may be expressed as having remaining arbitrariness. When the arbitrariness remains strong, it becomes difficult to match the feature points in step 2005.

[0073] The matching of feature points becomes simple when each point and the pattern correspond one-to-one (hereinafter, this may be expressed as the elimination of arbitrariness), that is, in a case where the same image is not taken at different points. In the projection pattern selection unit 1007, it is effective to select a pattern that eliminates arbitrariness.

[0074] The projection necessity determination unit 1005 can perform the above control by being configured to input the image captured by the camera 102 and be able to instruct the selection of the projection pattern based on the features of the camera image.

Example

[0075] Example 4 of the present invention will be described with reference to FIGS. 13 to 15. In Example 4, the interval between the feature points of the projected projection pattern as shown in FIGS. 5A to 5E will be exemplified. The appropriate interval between the feature points can be designed according to the shooting conditions of the camera, the resolution of the camera, and the resolution of the projector.

[0076] FIG. 13 is a schematic diagram showing the positional relationship between the shooting range and the projection pattern before and after the movement of the camera according to Example 4 of the present invention. The feature points of the projected projection pattern need to be designed so that the interval between the feature points is not too large in order to fit into both of two adjacent images taken while moving.

[0077] First, the movement of the camera will be described. As described with reference to FIG. 1, the camera 102 continuously shoots the object 101 while being moved by the photographer 103. FIG. 13 shows the shooting ranges of two consecutive images taken during the movement. After the camera 102 shoots at the pre-camera movement position 1200, it moves to the post-camera movement position 1201 and performs the next shooting.

[0078] Let the camera movement distance 1204 be X [m]. The camera is moved by the photographer 103 at a speed V [m / s]. When the shooting frame rate is f [fps], it can be expressed as X = V / f. At this time, the shooting range of the camera also moves by X as the camera moves.

[0079] In FIG. 13, it is illustrated that the pre-camera movement shooting range 1202 moves to the post-camera movement shooting range 1203. V is determined by the moving speed of the photographer. For example, according to the reference (Kunio Akutsu, "The Chemistry of Walking: To Overcome Lack of Exercise", Fushido Shinsho (1975)), the average speed of a 30 - 34-year-old male during normal walking is 1.5 [m / s], so V = 1.5 [m / s] can be considered. f can be determined according to the shooting conditions. For example, in a general three-dimensional scanner, many adopt about f = 8 [fps] in consideration of the computational resources and the amount of calculation that allows real-time processing. In this case,

[0080]

Mathematics

[0081] Next, the shooting range of the camera will be described. Let the size 1205 of the camera shooting range be W [m]. Assuming the subject distance 1206 is D [m] and the camera field angle 1207 is θ [°],

[0082]

Mathematics

[0083] …(Equation 2) can be expressed as. The subject distance 1206 is determined by the internal parameters of the camera including the focal length of the camera, etc. For example, in the case of a camera used in a portable three-dimensional scanner, D = about 1 [m] is often taken as the appropriate distance. The camera field angle 1207 is determined by the performance of the lens, etc. For example, it is conceivable to use a camera with θ = 46 [°]. In this case,

[0084]

Mathematics

[0085] The camera shoots while moving so that the projection pattern 105 fits within the shooting range. Here, in FIG. 13, a rectangular projection pattern 105 is shown as a simple projection pattern 105. The projection pattern 105 is not limited to a rectangle and may have other pattern shapes. As the feature points 202 in the rectangular projection pattern, for example, four vertices (202F, 202G, 202H, 202I) can be considered.

[0086] The distance between two feature points is defined as the feature point interval 1208, denoted as L [m]. To estimate the camera movement amount by the camera movement amount estimation unit 1003, it is necessary to match the same feature point 202 in two images before and after the camera moves in the feature point matching unit 1002. For this purpose, at least one feature point must be within the shooting range of the camera in the two images before and after the camera moves. To meet this condition, L may be determined such that two feature points are within the overlapping area 1209 of the pre-camera movement shooting range 1202 and the post-camera movement shooting range 1203. Expressed in a mathematical formula,

[0087]

Number

[0088]

Number

[0089] …(Equation 5) It can be expressed as. When (Equation 4) or (Equation 5) is satisfied, if one or more feature points 202 are within the pre-camera movement shooting range 1202, regardless of the positional relationship between the camera shooting range and the feature points, it is guaranteed that at least one or more feature points are within the post-camera movement shooting range 1203. Substituting (Equation 1) and (Equation 3) into (Equation 4),

[0090]

Number

[0091] Next, the lower limit value of the feature point interval is designed based on the resolution of the camera or the projector. The feature points of the projected projection pattern need to be designed so that the feature point interval is not too small so that they can be distinguished by the resolution of the camera. The camera resolution can generally be quantitatively evaluated using a resolution chart defined by ISO12233, etc.

[0092] FIG. 14 is a schematic diagram showing an example of a resolution chart for evaluating the resolution of the camera according to Embodiment 4 of the present invention. The resolution chart 1301 is described with a stripe pattern composed of black and white lines. The black and white lines can be not only horizontal lines but also vertical lines, diagonal lines, curves, and the like. The resolution chart 1301 is described with a plurality of stripe patterns having different intervals between the black and white lines. When evaluating the resolution of the camera, the resolution chart 1301 is photographed so that it appears in the entire photographing range.

[0093] Here, stripe patterns 1302A and 1302B that can be resolved, in which the black and white stripe patterns can be distinguished, and stripe patterns 1303 that cannot be resolved, in which the black and white stripe patterns cannot be distinguished, are photographed. The reason for the inability to distinguish is that the resolution of the camera is insufficient, resulting in the appearance of moiré patterns and the like. In the case of FIG. 14, the resolvable stripe pattern 1302B can be regarded as the resolution performance of the camera.

[0094] The resolution chart 1301 is described with the value of LW / PH (Line Width per Picture Height) indicating the resolution for each stripe pattern. LW / PH is defined as the number of black and white lines laid flat in the height direction of the resolution chart. For example, when the resolution chart 1301 is photographed to fit the entire height of the camera shooting range and the stripe pattern with LW / PH = 100 can be distinguished, the camera is evaluated to be able to resolve 50 white lines and 50 black lines in the vertical direction of the screen. In the case of FIG. 14, since the resolvable stripe pattern 1302B can be distinguished, it is evaluated as LW / PH = 200. Let the camera resolution 1304 be δ [LW / PH], then

[0095]

Equation

[0096] …(Equation 7) It can be set. δ is defined by the performance of the camera used. For example, δ of the rear camera of a general smartphone or tablet is about 200 [LW / PH].

[0097] In order for the feature points of the projected projection pattern to be distinguishable by the resolution of the camera, the feature point interval L needs to be larger than the minimum line width that can be resolved by the camera. The minimum line width that can be resolved by the camera can be expressed as the line width of the black and white stripe pattern obtained by dividing the camera shooting range W by the camera resolution δ. That is,

[0098]

Equation

[0099]

Equation

[0100] …(Equation 9) can be derived. From Equation (9), the lower limit value of the feature point interval L is determined. The projection pattern 105 may be designed so that the feature point interval becomes larger than 0.8 [mm] when projected.

[0101] FIG. 15 is a schematic diagram showing an example of a resolution chart for evaluating the resolution of the projector according to Embodiment 4 of the present invention. The resolution chart 1301 is generally a tool used to evaluate the resolution of a camera, but it can also be used in the same way to evaluate the resolution of a projector. By projecting the resolution chart 1301 at the full height of the projection range with the projector 104 and checking whether the projected black and white stripe pattern can be distinguished, the projector resolution 1305 can be evaluated. Let the projector resolution 1305 be ζ [LW / PH]. ζ is defined by the resolution of the projector, but generally, it can be considered that the projector resolution ζ is lower than the camera resolution. For example,

[0102]

Number

[0103] In order to distinguish the feature points of the projected pattern, the feature point interval L needs to be larger than the minimum line width that can be resolved by the projector 104. The minimum line width that can be resolved by the projector can be expressed as the line width of the black-and-white striped pattern obtained by dividing the projection range 1306 (let it be Z [m]) by the projector resolution ζ. That is,

[0104]

Number

[0105] …(Equation 11) needs to satisfy the condition. The projection range Z is determined by the positional relationship between the projector 104 and the object 101 to be projected. Widening the projection range Z allows the projection pattern to be projected onto a wider range of the object 101 with a single projector, so fewer projectors are required. On the other hand, when the projection range Z is widened, the brightness of the projection pattern 105 decreases, making it difficult to obtain contrast. For this reason, if a high-brightness projector is used, the projection range Z can be widened, but high-brightness projectors are generally expensive. For example, assuming a projector for a conference room,

[0106]

Number

[0107]

Number

[0108] From the above considerations, from the shooting conditions of the camera, the resolution of the camera, and the resolution of the projector, the feature point interval L satisfies (Equation 6), (Equation 9), and (Equation 13).

[0109]

Number

[0110] …(Equation 14) It can be seen that it is appropriate. The projection pattern 105 is preferably pattern-designed such that the feature point interval L when projected satisfies (Equation 14). That is, it is desirable that the interval between the feature points of the projected pattern is between 5 mm and 660 mm.

Example

[0111] Example 5 of the present invention will be described with reference to FIGS. 16A to 16D. In Example 5, a specific example of the projection pattern represented by alphanumeric symbols in the projection pattern shown in Example 1 is illustrated. In order to estimate the camera movement amount, it is important to design a projection pattern suitable for feature point extraction and feature point matching. As an effective projection pattern, there is an alphanumeric symbol pattern represented by alphanumeric symbols. The alphanumeric symbol pattern includes at least one of characters and symbols including at least a curve and an intersection. Specific examples of the alphanumeric symbol pattern will be described with reference to FIGS. 16A to 16D.

[0112] FIG. 16A is a pattern in which hiragana in black and white binary are arranged at equal intervals. It has a large number of vertices, and the arbitrariness of the pattern is eliminated, and it is considered an effective pattern.

[0113] FIG. 16B is a pattern in which the black and white of FIG. 16A are inverted. Similar to FIG. 16A, since it has a large number of vertices and the arbitrariness of the pattern is eliminated, it is considered an effective pattern.

[0114] FIG. 16C is a pattern in which specific single hiragana characters in black and white binary are arranged at equal intervals. Compared with FIG. 16A, since randomness of the pattern remains, it is considered to be a less effective pattern than FIG. 16A.

[0115] FIG. 16D is a pattern in which black and white binary alphabets, Greek characters, and symbols are arranged at equal intervals. Since the number of vertices is large and randomness of the pattern is eliminated, it is considered to be an effective pattern.

[0116] Characters and symbols are patterns that include many vertices and are easy for the photographer 103 to recognize as being projected onto the object 101. Therefore, from the viewpoint of improving the workability of the photographer 103, they are considered to be effective patterns.

[0117] Regarding the color of the pattern, basically, black and white binary is considered to be effective. In the feature point extraction algorithm in step 2004, there are many algorithms that search for feature points based on the change in pixel values with neighboring pixels after converting the RGB pixel values to grayscale. Since black and white binary is the color scheme with the largest change in pixel values, it is considered that the projection pattern 105 composed of black and white binary is easy to extract feature points.

Embodiment

[0118] Embodiment 6 of the present invention will be described with reference to FIGS. 17 and 18. In Embodiment 6, the size of the projected character symbol pattern is exemplified. The appropriate size of the character symbol pattern can be designed according to the shooting conditions of the camera, the line width of the character symbol, the resolution of the camera, and the resolution of the projector.

[0119] FIG. 17 is a schematic diagram showing the relationship between the shooting range before and after camera movement and the position of the character symbol pattern according to Embodiment 6 of the present invention. Here, the projection pattern is a pattern including the character symbol pattern 1401 as shown in Embodiment 5. The character symbol pattern 1401 to be projected needs to be designed so that the full form of the character symbol fits within both of the two images taken while moving and each character symbol is not too large.

[0120] The camera takes pictures while moving so that the character symbol pattern 1401 is within the shooting range. Here, in FIG. 17, two character symbol patterns (1401A and 1401B) composed of hiragana are shown. Let the size 1402 of each character symbol pattern be S [mm].

[0121] In order to estimate the camera movement amount by the camera movement amount estimation unit 1003, it is desirable to match the same character symbol pattern 1401 in two images before and after the camera moves by the feature point matching unit 1002. For this purpose, in two images before and after the camera moves, at least two character symbol patterns 1401 need to be within the shooting range of the camera. In order to satisfy this condition, it is advisable to determine L so that two character symbol patterns 1401 are within the overlapping area 1209 of the pre-camera movement shooting range 1202 and the post-camera movement shooting range 1203. Expressed by a mathematical formula,

[0122]

Equation

[0123] …(Equation 15) It can be expressed as. When (Equation 15) is satisfied, if one or more character symbol patterns 1401 are within the pre-camera movement shooting range 1202, regardless of the positional relationship between the camera shooting range and the character symbol pattern 1401, it is guaranteed that the whole form of at least one or more character symbol patterns 1401 is within the post-camera movement shooting range 1203. Substituting (Equation 1) and (Equation 3) into (Equation 15),

[0124]

Equation

[0125] …(Equation 16) It can be derived. From (Equation 16), the upper limit value of the size S of the character symbol pattern is determined. That is, considering the shooting conditions of a general camera, the vertical and horizontal lengths of the projected characters or symbols are preferably between 60 mm and 330 mm.

[0126] Next, the lower limit value of the size 1402 of the character symbol pattern is designed from the pattern resolution and the resolutions of the camera and the projector. The projected character symbol pattern needs to be designed so that it can be distinguished by the resolution of the camera and also by the resolution of the projector. The camera resolution and the projector resolution are represented by (Equation 7) and (Equation 10) as described in Example 4.

[0127] FIG. 18 is a schematic diagram showing the feature points and resolution of the character symbol pattern according to Example 6 of the present invention. As the character symbol pattern 1401C, the case of using hiragana characters (font name "MS P Gothic", using hiragana "あ") is illustrated.

[0128] In this case, the size 1402 of the character symbol pattern is the vertical or horizontal size of the hiragana character. The feature point 202 is, for example, the vertex part of the outline of the hiragana character is extracted. In order to distinguish the feature point 202, it is necessary to distinguish at least the character symbol pattern line width 1403. In the case of MS P Gothic, the character symbol pattern line width 1403 is the line width obtained by dividing the size 1402 of the character symbol pattern into about 12 parts. Let the number of lines that can be laid out for the line width of the line for drawing the character symbol with respect to the size S of the character symbol pattern be the character symbol pattern resolution 1404, and let this be ε. In the case of the MS P Gothic font,

[0129]

Equation

[0130] In order for the camera to distinguish the lines of the projected character symbol pattern, the camera resolution needs to be higher than the character symbol pattern resolution. Also, in order for the projector to distinguish and project the lines of the character symbol pattern, the projector resolution needs to be higher than the character symbol pattern resolution. Expressing each of them as mathematical formulas,

[0131]

Number

[0132]

Number

[0133]

Number

[0134]

Number

[0135] From the above considerations, from the shooting conditions of the camera, the line width of the character symbol, the resolution of the camera, and the projection resolution of the projector, the size S of the character symbol pattern satisfies (Equation 16), (Equation 20), and (Equation 21)

[0136]

Number

[0137] …(Equation 22) It can be seen that is appropriate. It is desirable that the projection pattern 105 is pattern-designed such that the size S of the projected character symbol pattern satisfies (Equation 22).

Example

[0138] Example 7 of the present invention will be described with reference to FIGS. 19A to 24C. In Example 7, in addition to the three-dimensional measurement method shown in Example 1, in the construction of a through-pipe, the position (virtual hole penetration position) where a hole will open on the opposite side of the wall before drilling a hole in the wall is estimated by three-dimensional measurement. According to Example 7, a construction procedure can be realized in which the straight pipes on both sides of the wall are installed in advance before drilling a hole in the wall, and finally a hole is drilled in the wall to install the through-pipe, and the construction period can be shortened.

[0139] FIG. 19A is a schematic diagram showing a cross-section of a wall for installing a through-pipe according to Example 7 of the present invention. In Example 7, in the construction of installing pipes in a plant, particularly, a construction of drilling a through-hole 303 in a wall 301 for installing a through-pipe 304 and connecting the straight pipe 305 and the through-pipe 304 via an adjustment pipe 306 (hereinafter, may be expressed as wall through-pipe construction) is dealt with. Three straight pipes 305 (305A, 305B, 305C) are shown on both sides of the wall 301, but the number is not limited to three, and any number may be installed. The straight pipe 305 may be installed in the air as shown in FIG. 19A, or may be installed on the floor 302.

[0140] Here, the plant includes all places where pipes are installed, such as power plants, chemical plants, food plants, assembly factories, construction sites, and building sites. Also, this embodiment can be applied to buildings, shopping malls, stations, airports, theaters, museums, etc.

[0141] The adjustment pipe 306A is installed to absorb and connect the misalignment between the straight pipe 305 and the through pipe 304. For example, since the right end of the straight pipe 305B and the left end of the through pipe 304 are at different heights, the adjustment pipe 306A is installed to absorb the height misalignment and make the connection. Since there is a limit to the magnitude of the misalignment that can be absorbed by the adjustment pipe 306, it is desirable that the misalignment be as small as possible. For example, assume that the adjustment pipe 306 can absorb a misalignment of 100 mm. There is no problem if the misalignment between the straight pipe 305 and the through pipe 304 is within 100 mm, but if it exceeds 100 mm, construction such as removing the already installed straight pipe 305 and correcting its position is required, which incurs a huge cost.

[0142] From the above viewpoints, the wall penetration pipe construction is carried out in the following procedure. First, a through hole 303 is drilled in the wall 301, and the through pipe 304 is installed. As a result, the positions of the left and right ends of the through pipe 304 are determined. Based on this, the straight pipes 305 are installed on both sides of the wall so that the final misalignment is within the absorption capacity of the adjustment pipe 306. Finally, the adjustment pipe 306 is installed, and all the pipes are connected.

[0143] The problem with this procedure is that the installation of the straight pipe 305 cannot be started until the through hole 303 is drilled. There are restrictions on the timing when the through hole 303 can be drilled in the plant. For example, when the through hole 303 is drilled, the risk of water overflow and fire risk in the rooms on both sides are mixed. Therefore, from the perspective of hazard management, the drillable time is determined depending on the operating status of the plant and the progress of other construction works. When the drilling time is delayed, the installation time of the straight pipe 305 is also delayed in the same way. Also, generally, a large number of through pipes 304 are installed in the plant. In the case of construction where the through pipe 304 is connected to the end of the straight pipe 305, the installation time is sequentially delayed, and the total construction period is prolonged.

[0144] Therefore, in order to shorten the construction period, it is required to install the straight pipe 305 before drilling the through hole 303. If the installation of the straight pipe 305 is completed in advance, the construction can be completed only by installing the through pipe 304 and the adjustment pipe 306A at a time when drilling is possible for convenience of hazard management, so that the construction period can be significantly shortened. For this purpose, before drilling the through hole 303, it is necessary to accurately grasp the positions at both ends of the through pipe 304 to be installed and install the straight pipe 305 so that it falls within the range that can be absorbed by the adjustment pipe 306.

[0145] FIG. 19B is a schematic diagram showing a cross-section of the wall three-dimensionally measured before drilling the through hole 303 according to Example 7 of the present invention. First, the hole construction position 307 before drilling is set. The adjustment required distance 311 on the first surface of the wall is set so as to fall within the absorption capacity of the adjustment pipe 306. Starting from the hole construction position 307, three-dimensional measurement is started, and it is desired to measure the position where the hole penetrates on the opposite side of the wall (hereinafter sometimes referred to as the virtual hole penetration position). A point on the opposite side obtained by drawing a straight line perpendicular to the wall 301 from the hole construction position 307 is defined as the true virtual hole penetration position 308. However, the three-dimensionally measured virtual hole penetration position 309 includes the measurement error 310 due to three-dimensional measurement. In order for the adjustment required distance 312 on the second surface of the wall to fall within the absorption capacity of the adjustment pipe 306, it is required that the measurement error 310 due to three-dimensional measurement be small.

[0146] FIG. 20 is a plan view of a plant for drilling a through hole according to Example 7 of the present invention. In order to install the through pipe 401, the photographer 103 who performs three-dimensional measurement performs three-dimensional measurement along a moving path 405 from the room 403 to 404 where the through pipe is to be installed.

[0147] FIG. 21 is a diagram showing the operation of a photographer who performs three-dimensional measurement before making a through hole according to Example 7 of the present invention. The photographer 103 sets the hole construction position 307 as the shooting starting point, performs scanning while passing through the movement path 405, and performs three-dimensional measurement up to the three-dimensionally measured virtual hole penetration position 309 on the opposite side of the wall 301. At this time, instead of the camera 102, the photographer 103 may perform three-dimensional scanning using the input interface-equipped camera 500 capable of setting the hole construction position 307 and displaying the virtual hole penetration position 309.

[0148] FIG. 22A is a diagram showing an example of an interface for setting a hole construction position according to Example 7 of the present invention. A touch monitor 502 is displayed on the input interface-equipped camera 500, and an image taken by the camera of the input interface-equipped camera 500 is displayed on the touch monitor 502. Here, the photographer 103 sets the hole construction position 307 using the finger 501 of the photographer, a touch pen, a mouse, a keyboard, or the like.

[0149] FIG. 22B is a diagram showing an example of an interface for displaying a hole penetration position according to Example 7 of the present invention. The three-dimensionally measured virtual hole penetration position 309 is displayed on the touch monitor 502. Here, since the wall thickness information has been measured by three-dimensional measurement, the depth direction of the wall that was not displayed in FIG. 22A is shown. The virtual hole penetration position 309 can be defined as the point where a straight line drawn perpendicular to the wall starting from the hole construction position 307 intersects the surface on the opposite side of the wall.

[0150] FIG. 23 is a flowchart showing an example of an operation procedure performed by a photographer who performs three-dimensional measurement before drilling a through hole according to Example 7 of the present invention. In step 3001, the first surface of the wall to be drilled with the through hole is photographed with a camera. In step 3002, the hole construction position 307 is input using an interface such as the touch monitor 502. The hole construction position 307 may be input by being calculated from separately input CAD (Computer Aided Design) data, building data, construction drawings, construction data, or the like. In step 3003, a scan inside the plant is performed while moving along the movement path 405. In step 3004, the second surface of the wall to be drilled with the through hole is reached. In step 3005, the virtual hole penetration position displayed on the interface is confirmed.

[0151] FIG. 24A is a block diagram showing a first example of the overall configuration of a three-dimensional measurement system that performs three-dimensional measurement before drilling a through hole according to Example 7 of the present invention. The information processing apparatus 1000 includes a virtual hole penetration position estimation unit 1100 that estimates the position of the virtual hole penetration position 309 using the three-dimensional model output by the model generation unit 1004. As shown in FIG. 16A, the camera 102 includes an input unit 1101 and a display unit 1102 as an interface. The virtual hole penetration position estimation unit 1100 calculates, for example, a point where a straight line drawn perpendicular to the wall from the hole construction position 307 intersects the surface on the opposite side of the wall, and sets it as the virtual hole penetration position 309.

[0152] FIG. 24B is a block diagram showing a second example of the overall configuration of a three-dimensional measurement system that performs three-dimensional measurement before drilling a through hole according to Example 7 of the present invention. The difference from the block diagram of FIG. 24A is that the input unit 1101 and the display unit 1102 are inside the information processing apparatus 1000. A form in which a keyboard, a mouse, a display, or the like included in the information processing apparatus 1000 is used as an interface can be considered.

[0153] FIG. 24C is a block diagram showing a third example of the overall configuration of a three-dimensional measurement system that performs three-dimensional measurement before drilling a through hole according to Example 7 of the present invention. The difference from the block diagram of FIG. 24A is that the input unit 1101 and the display unit 1102 are inside the interface terminal 1103. As the interface terminal 1103, a tablet, a tablet-type PC, a smartphone, or the like can be considered. The interface terminal 1103 is connected to the information processing apparatus 1000 by wire or wirelessly for communication.

Example

[0154] Example 8 of the present invention will be described with reference to FIG. 25. In Example 8, in addition to the method of estimating the position where a hole will open on the opposite side of the wall (virtual hole penetration position) by three-dimensional measurement before drilling the hole in the wall shown in Example 7, the virtual hole penetration position is displayed to the photographer 103 in an AR display method. According to Example 8, the photographer 103 can easily visually recognize the three-dimensionally measured virtual hole penetration position 309.

[0155] FIG. 25 is a diagram showing an example of glasses for AR-displaying a virtual hole penetration position according to Example 8 of the present invention. The AR glass type interface 600 has an AR glass display lens 601 and is displayed in an AR display method by superimposing the three-dimensionally measured virtual hole penetration position 309 on the actual wall surface. The AR glass type interface 600 may be equipped with an AR glass built-in camera 602. In that case, the camera 102 and the display unit of the interface can be integrated, and the number of measuring devices can be reduced.

Example

[0156] Example 9 of the present invention will be described with reference to FIG. 26. In Example 9, in addition to the three-dimensional measurement method shown in Example 1, a walking guide is displayed in the projection pattern. According to Example 9, the photographer 103 can easily visually recognize the path along which he / she moves during scanning.

[0157] FIG. 26 is a diagram showing how a three-dimensional measurement system onto which a projection pattern with a walking guide according to Example 9 of the present invention is projected is used. Since the photographer 103 focuses on operating the camera 102 to take pictures during scanning, his / her attention is concentrated on his / her hands, making it difficult to grasp the surrounding situation. At this time, by projecting the projection pattern 700 with a walking guide, it is effective because the path along which the photographer 103 should move can be easily visually recognized. The projection pattern 700 with a walking guide may be projected onto a wall as shown in 700A, or may be projected onto the floor as shown in 700B.

[0158] The walking guide may be determined after calculating the movement path 405 based on the in-plant floor plan, or may be determined after calculating so as to guide the photographer 103 so that the projection pattern projected from the projector 104 is not blocked by the photographer 103.

[0159] FIG. 27 is a diagram showing an example of a camera with a monitor that displays a projection pattern with a walking guide according to Example 9 of the present invention. It is also effective for the walking guide to be displayed on the camera held by the photographer 103, such as the camera 701 with a monitor.

Example

[0160] Example 10 of the present invention will be described with reference to FIG. 28. In Example 10, in the three-dimensional measurement method shown in Example 1, a camera and a projector are not included.

[0161] FIG. 28 is a block diagram showing an example of the overall configuration of a three-dimensional measurement system that does not include a camera and a projector according to Example 10 of the present invention. The difference from FIG. 2 is that the camera 102 and the projector 104 are not present. The information processing device 1000 performs processing using the image 801 input from the outside, and issues a projection command 802 to an external device.

[0162] According to the above embodiments, efficient construction or shortening of the construction period can be realized, so that energy consumption is low, carbon emissions are reduced, global warming can be prevented, and it can contribute to the realization of a sustainable society.

Explanation of Reference Numerals

[0163] 100 Three-dimensional measurement system, 101 Object, 102 Camera, 103 Photographer, 104 Projector, 105 Projection pattern, 106 Shooting range, 107 Three-dimensional model, 201 Image, 202 Feature point, 1000 Information processing device, 1001 Feature point extraction unit, 1002 Feature point matching unit, 1003 Camera movement amount estimation unit, 1004 Model generation unit, 1005 Projection necessity determination unit, 1006 Projection control unit, 1007 Projection pattern selection unit, 1100 Virtual hole penetration position estimation unit

Claims

1. A projection step of projecting a projection pattern onto an object, A photographing step of photographing at least two images of the object with a moving camera, A feature point extraction step of inputting the two images into an information processing apparatus and extracting feature points from the two images by the information processing apparatus, A feature point matching step of searching for pairs of the feature points indicating the same location by the information processing apparatus and calculating a feature point matching pair, A movement amount estimation unit step of estimating the movement amount of the camera from the feature point matching pair by the information processing apparatus, A model generation step of generating a three-dimensional model of the object from the movement amount of the camera by the information processing apparatus, A three-dimensional measurement method for executing the above steps.

2. At least one identical pattern in the projection pattern is included in the two images, The three-dimensional measurement method according to Claim 1.

3. The information processing apparatus, Based on the number of the feature points extracted in the feature point extraction step, determining whether to project the projection pattern, and based on the number of the feature point matching pairs searched in the feature point matching step, determining whether to project the projection pattern, and executing at least one of the above determinations, The three-dimensional measurement method according to Claim 1.

4. The information processing apparatus, Based on the number of the feature points extracted in the feature point extraction step, determining whether to switch the type of the projection pattern, based on the number of the feature point matching pairs searched in the feature point matching step, determining whether to switch the type of the projection pattern, and based on the images, determining whether to switch the type of the projection pattern, and executing at least one of the above determinations, The three-dimensional measurement method according to Claim 1.

5. The interval between the feature points of the projected pattern is between 5 mm and 660 mm for the projection pattern, The three-dimensional measurement method according to Claim 1.

6. The projection pattern is at least one of characters and symbols including at least a curve and an intersection point, The three-dimensional measurement method according to Claim 1.

7. The vertical and horizontal lengths of the projected character or symbol are between 60 mm and 330 mm, The three-dimensional measurement method according to Claim 6.

8. The projection pattern includes a guide indicating the moving direction of the camera, The three-dimensional measurement method according to Claim 1.

9. A three-dimensional measurement device that takes at least two images of an object onto which a projection pattern is projected, using a moving camera as input, comprising a feature point extraction unit, a feature point matching unit, a camera movement amount estimation unit, and a model generation unit, wherein the feature point extraction unit extracts feature points from the two images, the feature point matching unit searches for pairs of feature points indicating the same location and calculates a feature point matching pair, the camera movement amount estimation unit estimates the movement amount of the camera from the feature point matching pair, and the model generation unit generates a three-dimensional model of the object from the movement amount of the camera. Three-dimensional measurement device.

10. The two input images include the same pattern in the projection pattern. The three-dimensional measurement device according to claim 9.

11. further comprising a projection necessity determination unit, wherein the projection necessity determination unit determines whether to project the projection pattern based on the number of feature points extracted by the feature point extraction unit, and determines whether to project the projection pattern based on the number of feature point matching pairs searched by the feature point matching unit, and performs at least one of these. The three-dimensional measurement device according to claim 9.

12. further comprising a projection necessity determination unit, wherein the projection necessity determination unit determines whether to switch the type of the projection pattern based on the number of feature points extracted by the feature point extraction unit, determines whether to switch the type of the projection pattern based on the number of feature point matching pairs searched by the feature point matching unit, and determines whether to switch the type of the projection pattern based on the image, and performs at least one of these. The three-dimensional measurement device according to claim 9.

13. further comprising a virtual hole penetration position estimation unit, wherein the virtual hole penetration position estimation unit calculates a point where a straight line drawn perpendicular to the wall from a predetermined location on the wall intersects the surface on the opposite side of the wall, based on the three-dimensional model of the object, and sets it as the virtual hole penetration position. The three-dimensional measurement device according to claim 9.

14. further comprising a display unit, wherein the display unit is constituted by an AR display device and displays the virtual hole penetration position superimposed on a real wall surface. The three-dimensional measurement device according to claim 13.

15. A three-dimensional measurement system including a camera that captures an object while moving, a projector that projects a projection pattern onto the object, and an information processing apparatus that receives an image from the camera and issues a projection command to the projector. The information processing apparatus includes a feature point extraction unit that extracts feature points in the image from two images including the projection pattern captured before and after the camera moves, a feature point matching unit that searches for pairs of the feature points indicating the same location and calculates a feature point matching pair, a movement amount estimation unit that estimates the movement amount of the camera from the feature point matching pair, and a model generation unit that generates a three-dimensional model of the object from the movement amount of the camera. Three-dimensional measurement system.

Images