Program
A program corrects image distortions using point cloud data to facilitate easy and accurate measurement of surfaces like walls and ceilings by synthesizing scale images with captured images, addressing the limitations of existing surveying technologies.
Patent Information
- Application Number
- JP2023073977
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-04-28
AI Technical Summary
Existing technologies require advanced surveying skills and are limited in purpose to measuring tunnel interiors, making it difficult to easily measure surfaces like walls, ceilings, and floors in a house.
A program that inputs point cloud data from a three-dimensional shape measuring device, performs projective transformation on scale images to correct distortions, and synthesizes these images with captured images to enable easy measurement of target surfaces by projecting them onto a two-dimensional screen.
Enables users to accurately and easily measure dimensions of target surfaces by viewing composite images, allowing for real-time surveying and adjustment of scale images to match surface distortions.
Smart Images

Figure 2024161925000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a program. [Background technology]
[0002] Patent Document 1 discloses a system that uses a laser scanner to measure the three-dimensional shape inside a tunnel and uses a management device to manage the construction of the tunnel based on the measurement data from the laser scanner. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2023-003119 A Summary of the Invention [Problem to be solved by the invention]
[0004] The technology of Patent Document 1 is suitable for accurately measuring the shape of the inner surface of a tunnel, but it requires users to have advanced surveying skills, and its intended use is limited to the surveying of tunnels. The present invention has been made in consideration of the above circumstances, and an object of the present invention is to enable easy surveying of target surfaces such as the walls, ceilings, and floors of a house. [Means for solving the problem]
[0005] The following reference numerals in parentheses refer to FIGS. 1 to 9.
[0006] According to the invention of claim 1, a computer (11) receives an image captured by an imaging device (14) and a point cloud data consisting of three-dimensional coordinate values of a plurality of measurement target points measured by a three-dimensional shape measuring device (15), A recognition process for extracting and recognizing point cloud data of a target surface from the point cloud data input from the three-dimensional shape measuring device (15); and executing a synthesis process in which the scale image (21, 22 or 23) is projectively transformed onto a two-dimensional screen coordinate system of an image captured by the imaging device (14) based on the point cloud data of the target surface recognized by the recognition process, and the scale image (21, 22 or 23) is synthesized with the image captured by the imaging device (14) as the background and the scale image (21, 22 or 23) as the foreground. The present invention provides a program (17) comprising:
[0007] According to the invention of claim 1 as described above, the target surface in the image captured by the imaging device (14) is distorted from its actual shape due to perspective, and since the scale image (21, 22 or 23) is projected and transformed into a two-dimensional screen coordinate system of the image based on the point cloud data of the target surface, the scale image (21, 22 or 23) is also distorted according to the distortion of the target surface. Therefore, the length along the target surface can be accurately measured from the scale image (21, 22 or 23). Also, the user can easily measure the target surface by simply looking at the composite image of the image captured by the imaging device (14) and the scale image (21, 22 or 23).
[0008] According to the invention of claim 2, there is provided a program (17) according to claim 1, The scale image (21, 22 or 23) is an image of a grid pattern, an image in which a plurality of cross marks are regularly arranged two-dimensionally, or an image in which a plurality of second line segments (23b) shorter than a first line segment (23a) are arranged at regular intervals along the first line segment (23a). The present invention provides a program (17) comprising:
[0009] According to the invention of claim 2 as described above, if the scale image (21, 22 or 23) is an image of a grid pattern, the position, size, range, etc. of scratches and patterns present on the target surface can be easily measured from the grid. If the scale image (21, 22 or 23) is an image of a plurality of cross marks regularly arranged two-dimensionally, the number, position, and accuracy of measurement objects such as screws present on the target surface can be easily measured from the cross marks (22a, 22b, and 22c). If the scale image (21, 22 or 23) is an image of a first line segment (23a) extending straight along which a plurality of second line segments (23b) shorter than the first line segment (23a) are arranged at regular intervals, the length along the first line segment (23a) can be easily measured from the number of second line segments (23b), etc.
[0010] According to the invention of claim 3, there is provided a program (17) according to claim 1 or 2, The computer (11), A process of displaying in real time a live view screen in which the scale image (21, 22 or 23) is synthesized with the video by the synthesis process. Run The present invention provides a program (17) comprising:
[0011] According to the invention as defined in claim 3, the target surface can be surveyed in real time.
[0012] According to the invention of claim 4, A program (17) according to claim 1 or 2, The target surface is flat. The present invention provides a program (17) comprising:
[0013] According to the invention as defined in claim 4, the processing by the computer (11) is simple.
[0014] According to the invention of claim 5, A program (17) according to claim 1 or 2, The computer (11), A process for determining a translation amount of the scale image (21, 22 or 23) in a two-dimensional local coordinate system based on a signal from an input device (12). Run the command, The synthesis process includes a process of projectively mapping the scale image (21, 22 or 23) onto a two-dimensional screen coordinate system of an image captured by the imaging device (14) based on the point cloud data of the target surface and the translation amount. The present invention provides a program (17) comprising:
[0015] According to the invention as defined in claim 5, it is possible to measure the measurement object in accordance with the position of the measurement object on the target surface.
[0016] According to the invention of claim 6, there is provided a program (17) according to claim 1 or 2, The computer (11), Executing a process for determining a translation amount of the scale image (21, 22 or 23) in a two-dimensional local coordinate system based on a signal from an input device (12); The synthesis process includes a process of projectively mapping the scale image (21, 22 or 23) onto a two-dimensional screen coordinate system of an image captured by the imaging device (14) based on the point cloud data of the target surface and the translation amount. The present invention provides a program (17) comprising:
[0017] According to the invention as set forth in claim 6, the measurement object can be measured in accordance with the rotation angle of the measurement object on the target surface.
[0018] According to the invention of claim 7, there is provided a program (17) according to claim 1 or 2, The computer (11), Executing a process for determining a depth movement amount of the scale image (21, 22 or 23) in a direction perpendicular to the scale image (21, 22 or 23) based on a signal from an input device (12); The synthesis process includes a process of projectively mapping the scale image (21, 22 or 23) onto a two-dimensional screen coordinate system of an image captured by the imaging device (14) based on the depth movement amount in addition to the point cloud data of the target surface. The present invention provides a program (17) comprising:
[0019] According to the invention as defined in claim 7, the object to be measured can be measured in accordance with the positional deviation of the object to be measured in the direction perpendicular to the object surface.
[0020] According to the invention of claim 8, there is provided a program (17) according to claim 1 or 2, The computer (11), Executing a process to determine the transparency of the scale image (21, 22 or 23) based on a signal from an input device (12); The synthesis process includes a process of synthesizing the scale image (21, 22 or 23) with the image captured by the imaging device (14) at the transparency. The present invention provides a program (17) comprising:
[0021] According to the invention as defined in claim 8, it is possible to express that the scale image (21, 22 or 23) is in front of or behind the target surface.
[0022] According to the invention of claim 9, there is provided a program (17) according to claim 1, the scale image (23) is an image in which a first line segment (23a) extending straight is followed by a plurality of second line segments (23b) shorter than the first line segment (23a) and arranged at regular intervals; The computer (11), executing a process for determining a display range of the scale image (23) in a direction along the first line segment (23a) based on a signal from an input device (12); The synthesis process includes a process of synthesizing the scale image (23) in the display range with the image captured by the imaging device (14). The present invention provides a program (17) comprising:
[0023] According to the invention as set forth in claim 9, the display range of the scale image (23) can be adjusted to the length of the measurement object on the target surface. Effect of the Invention
[0024] According to the present invention, a user can easily survey a target surface by simply looking at a composite image of an image captured by an imaging device and a scale image. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a diagram showing an example of a usage state of a measurement support device. [Diagram 2] FIG. 2 is a block diagram of the measurement support device. [Diagram 3] FIG. 3 is a diagram showing the first scale image. [Figure 4] FIG. 4 is a diagram showing the second scale image. [Diagram 5] FIG. 5 is a diagram showing the third scale image. [Figure 6] FIG. 6 is a diagram showing an example of a live view screen displayed on a display device of the measurement support apparatus. [Figure 7] FIG. 7 is a diagram showing an example of a live view screen displayed on a display device of the measurement support apparatus. [Figure 8] FIG. 8 is a diagram showing an example of a live view screen displayed on a display device of the measurement support apparatus. [Figure 9] FIG. 9 is a diagram showing an example of a live view screen displayed on a display device of the measurement support apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Hereinafter, the embodiments will be described with reference to the drawings. The features and technical effects of the embodiments will be understood from the following detailed description and drawings. However, the scope of the present invention is not limited to the embodiments disclosed below. The drawings are provided for illustrative purposes only, and therefore the scope of the present invention is not limited to the illustrations in the drawings.
[0027] In the following description, when ordinal numbers such as "first" and "second" are attached to a common name, the ordinal numbers are used only to identify the object to which they are attached. They do not limit the object to which they are attached to a specific object, and they do not specify the order, rank, order, hierarchy, priority, or subordination of the object to which they are attached.
[0028] <1. Usage status of measurement support device> FIG. 1 is a diagram showing an example of a usage state of a measurement support device 10. As shown in FIG. A plurality of panels 95 are arranged in an upright state around the periphery of a room 91, thereby forming walls 92-94 of the room 91. The panels 95 are, for example, gypsum board, plywood, or structural panels, and the structural panels referred to here refer to those used in panel construction methods. In the example shown in Fig. 1, the walls 92-94 are not covered with wallpaper, but wallpaper may be attached thereto. Furniture 96 is placed in the room 91 close to the wall 92.
[0029] A user 90 uses the measurement support device 10 to measure the length along the walls 92 to 94 in a room 91. Specifically, when the user aims the measurement support device 10 at the walls 92 to 94 and the furniture 96 and photographs the walls 92 to 94 and the furniture 96 with the measurement support device 10, an image of the walls 92 to 94 and the furniture 96 is displayed on the measurement support device 10. When the user designates an image of one of the walls 92 to 94 in the image, the image of the scale is subjected to trapezoidal distortion processing by projective transformation to match the designated image in the image, and the image of the scale is superimposed on the designated image. The user can visually recognize the image of the scale in the image and measure the length along the designated image in the image in actual size. That is, the user can measure the actual length such as the height, width, and diagonal length of the walls 92 to 94 using the image of the scale in the image. In addition, since the furniture 96 is close to the wall 92, the user can measure the height, width, etc. of the furniture 96 from the image of the scale in the video.
[0030] The objects photographed by the measurement support device 10 are not limited to the walls 92 to 94 and the furniture 96. The length to be measured is not limited to the walls 92 to 94, but may be any length along a flat surface such as an inner wall, an outer wall, a ceiling, a floor, etc.
[0031] <2. Configuration of the measurement support device> FIG. 2 is a block diagram of the measurement support device 10. As shown in FIG.
[0032] The measurement support device 10 is a computer system such as a mobile phone, a smartphone, a tablet computer system, a laptop computer system, or a desktop computer system.
[0033] The measurement support device 10 includes a computer 11 , an input device 12 , a display device 13 , an imaging device 14 , a three-dimensional shape measuring instrument 15 , and a memory device 16 .
[0034] The computer 11 includes a main board, one or more hardware processors, a GPU (Graphics Processing Unit), and a RAM (Random Access Memory). The main board includes a bus, a bus controller, an interface circuit, and the like, and transmits information between the hardware processor, the GPU, the RAM, the input device 12, the display device 13, the imaging device 14, the three-dimensional shape measuring device 15, and the memory device 16. The hardware processor may be, for example, a CPU (Central Processing Unit). The hardware processor performs various types of arithmetic processing. The RAM provides the hardware processor with a storage area or a working area when the hardware processor performs arithmetic processing. The GPU performs processing that can be performed faster than the hardware processor (for example, image processing and matrix calculation processing) under the command of the hardware processor.
[0035] The input device 12 is an input device such as a keyboard, a mouse, a touch panel, a touch pad, a starry skype, a pointing device, a key, and a push button. The input device 12 outputs a signal according to the content of an operation performed by a user on the input device 12 to the computer 11. The computer 11 recognizes the input and command by the user according to the signal transferred from the input device 12.
[0036] The display device 13 may be, for example, a liquid crystal display device or an organic EL display device. The display device 13 displays an image according to a video signal input from the computer 11. A two-dimensional coordinate system for indicating the position of a point on the display screen of the display device 13 by coordinate values is called a screen coordinate system. When the input device 12 is a touch panel, the input device 12 is overlaid on the display device 13.
[0037] The imaging device 14 captures an image of an object, generates a video containing the image of the object, and transfers the video signal to the computer 11. The imaging device 14 has a lens, an imaging element, an image processing circuit, etc. The lens forms an image of the object on the imaging element. The imaging element captures images of the object one after another at a very short period, and outputs the images containing the image of the object one after another to the image processing circuit. The image processing circuit processes the images input one after another from the imaging element, generates a video signal at a predetermined frame rate from these images, and outputs the video signal to the computer 11.
[0038] The imaging device 14 and the three-dimensional shape measuring device 15 are fixed to each other. Therefore, the position of the three-dimensional shape measuring device 15 relative to the imaging device 14 remains unchanged.
[0039] The three-dimensional shape measuring instrument 15 is an optical range sensor, also called LiDAR (Light Detection And Ranging). The three-dimensional shape measuring instrument 15 measures the three-dimensional shape of an object and outputs the three-dimensional shape of the object to the computer 11 as point cloud data.
[0040] Specifically, the three-dimensional shape measuring instrument 15 projects a laser beam toward the imaging range of the imaging device 14 and receives the reflected light of the laser beam incident on the object, thereby detecting the distance from the three-dimensional shape measuring instrument 15 to the measurement target point. The measurement target point refers to the position where the laser beam is incident on the object. The three-dimensional shape measuring instrument 15 also detects the direction from the three-dimensional shape measuring instrument 15 to the measurement target point as a first deflection angle and a second deflection angle around two orthogonal axes by deflecting the laser beam around two orthogonal axes. In other words, the three-dimensional shape measuring instrument 15 detects the position of the measurement target point so that the three-dimensional position of the measurement target point is expressed by polar coordinate values in a polar coordinate system, that is, the distance, the first deflection angle, and the second deflection angle. The three-dimensional shape measuring instrument 15 may convert the three-dimensional position of the measurement target point from the polar coordinate system to a Cartesian coordinate system. Deflection of the laser beam refers to displacing the emission direction of the laser beam around two orthogonal axes. The two orthogonal axes refer to two axes that are orthogonal to each other and pass through the three-dimensional shape measuring instrument 15. The two orthogonal axes may be along a plane that is orthogonal to the optical axis of the imaging device .
[0041] The three-dimensional shape measuring instrument 15 measures the positions of the measurement target points periodically at a very short period while deflecting the laser beam, and outputs the coordinate values of the measurement target points one after another to the computer 11. The data string of the coordinate values of these measurement target points is called point cloud data. A collection of measurement target points is an object, and the point cloud data represents the three-dimensional shape of the object. A coordinate system for expressing the positions of the measurement target points with coordinate values is called a world coordinate system, and the origin of the world coordinate system is set, for example, in the three-dimensional shape measuring instrument 15. One scan of the three-dimensional shape measuring instrument 15 over the entire shooting range of the imaging device 14 is called one cycle.
[0042] The three-dimensional shape measuring instrument 15 may be a so-called stereo camera. A stereo camera uses the parallax of multiple cameras to measure the three-dimensional shape of an object from multiple images captured by these cameras, and outputs the three-dimensional shape of the object as point cloud data to the computer 11. When the three-dimensional shape measuring instrument 15 is a stereo camera, the imaging device 14 may be one of the multiple cameras of the three-dimensional shape measuring instrument 15.
[0043] The memory device 16 stores a general-purpose OS (Operating System), and the general-purpose OS is thereby installed in the measurement support device 10. The general-purpose OS is, for example, Windows (registered trademark), Android (registered trademark), iOS (registered trademark), macOS (registered trademark), Linux (registered trademark), or Unix (registered trademark). When the general-purpose OS is started, the computer 11 executes the general-purpose OS.
[0044] The memory device 16 stores a measurement support program 17. The measurement support program 17 is installed in a general-purpose OS. The measurement support program 17 is an application program that the computer 11 can execute on the general-purpose OS.
[0045] The memory device 16 stores the scale images 21-23. The first and second scale images 21, 22 may be in either raster format or vector format. The third scale image 23 is in vector format. The resolution and size of the scale images 21-23 are set so that when the scale images 21-23 are mapped onto a world coordinate system while maintaining their shapes, the lengths indicated by the scales of these scale images 21-23 are equal to their actual sizes.
[0046] The first scale image 21 is a grid image. A plurality of types of the first scale images 21 are stored in the memory device 16 as shown in the following table.
[0047] [Table 1]
[0048] Fig. 3 is a diagram showing a first scale image 21 with a square grid pattern. As shown in Fig. 3, grid lines 21a extend vertically and horizontally to form a square grid, and each square area 21b surrounded by the grid lines 21a is formed of a transparent background. In the first scale image 21 with a vertical grid pattern, the grid lines extend vertically and parallel to each other. In the first memory image 21 with a horizontal grid pattern, the grid lines extend horizontally and parallel to each other. The vertical direction is also called the vertical direction, and the horizontal direction is also called the horizontal direction.
[0049] As shown in FIG. 4, the second scale image 22 is an image in which a plurality of cross marks 22a to 22c are regularly arranged two-dimensionally. Specifically, a plurality of first cross marks 22a are arranged at regular intervals along two long sides of a rectangle similar to the panel 95, a plurality of second cross marks 22b are arranged at regular intervals along two short sides of the rectangle, and a plurality of third cross marks 22c are arranged at regular intervals along a median line connecting the midpoints of the two short sides of the rectangle. These cross marks 22a to 22c indicate, for example, screw fastening positions of the panel 95. An area 22d of the second scale image 22 other than the cross marks 22a to 22c is composed of a transparent background. A plurality of types of second scale images 22 are stored in the memory device 16. Specifically, second scale images 22 for a standard ceiling panel, a semi-fireproof ceiling panel, and a wall panel are stored in the scale device 16. For example, the intervals and arrangements of the multiple cross marks 22a to 22c differ for each type.
[0050] As shown in Fig. 5, the third scale image 23 is an image of a ruler scale. Specifically, a long line segment 23a extends vertically and a plurality of line segments 23b are arranged vertically at regular intervals along the line segment 23a. The line segments 23b are classified into long and short, and a number 23c indicating the length is added to the long line segment 23b. Because the third scale image 23 is in vector format, the thickness of the line segments 23a and 23b can be changed without degrading the third scale image 23, and display / non-display of the line segments 23a, 23b, and the numbers 23c can be selected.
[0051] A plurality of types of third scale images 23 are stored in the memory device 16 as shown in the following table.
[0052] [Table 2]
[0053] It is preferable that the measurement support device 10 is a portable smartphone or tablet computer system. In this case, the input device 12 is a touch panel, the computer 11, the input device 12, the display device 13, the imaging device 14, the three-dimensional shape measuring device 15, and the memory device 16 are housed in a thin housing, the input device 12 is superimposed on the display device 13, the input device 12 and the display device 13 face the transparent surface on the front side of the housing, and the imaging device 14 and the three-dimensional shape measuring device 15 face the transparent window on the back side and are arranged close to each other. In this case, the user faces the front side of the housing of the measurement support device 10 toward himself and the back side of the housing toward the object, and photographs the object with the measurement support device 10.
[0054] <3. Measurement support program> The flow of the processes that the measurement support program 17 causes the computer 11 to execute will be described below.
[0055] (1) Start the measurement support program First, when the user operates the input device 12 to instruct the computer 11 to start the measurement assistance program 17 , the computer 11 recognizes this and starts executing the measurement assistance program 17 .
[0056] (2) Activating the 3D shape measuring device and imaging device Thereafter, the computer 11 starts the three-dimensional shape measuring device 15. Then, the three-dimensional shape measuring device 15 continuously measures the shape of the object within the field of view of the imaging device 14, and continuously outputs point cloud data to the computer 11.
[0057] The computer 11 starts the imaging device 14 at the same time as starting the three-dimensional shape measuring instrument 15. Then, the imaging device 14 continuously captures images within the field of view and continuously outputs the captured images to the computer 11.
[0058] The computer 11 continues to operate the imaging device 14 and the three-dimensional shape measuring instrument 15 until the user operates the input device 12 to instruct the computer 11 to stop the measurement support program 17 .
[0059] (3) Displaying the Live View screen During the operation of the three-dimensional shape measuring instrument 15 and the imaging device 14, the computer 11 arranges the object in the image, with the image captured by the imaging device 14 as the background and the object as the foreground, and synthesizes them. The computer 11 outputs a signal of the synthesized image to the display device 13, thereby displaying a live view screen on the display device 13. FIG. 6 is a diagram showing an example of the live view screen. As shown in FIG. 6, the image captured by the imaging device 14 is displayed in real time on the live view screen of the display device 13 as the background, and the objects 101 to 108 are displayed on the live view screen of the display device 13 as the foreground, superimposed on the background. The objects 101 to 105 are gathered together and arranged at the bottom right of the live view screen, the objects 106 and 160 are arranged at the center of the screen, the object 107 is arranged to the right of the live view screen, and the object 108 is arranged at the top of the live view screen.
[0060] The object 101 is a GUI (Graphical User Interface) element such as a transition button for instructing the computer 11 to operate in the first mode. The object 102 is a GUI element such as a transition button for instructing the computer 11 to operate in the second mode. Object 103 is a GUI element such as a transition button for instructing computer 11 to operate in the third mode. Object 104 is a GUI element such as a transition button for instructing computer 11 to operate in the fourth mode. The object 105 is a GUI element such as a confirmation button for making the computer 11 recognize the decision to select any one of the objects 101 to 104. Since the objects 101 to 105 are arranged together at the bottom right of the live view screen, it is easy for a user to hold the measurement support device 10 with both hands and operate the input device 12 to select and confirm the objects 101 to 105. For example, if the input device 12 is a touch panel overlaid on the display device 13, it is easy to tap the objects 101 to 105 with the thumb of the right hand.
[0061] Object 106 is a rectangular frame. Object 160 is a circle with a smaller area than object 106. Object 160 is a target for determining a surface aligned with object 160 from the point cloud data as a recognition target for computer 11. Object 106 is a guide for determining the surface to be determined as a recognition target. The process by which computer 11 recognizes the target surface will be described in detail later. Object 107 is a GUI element such as a cancel button for canceling or resetting the recognition of the target surface by computer 11 . The object 108 is text indicating the distance from the target surface recognized by the computer 11 to the three-dimensional shape measuring instrument 15. The process by which the computer 11 calculates the distance will be described in detail later.
[0062] (4) Recognition of target surface and calculation of distance While viewing the live view screen, the user aims the imaging device 14 and the three-dimensional shape measuring instrument 15 at a target surface such as the walls 92 to 94, other walls, a floor, a ceiling, an exterior wall, the surface of a pillar, the surface of a beam, the surface of a door, and the surface of a piece of furniture 5, and aligns the object 160 with the image of the target surface in the live view screen, and then fixes the position and orientation of the imaging device 14 and the three-dimensional shape measuring instrument 15 as much as possible. Then, the computer 11 recognizes the point cloud data of the target surface from the point cloud data input from the three-dimensional shape measuring instrument 15. For example, the computer 11 calculates three-dimensional feature amounts for the point cloud data, and uses the three-dimensional feature amounts to extract and recognize point cloud data of a flat target surface from the point cloud data input from the three-dimensional shape measuring instrument 15.
[0063] Next, the computer 11 calculates the distance from the three-dimensional shape measuring instrument 15 to the target surface based on the point cloud data of the recognized target surface. For example, as follows. The computer 11 calculates the distance from a straight line parallel to the optical axis of the imaging device 14 that passes through the position of the three-dimensional shape measuring instrument 15 in the world coordinate system, i.e., the origin, to each measurement target point of the point cloud data of the target surface. Next, the computer 11 compares the distance to each measurement target point to identify the measurement target point that is closest to the straight line parallel to the optical axis of the imaging device 14 among the point cloud data of the target surface. Next, the computer 11 calculates the distance from the closest measurement target point to the origin, and this distance corresponds to the distance from the three-dimensional shape measuring instrument 15 to the target surface.
[0064] The computer 11 then updates the object 108 on the live view screen according to the calculated distance with text indicating the calculated distance.
[0065] The computer 11 applies a semi-transparent color to the image of the target surface on the live view screen in conjunction with the update of the object 108. Specifically, the computer 11 maps the point cloud data of the target surface from the world coordinate system to the screen coordinate system by central projection. That is, the computer 11 converts the coordinate values of each measurement target point of the point cloud data of the target surface from the world coordinate system to the screen coordinate system. Since the relative position of the three-dimensional shape measurement 15 with respect to the imaging device 14 is invariable, each element of the central projection transformation matrix for converting the coordinate values from the world coordinate system to the screen coordinate system may be preset as a constant in the program 17. The central projection is also called a perspective projection.
[0066] Next, the computer 11 identifies a closed area occupied by the point cloud data of the target surface in the screen coordinate system by edge extraction processing or the like. Next, the computer 11 assigns a color (i.e., RGB value) and a transparency to the closed area. The transparency is also called an alpha value. Next, the computer 11 alpha-blends the closed area to which the color and transparency have been assigned with the image captured by the imaging device 14 on the live view screen. As a result, the semi-transparent or opaque closed area is synthesized with the image of the target surface in the image captured by the imaging device 14, and the image of the target surface is displayed on the live view screen after being synthesized with the color of the closed area. In the example shown in FIG. 1 and FIG. 6, the wall 94 is the target surface, and the image 110 of the wall 94 is synthesized with the color of the closed area, and the pattern shown in FIG. 6 represents the color of the closed area.
[0067] On the live view screen, the object 108 is updated and the image of the target surface is given a semi-transparent color, allowing the user to visually grasp how the computer 11 is recognizing the target surface and calculating the distance.
[0068] Each time the computer 11 inputs one cycle of point cloud data from the three-dimensional shape measuring instrument 15, the computer 11 performs the above-mentioned series of processes. Therefore, even if the image capturing device 14 and the three-dimensional shape measuring instrument 15 are displaced due to camera shake or the like, the display of the object 108 is updated in real time according to the change in the distance from the three-dimensional shape measuring instrument 15 to the target surface, and the shape and position of the closed region are updated in real time according to the change in the relative position of the target surface with respect to the three-dimensional shape measuring instrument 15. After the computer 11 first recognizes the point cloud data of the target surface from one cycle of point cloud data, the computer 11 may perform a tracking process to extract and recognize the point cloud data of the target surface that matches the first recognized target surface from the point cloud data of each subsequent cycle. In this case, even if the image capturing device 14 and the three-dimensional shape measuring instrument 15 are displaced due to camera shake or the like, the target surface is tracked by the computer 11. Of course, even after computer 11 initially recognizes the point cloud data of the target surface from one cycle of point cloud data, computer 11 may not perform a tracking process, and computer 11 may determine as the recognition target the surface that is aligned with object 160 in the center of the live view screen.
[0069] After the recognition of the target surface and the calculation of the distance, the user selects the object 107 by operating the input device 12, and the computer 11 recognizes this. The computer 11 then cancels the above-mentioned alpha blending and updates the object 108 on the live view screen to text indicating "unknown." Furthermore, the computer 11 resets or cancels the recognized target surface and the calculated distance, and starts recognizing the target surface and calculating the distance again.
[0070] Even if the user selects the object 105 by operating the input device 12, the computer 11 ignores the selection of the object 105 until the computer 11 recognizes the target surface as described above. On the other hand, when the user selects one of the objects 101 to 104 by operating the input device 12 and then selects and confirms the object 105, the computer 11 recognizes the fact. When the user selects the object 101, the selection and confirmation of the object 105 causes the computer 11 to start an operation in the first mode. When the user selects the object 102, the selection and confirmation of the object 105 causes the computer 11 to start an operation in the second mode. When the user selects the object 103, the selection and confirmation of the object 105 causes the computer 11 to start an operation in the third mode. When the user selects the object 101, the selection and confirmation of the object 105 causes the computer 11 to start an operation in the fourth mode. The computer 11 continues to recognize the target surface and calculate the distance even during the operations in the first to fourth modes. The first to fourth modes will be described in detail below.
[0071] (5) First mode As described above, when the user selects the object 101 by the input device 12, the computer 11 erases the display of the objects 101 to 108 on the live view screen and cancels the composition of the colored closed area. Then, the computer 11 composes any type of first scale image 21 with the image captured by the imaging device 14, and arranges the object in the composed image with the composed image as the background and the object as the foreground, and composes them. The computer 11 outputs a signal of the image after composition to the display device 13, thereby displaying the live view screen on the display device 13. FIG. 7 is a diagram showing an example of the live view screen. As shown in FIG. 7, the image captured by the imaging device 14 is displayed in real time on the live view screen of the display device 13 as the background, and the objects 201 to 212, 221 to 233, 241 to 243, and 260 are displayed on the live view screen of the display device 13 as the foreground, superimposed on the background. Objects 201-212 are grouped together and placed at the bottom left of the live view screen, objects 221-233 are grouped together and placed at the bottom right of the live view screen, objects 241 and 242 are placed to the right of the live view screen, object 243 is placed at the top of the live view screen, and object 260 is placed in the center of the live view screen.
[0072] A user can view the live view screen and easily measure the position, size, range, etc. of a measurement target such as a scratch or pattern present on the target surface from the grid of the first scale image 21.
[0073] (5-1) Synthesis of the first scale image The first scale image 21 is synthesized with the image captured by the imaging device 14 in the following manner.
[0074] The computer 11 selects one type of the first scale image 21 from among a plurality of types of the first scale image 21. The type to be selected is determined by the user's selection of the objects 201 to 206. However, in the case of the first synthesis of the first scale image 21 after the start of the first mode, the default type is selected.
[0075] The computer 11 copies the first scale image 21 in the two-dimensional local coordinate system of the first scale image 21, and arranges the first scale images 21 closely together in the horizontal direction. However, the number of the copied scale images 21 is determined depending on whether or not the user has selected the objects 207-210, and the number of the copied scale images 21 may be zero. The initial value of the number of the copied scale images 21 is zero, and in the case of the first synthesis of the first scale images 21 after the start of the first mode, the number of the copied scale images 21 is always zero.
[0076] Next, the computer 11 determines the translation amount and rotation angle of the first scale image 21 in the local coordinate system of the first scale image 21 (when the first scale image 21 is duplicated, a plurality of first scale images 21 arranged horizontally). The translation amount refers to the vertical and horizontal movement amount of the first scale image 21 along the first scale image 21. The rotation angle refers to the rotation angle around an axis perpendicular to the first scale image 21. The translation amount and rotation angle are determined depending on whether or not the user selects the objects 221 to 226, and the translation amount and rotation angle may be zero. The initial values of the translation amount and rotation angle are zero, and in the case of the first synthesis of the first scale image 21 after the start of the first mode, the translation amount and rotation angle are always zero.
[0077] Next, the computer 11 determines the depth movement amount of the first scale image 21 (if the first scale image 21 is duplicated, the multiple first scale images 21 arranged in the horizontal direction). The depth movement amount means the movement amount in the axial direction perpendicular to the first scale image 21. The depth movement amount is determined by, for example, whether or not the object 230, 231 is selected by the user. The initial value of the depth movement amount is zero, and in the case of the first synthesis of the first scale image 21 after the start of the first mode, the depth movement amount is always zero.
[0078] Next, the computer 11 calculates a projective transformation matrix based on the point cloud data of the recognized target surface, the coordinate transformation matrix for central projection, the amount of parallel movement, the rotation angle, and the amount of depth movement. The projective transformation refers to a coordinate transformation matrix that generates a trapezoidal distortion in the first scale image 21 when the first scale image 21, which is mapped on the target surface of the world coordinate system while maintaining its shape, is projected onto the screen coordinate system by central projection.
[0079] Next, the computer 11 assigns a transparency to the first scale image 21. The transparency is determined by whether or not the user has selected the objects 230, 231, etc. The initial value of the transparency is zero, which indicates complete transparency, and in the case of the first synthesis of the first scale image 21 after the start of the first mode, the transparency is always zero. Note that the areas of the first scale image 21 other than the grid lines remain transparent.
[0080] Next, the computer 11 maps the first scale image 21 onto the screen coordinate system according to the projective transformation matrix. If the first scale image 21 after mapping protrudes from the outer edge of the closed region occupied by the point cloud data of the target surface in the screen coordinate system, the computer 11 may perform a trimming process to cut out the portion of the first scale image 21 after mapping that protrudes from the outer edge of the closed region and leave the remaining portion.
[0081] Next, the computer 11 alpha-blends the first scale image 21, which has a transparency, into the image captured by the imaging device 14. As a result, the first scale image 21, whose grid lines are semi-transparent or opaque, is synthesized with a trapezoidal distortion into the image of the target surface in the image captured by the imaging device 14, so that the image of the target surface is synthesized with the grid lines of the first scale image 21 and displayed on the live view screen. Note that if the optical axis of the imaging device 14 is perpendicular to the target surface, no trapezoidal distortion occurs in the grid lines in the live view screen. This is because the projection transformation matrix is calculated based on the point cloud data of the target surface.
[0082] Thereafter, the computer 11 performs the above-mentioned series of processes every time the computer 11 inputs one cycle of point cloud data from the three-dimensional shape measuring instrument 15. Therefore, even if the image capturing device 14 and the three-dimensional shape measuring instrument 15 are misaligned due to camera shake or the like, the position, size and distortion of the grid on the live view screen are updated in real time according to the position, size and distortion of the target surface on the live view screen.
[0083] (5-2) Object The computer 11 repeatedly executes the series of processes described in "(5-1) Combining first scale image", so that the objects 201 to 212, 221 to 233, 241 to 243, and 260 are displayed on the live view screen. The objects 201 to 212, 221 to 233, 241 to 243, and 260 will be described in detail below.
[0084] The object 241 is a GUI such as a button for recording the live view screen as a still image or a moving image. Object 242 is a GUI element such as a cancel button for canceling or resetting the recognition of the target surface by computer 11 . The object 243 is text indicating the distance from the target surface recognized by the computer 11 to the three-dimensional shape measuring instrument 15 . The object 260 is a target for determining, from among the point cloud data, a surface that is aligned with the object 260 as a recognition target for the computer 11.
[0085] When the user operates the input device 12 to select the object 241 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining first scale image", the computer 11 captures the live view screen as a still image or a video and records the still image or video in the memory device 16. In the case of recording a video, selecting the object 241 again causes the computer 11 to stop recording the video. In the case of recording a still image, the computer 11 associates point cloud data synchronized with the live view screen of the still image with the still image and records the point cloud data in the memory device 16. In the case of recording a video, the computer 11 associates time-series point cloud data synchronized with the live view screen of the video with the video and records the point cloud data in time series in the memory device 16.
[0086] While the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining the first scale image", the user operates the input device 12 to select the object 242. This causes the computer 11 to reset or cancel the recognized target surface and the calculated distance, and execute again the series of processes described in "(3) Displaying the live view screen" and "(4) Recognizing the target surface and calculating the distance".
[0087] The object 212 is a GUI element such as a window in which the objects 201 to 211 are arranged. The object 211 is a GUI element such as a button for minimizing the object 212 and releasing the minimization.
[0088] When the user operates the input device 12 to select object 211 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining first scale image", the computer 11 erases the display of objects 201-210 and 212, leaving the display of object 211. This minimizes object 212. Thereafter, when the user operates the input device 12 to select object 211, the computer 11 displays objects 201-212 again on the live view screen.
[0089] The object 233 is a GUI element such as a window in which the objects 221 to 232 are arranged. The object 232 is a GUI element such as a button for minimizing the object 233 and releasing the minimization.
[0090] When the user operates the input device 12 to select object 232 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining first scale image", the computer 11 erases the display of objects 221 to 231 and 233, leaving the display of object 232. This minimizes object 233. Thereafter, when the user operates the input device 12 to select object 232, the computer 11 displays objects 221 to 232 again on the live view screen.
[0091] The object 231 is a GUI element such as a button for changing the recognized target surface. When the computer 11 is repeatedly executing the series of processes described in “(5-1) Synthesis of the first scale image”, if the user operates the input device 12 to select object 231, the computer 11 stops tracking the target surface using the tracking process, recognizes and tracks another target surface, and calculates the distance from the other target surface to the three-dimensional shape measuring instrument 15.
[0092] Since the objects 201 to 211 are arranged close together at the bottom left of the live view screen, a user can easily hold the measurement support device 10 with both hands and tap the objects 201 to 211 with the thumb of his left hand. Since the objects 221 to 232 are arranged close together at the bottom right of the live view screen, a user can easily hold the measurement support device 10 with both hands and tap the objects 201 to 211 with the thumb of his right hand. Note that the objects 201 to 212 may be arranged close together at the bottom right of the live view screen, and the objects 221 to 233 may be arranged close together at the bottom left of the live view screen.
[0093] When the computer 11 is repeatedly executing the series of processes as described above, if the user operates the input device 12 to select one of the objects 201-210, 221-229, the display of the grid on the live view screen is changed. This will be explained in detail below.
[0094] (5-2-1) Changing the grid type The object 201 is a GUI element such as a selection button for selecting a first scale image 21 of a type in which the grid line interval is 1 mm. The object 202 is a GUI element such as a selection button for selecting the first scale image 21 of a type in which the grid line interval is 10 mm. The object 203 is a GUI element such as a selection button for selecting the first scale image 21 of a type in which the grid line interval is 20 mm. The object 204 is a GUI element such as a selection button for selecting the first scale image 21 of a type in which the grid pattern is a square lattice. The object 205 is a GUI element such as a selection button for selecting the first scale image 21 of the type whose grid pattern is a vertical grid. The object 206 is a GUI element such as a selection button for selecting the first scale image 21 of a type in which the grid pattern is a square lattice.
[0095] When the user operates the input device 12 to select one of the objects 201-203 and one of the objects 204-206 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining first scale image", the computer 11 recognizes this. In the next and subsequent series of processes, the computer 11 selects a first scale image 21 of a type according to the selection from among the objects 201-203 and the selection from among the objects 204-206. Therefore, a change in the selection from among the objects 201-203 causes a change in the grid pattern on the live view screen, and a change in the selection from among the objects 203-206 causes a change in the spacing of the grid lines on the live view screen.
[0096] (5-2-2) Adding and Removing Grids The object 207 is a GUI element such as a decrease button for decrementing by one the number of first scale images 21 to be duplicated to the left. Object 208 is a GUI element such as an increment button for incrementing by one the number of first scale images 21 to be duplicated to the left. Object 209 is a GUI element such as a decrease button for decrementing by one the number of first scale images 21 to be duplicated to the right. Object 210 is a GUI element such as an increment button for incrementing by one the number of first scale images 21 to be replicated to the right.
[0097] When the user selects any one of the objects 207-210 by operating the input device 12 while the computer 11 is repeatedly executing the series of processes as described above, the computer 11 recognizes this fact. In the next and subsequent series of processes, the computer 11 increases or decreases the number of first scale images 21 to be duplicated in accordance with the selection from among the objects 207-209. Thus, selection of object 207 increases the grid to the left on the live view screen, selection of object 208 decreases the grid from the left on the live view screen, selection of object 209 increases the grid to the right on the live view screen, and selection of object 208 decreases the grid from the right on the live view screen.
[0098] (5-2-3) Grid translation The object 221 is a GUI element such as an up button for incrementing the amount of upward translation of the first scale image 21 by a predetermined amount. The object 221 is also a GUI element for decrementing the amount of downward translation of the first scale image 21 by a predetermined amount. The object 222 is a GUI element such as a downward movement button for incrementing the downward translation amount of the first scale image 21 by a predetermined amount. The object 223 is a GUI element such as a left movement button for incrementing the amount of translation of the first scale image 21 to the left by a predetermined amount. The object 224 is a GUI element such as a right movement button for incrementing the amount of translation of the first scale image 21 to the right by a predetermined amount.
[0099] When the user selects any one of the objects 221-224 by operating the input device 12 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining the first scale image", the computer 11 recognizes this fact. In the next and subsequent series of processes, the computer 11 increases or decreases the amount of parallel movement of the first scale image 21 in accordance with the selection from among the objects 222-224. Thus, selection of the object 221 causes a predetermined amount of movement upward of the grid on the live view screen, selection of the object 222 causes a predetermined amount of movement downward of the grid on the live view screen, selection of the object 223 causes a predetermined amount of movement leftward of the grid on the live view screen, and selection of the object 224 causes a predetermined amount of movement rightward of the grid on the live view screen.
[0100] (5-2-4) Grid rotation The object 225 is a GUI element such as a rotation button for incrementing the counterclockwise rotation angle of the first scale image 21 by 1°. The object 225 is also a GUI element such as a rotation button for decrementing the clockwise rotation angle of the first scale image 21 by 1°. The object 226 is a GUI element such as a rotation button for incrementing the clockwise rotation angle of the first scale image 21 by 1°. The object 227 is a GUI element such as a rotation button for incrementing the counterclockwise rotation angle of the first scale image 21 by 45°. The object 228 is a GUI element such as a rotation button for incrementing the clockwise rotation angle of the first scale image 21 by 45°.
[0101] When the user selects one of the objects 225-227 by operating the input device 12 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining the first scale image", the computer 11 recognizes this fact. In the next and subsequent series of processes, the computer 11 increases or decreases the rotation angle of the first scale image 21 in accordance with the selection from among the objects 225-227. Thus, selection of the object 225 causes a 1° counterclockwise rotation of the grid on the live view screen, selection of the object 226 causes a 1° clockwise rotation of the grid on the live view screen, selection of the object 227 causes a 45° counterclockwise rotation of the grid on the live view screen, and selection of the object 228 causes a 45° clockwise rotation of the grid on the live view screen.
[0102] (5-2-5) Changing grid transparency and moving the grid in the depth direction The object 229 is a GUI element such as a button for incrementing the transparency of the first scale image 21 by a predetermined amount. The object 229 is also a GUI for incrementing the depth movement amount of the first scale image 21 by a predetermined amount. The object 230 is a GUI element such as a button for decrementing the transparency of the first scale image 21 by a predetermined amount. The object 229 is also a GUI for decrementing the depth movement amount of the first scale image 21 by a predetermined amount.
[0103] When the user selects either of the objects 229 and 230 by operating the input device 12 while the computer 11 is repeatedly executing the series of processes described in "(5-1) Combining the first scale image", the computer 11 recognizes the selection. In the next and subsequent series of processes, the computer 11 increases or decreases the transparency of the first scale image 21 and increases or decreases the depth movement amount of the first scale image 21 according to the selection from the objects 229 and 230. Therefore, the selection of the object 229 increases the transparency of the grid lines on the live view screen and moves the first scale image 21 toward the back of the target surface along a direction perpendicular to the target surface in the world coordinate system. The selection of the object 230 decreases the transparency of the grid lines on the live view screen and moves the first scale image 21 toward the front of the target surface along a direction perpendicular to the target surface in the world coordinate system. Increasing the transparency of the grid lines represents the grid lines moving deeper into the target surface on the live view screen, and decreasing the transparency of the grid lines represents the grid lines moving closer to the target surface on the live view screen. Moving the first scale image 21 deeper into the target surface along a direction perpendicular to the target surface in the world coordinate system represents the first scale image 21 shrinking on the live view screen, and moving the first scale image 21 closer to the target surface along a direction perpendicular to the target surface in the world coordinate system represents the first scale image 21 shrinking on the live view screen.
[0104] (6) Second mode As described above, when the user selects the object 102 by the input device 12, the computer 11 erases the display of the objects 101 to 108 on the live view screen and cancels the composition of the colored closed area. Then, the computer 11 composes any type of second scale image 22 with the image captured by the imaging device 14, and arranges the object in the composed image with the composed image as the background and the object as the foreground, and composes them. The computer 11 outputs a signal of the image after composition to the display device 13, thereby displaying the live view screen on the display device 13. FIG. 8 is a diagram showing an example of the live view screen. As shown in FIG. 8, the image captured by the imaging device 14 is displayed in real time on the live view screen of the display device 13 as the background, and the objects 301 to 303, 307 to 312, 321 to 333, 341 to 343, and 360 are displayed on the live view screen of the display device 13 as the foreground, superimposed on the background. The objects 301-303 and 307-312 are arranged together at the bottom left of the live view screen, the objects 321-333 are arranged together at the bottom right of the live view screen, the objects 341 and 342 are arranged to the right of the live view screen, the object 343 is arranged at the top of the live view screen, and the object 360 is arranged in the center of the live view screen. Note that the objects 301-303 and 307-312 may be arranged together at the bottom right of the live view screen, and the objects 321-333 may be arranged together at the bottom left of the live view screen.
[0105] A user can view the live view screen and easily measure the number, positions and accuracy of measurement objects such as screws present on the target surface from the cross marks of the second scale image 22. For example, by aligning the second scale image 22 in the live view screen with the image of the panel 95, the user can easily measure the number, positions and accuracy of the screws on the panel 95 from the cross marks.
[0106] The process by which the computer 11 combines the second scale image 22 with the image captured by the imaging device 14 is similar to the process by which the computer 11 combines the first scale image 21 with the image captured by the imaging device 14 .
[0107] Object 301 is a GUI element such as a selection button for selecting a second scale image 22 of a certain type for a wall panel. Object 302 is a GUI element such as a selection button for selecting a second scale image 22 of the type that is for a standard ceiling panel. Object 303 is a GUI element such as a selection button for selecting a second scale image 22 of a type that is for a semi-fire resistant ceiling panel.
[0108] When the user selects one of the objects 301-303 by operating the input device 12, the computer 11 recognizes this. The computer 11 selects the second scale image 22 of a type that corresponds to the selection from among the objects 301-303. Therefore, a change in the selection from among the objects 301-303 brings about a change in the cross mark on the live view screen.
[0109] Objects 307-310, 321-332, 341-343, and 360 in the second mode correspond to objects 207-210, 221-232, 241-243, and 260 in the first mode, respectively. The processes executed by computer 11 when the user selects objects 307-310, 321-332, 341, and 342 are similar to the processes executed by computer 11 when the user selects objects 207-210, 221-232, 241, and 242.
[0110] (6) Third mode As described above, when the user selects the object 103 by the input device 12, the computer 11 erases the display of the objects 101 to 108 on the live view screen and cancels the composition of the colored closed area. Then, the computer 11 composes any type of third scale image 3 with the image captured by the imaging device 14, and arranges the object in the composed image with the composed image as the background and the object as the foreground, and composes them. The computer 11 outputs a signal of the image after composition to the display device 13, thereby displaying the live view screen on the display device 13. FIG. 9 is a diagram showing an example of the live view screen. As shown in FIG. 9, the image captured by the imaging device 14 is displayed in real time on the live view screen of the display device 13 as the background, and the objects 401 to 405, 407 to 418, 421 to 433, 441 to 444, and 460 are displayed on the live view screen of the display device 13 as the foreground, superimposed on the background. Objects 401-405 and 407-418 are placed together at the bottom left of the live view screen, objects 421-433 are placed together at the bottom right of the live view screen, objects 441 and 442 are placed to the right of the live view screen, object 433 is placed at the top of the live view screen, and object 460 is placed in the center of the live view screen. Note that objects 401-405 and 407-418 may be placed together at the bottom right of the live view screen, and objects 421-433 may be placed together at the bottom left of the live view screen.
[0111] The user can view the live view screen and easily measure the length along the target surface from the number of line segments 23b of the third scale image 23. For example, the user can easily measure the height and width of the panel 95 and the height, width, length, etc. of an object brought close to the panel 95.
[0112] The process in which the computer 11 synthesizes the third scale image 23 with the image captured by the imaging device 14 is similar to the process in which the computer 11 synthesizes the first scale image 21 with the image captured by the imaging device 14. However, in the first mode, the computer 11 copies and arranges the multiple first scale images 21 in a common local coordinate system, whereas in the third mode, the computer 11 copies and arranges the multiple third scale images 23 in individual local coordinate systems. Therefore, the user can select, deselect, and edit each of the third scale images 23 by operating the input device 12 on the live view screen. The colors of the line segments 23a, 23b and the numbers 23c of the selected third scale image 23 are different from the colors of the line segments 23a, 23b and the numbers 23c of the unselected third scale images 23.
[0113] The object 307 is a GUI element such as a delete button for deleting the selected third scale image 23 . The object 308 is a GUI element such as an add button for additionally displaying the third scale image 23 .
[0114] The user operates the input device 12 to select any one of the third scale images 23 on the live view screen, and then selects the object 307. Then, the computer 11 deletes the selected third scale image 23, so that the selected third scale image 23 disappears from the live view screen.
[0115] The user selects the object 308 by operating the input device 12. Then, the computer 11 duplicates the third scale image 23 and adds it to the live view screen, so that the third scale image 23 is additionally displayed on the live view screen.
[0116] The objects 401 to 405 are GUI elements such as buttons for changing the type of the third scale image 23 selected on the live view screen. The object 401 is a GUI element such as a selection button for selecting the third scale image 23 of the type in which the interval between the long line segments 23b is 10 cm. The object 402 is a GUI element such as a selection button for selecting the third scale image 23 of the type in which the interval between the long line segments 23b is 15 cm. The object 403 is a GUI element such as a selection button for selecting the third scale image 23 of the type in which the interval between the long line segments 23b is 20 cm. The object 404 is a GUI element such as a selection button for selecting the third scale image 23 of the type in which the alignment of the line segment 23b with respect to the line segment 23a is to the left. The object 405 is a GUI element such as a selection button for selecting the third scale image 23 of the type in which the line segment 23b is positioned to the right with respect to the line segment 23a.
[0117] When the user selects one of the objects 401-403 by operating the input device 12 while the computer 11 is repeatedly executing a series of processes, the computer 11 recognizes this. Then, in the next and subsequent series of processes, the computer 11 selects the third scale image 23 of a type that corresponds to the selection from the objects 401-403. Therefore, a change in the selection from the objects 401-403 causes a change in the interval of the long line segments 23b labeled with the numbers 23c. When the user selects one of the objects 404, 405 by operating the input device 12 while the computer 11 is repeatedly executing a series of processes, the computer 11 recognizes this. Then, in the next and subsequent series of processes, the computer 11 selects the third scale image 23 of a type that corresponds to the selection from the objects 404, 405. Therefore, a change in the selection from the objects 404, 405 causes a change in the arrangement of the line segment 23b relative to the line segment 23a. In other words, a change in the selection from the objects 404, 405 causes the selected third scale image 23 to be displayed in a left-right inverted manner on the live view screen.
[0118] The objects 409, 410, 412, and 413 are GUI elements such as change buttons for changing the display area of the third scale image 23 selected on the live view screen along the line segment 23a. The object 409 is a GUI element such as a reduction button for narrowing the display area of the third scale image 23 selected on the live view screen by 1 cm along the line segment 23a. The object 409 is also a GUI element for expanding the non-display area of the third scale image 23 selected on the live view screen by 1 cm along the line segment 23a. An object 410 is a GUI element such as an increase button for expanding the display area of the third scale image 23 selected on the live view screen by 1 cm along the line segment 23a. The object 412 is a GUI such as a reduction button for narrowing the display area of the third scale image 23 selected on the live view screen by 50 cm along the line segment 23a. The object 413 is a GUI element such as an increase button for expanding the display area of the third scale image 23 selected on the live view screen by 50 cm along the line segment 23a.
[0119] When the user selects any one of the objects 409, 410, 412, and 413 by operating the input device 12 while the computer 11 is repeatedly executing a series of processes, the computer 11 recognizes this fact. In the next and subsequent series of processes, the computer 11 expands or reduces the display area of the selected third scale image 23 on the live view screen along the line segment 23a in accordance with the selection from among the objects 409, 410, 412, and 413. The selection of the object 409 or 412 causes the selected third scale image 23 on the live view screen to disappear from its edge. The selection of the object 409 or 412 causes the selected third scale image 23 on the live view screen to appear from its edge. Here, "towards the edge" refers to "the direction of the larger number 23c."
[0120] The object 414 is a GUI element such as a switch button for switching between displaying and hiding the number 23c of the third scale image 23 selected on the live view screen. When the user switches between selected and unselected states of the object 414 by operating the input device 12 while the computer 11 is repeatedly executing a series of processes, the computer 11 recognizes this. In the next and subsequent series of processes, the computer 11 switches between displaying and hiding the number 23c of the selected third scale image 23 on the live view screen depending on whether the object 414 is selected or unselected. Therefore, selection of the object 414 causes a display that shows the number 23c of the selected third scale image 23 on the live view screen, and deselection of the object 414 causes a display that erases the number 23c of the selected third scale image 23 on the live view screen.
[0121] The objects 415 to 417 are GUI elements such as change buttons for changing the thickness of the line segments 23a and 23b of the third scale image 23 selected on the live view screen. The object 415 is a GUI element for thinning the line segments 23a and 23b of the third scale image 23 selected on the live view screen. The object 415 is a GUI element for changing the line segments 23a and 23b of the third scale image 23 selected on the live view screen into medium-thick lines. The object 415 is a GUI element for thickening the line segments 23a and 23b of the third scale image 23 selected on the live view screen.
[0122] When the user selects one of the objects 415-417 by operating the input device 12 while the computer 11 is repeatedly executing a series of processes, the computer 11 recognizes this. In the next and subsequent series of processes, the computer 11 sets the thickness of the line segments 23a, 23b of the third scale image 23 selected on the live view screen in accordance with the selection from among the objects 415-417. Therefore, the selection of the objects 415-417 causes a display that changes the thickness of the line segments 23a, 23b of the third scale image 23 selected on the live view screen.
[0123] The object 418 is a GUI element such as a switch button for switching between displaying and hiding the identification numbers of the third scale images 23 on the live view screen. When the user switches between selected and non-selected object 418 by operating input device 12 while computer 11 is repeatedly executing a series of processes, computer 11 recognizes this. In the next and subsequent series of processes, computer 11 switches between displaying and hiding the identification number of the selected third scale image 23 on the live view screen depending on whether object 418 is selected or non-selected. Therefore, selection of object 418 causes a display in which an identification number is added near each third scale image 23 on the live view screen, and non-selection of object 418 causes a display in which the identification number is erased.
[0124] The objects 421-432, 441-443, and 460 in the fourth mode correspond to the objects 221-232, 241-243, and 260 in the first mode, respectively. The process executed by the computer 11 when the user selects the objects 421-432, 441, and 442 is the same as the process executed by the computer 11 when the user selects the objects 221-232, 241, and 242. However, in the first mode, when a plurality of first scale images 21 are displayed on the live view screen, the computer 11 executes the process according to the selection of the objects 221-232 on all the first scale images 21, whereas in the first mode, when a plurality of third scale images 23 are displayed on the live view screen, the computer 11 executes the process according to the selection of the objects 421-432 on the selected third scale image 23.
[0125] (6) Fourth mode As in the third mode, in the fourth mode, the third scale image 23 is displayed on the live view screen, and the processing of the computer 11 in the fourth mode is the same as the processing of the computer 11 in the third mode. However, while the initial value of the rotation angle of the third scale image 23 is zero in the third mode, the initial value of the rotation angle of the third scale image 23 is 90° or minus 90° in the fourth mode. Therefore, while the line segments 23b of the third scale image 23 that is displayed first are arranged in the vertical direction of the live view screen in the third mode, the line segments 23b of the third scale image 23 that is displayed first are arranged in the horizontal direction of the live view screen in the fourth mode.
[0126] <4. Beneficial Effects> The object surface in the image captured by the imaging device 14 is distorted from its actual shape due to perspective. In the first mode, as shown in FIG. 7, the scale image 21 is projected onto the two-dimensional screen coordinate system of the image captured by the imaging device 14 based on the point cloud data of the object surface, so that the first scale image 21 is also distorted according to the distortion of the object surface. Therefore, the position, size, range, etc. of the measurement object such as a scratch or pattern present on the object surface can be easily and accurately measured from the grid of the first scale image 21. In the second mode, the second scale image 22 is distorted according to the distortion of the object surface, so that the number, position, and accuracy of the measurement object such as a screw present on the object surface can be easily and accurately measured from the cross marks of the second scale image 22. In the third and fourth modes, the third scale image 23 is distorted according to the distortion of the object surface, so that the length along the object surface can be easily and accurately measured from the number of line segments 23b of the third scale image 23, etc.
[0127] By selecting the objects 221-224, 321-324, 421-424, the scale images 21, 22, 23 on the live view screen are translated along the object surface in accordance with the distortion of the object surface, so that an object at any position on the object surface can be surveyed.
[0128] By selecting the objects 225-228, 325-328, 425-428, the scale images 21, 22, 23 on the live view screen are aligned with the object surface and rotated around an axis perpendicular to the object surface, so that the object can be surveyed in any posture or direction on the object surface.
[0129] By selecting the objects 229, 230, 329, 330, 429, and 430, the measurement target can be measured in accordance with the positional deviation of the measurement target in the direction perpendicular to the target surface in the world coordinate system.
[0130] By selecting the object 229, 230, 329, 330, 429, 430, it is possible to express that the scale image 21, 22, 23) on the live view screen is in front of or behind the target surface.
[0131] By selecting the objects 201 to 203, the interval between the grid lines of the scale image 21 on the live view screen can be adjusted, and therefore the unit of measurement of length can be adjusted. Similarly, by selecting the objects 401 to 403, the unit of measurement of length can be adjusted.
[0132] By selecting object 204, the vertical and horizontal lengths of the live view screen can be measured. By selecting object 205, the horizontal length of the live view screen can be measured. By selecting object 206, the vertical length of the live view screen can be measured.
[0133] By selecting the objects 409, 410, 412, and 413, the display area of the third scale image 23 on the live view screen can be adjusted according to the length of the measurement target.
[0134] By selecting the objects 415 to 417, the thickness of the line segments 23a and 23b of the third scale image 23 on the live view screen can be adjusted.
[0135] By selecting the objects 404 and 405, the third scale image 23 can be inverted in accordance with the position and orientation of the measurement target on the live view screen.
[0136] The computer 11 of the measurement support device 10 is composed of a portable computer such as a smartphone, and portable computers have made remarkable progress in recent years, allowing them to be used for long periods of time and consuming little power. Therefore, surveying using the measurement support device 10 contributes to the realization of an energy-saving society, the reduction of carbon dioxide emissions, the promotion of carbon neutrality, the realization of a decarbonized society, and the promotion of the Sustainable Development Goals (SDGs). [Explanation of symbols]
[0137] 10. Measurement Support Device 11. Computers 12 Input Devices 13 Display Devices 14 Imaging device 15 3D shape measuring device 17 Measurement Support Program 21 First scale image 21a Grid Lines 21b square area 22 Second scale image 22a, 22b, 22c Cross mark 23 Third scale image 23a First line 23b Second line segment 23c numbers
Claims
1. A computer to which an image captured by an imaging device and point cloud data consisting of three-dimensional coordinate values of a plurality of measurement target points measured by a three-dimensional shape measuring instrument are inputted, A recognition process for extracting and recognizing point cloud data of a target surface from the point cloud data input from the three-dimensional shape measuring device; and executing a synthesis process in which the scale image is projectively transformed onto a two-dimensional screen coordinate system of an image captured by the imaging device based on point cloud data of the target surface recognized by the recognition process, and the image captured by the imaging device is used as a background and the scale image is used as a foreground to synthesize them. A program characterized by:
2. The program according to claim 1, The scale image is an image of a grid pattern, an image in which a plurality of cross marks are regularly arranged two-dimensionally, or an image in which a plurality of second line segments shorter than a first line segment are arranged at regular intervals along the first line segment. A program characterized by:
3. 3. The program according to claim 1 or 2, The computer includes: A process of displaying in real time a live view screen in which the scale image is synthesized with the video by the synthesis process. Run A program characterized by:
4. 3. The program according to claim 1 or 2, The target surface is flat. A program characterized by:
5. 3. The program according to claim 1 or 2, The computer includes: determining a translation amount of the scale image in a two-dimensional local coordinate system based on a signal from an input device; Run the command, The synthesis process includes a process of projectively mapping a scale image onto a two-dimensional screen coordinate system of an image captured by the imaging device based on the point cloud data of the target surface and the translation amount. A program characterized by:
6. 3. The program according to claim 1 or 2, The computer includes: Executing a process of determining an amount of translation of the scale image in a two-dimensional local coordinate system based on a signal from an input device; The synthesis process includes a process of projectively mapping a scale image onto a two-dimensional screen coordinate system of an image captured by the imaging device based on the point cloud data of the target surface and the translation amount. A program characterized by:
7. 3. The program according to claim 1 or 2, The computer includes: Executing a process of determining a depth movement amount of the scale image in a direction perpendicular to the scale image based on a signal from an input device; The synthesis process includes a process of projectively mapping a scale image onto a two-dimensional screen coordinate system of an image captured by the imaging device based on the depth movement amount in addition to the point cloud data of the target surface. A program characterized by:
8. 3. The program according to claim 1 or 2, The computer includes: Executing a process for determining a transparency of the scale image based on a signal from an input device; The synthesis process includes a process of synthesizing the scale image with the transparency on the image captured by the imaging device. A program characterized by:
9. The program according to claim 1, the scale image is an image in which a plurality of second line segments shorter than a first line segment are arranged at regular intervals along the first line segment extending straight; The computer includes: Executing a process of determining a display range of the scale image in a direction along the first line segment based on a signal from an input device; The synthesis process includes a process of synthesizing the scale image in the display range with the image captured by the imaging device. A program characterized by:
Citation Information
Patent Citations
Photographing system, and two-dimensional image pick-up device and three-dimensional measuring instrument used for the system
JP2001280933A
Imaging apparatus, control method and program for the same
JP2014155173A
Crane operation support system and operation support method
JP2020132361A
Display system and display method
JP2020197045A
Information processing apparatus, information processing method, and program
JP2021064300A
