Measuring instrument, system, method, and program

Abstract

To provide a measuring system which can contribute to improvement of picking accuracy without damaging visibility even in an environment where deviation between point cloud data and a real space occurs.SOLUTION: A measuring instrument comprises: a point cloud data acquisition unit configured to acquire point cloud data obtained by capturing a measurement object with a three-dimensional sensor; a filtering unit configured to extract nearby point cloud data existing in the vicinity of pointer coordinate data designated with a visualizing device by a user, from the point group data; and a measurement control unit configured to perform measurement processing based on selected point coordinate data selected with the visualizing device by the user out of the nearby point cloud data.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a measuring device, a system, a method, and a program.

Background Art

[0002] As a method for calculating the length, area, or volume of an object using a visualization device such as an AR (Augmented Reality) headset or a VR (Virtual Reality) headset, for example, the distance from the visualization device to the object is measured using a depth sensor of the visualization device, a 3D (3 Dimensions) mesh model of the object is generated based on the measured distance, the spatial coordinates of a plurality of points are determined by user input using the generated 3D mesh model, and the length, area, or volume is calculated by the visualization device based on the spatial coordinates of the determined plurality of points (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The following analysis is provided by the inventor of the present application.

[0005] However, since high-precision alignment technology is required to superimpose a 3D mesh model on the real space, it is difficult to precisely superimpose the 3D mesh model on the real space by the method of Patent Document 1. In an environment where there is a deviation between the 3D mesh model and the real space, it is difficult to compare the 3D mesh model with the real space, leading to a deterioration in the accuracy of determining (picking) the spatial coordinates of multiple points during calculation. In particular, when picking is performed by user input using hand tracking (e.g., gesture recognition), eye tracking (e.g., gaze tracking), etc., in addition to the influence of deviation due to the accuracy of the alignment of the above 3D mesh model, there is a possibility of deterioration in picking accuracy due to hand tremors, gaze fluctuations, etc. Further, the visibility may be impaired by the 3D mesh model being displayed at a position deviated from the real space. These possibilities are not limited to 3D mesh models, but also apply to point cloud data, which is 3D data.

[0006] The main problem of the present invention is to provide a measuring device, a system, a method, and a program that can contribute to improving the picking accuracy without impairing the visibility even in an environment where there is a deviation between the point cloud data and the real space.

Means for Solving the Problem

[0007] The measuring device according to the first aspect includes a point cloud data acquisition unit configured to acquire point cloud data obtained by photographing a measurement object with a three-dimensional sensor, a filtering unit configured to extract neighborhood point cloud data existing in the vicinity of pointer coordinate data specified by a user with a visualization device from the point cloud data, and a measurement control unit configured to perform measurement processing based on selection point coordinate data selected by the user with the visualization device among the neighborhood point cloud data.

[0008] The measurement system according to the second perspective includes the measurement device according to the first perspective, the three-dimensional sensor configured to output the point cloud data obtained by photographing the measurement target toward the measurement device, a pointer movable by a tracking operation is displayed in the visual field, the vicinity point cloud data is visualized according to the real space visible from the visual field, the selection point coordinate data for performing the measurement process is selected from the vicinity point cloud data by the tracking operation, and the visualization device configured to visualize the measurement result obtained by the measurement process.

[0009] The measurement method according to the third perspective includes a step in which a measurement device acquires point cloud data obtained by photographing a measurement target with a three-dimensional sensor, a step in which the measurement device extracts vicinity point cloud data existing in the vicinity of a pointer specified by a user with a visualization device from the point cloud data, and a step in which the measurement device performs a measurement process based on the selection point coordinate data selected by the user with the visualization device from the vicinity point cloud data.

[0010] The program according to the fourth perspective causes a measurement device to execute a process of acquiring point cloud data obtained by photographing a measurement target with a three-dimensional sensor, a process of extracting vicinity point cloud data existing in the vicinity of a pointer specified by a user with a visualization device from the point cloud data, and a process of performing a measurement process based on the selection point coordinate data selected by the user with the visualization device from the vicinity point cloud data.

[0011] Note that the program can be recorded on a computer-readable storage medium. The storage medium can be non-transitory, such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. Also, in the present disclosure, it is possible to embody it as a computer program product. The program is input into a computer device through an input device or from the outside via a communication interface, stored in a storage device, drives a processor according to predetermined steps or processes, and can display the processing results including intermediate states step by step via a display device as needed, or can communicate with the outside via a communication interface. A computer device for this purpose typically includes, as an example, a processor, a storage device, an input device, a communication interface, and a display device as needed, which can be connected to each other by a bus.

Advantages of the Invention

[0012] According to the first to fourth viewpoints, it is possible to contribute to improving the picking accuracy without impairing the visibility even in an environment where there is a deviation between the point cloud data and the real space.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0014] Hereinafter, the embodiments will be described with reference to the drawings. In the present application, when reference numerals are attached to the drawings, they are solely for the purpose of assisting understanding and are not intended to be limited to the illustrated embodiments. Further, the following embodiments are merely examples and do not limit the present invention. Also, the connection lines between the blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional ones. The one-way arrow schematically shows the flow of the main signal (data) and does not exclude bidirectionality. Furthermore, in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. shown in the present application disclosure, although not explicitly shown, input ports and output ports exist at the input end and output end of each connection line, respectively. The same applies to the input / output interface. The program is executed via a computer device, and the computer device includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as required. The computer device is configured to be able to communicate with devices inside or outside the device (including computers) via the communication interface, whether wired or wireless.

[0015] [Embodiment 1] The measurement system according to Form 1 will be described with reference to the drawings. FIG. 1 is an image diagram schematically showing an example of the usage mode of the measurement system according to the present disclosure. FIG. 2 is a block diagram schematically showing a first example of the configuration of the measurement system according to the present disclosure. FIG. 3 is an image diagram schematically showing a first example of the display in the field of view of the visualization device in the measurement system according to the present disclosure. FIG. 4 is an image diagram schematically showing a second example of the display in the field of view of the visualization device in the measurement system according to the present disclosure. FIG. 5 is an image diagram schematically showing a third example of the display in the field of view of the visualization device in the measurement system according to the present disclosure.

[0016] The measurement system 1 selectively displays the point cloud data 50 obtained by photographing the measurement object with the three-dimensional sensor 10 on the visualization device 30, measures the metrological characteristics of the object (any of 2a to 2c, 3a to 3c) related to the selected point cloud with the measuring device 20, and visualizes the measurement results on the visualization device 30 (see FIGS. 1 and 2). Examples of metrological characteristics include height, width, depth, length, area, volume, angle, curvature, radius, focus, bisecting point, etc. The measurement system 1 can be used, for example, in measurement work at work sites in the manufacturing and construction industries, and measurement work on high-place facilities in the power and railway industries. The measurement system 1 includes a three-dimensional sensor 10, a measuring device 20, and a visualization device 30. Here, although it is divided into three, namely, the three-dimensional sensor 10, the measuring device 20, and the visualization device 30, an integrated one of the three-dimensional sensor 10 and the measuring device 20 (for example, a smartphone with a three-dimensional sensor), or an integrated one of the three-dimensional sensor 10 and the visualization device 30 (for example, AR glasses with a three-dimensional sensor) may be used.

[0017] The 3D sensor 10 is a sensor that senses and photographs the surfaces of objects 2a to 2c and 3a to 3c to be measured (see FIGS. 1 and 2). The 3D sensor 10 is communicably connected (by wired communication or wireless communication) to the measuring device 20. The 3D sensor 10 generates point cloud data 50 by photographing objects 2a to 2c and 3a to 3c, and outputs the generated point cloud data 50 toward the measuring device 20. The 3D sensor 10 can use, for example, a ToF (Time of Flight) camera, a stereo camera, 3D-LIDAR (Laser Imaging Detection And Ranging), a depth sensor, a distance measuring sensor, a distance camera, etc. The 3D sensor 10 can be operated by the operator 4. Here, the point cloud data 50 is data generated by the 3D sensor 10 and is data drawn as a point cloud (a large number of points with three-dimensional coordinates). The 3D sensor 10 can be changed to sensors of various output formats according to customer requirements. The 3D sensor 10 is installed on-site for the purpose of photographing objects 2a to 2c and 3a to 3c (assuming 2a to 2c are houses and 3a to 3c are utility poles) existing in the real space and their surroundings. The 3D sensor 10 can be installed anywhere as long as objects 2a to 2c and 3a to 3c and their surroundings can be photographed. It can be fixed to a wall or fixed to a tripod installed on the ground. Not limited to installation, it can also be mounted on a moving body such as a drone or a robot to move to the site, and is not limited to these. The point cloud data 50 obtained by photographing the measurement target with the 3D sensor 10 is transmitted to the measuring device 20. The point cloud data 50 can include data related to the installation position (three-dimensional coordinates) and installation angle (azimuth, tilt, twist of the line of sight) of the 3D sensor 10. Regarding the installation position of the 3D sensor 10, for example, it can be specified by using a positioning method such as GNSS (Global Navigation Satellite System), but is not limited to this. Also, regarding the installation angle of the 3D sensor 10, for example, it can be specified by using a measurement method by an IMU (Inertial Measurement Unit) sensor, but is not limited to this.Alternatively, an AR marker (which can also be recognized by a camera) may be attached to a predetermined position and recognized by the camera and software attached to the three-dimensional sensor 10, and the position and angle of the three-dimensional sensor 10 may be specified based on the form (size, shape, etc.) of the recognized AR marker.

[0018] The measuring device 20 is a device that performs measurement processing on a selected object based on the selection point coordinate data 52 and 53 related to an object (any one of 2a to 2c and 3a to 3c in FIG. 1) selected by the user with the visualization device 30 from among the point cloud data 50 obtained by photographing the measurement object with the three-dimensional sensor 10 (see FIGS. 1 and 2). The measuring device 20 is communicably connected to the three-dimensional sensor 10 and the visualization device 30 (wired communication or wireless communication). As the measuring device 20, a device (computer device) having functional units (for example, a processor, a storage device, an input device, a communication interface, and a display device) that constitute a computer can be used. For example, a personal computer, a notebook personal computer, a smartphone, a tablet terminal, etc. can be used. The measuring device 20 can realize a configuration including a point cloud data acquisition unit 21, a filtering unit 22, and a measurement control unit 23 virtually by executing a predetermined program.

[0019] The point cloud data acquisition unit 21 is a functional unit that acquires the point cloud data 50 obtained by photographing the measurement object with the three-dimensional sensor 10 (see FIG. 2). The acquired point cloud data 50 may be not only real-time data from the three-dimensional sensor 10 but also past data from a storage device (not shown) that stores data from the three-dimensional sensor 10. The point cloud data acquisition unit 21 outputs the acquired point cloud data 50 toward the filtering unit 22.

[0020] The filtering unit 22 is a functional unit that extracts a neighboring point cloud existing in the vicinity of a pointer (5 in FIG. 3; pointer coordinate data 40) specified by the user with the visualization device 30 from among the point cloud data 50 (FIG. 2). The filtering unit 22 includes a pointer coordinate acquisition unit 22a, a neighboring point cloud search unit 22b, and a search result output unit 22c.

[0021] The pointer coordinate acquisition unit 22a is a functional unit that acquires pointer coordinate data 40 related to the pointer 5 from the pointer coordinate output unit 32d of the tracking operation unit 32 of the visualization device 30 (see FIG. 2).

[0022] The vicinity point cloud search unit 22b is a functional unit that searches for and extracts a vicinity point cloud existing in the vicinity of the pointer 5 related to the pointer coordinate data 40 from the point cloud data 50 (see FIG. 2). As a method for searching for the vicinity point cloud, for example, as shown in FIG. 3, only the point cloud (vicinity point cloud data 51) of the object 3c existing closest to the pointer 5 (pointer coordinate data 40) can be extracted. When extracting the vicinity point cloud data 51, for example, data related to the installation position and installation angle of the 3D sensor 10 included in the point cloud data 50 and data related to the position and angle of the visualization device 30 included in the pointer coordinate data 40 are used to align the point cloud data 50 and the pointer coordinate data 40, and the vicinity point cloud data 51 near the pointer coordinate data 40 may be extracted. Also, as shown in FIG. 4, only one or a plurality of point clouds (vicinity point cloud data 54, 55) existing within a specific distance centered on the pointer 5 (pointer coordinate data 40) may be extracted. Also, instead of extracting the vicinity point cloud data 51 (54, 55) near the pointer 5, only the point cloud of a specific object (for example, a utility pole) may be extracted. For example, when the user wants to measure a utility pole (specific object), clustering may be performed on the point cloud data 50, and only the point cloud data related to the cluster corresponding to the utility pole existing closest to the pointer 5 (pointer coordinate data 40) may be extracted. Also, when the user wants to measure only a specific house, in an environment where a plurality of houses are concentrated, clustering may be performed on the point cloud data 50, and only the point cloud data of the cluster of the specific house existing closest to the pointer 5 (pointer coordinate data 40) may be extracted. The method for searching for the vicinity point cloud is not limited to these, and other methods may be used.

[0023] The search result output unit 22c is a functional unit that outputs the neighborhood point cloud data (51 in FIG. 3, 54 and 55 in FIG. 4) extracted by the search of the neighborhood point cloud search unit 22b as search results to the point cloud data display unit 33a of the point cloud data control unit 33 of the visualization device 30 (see FIG. 2). The neighborhood point cloud data 51 (54, 55) can include the installation position and installation angle of the 3D sensor 10.

[0024] The measurement control unit 23 is a functional unit that performs measurement processing based on the selected points selected by the user on the visualization device 30 among the neighborhood point cloud data 51 (see FIG. 2). The measurement control unit 23 includes a selected point acquisition unit 23a, a measurement processing unit 23b, and a measurement result output unit 23c.

[0025] The selected point acquisition unit 23a is a functional unit that acquires the selected point coordinate data 52, 53 from the selected point output unit 33c of the point cloud data control unit 33 of the visualization device 30 (see FIG. 2).

[0026] The measurement processing unit 23b is a functional unit that performs a desired measurement process based on the acquired selected point coordinate data 52, 53 (see FIG. 2). In the measurement process, the above-mentioned metrological characteristics can be measured. For example, when measuring the distance between two points, two points can be selected by a picking operation to measure the distance between the two points. Also, when the measurement process cannot be performed only with the selected point coordinate data 52, 53, the point cloud data 50 can be used.

[0027] The measurement result output unit 23c is a functional unit that outputs the measurement result 60 obtained by the measurement process of the measurement processing unit 23b to the measurement result display unit 34 of the visualization device 30 (see FIG. 2). The measurement result 60 stores information on the measurement result. Examples of the stored information include the value of the measurement result, the 3D coordinates of the display position, the font of the text, the color of the text, the size of the text, etc., but other items can be added and are not limited to this.

[0028] The visualization device 30 is a device that visualizes a point group (neighborhood point group data 51 (54, 55)) and measurement results 60 related to an object (any of 2a to 2c, 3a to 3c) in the real space (see FIGS. 1 and 2). The visualization device 30 is worn (attached) so as to cover the eyes of the head of the operator 4. As the visualization device 30, a device having functional parts constituting a computer (for example, a processor, a storage device, an input device (camera, sensor, etc.), a communication interface, and a display device), for example, a glasses-type display device such as smart glasses or AR glasses can be used.

[0029] The visualization device 30 is communicably connected to the measurement device 20 (wired communication or wireless communication). The visualization device 30 can display a pointer (5 in FIG. 3) in the visual field (70 in FIG. 3). The visualization device 30 can detect a tracking operation (hand tracking, eye tracking, etc.) and move the display position of the pointer 5 in the visual field 70. The visualization device 30 can calculate the coordinates of the displayed pointer 5 and output pointer coordinate data 40 related to the calculated coordinates to the measurement device 20. The visualization device 30 can acquire neighborhood point group data (51 in FIG. 3, 54, 55 in FIGS. 4 and 5; a part of the point group data 50) near the pointer coordinate data 40 from the measurement device 20. The visualization device 30 can superimpose and display the acquired neighborhood point group data 51 (54, 55) on the real space visible from the visual field 70. The visualization device 30 can detect a tracking operation, select points of the neighborhood point group data 51 (54, 55), and output selection point coordinate data (52, 53 in FIG. 3) related to the coordinates of the selected points to the measurement device 20. The visualization device 30 can acquire measurement results 60 based on the selection point coordinate data 52, 53 from the measurement device 20 and superimpose and display them on the real space visible from the visual field 70.

[0030] By executing a predetermined program, the visualization device 30 can virtually realize a configuration including a self-position angle specifying unit 31, a tracking operation unit 32, a point group data control unit 33, and a measurement result display unit 34.

[0031] The self-position and angle specifying unit 31 is a functional unit that specifies the position (3D coordinates) and angle (direction, inclination, and twist of the line of sight) of the self (the visualization device 30) (see Fig. 2). As a method for specifying the position of the self, for example, position measurement by GNSS or the like can be used, but it is not limited to this. Further, as a method for specifying the angle of the self, for example, angle measurement by an IMU sensor or the like can be used, but it is not limited to this. Alternatively, an AR marker (which can also be identified by a camera) may be attached at a predetermined position and recognized by the self-position and angle specifying unit 31, and the position and angle of the self may be specified based on the form (size, shape, etc.) of the recognized AR marker.

[0032] The tracking operation unit 32 is a functional unit that detects tracking operations such as hand tracking and eye tracking and moves the display position of the pointer 5 and selects a selection point in the nearby point cloud data 51 (54, 55) (see Fig. 2). The tracking operation unit 32 detects a tracking operation, moves the display position of the pointer 5, and outputs the pointer coordinate data 40 related to the pointer 5 to the pointer coordinate acquisition unit 22a of the filtering unit 22 of the measuring device 20. The tracking operation unit 32 includes a method determination unit 32a, a tracking detection unit 32b, a pointer display unit 32c, and a pointer coordinate output unit 32d.

[0033] The method determination unit 32a is a function for determining the tracking method (see Fig. 2). The method determination unit 32a can select the tracking method by the operation of the user (worker 4). Examples of the tracking method include hand tracking by hand movement, eye tracking by eye movement, etc., but other methods may be included in the options and are not limited to this.

[0034] The tracking detection unit 32b is a functional unit that detects the tracking operation of the user (worker 4) according to the method determined by the method determination unit 32a (see Fig. 2). The tracking detection unit 32b can perform various processes such as moving the display position of the pointer 5 and selecting points in the nearby point cloud data 51 (54, 55) according to the detected tracking operation.

[0035] The pointer display unit 32c is a functional unit that displays the pointer 5 so as to be movable by the tracking operation detected by the tracking detection unit 32b in the visual field 70 (see FIG. 2).

[0036] The pointer coordinate output unit 32d is a functional unit that calculates the coordinates (two-dimensional coordinates) of the pointer 5 displayed in the visual field 70 and outputs pointer coordinate data 40 related to the calculated coordinates of the pointer 5 to the pointer coordinate acquisition unit 22a of the filtering unit 22 of the measuring device 20 (see FIG. 2). The pointer coordinate data 40 can include data related to the position and angle of the self (visualization device 30) specified by the self-position angle specifying unit 31.

[0037] The point cloud data control unit 33 is a functional unit that controls the point cloud data 50 displayed in the visual field 70 (see FIG. 2). The point cloud data control unit 33 acquires the nearby point cloud data 51 (54, 55) from the search result output unit 22c of the filtering unit 22 of the measuring device 20 and controls the position displayed in the visual field 70. The point cloud data control unit 33 outputs the selection point coordinate data 52, 53 related to the selected point to the selection point acquisition unit 23a of the measurement control unit 23 of the measuring device 20. The point cloud data control unit 33 includes a point cloud data display unit 33a, a pointer correction unit 33b, and a selection point output unit 33c.

[0038] The point cloud data display unit 33a is a functional unit that acquires the nearby point cloud data (51 in FIG. 3, 54 and 55 in FIG. 4) from the search result output unit 22c of the filtering unit 22 of the measuring device 20 and displays it according to the real space visible from the visual field 70 (see FIG. 2). When displaying the nearby point cloud data 51 (54, 55), for example, the position and angle of the self (visualization device 30) specified by the self-position angle specifying unit 31 and the installation position and installation angle of the three-dimensional sensor 10 included in the nearby point cloud data 51 (54, 55) may be used to display the nearby point cloud data 51 (54, 55) in accordance with the real space. Further, when displaying the nearby point cloud data 54 and 55, the nearby point cloud data 54 and 55 may be enlarged as shown in FIG. 5 by the user's tracking operation, and may also be reduced or rotated.

[0039] The pointer correction unit 33b is a functional unit that automatically corrects and displays the display position of the pointer 5 to the position of the point closest to the pointer 5 or the position of the point of a specific object closest to the pointer 5 when there is no neighborhood point cloud data 51 (54, 55) at the display position of the pointer 5 in the process of selecting a selection point related to the object in the neighborhood point cloud data 51 (54, 55) by a tracking operation (see Fig. 2). This function is used in situations where it is difficult to accurately select a point related to an object, such as when the pointer 5 moves slightly due to hand tremors in hand tracking or gaze fluctuations in eye tracking, making it difficult to select a point, or when the accuracy of point cloud alignment is poor and the point cloud data 50 is displaced with respect to the real space, making it difficult to select a point. The pointer correction unit 33b may be configured to calculate the deviation between the object in the real space and the displayed point cloud data based on the coordinates of the pointer 5 by the tracking operation and the coordinates of the selected point, and feedback it to the next alignment of the object and the point cloud data in the real space to improve the alignment accuracy. The pointer correction unit 33b may prevent the display position of the pointer 5 from being corrected to the location of the noisy point cloud. Also, the pointer correction unit 33b may remove the point cloud that becomes noise so as not to correct the display position of the pointer 5 to the location of the noisy point cloud.

[0040] The selection point output unit 33c is a functional unit that outputs the selection point coordinate data 52, 53 related to the coordinates of the point selected by the user's tracking operation from the displayed neighborhood point cloud data 51 (54, 55) to the selection point acquisition unit 23a of the measurement control unit 23 of the measurement device 20 (see Fig. 2).

[0041] The measurement result display unit 34 is a functional unit that acquires the measurement result 60 from the measurement result output unit 23c of the measurement control unit 23 of the measurement device 20 and displays it in the visual field 70 (see Fig. 2).

[0042] In the measurement system 1 configured as described above, the worker 4 moves the pointer 5 displayed on the visualization device 30 by a tracking operation, and causes the visualization device 30 to display limited point cloud data (in FIG. 3, the neighborhood point cloud data 51; a part of the point cloud data 50) near the pointer 5. If the displayed neighborhood point cloud data 51 is small, the neighborhood point cloud data 51 can be enlarged and displayed by a tracking operation. Thereby, the selection accuracy of the point cloud can be improved. Further, if the displayed neighborhood point cloud data 51 is large, the neighborhood point cloud data 51 can be reduced and displayed, or only a partial area of the neighborhood point cloud data 51 can be restricted and displayed. Thereby, the visual field is simplified and the visibility can be improved. While looking at the displayed neighborhood point cloud data 51, the worker 4 checks whether there are points to be measured in the neighborhood point cloud data 51 related to the object (3c in FIG. 3) by a tracking operation, and selects (point picking) the selected point coordinate data (52, 53 in FIG. 3). When the selected point coordinate data 52 and 53 are selected, the measurement process is carried out. In the measurement process, for example, by selecting two selected point coordinate data 52 and 53 from the neighborhood point cloud data 51 by a tracking operation, the distance between two points can be measured. The measurement result 60 obtained by the measurement process is displayed on the visualization device 30.

[0043] Next, the operation of the measuring device in the measurement system according to Form 1 will be described with reference to the drawings. FIG. 6 is a flowchart schematically showing an example of the operation of the measuring device in the measurement system according to the present disclosure. For the configuration of the measurement system, refer to FIG. 2.

[0044] First, the point cloud data acquisition unit 21 of the measuring device 20 acquires the point cloud data 50 of the work site obtained by photographing with the three-dimensional sensor 10 (step A1). The point cloud data 50 can include data related to the installation position and installation angle of the three-dimensional sensor 10.

[0045] Next, the pointer coordinate acquisition unit 22a of the filtering unit 22 of the measuring device 20 acquires pointer coordinate data 40 from the pointer coordinate output unit 32d of the tracking operation unit 32 of the visualization device 30 (step A2). The pointer coordinate data 40 is data related to the coordinates of the pointer 5 when the pointer 5 is moved by the user's tracking operation. The pointer coordinate data 40 can include data related to the position and angle of the visualization device 30.

[0046] Next, the nearby point cloud search unit 22b of the filtering unit 22 of the measuring device 20 aligns the point cloud data 50 with the pointer coordinate data 40 using the data related to the installation position and installation angle of the three-dimensional sensor 10 included in the point cloud data 50 and the data related to the position and angle of the visualization device 30 included in the pointer coordinate data 40 (step A3). The alignment here is performed for the purpose of displaying the nearby point cloud data 51 (54, 55) on the visualization device 30 at an appropriate position and angle. In the alignment, when the three-dimensional sensor 10 is fixed and used, only the data related to the installation position and installation angle of the three-dimensional sensor 10 included in the initial point cloud data 50 may be used. When the three-dimensional sensor 10 is used while being moved, the data related to the installation position and installation angle of the three-dimensional sensor 10 included in the latest point cloud data 50 may be used periodically.

[0047] Next, the nearby point cloud search unit 22b of the filtering unit 22 of the measuring device 20 searches for a nearby point cloud existing near the pointer coordinate data 40 from the aligned point cloud data 50 and extracts nearby point cloud data 51 (54, 55) (step A4). A detailed search method will be described later (see FIG. 7).

[0048] Next, the search result output unit 22c of the filtering unit 22 of the measuring device 20 outputs the extracted neighboring point cloud data 51 (54, 55) as search results to the point cloud data display unit 33a of the point cloud data control unit 33 of the visualization device 30 (step A5). As a result, the neighboring point cloud data 51 (54, 55) is displayed on the visualization device 30, and the user determines whether the displayed point cloud is the measurement target. If the displayed neighboring point cloud data 51 (54, 55) is not the measurement target, the pointer coordinate data 40 is output again from the visualization device 30 by the user's tracking operation, and the process returns to step A2 to acquire the pointer coordinate data 40. On the other hand, if the displayed neighboring point cloud data 51 (54, 55) is the measurement target, the selection point coordinate data 52, 53 related to the coordinates of the points selected by the user's tracking operation is output from the neighboring point cloud data 51 (54, 55) toward the measuring device 20.

[0049] Next, the selection point acquisition unit 23a of the measurement control unit 23 of the measuring device 20 acquires the selection point coordinate data 52, 53 from the selection point output unit 33c of the point cloud data control unit 33 of the visualization device 30 (step A6).

[0050] Next, the measurement processing unit 23b of the measurement control unit 23 of the measuring device 20 performs a desired measurement process based on the selection point coordinate data 52, 53 (step A7). During the measurement process, if the measurement cannot be performed only with the selection point coordinate data 52, 53, the point cloud data 50 can be used. For example, in the case of volume measurement, the point cloud data related to the object 3c corresponding to the selection point coordinate data 52, 53 (corresponding to the neighboring point cloud data 51 in FIG. 3; a part of the point cloud data 50) can be used to perform volume measurement by a convex hull, but other measurement methods may also be used.

[0051] Next, the measurement result output unit 23c of the measurement control unit 23 of the measuring device 20 outputs the measurement result 60 obtained by the measurement process of the measurement processing unit 23b to the measurement result display unit 34 of the visualization device 30 (step A8), and then ends. As a result, the measurement result 60 is displayed on the visualization device 30.

[0052] Next, the details of the operation of searching for a nearby point group of the measuring device in the measurement system according to Form 1 (step A4 in FIG. 6) will be described with reference to the drawings. FIG. 7 is a flowchart schematically showing an example of the operation of searching for a nearby point group of the measuring device in the measurement system according to the present disclosure. For the configuration of the measurement system, refer to FIG. 2. Here, as an example of the nearby point group search method, the case of extracting the nearby point group data 51 related to the object 3c closest to the pointer coordinate data 40 related to the pointer 5 will be described.

[0053] After aligning the point group data 50 and the pointer coordinate data 40 in step A3 of FIG. 6, the nearby point group search unit 22b of the filtering unit 22 of the measuring device 20 searches for the point closest to the pointer coordinate data 40 from among the aligned point group data 50 (step B1).

[0054] Next, the nearby point group search unit 22b of the filtering unit 22 of the measuring device 20 performs clustering processing on the point group data 50 (step B2).

[0055] Next, the nearby point group search unit 22b of the filtering unit 22 of the measuring device 20 divides the clustered cluster (point group) into object unit clusters (point groups) (step B3).

[0056] Next, the nearby point group search unit 22b of the filtering unit 22 of the measuring device 20 extracts, as the nearby point group data 51, the cluster (the point group of the object 3c in FIG. 3) including the point closest to the searched pointer coordinate data 40 from among the divided object unit clusters (point groups) (step B4), and then proceeds to step A5 in FIG. 6.

[0057] According to Form 1, from the point cloud data 50 obtained by photographing the measurement target with the three-dimensional sensor 10, the vicinity point cloud data 51 (54, 55) existing in the vicinity of the pointer 5 that can be moved by the tracking operation and is displayed on the visualization device 30 is extracted and displayed on the visualization device 30. Therefore, even in an environment where there is a deviation between the point cloud data 50 and the real space, it can contribute to improving the picking accuracy (selection accuracy of the point cloud) without impairing the visibility.

[0058] Also, according to Form 1, since the area of the point cloud data 50 displayed on the visualization device 30 is restricted, the visual field is simplified and the visibility can be improved.

[0059] Also, according to Form 1, since the amount of the point cloud data 50 displayed on the visualization device 30 is suppressed, the communication data between the measurement device 20 and the visualization device 30 is reduced, and the followability of the vicinity point cloud data 51 (54, 55) with respect to the change in the real space can be improved.

[0060] Also, according to Form 1, when displaying the point cloud data on the visualization device 30 in the tracking operation, even if there is a deviation between the position of the pointer indicated by the tracking operation and the position of the point cloud data, the vicinity point cloud data can be displayed. Therefore, a technique for highly accurate alignment of the point cloud data is not required, and the system can be simplified.

[0061] As a comparative example, since the point cloud data obtained by shooting with a 3D sensor is a set of points with 3D (3-axis) coordinates, various measurement processes can be performed using its values. For example, it is possible to measure the volume of a specific object in the point cloud data or measure the distance between two points in the point cloud data. These measurement processes can be executed by a measurement device such as a personal computer, a smartphone, or a tablet terminal. When executing the measurement process, the user selects the point cloud of the object to be the measurement target while referring to the point cloud data displayed on the measurement device. However, as described above, since the point cloud data is data consisting of a set of points, if the point cloud data is displayed on the measurement device in its original state, it is difficult to understand what data was shot just by looking at the video of the point cloud data. For example, the point cloud data obtained by shooting with a 3D camera such as 3D-LIDAR (Three dimensions - Light Detection and Ranging) cannot reproduce the color tone of an object like the image data shot with an RGB (Red Green Blue) camera, and there is a problem that it is difficult to distinguish an object by color. Also, even in a sparse situation where the number of points in the point cloud data is small in the captured video, there is a problem that it is difficult to distinguish an object from the shape of the point cloud. As a method for solving these problems, for example, a method of combining each video of a 3D camera and an RGB camera to colorize the point cloud data can be considered. However, in this method, the hardware configuration becomes complicated, and it is necessary to synchronize each video of the 3D sensor and the RGB camera, and it is difficult to ensure the visibility of the point cloud data. Therefore, as one of the methods for ensuring the visibility of such point cloud data, the utilization of a visualization device such as AR glasses or smart glasses that can superimpose and display a virtual video on the real space can be mentioned. For example, by projecting the point cloud data obtained by shooting with a 3D sensor onto the visualization device according to the real space seen through the visualization device, the real space and the point cloud data can be seen superimposed, and the comparison between the point cloud data and the real space can be facilitated.However, since high-precision alignment technology is required to superimpose point cloud data onto the real space, it is difficult to precisely superimpose the point cloud data onto the real space. In an environment where there is a deviation between the point cloud data and the real space, it is difficult to compare the point cloud data with the real space, leading to a deterioration in the accuracy of point picking during measurement. In particular, in point picking by tracking operations such as hand tracking and eye tracking, in addition to the deviation caused by the accuracy of the alignment of the above-mentioned point cloud data, a deterioration in picking accuracy due to hand tremors, eye movement, etc. can be considered. Also, there is a problem that the visual field becomes complicated when the point cloud data is displayed at a position deviated from the real space. On the other hand, according to Form 1, among the point cloud data 50 obtained by photographing the measurement object with the three-dimensional sensor 10, the vicinity point cloud data 51 (54, 55) existing in the vicinity of the pointer 5 that can be moved by the tracking operation and is displayed on the visualization device 30 is extracted and displayed on the visualization device 30. Therefore, the picking accuracy is improved and the complication of the visual field is suppressed.

[0062] [Form 2] The measuring device according to Form 2 will be described with reference to the drawings. FIG. 8 is a block diagram schematically showing an example of the configuration of the measuring device according to the present disclosure.

[0063] The measuring device 20 is a device that performs measurement processing of the selected object based on the selected point data related to the object selected by the user on the visualization device 30 from among the point cloud data obtained by photographing the measurement object with the three-dimensional sensor 10. The measuring device 20 includes a point cloud data acquisition unit 21, a filtering unit 22, and a measurement control unit 23.

[0064] The point cloud data acquisition unit 21 is configured to acquire the point cloud data obtained by photographing the measurement object with the three-dimensional sensor 10. The filtering unit 22 is configured to extract the vicinity point cloud data existing in the vicinity of the pointer coordinate data specified by the user on the visualization device 30 from among the point cloud data 50. The measurement control unit 23 is configured to perform measurement processing based on the selected point coordinate data selected by the user on the visualization device 30 among the vicinity point cloud data.

[0065] According to the second form, from the point cloud data obtained by photographing the measurement target with the three-dimensional sensor 10, the nearby point cloud data existing in the vicinity of the pointer coordinate data specified by the visualization device 30 is extracted, and the visualization device 30 selects the selection point coordinate data for measurement processing. Therefore, even in an environment where there is a deviation between the point cloud data and the real space, it can contribute to improving the picking accuracy (selection accuracy of the point cloud) without impairing the visibility.

[0066] Note that the measuring devices according to the first and second forms can be configured by so-called hardware resources (information processing devices, computers), and those having the configuration exemplified in FIG. 9 can be used. For example, the hardware resource 100 includes a processor 101, a memory 102, a network interface 103, etc., which are interconnected by an internal bus 104.

[0067] Note that the configuration shown in FIG. 9 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware not shown (for example, an input / output interface). Alternatively, the number of units such as the processor 101 included in the device is not intended to be limited to the example shown in FIG. 9. For example, a plurality of processors 101 may be included in the hardware resource 100. As the processor 101, for example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), etc. can be used.

[0068] As the memory 102, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. can be used.

[0069] For the network interface 103, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, etc. can be used.

[0070] The functions of the hardware resources 100 are realized by the above-described processing modules. The processing modules are realized, for example, by the processor 101 executing a program stored in the memory 102. Further, the program can be downloaded via a network or updated using a storage medium storing the program. Furthermore, the above processing module may be realized by a semiconductor chip. That is, the functions performed by the above processing module may be realized as long as software is executed in some hardware.

[0071] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto.

[0072] [Supplementary Note 1] A point cloud data acquisition unit configured to acquire point cloud data obtained by photographing a measurement object with a three-dimensional sensor, A filtering unit configured to extract neighborhood point cloud data existing in the vicinity of pointer coordinate data specified by a user with a visualization device from the point cloud data, A measurement control unit configured to perform measurement processing based on selection point coordinate data selected by the user with the visualization device among the neighborhood point cloud data, A measuring device comprising: [Supplementary Note 2] The filtering unit A pointer coordinate acquisition unit configured to acquire the pointer coordinate data from the visualization device, A neighborhood point cloud search unit configured to search for and extract the neighborhood point cloud data existing in the vicinity of the pointer coordinate data from the point cloud data, A search result output unit configured to output the neighborhood point cloud data as a search result to the visualization device Comprising The measuring device according to Appendix 1. [Appendix 3] When extracting the nearby point cloud data, the nearby point cloud search unit uses the data related to the installation position and installation angle of the 3D sensor included in the point cloud data and the data related to the position and angle of the visualization device included in the pointer coordinate data to align the point cloud data with the pointer coordinate data, and is configured to extract the nearby point cloud data near the pointer coordinate data. The measuring device according to Appendix 2. [Appendix 4] The nearby point cloud search unit Extracts only the nearby point cloud data related to the object closest to the pointer coordinate data, or Extracts only one or more of the nearby point cloud data existing within a specific distance centered on the pointer coordinate data, or Extracts only the nearby point cloud data related to a specific object existing near the pointer coordinate data. Is configured as The measuring device according to Appendix 2 or 3. [Appendix 5] When extracting only the nearby point cloud data related to the specific object, the nearby point cloud search unit performs clustering on the point cloud data and is configured to extract only the nearby point cloud data related to the cluster corresponding to the specific object closest to the pointer coordinate data. The measuring device according to Appendix 4. [Appendix 6] The measurement control unit A selection point acquisition unit configured to acquire the selection point coordinate data from the visualization device, A measurement processing unit configured to perform the measurement processing based on the selection point coordinate data, A measurement result output unit configured to output the measurement result obtained by the measurement processing toward the visualization device. Comprising The measuring device according to any one of Supplementary Notes 1 to 5. [Supplementary Note 7] The measurement result includes any one of the three-dimensional coordinates of the display position on the visualization device, the font of the character, the color of the character, and the size of the character. The measuring device according to Supplementary Note 6. [Supplementary Note 8] In the measurement process, measuring the distance between two points related to the selected point coordinate data, or measuring the volume by a convex hull using the point cloud data related to the object corresponding to the selected point coordinate data. The measuring device according to any one of Supplementary Notes 1 to 7. [Supplementary Note 9] The point cloud data is real-time data from the three-dimensional sensor or past data from a storage device that stores the data from the three-dimensional sensor. The measuring device according to any one of Supplementary Notes 1 to 8. [Supplementary Note 10] The measuring device according to any one of Supplementary Notes 1 to 9, the three-dimensional sensor configured to output the point cloud data obtained by photographing the measurement object toward the measuring device, a visualization device configured to display a pointer movable by a tracking operation in the visual field, visualize the nearby point cloud data according to the real space visible from the visual field, select the selected point coordinate data for performing the measurement process from the nearby point cloud data by the tracking operation, and visualize the measurement result obtained by the measurement process. A measurement system comprising the above. [Supplementary Note 11] The visualization device is configured to specify its own position and angle, a tracking operation unit configured to detect the tracking operation, move the display position of the pointer, and output the pointer coordinate data related to the pointer toward the measuring device. A point cloud data control unit configured to acquire the nearby point cloud data from the measurement device and control the position of the nearby point cloud data displayed in the visual field using data related to the position and angle of the self identified by the self-position angle identification unit and data related to the installation position and installation angle of the three-dimensional sensor included in the nearby point cloud data; A measurement result output unit configured to acquire the measurement result from the measurement device and display it in the visual field; Comprising; The tracking operation unit is configured to detect the tracking operation and select the selected point coordinate data among the nearby point cloud data; The point cloud data control unit is configured to output the selected point coordinate data toward the measurement device. The measurement system according to Supplementary Note 10. [Supplementary Note 12] The tracking operation unit A method determination unit configured to determine a tracking method; A tracking detection unit configured to detect the user's tracking operation according to the method determined by the method determination unit; A pointer display unit configured to display the pointer movable by the tracking operation in the visual field; A pointer coordinate output unit configured to calculate the pointer coordinate data related to the pointer displayed in the visual field and output the calculated pointer coordinate data toward the measurement device; The measurement system according to Supplementary Note 11, comprising. [Supplementary Note 13] The point cloud data control unit A point cloud data display unit configured to acquire the nearby point cloud data from the measurement device and display it in accordance with the real space visible from the visual field; In the process of selecting the selected point coordinate data related to the object in the vicinity point cloud data by the tracking operation, a pointer correction unit configured to automatically correct and display the display position of the pointer to the position of the point closest to the pointer or the position of the point of the specific object closest to the pointer. A selection point output unit configured to output the selected point coordinate data selected by the user's tracking operation from the vicinity point cloud data to the measuring device. The measuring system according to any one of Appendices 10 to 12, comprising: [Appendix 14] A step of the measuring device acquiring point cloud data obtained by photographing a measurement object with a three-dimensional sensor. A step of the measuring device extracting vicinity point cloud data existing in the vicinity of a pointer designated by the user with a visualization device from the point cloud data. A step of the measuring device performing measurement processing based on the selected point coordinate data selected by the user with the visualization device from the vicinity point cloud data. A measurement method including: [Appendix 15] A process of acquiring point cloud data obtained by photographing a measurement object with a three-dimensional sensor. A process of extracting vicinity point cloud data existing in the vicinity of a pointer designated by the user with a visualization device from the point cloud data. A process of performing measurement processing based on the selected point coordinate data selected by the user with the visualization device from the vicinity point cloud data. A program for causing a measuring device to execute:

[0073] Note that Appendices 14 and 15 can be expanded as in Appendices 2 to 9.

[0074] Note that the disclosures of the above patent documents are hereby incorporated by reference into this document and can be used as the basis or part of the present invention as necessary. Within the scope of the entire disclosure of the present invention (including the claims and the drawings), further modifications and adjustments of the form or embodiments can be made based on the basic technical idea. Also, within the scope of the entire disclosure of the present invention, various combinations or selections (and non-selections if necessary) of various disclosure elements (including each element of each claim, each element of each form or embodiment, each element of each drawing, etc.) are possible. That is, the present invention naturally includes all the disclosures including the claims and the drawings, as well as various modifications and corrections that could be made by those skilled in the art according to the technical idea. In addition, regarding the numerical values and numerical ranges described in this application, even if not explicitly stated, any intermediate value, lower numerical value, and small range are considered to be described. Furthermore, each disclosure item of the above-cited documents is, as necessary, in accordance with the spirit of the present invention of this application, and is considered to be included (belonging) in the disclosure of the present invention of this application, and can be used in combination with the description items of this document, either in part or in whole, as part of the disclosure of the present invention of this application.

Explanation of Reference Signs

[0075] 1 Measurement system 2a - 2c, 3a - 3c Object 4 Operator 5 Pointer 10 3D sensor 20 Measuring device 21 Point cloud data acquisition unit 22 Filtering unit 22a Pointer coordinate acquisition unit 22b Near - point cloud search unit 22c Search result output unit 23 Measurement control unit 23a Selected point acquisition unit 23b Measurement processing unit 23c Measurement result output unit 30 Visualization device 31 Self - position and angle determination unit 32 Tracking operation unit 32a Method determination unit 32b Tracking Detection Unit 32c Pointer Display Unit 32d Pointer Coordinate Output Unit 33 Point Cloud Data Control Unit 33a Point Cloud Data Display Unit 33b Pointer Correction Unit 33c Selected Point Output Unit 34 Measurement Result Display Unit 40 Pointer Coordinate Data 50 Point Cloud Data 51, 54, 55 Neighboring Point Cloud Data 52, 53 Selected Point Coordinate Data 60 Measurement Result 70 Field of View 100 Hardware Resources 101 Processor 102 Memory 103 Network Interface 104 Internal Bus

Claims

1. A point cloud data acquisition unit configured to acquire point cloud data obtained by photographing a measurement target with a three-dimensional sensor, A filtering unit configured to extract neighborhood point cloud data existing in the vicinity of pointer coordinate data specified by a user using a visualization device from among the point cloud data, A measurement control unit configured to perform a measurement process based on selection point coordinate data selected by the user using the visualization device from among the neighborhood point cloud data, A measurement device comprising the above.

2. The filtering unit, A pointer coordinate acquisition unit configured to acquire the pointer coordinate data from the visualization device, A neighborhood point cloud search unit configured to search for and extract the neighborhood point cloud data existing in the vicinity of the pointer coordinate data from among the point cloud data, A search result output unit configured to output the neighborhood point cloud data as a search result to the visualization device, Comprising, The measurement device according to Claim 1.

3. When extracting the neighborhood point cloud data, the neighborhood point cloud search unit performs alignment between the point cloud data and the pointer coordinate data using data related to the installation position and installation angle of the three-dimensional sensor included in the point cloud data and data related to the position and angle of the visualization device included in the pointer coordinate data, and is configured to extract the neighborhood point cloud data in the vicinity of the pointer coordinate data, The measurement device according to Claim 2.

4. The neighborhood point cloud search unit, Extracts only the neighborhood point cloud data related to the object closest to the pointer coordinate data, or, Extracts only one or a plurality of the neighborhood point cloud data existing within a specific distance centered on the pointer coordinate data, or, Extracts only the neighborhood point cloud data related to a specific object existing in the vicinity of the pointer coordinate data, Is configured as such, The measurement device according to Claim 2.

5. When extracting only the neighborhood point cloud data related to the specific object, the neighborhood point cloud search unit performs clustering on the point cloud data and is configured to extract only the neighborhood point cloud data related to the cluster corresponding to the specific object closest to the pointer coordinate data, The measurement device according to Claim 4.

6. The measurement control unit, A selection point acquisition unit configured to acquire the selection point coordinate data from the visualization device, A measurement processing unit configured to perform the measurement processing based on the selected point coordinate data; A measurement result output unit configured to output the measurement result obtained by the measurement processing to the visualization device; Comprising; The measurement device according to claim 1.

7. The measurement device according to any one of claims 1 to 6, and The three-dimensional sensor configured to output the point cloud data obtained by photographing the measurement object to the measurement device; A visualization device configured to display a pointer movable by a tracking operation in a visual field, visualize the nearby point cloud data in accordance with the real space visible from the visual field, select the selected point coordinate data for performing the measurement processing from among the nearby point cloud data by the tracking operation, and visualize the measurement result obtained by the measurement processing; Comprising a measurement system.

8. The visualization device A self-position and angle specifying unit configured to specify its own position and angle; A tracking operation unit configured to detect the tracking operation, move the display position of the pointer, and output the pointer coordinate data related to the pointer to the measurement device; A point cloud data control unit configured to acquire the nearby point cloud data from the measurement device, and control the position of the nearby point cloud data displayed in the visual field using data related to the position and angle of itself specified by the self-position and angle specifying unit, and data related to the installation position and installation angle of the three-dimensional sensor included in the nearby point cloud data; A measurement result output unit configured to acquire the measurement result from the measurement device and display it in the visual field; Comprising The tracking operation unit is configured to detect the tracking operation and select the selected point coordinate data among the nearby point cloud data; The point cloud data control unit is configured to output the selected point coordinate data to the measurement device; The measurement system according to claim 7.

9. A step in which the measurement device acquires point cloud data obtained by photographing a measurement object with a three-dimensional sensor; A step in which the measurement device extracts nearby point cloud data existing in the vicinity of a pointer designated by a user with a visualization device from among the point cloud data; The step in which the measurement device performs measurement processing based on the selected point coordinate data selected by the user with the visualization device from among the neighborhood point group data; A measurement method including the above.

10. A process of acquiring point group data obtained by photographing a measurement target with a three-dimensional sensor; A process of extracting neighborhood point group data existing in the vicinity of a pointer specified by a user with a visualization device from among the point group data; A process of performing measurement processing based on the selected point coordinate data selected by the user with the visualization device from among the neighborhood point group data; A program for causing a measurement device to execute the above.

Images