Measurement system and measurement program

Abstract

To provide a measurement system and a measurement program capable of easily and accurately making alignment of a measured point group and an ideal point group.SOLUTION: A measurement system includes: an image comparison unit; an aligning unit; a shape difference calculation unit; and a display control unit. The image comparison unit selects a data set from a database on the basis of a result of comparison of similarity between each of ideal images and measured images acquired from the database which stores a data set regarding each of the measurement objects containing ideal images, a point group for alignment, and a point group for shape comparison corresponding to a case in which the ideal measurement object are viewed from a plurality of visual fields. The aligning unit aligns the aligning point group of the selected data set and the measurement point group. The shape difference calculation unit calculates shape difference between the shape comparison point group and the measurement point group of the selected data set on the basis of the result of the aligning. The display control unit visibly displays on the display device on the basis of the calculation result of the shape difference.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments relate to a measurement system and a measurement program. [Background technology]

[0002] One known technology that supports the manufacturing of large structures is to visualize the deviation of the shape of a part being manufactured from its ideal shape by overlaying a 3D image of the part with an ideal shape based on 3D measurement of the part. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6673504 Summary of the Invention [Problem to be solved by the invention]

[0004] In the visualization technology described above, it is required to be able to align the measured point cloud with the ideal point cloud with high accuracy. Furthermore, in order to align the measured point cloud with high accuracy in a short time, it is desirable to obtain a measured point cloud similar to the ideal point cloud, but it is generally difficult to prepare a measured point cloud similar to the ideal point cloud.

[0005] The embodiments provide a measurement system and a measurement program that can easily and accurately align a measured point cloud with an ideal point cloud. [Means for solving the problem]

[0006] A measurement system according to one embodiment includes an image comparison unit, a registration unit, a shape deviation calculation unit, and a display control unit. The image comparison unit selects a data set from the database based on a comparison result of similarity between each ideal image, which is an image of an ideal measurement object corresponding to a case where an ideal measurement object having an ideal shape for each measurement object is viewed from a plurality of fields of view, a registration point cloud for registration with a measurement point cloud, which is a point cloud obtained by measuring the measurement object, and a shape comparison point cloud for shape comparison with the measurement point cloud, which are acquired from a database storing data sets for each measurement object. The registration unit aligns the registration point cloud and the measurement point cloud included in the data set selected by the image comparison unit. The shape deviation calculation unit calculates a shape deviation between the shape comparison point cloud and the measurement point cloud included in the data set selected by the image comparison unit based on the registration result of the registration unit. The display control unit visualizes and displays on a display device based on the calculation result of the shape deviation. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing a configuration of an example of a measurement system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the measurement system. [Diagram 3] FIG. 3 is a flowchart showing a shape deviation visualization process as an operation of the measurement system according to the first embodiment. [Figure 4] FIG. 4 is a conceptual diagram of comparison of depth images. [Diagram 5] FIG. 5 is a conceptual diagram of alignment. [Figure 6] FIG. 6 is a conceptual diagram of the calculation of the shape deviation. [Figure 7] FIG. 7 is a block diagram showing a configuration of an example of a measurement system according to a modified example of the first embodiment. As shown in FIG. [Figure 8]FIG. 8 is a flowchart showing a shape deviation visualization process as an operation of the measurement system in the modified example of the first embodiment. [Figure 9] FIG. 9 is a conceptual diagram of point cloud comparison. [Figure 10] FIG. 10 is a conceptual diagram of a data set stored in the shape DB of the second embodiment. [Figure 11] FIG. 11 is a flowchart showing the operation of the measurement system in the second embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a guide display. [Figure 13] FIG. 13 is a flowchart showing the operation of the measurement system in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment will be described with reference to the drawings.

[0009] (First embodiment) First, a first embodiment will be described. Fig. 1 is a block diagram showing a configuration of an example of a measurement system according to the first embodiment. The measurement system 1 can be used to measure the three-dimensional shape of a measurement target.

[0010] The measurement system 1 in the embodiment compares the shape deviation between the three-dimensional shape of the measurement object O measured by the camera 2 and the three-dimensional shape of an ideal measurement object that has an ideal three-dimensional shape for the measurement object O, which is prepared in advance, and presents the shape deviation to a user. The measurement object is not limited to this, but may be, for example, a part to be welded to a large structure. The measurement object does not need to be of one type, and may be of multiple types.

[0011] As shown in FIG. 1, the measurement system 1 of the first embodiment has a shape database (DB) 11, an image comparison unit 12, a registration unit 13, a shape deviation calculation unit 14, and a display control unit 15. The measurement system 1 is configured to be able to communicate with a camera 2. The communication between the measurement system 1 and the camera 2 may be wireless or wired. The measurement system 1 is also configured to be able to communicate with a display device 3. The communication between the measurement system 1 and the display device 3 may be wireless or wired.

[0012] Camera 2 is a camera configured to measure information related to a point cloud of a measurement target. Camera 2 is, for example, an RGB-D camera. The RGB-D camera is a camera configured to be able to measure an RGB-D image. The RGB-D image includes a depth image and a color image (RGB color image). The depth image is a two-dimensional image having the depth of each point of the measurement target O as a pixel value. The color image is a two-dimensional image having the RGB value of each point of the measurement target O as a pixel value.

[0013] The display device 3 is a display device such as a liquid crystal display or an organic EL display. The display device 3 displays various images based on the data transferred from the measurement system 1.

[0014] The shape DB 11 is a database that stores a data set for an ideal measurement target having an ideal three-dimensional shape for each measurement target. In the first embodiment, the data set includes an image 111, a point cloud for alignment 113, and a point cloud for shape comparison 114. Here, the image 111 includes a plurality of images corresponding to the ideal measurement target when viewed from a plurality of fields of view. Similarly, the point cloud for alignment 113 and the point cloud for shape comparison 114 each include a plurality of point clouds corresponding to the ideal measurement target when viewed from a plurality of fields of view. The image 111 corresponding to the same ideal measurement target when viewed from the same field of view, the point cloud for alignment 113, and the point cloud for shape comparison 114 constitute one data set.

[0015] The image 111 is an image of each ideal measurement target. The image 111 includes one or both of a depth image and a color image of the ideal measurement target. Hereinafter, the depth image of the ideal measurement target may be referred to as an ideal depth image, and the color image of the ideal measurement target may be referred to as an ideal color image.

[0016] The alignment point cloud 113 is a point cloud of an ideal measurement target for alignment with the point cloud of the measurement target O obtained from a depth image measured by the camera 2. The alignment point cloud 113 is a point cloud obtained by excluding a point cloud corresponding to a shape comparison portion with the measurement target O from the shape comparison point cloud 114.

[0017] The shape comparison portion is a portion where a shape deviation may occur between the point cloud of the measurement object O and the point cloud of the ideal measurement object. For example, in the case of a part to be welded to a large structure, a weld may generate excess weld metal due to welding. The excess weld metal is a portion where an excess weld metal is deposited in excess of the dimensions of the weld metal defined as an ideal shape. The portion where such excess weld metal may occur corresponds to the shape comparison portion. The difference in shape between the point cloud of the measurement object O and the point cloud of the ideal measurement object at the shape comparison portion may be an error factor in the alignment of the point clouds for comparing the shape deviation. In order to eliminate such an error factor, alignment is performed using the alignment point cloud 113 from which the shape comparison portion has been removed in advance. Here, since the alignment of the point clouds can be performed with high accuracy even between point clouds with a lower density than the comparison of the shape deviation, the alignment point cloud 113 may be a point cloud obtained by downsampling the shape comparison point cloud 114 and further removing the point cloud corresponding to the shape comparison portion.

[0018] The shape comparison point cloud 114 is a point cloud of an ideal measurement object for comparing shape deviation with the point cloud of the measurement object O obtained from a depth image measured by the camera 2. It is desirable that the shape comparison point cloud 114 is a point cloud with the same density as the point cloud of the measurement object O. In addition, as described above, the shape comparison point cloud 114 also includes a point cloud corresponding to a shape comparison portion.

[0019] The image 111, the alignment point cloud 113, and the shape comparison point cloud 114 can be generated, for example, from 3D CAD (Computer Aided Design) data of an ideal measurement target. As an example, the image 111 and the shape comparison point cloud 114 are obtained by disposing a virtual RGB-D camera at a predetermined position based on a 3D CAD model of the ideal measurement target, and generating an image and a point cloud to be acquired by the virtual RGB-D camera from each position from the 3D CAD data while changing the position of the virtual RGB-D camera by a certain width in the horizontal and vertical directions based on this position. Furthermore, the alignment point cloud 113 is obtained by removing the shape comparison part from the shape comparison point cloud 114. When position information indicating that the shape comparison part is a shape comparison part such as a welded part is embedded in the 3D CAD data, the shape comparison part is identified based on the position information. Alternatively, the shape comparison part may be identified by being designated by an operator who creates the data set.

[0020] Here, the image 111, the alignment point cloud 113, and the shape comparison point cloud 114 may be obtained by actually photographing an actual ideal measurement target using an RGB-D camera.

[0021] Moreover, the shape DB 11 may be provided outside the measurement system 1. In this case, the measurement system 1 acquires information from the shape DB 11 as necessary.

[0022] The image comparison unit 12 compares the similarity between a measurement image, which is an image of the measurement object O obtained by the camera 2, and an image 111 stored in the shape DB 11, and selects a data set based on the result of the comparison of the similarity. The similarity may be, for example, a result of a correlation calculation between images. The image comparison unit 12 may compare the similarity in the depth image, may compare the similarity in the color image, or may compare the similarity in both. Furthermore, the image comparison unit 12 may generate color histograms for each of the measurement image and the color image, and characterize dissimilar colors by comparing the color histograms.

[0023] The alignment unit 13 aligns the alignment points 113 included in the data set selected by the image comparison unit 12 with the measurement point cloud, which is a point cloud of the measurement target O obtained by the camera 2. The alignment unit 13 performs alignment using a matching method such as the ICP (Iterative Closest Point) method or the CPD (Coherent Point Drift) method. Furthermore, when a dissimilar color is identified by the image comparison unit 12, the alignment unit 13 can filter the point cloud of that color from the measurement point cloud and then perform alignment.

[0024] The shape deviation calculation unit 14 calculates the shape deviation between the shape comparison point cloud 114 and the measurement point cloud included in the data set selected by the image comparison unit 12. The shape deviation calculation unit 14 calculates the shape deviation from the difference between points at the same position between the measurement point cloud aligned based on the alignment result by the alignment unit 13 and the shape comparison point cloud 142. The shape deviation can be, for example, the difference between values ​​at the same position between the shape comparison point cloud 114 and the measurement point cloud.

[0025] The display control unit 15 visualizes the comparison result by the shape deviation calculation unit 14 on the display device 3. The visualization is performed, for example, by superimposing a 3D image based on the measurement point cloud and a 3D image based on the shape comparison point cloud 114, and emphasizing a portion with a large shape deviation between the measurement point cloud and the shape comparison point cloud 114. The highlighting can be performed by any method, such as changing the density of the displayed color according to the magnitude of the shape deviation. Furthermore, the display control unit 15 can perform a guide display, which will be described later.

[0026] Fig. 2 is a diagram showing an example of a hardware configuration of the measurement system 1. The measurement system 1 can be various terminal devices such as a personal computer (PC) or a tablet terminal. As shown in Fig. 2, the measurement system 1 has a processor 201, a ROM 202, a RAM 203, a storage 204, an input interface 205, and a communication device 206 as hardware.

[0027] The processor 201 is a processor that controls the overall operation of the measurement system 1. The processor 201 operates as the image comparison unit 12, the alignment unit 13, the shape deviation calculation unit 14, and the display control unit 15 by executing a measurement program stored in the storage 204, for example. The processor 201 is, for example, a CPU (Central Processing Unit). The processor 101 may be an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. The processor 201 may be a single CPU, or a plurality of CPUs, or the like.

[0028] The ROM (Read Only Memory) 202 is a non-volatile memory. The ROM 202 stores a startup program and the like of the measurement system 1. The RAM (Random Access Memory) 203 is a volatile memory. The RAM 203 is used as a working memory during processing in the processor 201, for example.

[0029] The storage 204 is, for example, a storage such as a hard disk drive or a solid state drive. The storage 204 stores various programs executed by the processor 201, such as a measurement program. The storage 204 may also store the shape DB 11. The shape DB 11 does not necessarily have to be stored in the storage 204.

[0030] The input interface 205 includes input devices such as a touch panel, a keyboard, a mouse, etc. When an input device of the input interface 205 is operated, a signal corresponding to the operation content is input to the processor 201. The processor 201 performs various processes according to this signal.

[0031] The communication device 206 is a communication device for allowing the measurement system 1 to communicate with external devices such as the camera 2 and the display device 3. The communication device 206 may be a communication device for wired communication or a communication device for wireless communication.

[0032] Next, an operation of the measurement system 1 in the modified example of the first embodiment will be described. Fig. 3 is a flowchart showing a shape deviation visualization process as the operation of the measurement system 1 in the first embodiment. The process in Fig. 3 is executed by the processor 201.

[0033] In step S1, the processor 201 controls the camera 2 to measure the measurement object O. Then, the processor 201 acquires an RGB-D image from the camera 2. Here, the measurement of the measurement object O may be performed by a user. In this case, the user holds the camera 2 in his / her hand and measures the measurement object O.

[0034] In step S2, the processor 201 reads one ideal depth image from the shape DB 11. For example, the processor 201 reads the ideal depth images in the order of numbers assigned to the data sets. Here, in step S2, the processor 201 may read an ideal color image instead of the ideal depth image.

[0035] In step S3, the processor 201 compares the read ideal depth image with a measured depth image, which is a depth image of the measurement object O, and calculates the similarity. FIG. 4 is a conceptual diagram of the comparison of depth images. The similarity can be calculated as a correlation value between the ideal depth image Ic and the measured depth image I shown in FIG. 4, for example. The correlation value is the sum of the differences in depth values ​​between corresponding pixels in the ideal depth image Ic and the measured depth image I. The smaller the correlation value calculated in this way, the higher the similarity between the two images. The similarity may be calculated by various methods other than correlation calculation. Also, in step S3, the processor 201 may perform the comparison using color images.

[0036] In step S4, the processor 201 determines whether the similarity is high. For example, the processor 201 determines that the similarity is high when the calculated similarity is the highest similarity. Alternatively, the processor 201 determines that the similarity is high when the calculated similarity is equal to or higher than a threshold. If it is determined that the similarity is high in step S4, the process proceeds to step S5. If it is not determined that the similarity is high in step S4, the process returns to step S2. In this case, the processor 201 reads out another ideal depth image from the shape DB 11 and performs a comparison again.

[0037] In step S5, the processor 201 selects a data set including an ideal depth image determined to have a high similarity as a data set corresponding to the measurement object O.

[0038] In step S6, the processor 201 reads out from the shape DB 11 a group of points for registration of the selected data set.

[0039] In step S7, the processor 201 generates a measurement point cloud from the depth image obtained from the camera 2, and downsamples the measurement point cloud to the same density as the alignment point cloud from which the measurement point cloud was read out. Downsampling may be performed by any method, such as averaging multiple adjacent point clouds.

[0040] In step S8, the processor 201 aligns the measurement point cloud and the alignment point cloud. FIG. 5 is a conceptual diagram of alignment. The alignment is performed so that the position of the measurement point cloud P shown in FIG. 5 is aligned with the position of the alignment point cloud Pm. Here, if the measurement target O is a part to be welded to a large structure, the measurement point cloud P may include a welded portion A for the large structure. As described above, the shape of the welded portion A is not constant and may vary due to excess fillet, etc. The variation in shape due to excess fillet, etc. may be a cause of error during alignment. Therefore, a portion corresponding to the portion A as a shape comparison portion is removed from the alignment point cloud Pm. Even if the portion A is removed, a corner portion that can be a reference for alignment remains, so that accurate alignment can be performed. Here, the alignment may be performed by any method such as an ICP method.

[0041] In step S9, the processor 201 reads out from the shape DB 11 the point group for shape comparison of the selected data set.

[0042] In step S10, the processor 201 calculates the shape deviation between the measurement point group and the shape comparison point group. FIG. 6 is a conceptual diagram of the calculation of the shape deviation. The calculation of the shape deviation is performed by calculating the deviation between corresponding points between the measurement point group P and the shape comparison point group Pr. In the calculation of the shape deviation, the measurement point group P is a point group that has not been downsampled. The shape comparison point group Pr is a point group that includes a portion where a shape deviation may occur with the measurement point group P, such as the above-mentioned welded portion. The processor 201 aligns the measurement point group P and the shape comparison point group Pr based on the result of the alignment, and calculates the deviation between the corresponding points.

[0043] In step S11, the processor 201 performs visualization display on the display device 3 based on the calculation result of the shape deviation. For example, the processor 201 performs display such as superimposing a 3D image based on the shape comparison point cloud on a 3D image based on the measurement point cloud, and changing the density of the color displayed in the superimposed image according to the magnitude of the shape deviation. Then, the process of FIG. 3 ends.

[0044] As described above, according to the first embodiment, the shape DB stores a data set for an ideal measurement target having an ideal three-dimensional shape for each measurement target. The data set includes an image, a point cloud for alignment, and a point cloud for shape comparison. The image includes a plurality of images corresponding to the ideal measurement target when viewed from a plurality of fields of view. The point cloud for alignment and the point cloud for shape comparison include a plurality of point clouds corresponding to the ideal measurement target when viewed from a plurality of fields of view. By selecting a data set by comparing an image obtained by the camera with an image included in the data set, a point cloud for alignment and a point cloud for shape comparison corresponding to the ideal measurement target when viewed from a field of view close to the measurement point cloud based on the measurement result by the camera 2 can be selected. This is expected to make the alignment and shape comparison of the point clouds easy and highly accurate. In addition, since the comparison of the similarity of images is generally performed with a lower load than the comparison of the similarity of point clouds, the comparison of the similarity for selecting a data set is also expected to be performed with a lower load.

[0045] Moreover, the alignment point cloud is a point cloud in which the shape comparison portion is removed from the shape comparison point cloud. By removing the shape comparison portion, alignment can be performed with high accuracy. Moreover, the alignment point cloud is a point cloud with a lower density than the shape comparison point cloud. Aligning point clouds only requires a point that serves as a reference for alignment, and does not require a high-density point cloud as in shape comparison. Therefore, by making the density of the alignment point cloud lower than that of the shape comparison point cloud, an increase in the capacity of the shape DB can be suppressed.

[0046] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described. Fig. 7 is a block diagram showing the configuration of an example of a measurement system according to a modified example of the first embodiment. In the following description, descriptions that overlap with those in the first embodiment will be omitted.

[0047] As shown in Figure 7, the measurement system 1 of the modified example of the first embodiment has a shape database (DB) 11, an image comparison unit 12, an alignment unit 13, a shape deviation calculation unit 14, and a display control unit 15, as well as a downsampling unit 16 and a point cloud comparison unit 17.

[0048] The shape DB 11 of the modified example is a database that stores a data set of an ideal measurement target for each measurement target, similar to the shape DB 11 of the first embodiment. In the modified example of the first embodiment, the data set includes an image 111, a similarity evaluation point cloud 112, a positioning point cloud 113, and a shape comparison point cloud 114. The image 111, the positioning point cloud 113, and the shape comparison point cloud 114 are the same as those described in the first embodiment. In addition, the similarity evaluation point cloud 112 includes a plurality of point clouds corresponding to the ideal measurement target when viewed from a plurality of fields of view, similar to the positioning point cloud 113 and the shape comparison point cloud 114.

[0049] The similarity evaluation point cloud 112 is a point cloud of an ideal measurement target for evaluating the similarity with the point cloud of the measurement target O obtained from a depth image measured by the camera 2. The similarity evaluation point cloud 112 is a point cloud obtained by downsampling the shape comparison point cloud 114.

[0050] The downsampling unit 16 downsamples the measurement point cloud to the same density as the similarity evaluation point cloud 112. The downsampling may be performed by any method such as averaging a plurality of adjacent point clouds.

[0051] The image comparison unit 12 in the modified example compares the similarity between a measurement image, which is an image of the measurement object O obtained by the camera 2, and an image 111 stored in the shape DB 11, and selects a data set of a predetermined field of view range based on the result of the comparison of similarity. The predetermined field of view range is, for example, the range of several sets of fields of view including the position of the image 111 determined to have a high similarity.

[0052] The point cloud comparison unit 17 compares the similarity between each similarity evaluation point cloud 112 included in the data set in the predetermined range selected by the image comparison unit 12 and the measurement point cloud downsampled by the downsampling unit 16, and selects a data set based on the similarity comparison result. The similarity may be, for example, a correlation calculation result between point clouds.

[0053] The registration unit 13 performs registration between the registration point cloud 113 included in the data set selected by the point cloud comparison unit 17 and the measurement point cloud, which is the point cloud of the measurement object O obtained by the camera 2.

[0054] Next, an operation of the measurement system 1 in the modified example of the first embodiment will be described. Fig. 8 is a flowchart showing a shape deviation visualization process as an operation of the measurement system 1 in the modified example of the first embodiment. The process in Fig. 8 is executed by the processor 201.

[0055] In step S101, the processor 201 controls the camera 2 to measure the measurement object O. Then, the processor 201 acquires an RGB-D image from the camera 2. Here, the measurement of the measurement object O may be performed by a user. In this case, the user holds the camera 2 in his / her hand and measures the measurement object O.

[0056] In step S102, the processor 201 reads one ideal depth image from the shape DB 11. For example, the processor 201 reads the ideal depth images in the order of numbers assigned to the data sets. Here, in step S102, the processor 201 may read an ideal color image instead of the ideal depth image.

[0057] In step S103, the processor 201 compares the read ideal depth image with the measured depth image, which is a depth image of the measurement object O, and calculates the similarity. In step S103, the processor 201 may perform the comparison using color images.

[0058] In step S104, the processor 201 determines whether the similarity is high. For example, the processor 201 determines that the similarity is high when the calculated similarity is the highest similarity. Alternatively, the processor 201 determines that the similarity is high when the calculated similarity is equal to or higher than a threshold. When it is determined that the similarity is high in step S104, the process proceeds to step S105. When it is not determined that the similarity is high in step S104, the process returns to step S102. In this case, the processor 201 reads out another ideal depth image from the shape DB 11 and performs a comparison again.

[0059] In step S105, the processor 201 selects a dataset of a predetermined range of field of view based on a dataset including an ideal depth image determined to have a high similarity as a dataset corresponding to the measurement target O. The predetermined range of field of view is, for example, a range of several sets of fields of view around the position of the ideal depth image determined to have a high similarity. If the range is wide, the accuracy of the point cloud comparison can be improved, but the processing load required for the point cloud comparison also increases. Conversely, if the range is narrow, the accuracy of the point cloud comparison decreases, but the processing load required for the point cloud comparison also decreases.

[0060] In step S106, the processor 201 generates a measurement point cloud from the depth image obtained from the camera 2, and downsamples the measurement point cloud to the same density as the read similarity evaluation point cloud. The downsampling may be performed by any method, such as averaging multiple adjacent point clouds.

[0061] In step S107, the processor 201 reads one similarity evaluation point group of the selected data sets from the shape DB 11. The processor 201 reads the similarity evaluation point groups in the order of numbers assigned to the data sets, for example.

[0062] In step S108, the processor 201 compares the downsampled measurement point group with the read similarity evaluation point group to calculate the similarity. FIG. 9 is a conceptual diagram of point group comparison. The similarity can be calculated, for example, as a correlation value between the similarity evaluation point group Pc and the measurement point group P shown in FIG. 9. The correlation value is the sum of differences between corresponding points in the similarity evaluation point group Pc and the measurement point group P. The smaller the correlation value calculated in this way, the higher the similarity between the two images. The similarity may be calculated by various methods other than correlation calculation.

[0063] In step S109, the processor 201 determines whether the similarity is high. For example, the processor 201 determines that the similarity is high when the calculated similarity is the highest similarity. Alternatively, the processor 201 determines that the similarity is high when the calculated similarity is equal to or higher than a threshold. When it is determined that the similarity is high in step S109, the process proceeds to step S110. When it is not determined that the similarity is high in step S109, the process returns to step S107. In this case, the processor 201 reads out another shape evaluation point group from the selected data set from the shape DB 11 and performs a comparison again.

[0064] In step S110, the processor 201 selects a data set including the shape evaluation point cloud determined to have a high similarity as a data set corresponding to the measurement object O. The subsequent processes of steps S111-S116 are the same as steps S6-S11 in Fig. 3. Therefore, the description will be omitted.

[0065] As described above, according to the modified example of the first embodiment, the data set includes a similarity evaluation point cloud in addition to an image, a positioning point cloud, and a shape comparison point cloud. A comparison of the similarity of the point clouds is performed using the similarity evaluation point cloud of the data set within a predetermined field of view selected based on the result of comparing the similarity of the images. This is expected to select a positioning point cloud and a shape comparison point cloud that correspond to the ideal measurement target when viewed from a field of view that is closer to the measurement point cloud based on the measurement result by the camera 2 than in the first embodiment.

[0066] Second embodiment Next, a second embodiment will be described. In the above-mentioned first embodiment and the modified example, a data set is prepared for each measurement target. Here, in an operation such as mounting parts in a large structure, certain parts are often mounted in a certain order. When considering visualization of shape deviation during such an operation, it is not necessary to perform image comparison and point cloud comparison using a data set of a measurement target that is not related to the operation.

[0067] In the second embodiment, a data set is prepared for each operation. Fig. 10 is a conceptual diagram of a data set stored in the shape DB 11 of the second embodiment. As shown in Fig. 10, in the second embodiment, the data set is divided by operation number, such as operation number 1, operation number 2, .... The operation number is a unique number assigned to each operation. An operation is a series of part installation, part processing, etc., and is not limited to a specific operation.

[0068] The data set for each operation number is further divided into data sets for each location number. The location number is a unique number that indicates the installation location of the part to be measured in the operation of the corresponding operation number. Instead of the location number, a number indicating the order of the operation process may be used.

[0069] The data set of each location number includes an image, a similarity evaluation point cloud, a positioning point cloud, and a shape comparison point cloud, as described in the modified example of the first embodiment. The data set of each location number does not need to include the similarity evaluation point cloud. The image, similarity evaluation point cloud, positioning point cloud, and shape comparison point cloud included in the data set of each location number include multiple images and multiple point clouds corresponding to the ideal measurement target being viewed from multiple fields of view, as described in the first embodiment and the modified example.

[0070] The configuration of the measurement system in the second embodiment is the same as that described in the first embodiment and the modified example, except that a data set stored in the shape DB 11 is prepared for each task. Therefore, a detailed description will be omitted.

[0071] Next, the operation of the measurement system 1 in the second embodiment will be described. Fig. 11 is a flowchart showing the operation of the measurement system 1 in the second embodiment. The process of Fig. 11 is executed by the processor 201.

[0072] In step S201, the processor 201 sets a task number. The task number is set, for example, in response to an input from a user. The user selects the task number of the task to be performed from, for example, a list of tasks displayed on the display device 3. In response to this, the processor 201 sets the task number.

[0073] In step S202, the processor 201 reads out the ideal color image and / or the ideal depth image of one location number in the data set of the set work number. The processor 201 may read out the ideal color image and / or the point cloud in the order of the location numbers, for example.

[0074] In step S203, the processor 201 performs guide display. Fig. 12 is a diagram showing an example of the guide display. In the example of the guide display, the processor 201 displays the image of the measurement object O obtained by the camera 2 on the display device 3, while displaying the read ideal color image as a guide image GI, for example, superimposed on the upper left corner of the image of the measurement object O. Furthermore, the processor 201 displays a point cloud generated from the read ideal depth image as a guide point cloud GP, for example, superimposed on the measurement object O.

[0075] In step S204, the processor 201 performs a shape deviation visualization process. The shape deviation visualization process is the operation of FIG. 3 described in the first embodiment or the operation of FIG. 8 described in the modified example of the first embodiment. Here, when the user performs RGB-D measurement of the measurement target O while holding the camera 2, the user may adjust the attitude of the camera 2 while looking at the guide image GI and / or the guide point cloud GP. This allows measurement of the measurement target O having a similar attitude to the measurement target O shown in the guide image GI and / or the guide point cloud GP. It is expected that image comparison and / or point cloud comparison will be performed easily and with high accuracy by measuring the measurement target O having a similar attitude to the measurement target O shown in the guide image GI and / or the guide point cloud GP by the camera 2. After the shape deviation visualization process, the process proceeds to step S205. The transition from step S204 to step S205 may be performed in response to an instruction from the user.

[0076] In step S205, the processor 201 determines whether the work has been completed. For example, the work is determined to be completed when the shape deviation visualization process has been performed for all location numbers of the set work number. If it is determined in step S205 that the work has been completed, the process of FIG. 11 ends. If it is not determined in step S205 that the work has been completed, the process returns to step S202. In this case, the processor 201 reads the point cloud generated from the ideal color image and / or ideal depth image of the next location number and performs the guide display and shape deviation visualization process for the next location number.

[0077] As described above, in the second embodiment, a data set is prepared for each task. Then, a shape deviation visualization process is performed for the set task. In other words, in the second embodiment, image comparison and point cloud comparison are not performed using a data set of a measurement target that is not related to the task, so that a positioning point cloud and a shape comparison point cloud that correspond to a case where the measurement target is viewed from a field of view close to the measurement point cloud based on the measurement result by the camera 2 can be selected in a shorter time than in the first embodiment.

[0078] In the second embodiment, the user can adjust the attitude of the camera 2 while viewing the guide image GI and / or the guide point cloud GP during RGB-D measurement. This allows the alignment point cloud and the shape comparison point cloud, which correspond to the ideal measurement target viewed from a field of view close to that of the measurement point cloud based on the measurement result by the camera 2, to be selected in a shorter time than in the first embodiment. Furthermore, in the second embodiment, the orientation of the camera 2 is guided by the guide image GI and / or the guide point cloud GP, so that the range of the field of view of the image, similarity evaluation point cloud, alignment point cloud, and shape comparison point cloud that need to be prepared for one measurement target can be narrower than in the first embodiment and the modified example. Therefore, the capacity of the data set for one measurement target can be reduced.

[0079] (Third embodiment) Next, a third embodiment will be described. Here, the configuration of the measurement system in the third embodiment is basically the same as that described in the first embodiment and the modified example. However, in the third embodiment, the image 111 includes at least an ideal color image. The image 111 may or may not include an ideal depth image.

[0080] 13 is a flowchart showing the operation of the measurement system 1 in the third embodiment. The process in FIG.

[0081] In step S301, the processor 201 controls the camera 2 to measure the measurement object O. Then, the processor 201 acquires an RGB-D image from the camera 2. Here, the measurement of the measurement object O may be performed by a user.

[0082] In step S302, the processor 201 reads one ideal depth image from the shape DB 11. For example, the processor 201 reads the ideal depth images in the order of the data set numbers. Here, in step S302, the processor 201 may read an ideal color image instead of the ideal depth image.

[0083] In step S303, the processor 201 compares the read ideal depth image with the measurement depth image, which is a depth image of the measurement object O, and calculates the similarity. Here, in step S303, the processor 201 may perform the comparison using color images.

[0084] In step S304, the processor 201 determines whether the similarity is high. For example, the processor 201 determines that the similarity is high when the calculated similarity is the highest similarity. Alternatively, the processor 201 determines that the similarity is high when the calculated similarity is equal to or higher than a threshold. When it is determined that the similarity is high in step S304, the process proceeds to step S305. When it is not determined that the similarity is high in step S304, the process returns to step S302. In this case, the processor 201 reads out another ideal depth image from the shape DB 11 and performs a comparison again.

[0085] In step S305, the processor 201 selects a data set including an ideal depth image determined to have a high similarity as a data set corresponding to the measurement object O.

[0086] In step S306, the processor 201 reads out the ideal color image of the selected data set from the shape DB 11.

[0087] In step S307, the processor 201 compares the color histograms of the read ideal color image and the measured color image, which is a color image of the measurement object O, and calculates the similarity between the color histograms. A color histogram is a frequency distribution of the RGB values ​​of each pixel in an image.

[0088] In step S308, the processor 201 identifies dissimilar colors between the ideal color image and the measured color image. For example, the processor 201 calculates the difference between the two color histograms and identifies colors having a difference equal to or greater than a threshold as dissimilar colors.

[0089] In step S309, the processor 201 filters out points of dissimilar colors from the measurement point cloud, i.e., removes points of colors that do not exist in the registration point cloud from the measurement point cloud.

[0090] In step S310, the processor 201 reads out from the shape DB 11 a group of points for registration of the selected data set.

[0091] In step S311, the processor 201 generates a measurement point cloud from the depth image obtained from the camera 2, and downsamples the measurement point cloud to the same density as the read-out alignment point cloud. Downsampling may be performed by any method, such as averaging multiple adjacent point clouds.

[0092] In step S312, the processor 201 aligns the measurement point cloud with the alignment point cloud. As described above, since the measurement point cloud has been removed from the alignment point cloud a point cloud of a color that does not exist in the alignment point cloud, accurate alignment can be performed. Here, the alignment may be performed by any method such as the ICP method.

[0093] In step S313, the processor 201 reads out from the shape DB 11 the group of shape comparison points of the selected data set.

[0094] In step S314, the processor 201 calculates the shape deviation between the measurement point cloud and the shape comparison point cloud.

[0095] In step S315, the processor 201 performs visualization display on the display device 3 based on the calculation result of the shape deviation. For example, the processor 201 performs display such as superimposing a 3D image based on the shape comparison point cloud on a 3D image based on the measurement point cloud, and changing the density of the color displayed in the superimposed image according to the magnitude of the shape deviation. Then, the process of FIG. 13 ends.

[0096] As described above, according to the third embodiment, a point cloud of dissimilar colors is removed from the measurement point cloud based on the comparison result between the ideal color image and the measurement color image. Dissimilar colors are considered to be colors that do not exist in the ideal measurement target. Since such a point cloud of dissimilar colors can be a cause of errors during alignment, it is expected that the alignment of the point clouds can be performed with high accuracy by removing such a point cloud of dissimilar colors from the measurement point cloud.

[0097] Here, Fig. 13 shows an example of application to the shape deviation visualization process of the first embodiment shown in Fig. 3. On the other hand, the third embodiment can also be applied to the modified example of the first embodiment shown in Fig. 8.

[0098] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0099] 1 Measurement system, 2 Camera, 3 Display device, 11 Shape database (DB), 12 Image comparison unit, 13 Alignment unit, 14 Shape deviation calculation unit, 15 Display control unit, 16 Downsampling unit, 17 Point cloud comparison unit, 201 Processor, 202 ROM, 203 RAM, 204 Storage, 205 Input interface, 206 Communication device.

Claims

1. an image comparison unit that selects a data set from the database based on a comparison result of the degree of similarity between each of the ideal images, which are acquired from a database that stores data sets for each measurement object, including an ideal image, which is an image of the ideal measurement object corresponding to an ideal measurement object having an ideal shape for each measurement object when viewed from a plurality of fields of view, a registration point cloud for registration with a measurement point cloud, which is a point cloud obtained by measuring the measurement object, and a shape comparison point cloud for shape comparison with the measurement point cloud; and a registration unit that performs registration between the registration point cloud included in the data set selected by the image comparison unit and the measurement point cloud; a shape deviation calculation unit that calculates a shape deviation between the shape comparison point cloud and the measurement point cloud included in the data set selected by the image comparison unit based on the alignment result of the alignment unit; a display control unit that displays a visualization on a display device based on the calculation result of the shape deviation; A measurement system comprising:

2. the alignment point cloud is a point cloud having a lower density than the measurement point cloud, The shape comparison point cloud is a point cloud having the same density as the measurement point cloud. The measurement system of claim 1 .

3. The alignment point cloud is a point cloud obtained by removing a shape comparison portion, which is a portion where a shape deviation may occur between the shape comparison point cloud and the measurement point cloud, from the shape comparison point cloud. The measurement system of claim 1 .

4. The ideal image includes one or both of a depth image and a color image. The measurement system of claim 1 .

5. The database further includes a similarity evaluation point cloud, which is a point cloud for comparing a similarity with the measurement point cloud, the point cloud corresponding to a case where the ideal measurement target is viewed from a plurality of visual fields, the image comparison unit selects a plurality of data sets within a predetermined field of view from the database based on a comparison result between the measurement image and each of the ideal images; a point cloud comparison unit that selects a data set from the database based on a comparison result of the similarity between the similarity evaluation point cloud of each data set selected by the image comparison unit and the measurement point cloud, The registration unit aligns the registration point cloud included in the data set selected by the point cloud comparison unit with the measurement point cloud, the shape deviation calculation unit calculates a shape deviation between the shape comparison point cloud included in the data set selected by the point cloud comparison unit and the measurement point cloud based on the alignment result of the alignment unit. The measurement system of claim 1 .

6. the database stores the data set for each measurement target for each task, The image comparison unit compares the similarity between the ideal image and the measurement image included in the data set for the set task. The measurement system of claim 1 .

7. the display control unit performs a guide display for the measurement of the measurement object on the display device based on the ideal image included in the data set for the set task. The measurement system of claim 6.

8. The image comparison unit identifies dissimilar colors between the measurement image and the ideal image, the alignment unit aligns a measurement point cloud obtained by excluding the point cloud of the dissimilar color from the measurement point cloud with the alignment point cloud; The measurement system of claim 1 .

9. a database storing a data set for each measurement object, the data set including a plurality of ideal images for each measurement object, which corresponds to an ideal measurement object having an ideal shape for each measurement object, viewed from a plurality of fields of view, a registration point cloud for registration with a measurement point cloud, which is a point cloud obtained by measuring the measurement object, and a shape comparison point cloud for shape comparison with the measurement point cloud; and selecting a data set from the database based on a comparison result between each of the ideal images and a measurement image, which is an image of the measurement object obtained by measuring the measurement object; Aligning the registration point cloud and the measurement point cloud included in the selected data set; Calculating a shape deviation between the shape comparison point cloud and the measurement point cloud included in the selected data set based on the registration result; visually displaying the shape deviation on a display device based on the calculation result of the shape deviation; A measurement program for causing a processor to execute the above.

Images