Three-dimensional scanner, three-dimensional measurement method, and storage medium storing three-dimensional measurement program

Abstract

The invention provides a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program. According to the present invention, alignment between a plurality of pieces of three-dimensional data acquired by scanning a workpiece can be performed with high precision while reducing the burden of a user. A three-dimensional scanner includes: an attitude calculation unit that calculates a placement attitude different from a first placement attitude based on first three-dimensional data acquired by a data acquisition unit; an overlap region estimation unit that estimates an overlap region between the three-dimensional data acquired by the data acquisition unit and the first three-dimensional data acquired by the data acquisition unit in a state in which the placement posture calculated by the posture calculation unit is placed; and a display control unit that displays, on a display unit, an evaluation index based on the amount of the overlap region estimated by the overlap region estimation unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program. BACKGROUND

[0002] For example, JP 2024-24328 A discloses a three-dimensional scanner that scans a workpiece to generate three-dimensional data.

[0003] This type of three-dimensional scanner is configured to be able to measure the three-dimensional shape of a workpiece by irradiating the workpiece with mainly structured illumination light and capturing and analyzing the distortion of the illumination light by a camera.

[0004] Since the shape of the workpiece obtained by one scan by the three-dimensional scanner is limited to the range captured by the camera, in order to acquire the shape of a portion not captured by the camera, it is necessary to scan the workpiece multiple times while changing the relative positional relationship between the workpiece and the three-dimensional scanner or to operate the three-dimensional scanner multiple times, align and combine a plurality of pieces of scan data.

[0005] However, for example, in order to scan the entire workpiece such as the front surface and the rear surface of the workpiece, it is necessary to repeat multiple scans while changing the position and orientation of the workpiece. However, the position and orientation at which the workpiece is installed are determined by the user, and a plurality of pieces of scan data can not be properly combined.

[0006] That is, as a method for aligning a plurality of pieces of scan data, there are a method for mechanically aligning a plurality of pieces of scan data by using a high-precision stage or a robot arm, a method for aligning a plurality of pieces of scan data by performing pattern matching using the brightness or shape features of a plurality of pieces of scan data, and a hybrid method for mechanically aligning approximate positions and precisely aligning a plurality of pieces of scan data by matching. In addition, there is also a method for aligning the position of a three-dimensional scanner by capturing the position without moving the object by another camera or the like.

[0007] The mechanical method is difficult to achieve high-precision alignment. In addition, for example, when the entire periphery of the workpiece is to be scanned, the surface in contact with the stage surface cannot be scanned in a normal stage, and the workpiece needs to be reinstalled.

[0008] The matching method has an advantage in that alignment accuracy corresponding to the accuracy of the scan data can be achieved regardless of the size of the object, but requires an overlapping region to be provided between a plurality of pieces of data to be aligned, and the position is not determined unless there are some markers or feature points in the brightness and shape of the region. Therefore, at the time of scanning, it is necessary to consider which region is acquired by which scan, whether a feature marker can be arranged in the overlapping region between a plurality of pieces of scan data, and whether the work and the camera can be physically arranged in a relative positional relationship to be scanned, and there is a problem that not only the alignment is difficult but also an advanced technique is required to make an accurate measurement.

[0009] In addition, regarding whether a feature marker can be arranged in the overlapping region between a plurality of pieces of scan data, it is necessary to actually scan how many features are to be arranged, collate the measurement value with the data of other measuring instruments and the design value, and verify the result, and repeated trials are required. SUMMARY

[0010] The present disclosure has been made in view of such a point, and the object is to enable alignment of a plurality of pieces of three-dimensional data acquired by scanning a work with high accuracy while reducing the burden on the user.

[0011] To achieve the above object, according to one embodiment of the present disclosure, a three-dimensional scanner that generates combined three-dimensional data of a work by generating three-dimensional data of the work placed in different placement postures and combining a plurality of pieces of three-dimensional data can be assumed. The three-dimensional scanner includes a data acquisition unit that acquires first three-dimensional data of a work placed in a first placement posture, a posture calculation unit that calculates a placement posture different from the first placement posture based on the first three-dimensional data acquired by the data acquisition unit, an overlapping region estimation unit that estimates an overlapping region between three-dimensional data to be acquired by the data acquisition unit in a state where the work is placed in the placement posture calculated by the posture calculation unit and the first three-dimensional data acquired by the data acquisition unit, and a display control unit that displays an evaluation index based on the amount of the overlapping region estimated by the overlapping region estimation unit on a display unit.

[0012] According to this configuration, the overlapping region estimation unit estimates an overlapping region between the first three-dimensional data of the work placed in the first placement posture and the three-dimensional data of the work placed in the placement posture different from the first placement posture. Since the evaluation index based on the amount of the overlapping region is displayed on the display unit, the user can specify the placement posture of the work, in which alignment can be performed with high accuracy without repeated trials, with reference to the evaluation index.

[0013] According to another embodiment of the present disclosure, a three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating a plurality of pieces of three-dimensional data of the workpiece placed in different placement postures and combining the plurality of pieces of three-dimensional data can be premised. The three-dimensional measurement method can include acquiring first three-dimensional data of a workpiece placed in a first placement posture, calculating a placement posture different from the first placement posture based on the acquired first three-dimensional data, estimating an overlapping region between three-dimensional data acquired by a data acquisition unit in a state of being placed in the calculated placement posture and the first three-dimensional data acquired by the data acquisition unit, and displaying an evaluation index on a display unit based on an amount of the estimated overlapping region.

[0014] According to still another embodiment of the present disclosure, a storage medium storing a three-dimensional measurement program for causing a computer to execute a three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating a plurality of pieces of three-dimensional data of the workpiece placed in different placement postures and combining the plurality of pieces of three-dimensional data can be provided.

[0015] As described above, since an overlapping region between first three-dimensional data of a workpiece placed in a first placement posture and three-dimensional data acquired in a state of being placed in a different placement posture can be estimated, and an evaluation index based on an amount of the estimated overlapping region can be provided to a user, alignment between a plurality of pieces of three-dimensional data acquired by scanning a workpiece can be performed with high precision while reducing a burden on the user. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a diagram illustrating an overall configuration of a three-dimensional scanner according to an embodiment of the present invention;

[0017] Figure 2 is a block diagram of a three-dimensional scanner;

[0018] Figure 3 is a side view of a measurement unit and a base;

[0019] Figure 4 is a block diagram of a measurement unit;

[0020] Figure 5 is a diagram illustrating a configuration example of a module;

[0021] Figure 6 is a flowchart illustrating an example of a scanning process in a case where there is no CAD data of a workpiece;

[0022] Figure 7 is a diagram illustrating an example of a user interface screen displayed at the start of measurement;

[0023] Figure 8FIG. 1 is a diagram showing an example of a user interface screen displayed when presenting posture candidates;

[0024] Figure 9 FIG. 2 is a diagram showing an example of a user interface screen displayed when accepting selection of a posture candidate;

[0025] Figure 10 FIG. 3 is a diagram showing an example of a user interface screen in which a model of a determined placement posture is superimposed and displayed on a real-time image;

[0026] Figure 11 FIG. 4 is a flowchart showing an example of an alignment process;

[0027] Figure 12 FIG. 5 is a schematic view related to hidden surface removal;

[0028] Figure 13 FIG. 6 is a flowchart showing an example of a process of extracting shape features and image features;

[0029] Figure 14 FIG. 7 is a flowchart showing an example of an extraction process in a case where a vector in which an image feature is added is input to a shape feature extraction unit;

[0030] Figure 15 FIG. 8 is a flowchart showing an example of a process of calculating a low-precision alignment parameter;

[0031] Figure 16 FIG. 9 is a flowchart showing an example of a process of calculating a high-precision alignment parameter;

[0032] Figure 17 FIG. 10 is a flowchart showing an example of a scanning process in a case where CAD data of a workpiece is present;

[0033] Figure 18 FIG. 11 is a diagram showing a user interface screen for displaying first three-dimensional data, second three-dimensional data, and combined three-dimensional data;

[0034] Figure 19 FIG. 12 is a diagram showing a workflow of generating combined three-dimensional data;

[0035] Figure 20 FIG. 13 is a diagram showing a confirmation screen of combined three-dimensional data;

[0036] Figure 21 FIG. 14 is a diagram showing a confirmation screen of second three-dimensional data;

[0037] Figure 22 FIG. 15 is a diagram showing a confirmation screen of first three-dimensional data;

[0038] Figure 23 FIG. 16 is a diagram showing a user interface screen for data editing;

[0039] Figure 24 is a flowchart showing an example of the processing of the alignment function;

[0040] Figure 25 is a flowchart showing an example of the virtual object rotation processing by mouse drag;

[0041] Figure 26 is a flowchart showing an example of the processing of adjusting the positional relationship between the virtual object and the virtual ground;

[0042] Figure 27 is a flowchart showing an example of the stage surface detection processing;

[0043] Figure 28 is a flowchart showing an example of the processing of the alignment function in the case where rotation and movement of the virtual object are performed;

[0044] Figure 29 is a flowchart showing an example of the processing in the case where rotation and movement of the virtual object are performed;

[0045] Figure 30 is a flowchart showing another example of the processing in the case where rotation and movement of the virtual object are performed;

[0046] Figure 31 is a flowchart showing an example of the processing in the case where the positional relationship between the virtual object, the virtual ground, and the virtual tilt platform is adjusted;

[0047] Figure 32 is a color map based on the dimensional difference between the scan data of the workpiece and the CAD data;

[0048] Figure 33A is a graph showing an example of cross-sectional measurement of the scan data of the workpiece; and

[0049] Figure 33B is a graph showing an example of performing cross-sectional measurement on the CAD data of the workpiece. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings. Note that the following descriptions of preferred embodiments are merely examples in nature and are not intended to limit the present application, its application, or its uses.

[0051] Figure 1is a diagram showing the overall configuration of a three-dimensional scanner 1 according to an embodiment of the present application. The three-dimensional scanner 1 is a device capable of acquiring three-dimensional data by measuring the shape of a workpiece (measurement object) W, converting the three-dimensional data into mesh data of the workpiece W, and outputting the mesh data. The three-dimensional scanner 1 can also convert the mesh data of the workpiece W into CAD data and output the CAD data, or convert the mesh data into surface data and output the surface data.

[0052] In the following description, when the shape of the workpiece W is measured, in a process of acquiring coordinate information of a front surface of the workpiece W, the workpiece W is irradiated with measurement light of a predetermined pattern, and the coordinate information is acquired by using a signal obtained from reflected light reflected by the front surface of the workpiece W. For example, a measurement method using triangulation using a fringe projection image obtained from the reflected light by projecting measurement light onto the workpiece W using structured illumination as the measurement light of the predetermined pattern can be used. However, in the present application, the principle and configuration for acquiring the coordinate information of the workpiece W are not limited thereto, and other methods can also be applied.

[0053] The three-dimensional scanner 1 includes a measurement unit 100 that measures the shape of the workpiece W, a base 600 on which the workpiece W can be mounted, a controller 200, a light source unit 300, a display unit 400, and the like. The controller 200 can be incorporated in the measurement unit 100, the light source unit 300 can be incorporated in the measurement unit 100, or the display unit 400 can be incorporated in the measurement unit 100. In addition, the controller 200 and the light source unit 300 can be integrated, or the controller 200 and the display unit 400 can be integrated.

[0054] The three-dimensional scanner 1 performs structured illumination on the workpiece W by the light source unit 300, captures a fringe projection image to generate a depth image having coordinate information, and can measure the three-dimensional dimensions and shape of the workpiece W based on the depth image. Measurement using such fringe projection has the advantage that the measurement time can be shortened, because three-dimensional measurement can be performed without moving the workpiece W or an optical system such as a lens in the Z direction (height direction).

[0055] Figure 2 A block diagram of the three-dimensional scanner 1 according to an embodiment of the present application is shown. As shown in this diagram, the measurement unit 100 includes a pattern light projection unit (first light projection unit) 110 that projects pattern light for measurement onto the workpiece W, a light receiving unit 120, a measurement control unit 150, and an illumination light output unit 130. The light projection unit 110 is a component that irradiates the workpiece W mounted on a mounting unit 140 described later with measurement light of a predetermined pattern. The mounting of the workpiece W on the mounting unit 140 is the same as the placement of the workpiece W on the mounting unit 140.

[0056] The light-receiving unit 120 is fixed in a posture inclined with respect to a mounting surface 142 of a later-described rotary stage 143. The light-receiving unit 120 receives measurement light emitted by the light-projecting unit 110 and reflected by the workpiece W. When receiving the measurement light as reflected light from the workpiece W, the light-receiving unit 120 generates and outputs a first light-receiving signal for measurement indicating the amount of the received measurement light. The light-receiving unit 120 can generate an observation image for observing the entire shape of the workpiece W by capturing the workpiece W mounted on the mounting unit 140. In this example, the illumination light output unit 130 is provided, but the workpiece W can be irradiated with uniform light from the light-projecting unit 110. In this case, the light-projecting unit 110 is a member that irradiates the workpiece W with the measurement light and the uniform light at different timings. The light-receiving unit 120 can also receive the uniform light emitted from the light-projecting unit 110 and output a second light-receiving signal for texture acquisition. For example, uniform light having the same wavelength as the measurement light can be emitted from the measurement light source, and a light-receiving signal including uniaxial color information can be output. Note that although not shown, a calibrated first camera and a second camera can also be prepared, the shape is acquired by the first camera, and the texture information is acquired by the second camera. The texture information includes color information and brightness information of the workpiece W.

[0057] The light-receiving unit 120 according to the present embodiment includes a high-magnification light-receiving unit and a low-magnification light-receiving unit. The high-magnification light-receiving unit is a member capable of capturing the workpiece W in a magnified manner as compared with the low-magnification light-receiving unit. On the other hand, the low-magnification light-receiving unit is a light-receiving unit having a wider field of view range than the high-magnification light-receiving unit.

[0058] The base 600 includes a bottom plate 602, the mounting unit 140, and a movement control unit (stage control unit) 144. The mounting unit 140 is supported on the bottom plate 602 of the base 600. The movement control unit 144 is a member that controls movement and rotation operation of the rotary stage 143 on which the workpiece W is mounted. The movement control unit 144 can be provided on the controller 200 side in addition to being provided on the base 600 side.

[0059] The light source unit 300 is connected to the measurement unit 100. The light source unit 300 is a member that generates measurement light and supplies the measurement light to the measurement unit 100. The controller 200 is a member that controls the measurement unit 100 and the like. The display unit 400 is connected to the controller 200 and is configured to display an image generated by the measurement unit 100 and perform necessary settings, inputs, selections, and the like.

[0060] The mounting unit 140 includes the rotary stage 143 having a top surface on which the mounting surface 142 on which the workpiece W is mounted is formed. As Figure 4As shown, two directions orthogonal to each other on the mounting surface 142 of the rotary stage 143 are defined as the X direction and the Y direction, and are indicated by arrows X and Y, respectively. A direction orthogonal to the mounting surface 142 of the mounting unit 140 is defined as the Z direction, and is indicated by an arrow Z. A direction of rotation about an axis parallel to the Z direction is defined as the Θ direction, and is indicated by an arrow Θ.

[0061] The mounting unit 140 includes the rotary stage 143 that rotates the mounting surface 142 about an axis extending in the Z direction, and the translation stage 141 that moves the mounting surface 142 in the horizontal directions (the X direction and the Y direction). The translation stage 141 includes an X-direction moving mechanism and a Y-direction moving mechanism. The rotary stage 143 has a Θ-direction rotating mechanism. The mounting unit 140 can include a workpiece holding member (jig or the like) that holds the workpiece W on the mounting surface 142. Further, the mounting unit 140 can include a tilt stage having a mechanism that can rotate about an axis parallel to the mounting surface 142.

[0062] The movement control unit 144 controls the rotational movement of the rotary stage 143 and the translation of the translation stage 141 in accordance with the measurement conditions set by the measurement condition setting unit 261 described later. In addition, the movement control unit 144 controls the movement action of the mounting unit 140 by the mounting movement unit based on the measurement region set by the measurement condition setting unit 261 described later.

[0063] The controller 200 includes a central processing unit (CPU) 210, a read only memory (ROM) 220, a work memory 230, a storage device (storage unit) 240, an operation unit 250, and the like. For example, a personal computer (PC) or the like can be used as the controller 200.

[0064] In Figure 4 A configuration of the measurement unit 100 is shown in a block diagram. The measurement unit 100 includes the light projection unit 110, the light reception unit 120, the illumination light output unit 130, the measurement control unit 150, and a main body case 101 that houses these units. The light projection unit 110 includes a measurement light source 111, a pattern generation unit 112, and a plurality of lenses 113, 114, and 115. The light reception unit 120 includes a camera 121 and a plurality of lenses 122 and 123. In a case where measurement is performed at different magnifications by providing a plurality of light reception units, a light reception unit 120a including the camera 121 for low magnification and the lens for low magnification, and a light reception unit 120b including the camera 121 for high magnification and the lens for high magnification can be installed. Note that the present application is not limited to this configuration, and the magnification can be changed by switching between a plurality of lenses for one camera 121, or the magnification can be changed by providing a zoom lens for one camera 121.

[0065] The light projection units 110 are obliquely arranged above the mounting unit 140. In the example shown, the light projection units 110 are arranged so that the center axes of the light projection units 110 and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140. In this way, the light projection units 110 and the light receiving unit 120 are arranged so that the center axes of the light projection units 110 and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 can include a plurality of light projection units 110. Here, a first measurement light projection unit 110A (right side in FIG. 1) capable of irradiating the workpiece W with first measurement light ML1 from a first direction and a second measurement light projection unit 110B (left side in FIG. 1) capable of irradiating the workpiece W with second measurement light ML2 from a second direction different from the first direction are provided. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 can include a plurality of light projection units 110. Here, a first measurement light projection unit 110A (right side in FIG. 1) capable of irradiating the workpiece W with first measurement light ML1 from a first direction and a second measurement light projection unit 110B (left side in FIG. 1) capable of irradiating the workpiece W with second measurement light ML2 from a second direction different from the first direction are provided. Figure 4 The first measurement light projection unit 110A and the second measurement light projection unit 110B are arranged symmetrically with respect to the center axis of the light receiving unit 120. Note that although not shown, three or more light projection units 110 can be included, or the light projection units 110 and the mounting unit 140 can be relatively moved to project light onto the workpiece W in different irradiation directions while using a common light projection unit 110. In addition, in the example described above, a plurality of light projection units 110 are prepared and the light rays are received by a common light receiving unit 120, but conversely, a plurality of light receiving units 120 can be prepared for a common light projection unit 110, and the light rays can be received by the plurality of light receiving units. Furthermore, in this example, the irradiation angle of the illumination light projected by the light projection units 110 with respect to the Z direction is fixed, but this can be variable.

[0066] Each of the first measurement light projection unit 110A and the second measurement light projection unit 110B includes a first measurement light source and a second measurement light source as the measurement light source 111. The measurement light source 111 is, for example, a halogen lamp that emits white light. The measurement light source 111 can be a light source that emits monochromatic light, for example, such as a blue light emitting diode (LED) or another light source that emits blue light, such as an organic EL. The light emitted from the measurement light source 111 (hereinafter referred to as "measurement light") is appropriately condensed by a lens 113 and then incident on a pattern generation unit 112.

[0067] The relative positional relationship between the light receiving unit 120, the light projection units 110A and 110B, and the mounting unit 140 is determined so that the center axes of the light projection units 110A and 110B and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140, at which the depth of field of the light projection units 110 and the light receiving unit 120 is appropriate. In addition, since the center of the rotation axis in the θ direction coincides with the center axis of the light receiving unit 120, when the mounting unit 140 is rotated in the θ direction, the workpiece W does not deviate from the field of view and rotates around the rotation axis in the field of view.

[0068] The pattern generating unit 112 reflects light emitted from the measurement light source 111 to project the measurement light onto the workpiece W. The measurement light incident on the pattern generating unit 112 is converted into a preset pattern and a preset intensity (brightness) and emitted. The measurement light emitted by the pattern generating unit 112 is converted into light having an observable and measurable field of view larger than the light receiving unit 120 by the plurality of lenses 114 and 115, and then the workpiece W on the mounting unit 140 is irradiated with the converted light.

[0069] The pattern generating unit 112 is a member capable of switching between a light projection state in which the measurement light is projected onto the workpiece W and a non-light projection state in which the measurement light is not projected onto the workpiece W. For example, for such a pattern generating unit 112, a digital micromirror device (DMD) or the like can be used. The pattern generating unit 112 using the DMD can be controlled by the measurement control unit 150 to be capable of switching between a reflection state in which the measurement light is reflected on the optical path to become the light projection state and a light shielding state in which the measurement light is shielded to become the non-light projection state.

[0070] Note that, in the above example, an example in which the DMD is used for the pattern generating unit 112 has been described, but the pattern generating unit 112 is not limited to the DMD in the present application, and other members can also be used. For example, a liquid crystal on silicon (LCOS) can be used as the pattern generating unit 112. Alternatively, the amount of transmitted measurement light can be adjusted by using a transmissive member instead of a reflective member. In this case, the pattern generating unit 112 is provided on the optical path of the measurement light to switch between a light projection state in which the measurement light is transmitted and a light shielding state in which the measurement light is shielded. For example, a liquid crystal display (LCD) can be used as the pattern generating unit 112. Alternatively, the pattern generating unit 112 can be formed by using a projection method of a plurality of line LEDs, a projection method using a plurality of optical paths, an optical scanner method including a laser and a galvanometer mirror, an accordion fringe interferometry (AFI) method using interference fringes generated by superimposing beams split by a beam splitter, a projection method using an actual grating and a moving mechanism including a piezoelectric stage, a high-resolution encoder, or the like.

[0071] The light receiving unit 120 is provided above the mounting unit 140. The measurement light reflected upward from the mounting unit 140 by the workpiece W is collected and captured by the plurality of lenses 122 and 123 of the light receiving unit 120, and then received by the camera 121.

[0072] The camera 121 is, for example, a charge-coupled device (CCD) camera including an imaging element 121a. The imaging element 121a is, for example, a monochrome charge-coupled device (CCD). The imaging element 121a can be another imaging element, such as a complementary metal-oxide semiconductor (CMOS) image sensor. In a color imaging element, since each pixel needs to receive light corresponding to red, green, and blue colors, the measurement resolution is lower than that of a monochrome imaging element. Since a color filter needs to be provided in each pixel, the sensitivity is reduced. Therefore, in the present embodiment, a color image is acquired by employing a monochrome CCD as the imaging element and causing the illumination light output unit 130 described later to emit illuminations corresponding to each RGB color in a time-division manner to take an image. With this configuration, it is possible to acquire a color image of the measurement object without reducing the measurement accuracy. The illumination light output unit 130 is an example of a second light projection unit that irradiates the workpiece W with illumination light. The illumination light can be uniform light.

[0073] Note that a color imaging element can be used as the imaging element 121a. In this case, although the measurement accuracy and the sensitivity are lower than those of a monochrome imaging element, it is not necessary to emit illuminations corresponding to each RGB color from the illumination light output unit 130 in a time-division manner, and it is possible to acquire a color image by emitting only white light, so the illumination optical system can be simply formed. An analog electric signal corresponding to the amount of received light (hereinafter referred to as a "light reception signal") is output from each pixel of the imaging element 121a to the measurement control unit 150.

[0074] An analog / digital converter (A / D converter) and a first-in first-out (FIFO) memory (both not shown) are installed on the measurement control unit 150. The light reception signal output from the camera 121 is sampled at a constant sampling period, and converted into a digital signal by the A / D converter of the measurement control unit 150 under the control of the light source unit 300. The digital signal output from the A / D converter is sequentially accumulated in the FIFO memory. The digital signal accumulated in the FIFO memory is sequentially transferred to the controller 200 as pixel data.

[0075] The operation unit 250 of the controller 200 can include, for example, a keyboard, a pointing device, and the like. For example, a mouse, a joystick, and the like are used as the pointing device.

[0076] The ROM 220 of the controller 200 stores a system program or the like. The work memory 230 of the controller 200 includes, for example, a random access memory (RAM), and is used to process various types of data. The storage device 240 includes a solid state drive, a hard disk drive, or the like. The storage device 240 stores a reverse engineering program. In addition, the storage device 240 is used to store various types of data, such as pixel data (image data), setting information, and a measurement condition given from the measurement control unit 150. The measurement condition includes, for example, various settings set by the scanning module 260, such as a setting of the light projection unit 110 (pattern frequency or pattern type) and a type of the light receiving unit 120 (low magnification light receiving unit or high magnification light receiving unit), when the shape of the workpiece W is measured, which will be described later. Furthermore, the storage device 240 can also store brightness information, coordinate information, and attribute information of each pixel constituting a measurement image.

[0077] The CPU 210 is a control circuit or a control element that processes a given signal or data, performs various arithmetic operations, and outputs an arithmetic operation result. In the present specification, the CPU refers to an element or a circuit that performs arithmetic operations, and is not limited to a processor (such as a CPU, MPU, GPU, or TPU) for a general-purpose PC regardless of the name, and is used in the sense of including a processor (such as an FPGA, ASIC, or LSI), a microcomputer, or a chipset (such as a SoC).

[0078] The CPU 210 generates image data based on pixel data given from the measurement control unit 150. In addition, the CPU 210 performs various types of processing on the generated image data by using the work memory 230. For example, the CPU 210 generates measurement data indicating a three-dimensional shape of the workpiece W included in the field of view of the light receiving unit 120 at a certain position of the mounting unit 140 based on a light receiving signal output from the light receiving unit 120. The measurement data is an image itself acquired by the light receiving unit 120, and, for example, in a case where the shape of the workpiece W is measured by a phase shift method, a plurality of images constitute one piece of measurement data. Note that the measurement data can be point cloud data that is a set of points having three-dimensional position information, and the measurement data of the workpiece W can be acquired from the point cloud data. The point cloud data is data represented by an aggregation of a plurality of points having three-dimensional coordinates.

[0079] The movement control unit 144 determines whether to perform only the rotation operation of the rotary stage 143 or to perform both the rotation operation of the rotary stage 143 and the translation operation of the translation stage 141 based on the measurement data of at least a part of the workpiece W. As a result, the imaging range is automatically determined without the user's awareness in accordance with the outer shape of the workpiece W, and thus, three-dimensional measurement becomes easy. Note that the movement control unit 144 can control the rotary stage 143 to rotate in a state where the movement of the translation stage 141 in the XY direction is stopped after the translation stage 141 moves in the XY direction, and thus, the shape around the workpiece W can also be acquired. Note that scanning can also be performed by relatively moving and rotating the workpiece W with respect to the measurement unit 100 in a state where the measurement unit 100 is fixed.

[0080] The display unit 400 is a member for displaying a fringe projection image acquired by the measurement unit 100, a depth image generated based on the fringe projection image, a texture image captured by the measurement unit 100, various user interface screens, and the like. The display unit 400 includes, for example, an LCD panel or an organic electroluminescence (EL) panel. Further, a touch panel is used for the display unit 400, and thus, it can also be used as the operation unit 250. Further, the display unit 400 can also display an image generated by the light receiving unit 120.

[0081] The light source unit 300 includes a control board 310 and an observation illumination light source 320. A CPU (not shown) is mounted on the control board 310. The CPU of the control board 310 controls the light projection unit 110, the light receiving unit 120, and the measurement control unit 150 based on a command from the CPU 210 of the controller 200. Note that this configuration is an example, and other configurations can be used. For example, the control board can be omitted by controlling the light projection unit 110 and the light receiving unit 120 by the measurement control unit 150 or by the controller 200. Alternatively, a power supply circuit for driving the measurement unit 100 can be provided in the light source unit 300.

[0082] The observation illumination light source 320 includes, for example, LEDs that emit three colors of red, green, and blue light. The brightness of the light emitted from each LED is controlled, and thus, any color of light can be generated from the observation illumination light source 320. Illumination light IL generated from the observation illumination light source 320 is output from the illumination light output unit 130 of the measurement unit 100 through a light guide member (light guide). Note that, as the observation illumination light source, other light sources such as a semiconductor laser (LD), a halogen lamp, and an HID can be appropriately used in addition to the LED. Specifically, in a case where an element capable of color shooting is used as the imaging element, a white light source can be used as the observation illumination light source.

[0083] The illumination light IL output from the illumination light output unit 130 irradiates the workpiece W with red light, green light, and blue light in a time-division manner. As a result, a color texture image can be obtained by combining texture images respectively captured by these RGB lights, and the texture image is displayed on the display unit 400.

[0084] A three-dimensional measurement program and an application for realizing the functions of the three-dimensional scanner 1 by the controller 200 are installed on the controller 200. Therefore, the three-dimensional measurement method according to the present application can be executed by using the three-dimensional scanner 1. The three-dimensional measurement method is a method for measuring the three-dimensional shape of the workpiece W, and is executed by a computer included in the controller 200. The three-dimensional measurement program for causing the computer to execute the three-dimensional measurement method can be recorded in the storage medium 1000. The storage medium 1000 can be, for example, an optical disc such as a CD-ROM or a DVD-ROM, or can be a semiconductor memory such as a memory card.

[0085] In the controller 200 in which the three-dimensional measurement program and the application are installed, the CPU 210, the ROM 220, the work memory 230, the storage device 240, and the like constitute Figure 5 the scanning module 260, the conversion module 270, the integration module 280, and the analysis module 290 illustrated in FIG. 8. In this embodiment, the scanning module 260, the conversion module 270, the integration module 280, and the analysis module 290 are divided into four modules, but any two or more of the modules 260, 270, 280, and 290 can be integrated to constitute one module. In addition, a part of each of the modules 260, 270, 280, and 290 can be incorporated into another module. That is, Figure 5 The configuration example illustrated in FIG. 8 is an example, and is not limited to Figure 5 the configuration example illustrated in FIG. 8.

[0086] The scanning module 260 is a component that acquires image data of the workpiece W by measuring the shape of the workpiece W and creates mesh data of the workpiece W based on the image data. The conversion module 270 is a component that converts the mesh data created by the scanning module 260 into CAD data. The CAD data is three-dimensional shape information constituted by analysis surfaces and free-form surfaces, and includes surface data, solid data, data for design, and the like. The surface data is data of a shape surface including a free-form surface and an analysis curved surface, such as side surface data of a cylinder and plane data.

[0087] The integration module 280 is a component that transmits signals and data from the scanning module 260 to the conversion module 270 and the analysis module 290 and transmits signals and data from the conversion module 270 to the scanning module 260. In this example, the module can perform a plurality of arithmetic processes in one unit, and can also be referred to as, for example, a functional unit, a functional block, or the like.

[0088] The scanning module 260 includes, for example, a measurement condition setting unit 261, a scanning control unit 262, a point cloud acquisition unit 263a, a mesh data generation unit 263b, a scanning output unit 264, and the like. The measurement condition setting unit 261 is a means for setting a measurement condition of a shape of a workpiece. The scanning control unit 262 is a means for controlling the measurement unit 100 to generate image data in accordance with the measurement condition set by the measurement condition setting unit 261 and acquire measurement data of the workpiece W based on the generated image data.

[0089] The point cloud acquisition unit 263a is a means for acquiring point cloud data of the workpiece W based on the image data of the workpiece W acquired by the scanning control unit 262. The mesh data generation unit 263b is a means for acquiring the point cloud data acquired by the point cloud acquisition unit 263a, processing the acquired point cloud data, and converting the data into mesh data.

[0090] The scanning output unit 264 is a means for outputting the mesh data created by the mesh data generation unit 263b and additional data to the conversion module 270. The additional data is, for example, data including at least one of a measurement condition and data calculated from the measurement data of the workpiece W.

[0091] The scanning module 260 controls the measurement unit 100, generates a condition for measuring a shape of the workpiece W (a measurement device model, a measurement magnification, a resolution, and the like) and raw data (for example, image data) at the time of measurement, and three-dimensional data. The three-dimensional data is mesh data including a plurality of polygons, and can also be referred to as polygon data. A polygon is data including information specifying a plurality of points and information indicating a polygon surface formed by connecting the points, and can include, for example, information specifying three points and information indicating a triangular surface formed by connecting the three points. The mesh data and the polygon data can also be defined as data represented by aggregation of a plurality of polygons.

[0092] In the conversion module 270, the mesh data is converted into CAD data, and a conversion process is determined based on the measurement condition and the raw data. Specifically, the conversion module 270 includes, for example, a data input unit 271, a processing parameter determination unit 272, a CAD conversion unit 273, a CAD output unit 274, and the like. The data input unit 271 is a means for accepting the mesh data and the additional data output from the scanning output unit 264. The processing parameter determination unit 272 is a means for determining a processing parameter at the time of converting the mesh data into CAD data in accordance with the additional data accepted by the data input unit 271. The CAD conversion unit 273 is a means for converting the mesh data into CAD data in accordance with the processing parameter determined by the processing parameter determination unit 272. The CAD output unit 274 is a means for outputting the CAD data converted by the CAD conversion unit 273.

[0093] The analysis module 290 of the three-dimensional scanner 1 is a module that generates combined three-dimensional data of the workpiece W by generating a plurality of pieces of three-dimensional data of the workpiece W placed in different placement attitudes and combining the plurality of pieces of three-dimensional data. The analysis module 290 includes, for example, a data acquisition unit 291 that acquires three-dimensional data of the workpiece W mounted on the rotary stage 143. The user can mount the workpiece W on the rotary stage 143 in any attitude. For example, in a case where the three-dimensional shapes of the front surface and the back surface of the workpiece W are acquired, the three-dimensional data can be acquired by mounting the workpiece W on the rotary stage 143 in a placement attitude in which the front surface of the workpiece W faces upward so as to acquire the three-dimensional data, and then mounting the workpiece W on the rotary stage 143 in a placement attitude in which the back surface of the workpiece W faces upward so as to acquire the three-dimensional data. In addition, in a case where the three-dimensional shape of the side surface of the workpiece W is acquired, the three-dimensional data can be acquired by mounting the workpiece W on the rotary stage 143 in a placement attitude in which the side surface of the workpiece W faces upward. For example, the placement attitude in which the front surface of the workpiece W faces upward can be set as a first placement attitude, and the placement attitude in which the back surface of the workpiece W faces upward can be set as a second placement attitude. In addition, the placement attitude in which the side surface of the workpiece W faces upward can be a third placement attitude. The definition of the placement attitude is an example, and the placement attitudes can be different from each other. For example, the first placement attitude, the second placement attitude, and the third placement attitude can be defined in accordance with the shape of the workpiece W, the range in which it is desired to acquire the three-dimensional data, or the like. In addition, a fourth placement attitude and a fifth placement attitude can be defined, and the number of placement attitudes is not particularly limited.

[0094] The data acquisition unit 291 acquires first three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the first placement attitude, and second three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the second placement attitude. Similarly, the data acquisition unit 291 also acquires third three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the third placement attitude, fourth three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the fourth placement attitude, and the like.

[0095] The data acquisition unit 291 acquires three-dimensional data measured by the scanning module 260. The three-dimensional data acquired by the data acquisition unit 291 includes shape information and texture information of the workpiece W, and the data acquisition unit 291 acquires shape data with texture. Therefore, the data acquisition unit 291 acquires first three-dimensional data that is three-dimensional data including the shape information and the texture information of the workpiece W placed in the first placement attitude, and second three-dimensional data that is three-dimensional data including the shape information and the texture information of the workpiece W placed in the second placement attitude.

[0096] The data acquisition unit 291 receives the light-reception signal generated by the light-reception unit 120 of the measurement unit 100, generates a real-time image of the work W on the basis of the received light-reception signal, and acquires the generated real-time image.

[0097] The mesh data generation unit 263b generates first mesh data that is mesh data of the work W placed in the first placement attitude and second mesh data that is mesh data of the work W placed in the second placement attitude. Similarly, the mesh data generation unit 263b can also generate third mesh data that is mesh data of the work W placed in the third placement attitude and fourth mesh data that is mesh data of the work W placed in the fourth placement attitude.

[0098] In a case where the mesh data generation unit 263b generates mesh data, the data acquisition unit acquires the first mesh data and the second mesh data generated by the mesh data generation unit 263b as first three-dimensional data and second three-dimensional data. Similarly, the third mesh data and the fourth mesh data can also be acquired.

[0099] For example, the three-dimensional data of the work W placed in the first placement attitude (first three-dimensional data) can be stored in the storage device 240. In this case, the read unit 292 included in the analysis module 290 reads the first three-dimensional data stored in the storage device 240. Similarly, the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data can be stored in the storage device 240. In this case, the read unit 292 reads the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data from the storage device 240.

[0100] In a case where the CAD data of the work W exists, the CAD data of the work W can be stored in the storage device 240. In this case, the read unit 292 reads the CAD data stored in the storage device 240 from the storage device 240.

[0101] Hereinafter, a scanning process in a case where the CAD data of the work W does not exist and a scanning process in a case where the CAD data of the work W exists will be described. Figure 6 An example of the scanning process in a case where the CAD data of the work W does not exist will be described. Before or after the start of this flow and before proceeding to Step SA1, the work W is mounted on the rotary stage 143 in the first placement attitude. In Step SA1, the scanning of the work W is started. In Step SA2, the measurement unit 100 scans the work W placed in the first placement attitude. For example, in a case where the work W is placed so that the front surface of the work W faces upward, since the back surface of the work W cannot be scanned, the scanning in Step SA2 is referred to as "one-side scanning". Figure 7A user interface screen 700 displayed at the start of measurement is shown. The user interface screen 700 is generated by the controller 200 and displayed on the display unit 400.

[0102] A real-time image display region 701 and a model display region 702 are provided on the user interface screen 700. In the real-time image display region 701, a real-time image generated by the data acquisition unit 291 is displayed. In the real-time image, the rotary stage 143 and the workpiece W mounted on the rotary stage 143 are displayed. Further, the measurement unit 100 can also be referred to as a scan head, and the scan head includes the light projection unit 110, the light reception unit 120, the illumination light output unit 130, and the measurement control unit 150. The display control unit 255 can also display the scan head and the workpiece W on the display unit 400. This display is effective, for example, in a case where the workpiece W is placed on any stage and the workpiece W is scanned from different angles by moving the scan head side.

[0103] In step SA2, the light projection unit 110 of the measurement unit 100 irradiates the workpiece W placed in the first placement attitude with measurement light. The light reception unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light reception signal output from the light reception unit 120 is received by the point cloud acquisition unit 263a, and first point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the first point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired first point cloud data, and converts the first point cloud data into first mesh data. Note that here, the processing of the point cloud data is the sparsification of the point cloud, the removal of point clouds outside the measurement region, the removal of noise point clouds, and the like. In step SA3, the first mesh data obtained by the processing in step SA2 is acquired as first three-dimensional data. The model of the workpiece W based on the first three-dimensional data is displayed in the model display region 702 of the user interface screen 700 shown. The user interface screen 700 is generated by the display control unit 255 included in the controller 200 and displayed on the display unit 400. Figure 7 The model display region 702 of the user interface screen 700 shown. The user interface screen 700 is generated by the display control unit 255 included in the controller 200 and displayed on the display unit 400.

[0104] In step SA4, the three-dimensional scanner 1 calculates an evaluation value and proposes a candidate attitude. The evaluation value in step SA4 is an example of an evaluation index that will be described later. When the user operates the attitude suggestion button 702a in the model display region 702, the controller 200 detects the operation. Then, as Figure 8 The display control unit 255 generates a candidate display region 703 that displays the next scannable attitude candidate and displays the generated candidate display region on the user interface screen 700, as shown. The user can select the attitude candidate on the user interface screen 700 in step SA5.

[0105] When the three-dimensional scanner 1 proposes a candidate posture, the candidate posture is calculated before the evaluation value is calculated. That is, as shown in Figure 5 The posture calculation unit 293 is a component that calculates a placement posture different from the first placement posture based on the first three-dimensional data acquired by the data acquisition unit 291, and specifically, calculates a recommended placement posture (hereinafter also simply referred to as "placement posture") different from the first placement posture based on the three-dimensional data read by the reading unit 292 from the work memory 230 or the storage device 240. The display control unit 255 superimposes and displays the recommended placement posture calculated by the posture calculation unit 293 on the real-time image acquired by the data acquisition unit 291.

[0106] When a placement posture different from the first placement posture is calculated, the posture calculation unit 293 first specifies the first placement posture of the workpiece W based on the first three-dimensional data. The first placement posture is specified, so the posture calculation unit 293 can calculate a placement posture different from the first placement posture. The distance between the measurement unit 100 and the workpiece can be set based on the focal length of the lens of the measurement unit 100. In the present embodiment, since the measurement unit 100 includes the rotary stage 143, for example, the posture calculation unit 293 can calculate a plurality of placement postures by virtually rotating the first three-dimensional data around the rotation axis of the rotary stage 143. When the first three-dimensional data is rotated, the first three-dimensional data can not be rotated once, and can be rotated at a rotation angle smaller than 360°.

[0107] Further, in the case where the workpiece W is large, there is also a case where scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times. Whether the measurement unit is moved with respect to the workpiece a plurality of times to perform scanning can be determined by the analysis module 290 based on whether the first three-dimensional data exceeds the measurable range (length, width, and height) of the measurement unit 100. In the case where the first three-dimensional data exceeds the measurable range of the measurement unit 100, it can be determined that scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times, and conversely, in the case where the first three-dimensional data is within the measurable range of the measurement unit 100, it can be determined that scanning is performed without employing a plurality of times of translating the measurement unit 100.

[0108] In the case where scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times, the measurement range of the measurement unit 100 overlaps at a certain level or higher, and the measurement unit 100 is virtually arranged in a plurality of directions based on the center point of the translation so as to include the workpiece to the greatest extent. Note that, although a method for virtually moving the measurement unit 100 has been described, the present application is not limited thereto, and the workpiece can be moved and rotated.

[0109] The analysis module 290 includes an arithmetic unit 294. The arithmetic unit 294 calculates a relative position attitude of the recommended placement attitude with respect to the first placement attitude. In a case where the attitude calculation unit 293 calculates a plurality of placement attitudes, the arithmetic unit 294 calculates a relative position attitude with respect to the first placement attitude for each of the plurality of placement attitudes. For example, the arithmetic unit 294 calculates a conversion expression for converting a relative positional relationship between the first placement attitude and the recommended placement attitude. The arithmetic unit 294 applies the conversion expression to the three-dimensional data of the first placement attitude to convert the first placement attitude into the recommended placement attitude.

[0110] Since the placement attitude is different from the first placement attitude, the attitude calculation unit 293 can also calculate the candidate attitude based on the size of the contact area with the mounting surface 142 at each of the plurality of placement attitudes. When the contact area with the mounting surface 142 is too small, it can be difficult to mount the workpiece W on the mounting surface 142. However, when the placement attitude is calculated, it is possible to stabilize the workpiece W when the workpiece W is mounted on the mounting surface 142 by calculating a placement attitude that can ensure a predetermined or more contact area with the mounting surface 142. Although Figure 8 An example in which four placement attitudes are calculated is shown, but the number of placement attitudes calculated by the attitude calculation unit 293 is not limited to four, and can be any number of one or two or more.

[0111] Figure 9 A user interface screen 710 displayed when the selection of the candidate attitude is accepted is shown, which is generated by the display control unit 255 and displayed on the display unit 400. A model display area 711 in which a model of the workpiece W based on the first three-dimensional data is displayed and a candidate display area 712 are provided on the user interface screen 710. In the candidate display area 712, four placement attitudes calculated by the attitude calculation unit 293 are displayed. The placement attitudes are displayed in the candidate display area 712, so it is possible to present the next scannable placement attitude to the user.

[0112] An evaluation index display area 712a in which an evaluation index indicating whether the placement attitude of the workpiece presented to the user is a placement attitude suitable for the next scan is provided in the candidate display area 712. The evaluation index display area 712a is also generated by the display control unit 255 and displayed on the display unit 400.

[0113] The evaluation index is based on the amount of additional data to be added to the first 3D data by performing the next scan and the amount of overlap between the 3D data acquired by performing the next scan and the first 3D data. That is, the analysis module 290 includes an overlap region estimation unit 295 and an additional data volume estimation unit 296. The overlap region estimation unit 295 estimates the overlap region between the 3D data to be acquired by the data acquisition unit 291 and the first 3D data acquired by the data acquisition unit 291, in a placement state calculated by the attitude calculation unit 293. The overlap region estimation unit 295 can also estimate the degree of feature based on the distribution of points in the overlap region.

[0114] An example of a method for estimating the overlapping region using the overlapping region estimation unit 295 will be described. Here, in the presence of 3D CAD data for the workpiece W, the outermost surface when projecting the 3D CAD data relative to the outermost surface when projecting the first 3D data acquired by the data acquisition unit 291 can be further estimated as the overlapping region. However, as... Figure 6 The flowchart shown illustrates that, in the absence of 3D CAD data for workpiece W, since estimation based on 3D CAD data cannot be performed, the front surface of the existing first 3D data is used as the overlapping region. When workpiece W has a complex shape, there is a possibility that a portion of the front surface of the existing first 3D data is hidden by the second additional scan and subsequent additional scans. When 3D CAD data exists, the effect of hiding a portion of the front surface of the existing first 3D data can be considered when estimating the overlapping region; however, this effect is not considered when 3D CAD data is unavailable. However, when the workpiece has a shape close to a convex polygon, the effect of hiding a portion of the front surface of the existing first 3D data is absent, and substantially good results can be obtained. Therefore, the quality of the overlapping region can be determined by evaluating the area (amount of overlapping region) and the degree of feature. The degree of feature is obtained by analyzing the distribution of points in the overlapping region. Specifically, the deviation of coordinates or the normal of a point can be used as an evaluation value.

[0115] Furthermore, it is necessary to consider whether the workpiece W can be placed on the mounting surface 142 of the rotating stage 143 in the placement posture calculated by the posture calculation unit 293 during actual measurement. When the measuring unit 100 moves relative to the workpiece W, the analysis module 290 can perform conflict determination between the three-dimensional shapes of the measuring unit 100 and the base 600 and the three-dimensional CAD data or first three-dimensional data of the workpiece in computer graphics space.

[0116] Further, of course, since the workpiece W cannot be placed below the mounting surface on which the measurement unit 100 is mounted, the analysis module 290 performs a collision determination with the mounting surface (Z coordinate < 0). Further, since it is difficult to arrange the measurement unit 100 at an angle from directly above the workpiece or at a low angle with respect to the workpiece, the analysis module 290 can evaluate the "arrangement easiness" depending on the arrangement angle of the measurement unit 100.

[0117] In the case where the workpiece W is moved and rotated, the analysis module 290 also determines the easiness of the placement of the workpiece W on the mounting surface 142. When the three-dimensional data is approximated by a convex polygon, the analysis module 290 calculates the area of the three-dimensional data or the bottom surface in the placement attitude calculated by the attitude calculation unit 293, and determines that the larger the calculated area of the bottom surface, the more stable the placement on the mounting surface 142 can be performed. Further, in the case where the three-dimensional CAD data of the workpiece W exists, the analysis module 290 acquires the coordinates of the center of gravity and the center point of the bottom surface, and calculates the distance between the center of gravity and the center point of the bottom surface. The analysis module 290 can determine the arrangement easiness of the workpiece W based on the distance between the center of gravity point and the center point of the bottom surface.

[0118] Next, the additional data amount will be described. The additional data amount estimation unit 296 is a component that estimates the additional data amount to be added to the first three-dimensional data by acquiring the three-dimensional data by the data acquisition unit 291 in a state where arrangement is performed in one arrangement attitude, for each of the plurality of arrangement attitudes calculated by the attitude calculation unit 293. The additional data amount estimation unit 296 can move the first three-dimensional data so as to have each placement attitude, and estimate the additional data amount based on the orientation of the normal vector after the movement, and for example, can estimate the additional data amount based on the positional relationship between the normal vector after the movement from a predetermined viewpoint and the line-of-sight direction of the measurement unit 100. For example, the inner product of the normal vector after the movement from the predetermined viewpoint and the line-of-sight direction of the measurement unit 100 is calculated, and thus the three-dimensional data facing the measurement unit 100 is specified, and the larger the amount of three-dimensional data facing the measurement unit 100, the larger the amount of additional data is estimated.

[0119] An example of the method for estimating the additional data amount by the additional data amount estimation unit 296 will be described. Assuming that scanning is performed with the attitude candidate calculated by the attitude calculation unit 293, the additional data amount is obtained by estimating how much additional data can be acquired with respect to the first three-dimensional data that has already been acquired (this can also be referred to as a "useable footage ratio"). Here, in the case where the three-dimensional CAD data of the workpiece W exists, the points that exist in the three-dimensional CAD data but do not exist in the first three-dimensional data become additional data, and the amount of additional data increases as the number of points increases.

[0120] However, as shown in the flowchart Figure 6 In the absence of three-dimensional CAD data of the workpiece W, since estimation based on three-dimensional CAD data cannot be performed, the point cloud included in the first three-dimensional data is analyzed, and the number of points of which the back surface of the scanned surface is visible is set as the additional data. This is because the back surface is not visible in the fully scanned object, and thus the portion of the back surface that is visible can be determined as the unscanned portion.

[0121] The back surface of the workpiece W can be detected by detecting that the normal of the point included in the first three-dimensional data points in the direction opposite to the measurement unit 100. With this detection process, the amount of additional data can be estimated in the absence of three-dimensional CAD data of the workpiece W. Note that the analysis module 290 can calculate the percentage of the area in which the remaining scanning needs to be performed or the scanned area of the workpiece W based on the area of the already scanned area and the amount of additional data when scanning in each direction. The analysis module 290 can also automatically determine that the scanning is complete if the scanned area exceeds a certain percentage. Either of the order of estimating the overlapping area and estimating the amount of additional data can be in the front.

[0122] After estimating the overlapping area and the amount of additional data for each placement attitude as described above, the evaluation unit 299 included in the analysis module 290 calculates an evaluation index. When calculating the evaluation index, the evaluation index can be calculated based on, for example, the following expression. Evaluation index = (area of overlapping area) x (amount of additional data) x (dispersion of points in overlapping area) x (ease of arrangement)

[0123] As described above, the evaluation unit 299 calculates the evaluation index in each of the plurality of placement attitudes based on the amount of additional data and the amount of overlapping area in each of the plurality of placement attitudes calculated by the attitude calculation unit 293, and calculates a comprehensive evaluation index based on each evaluation index. Note that in the case of using texture information, integration, average, maximum value, median, or the like of a texture feature amount based on the local contrast or differential value of the texture information can be incorporated.

[0124] The display control unit 255 displays the calculated evaluation index in numerical form or graphical form on the display unit 400. Figure 9The example illustrates the graphical display of evaluation indices in the evaluation index display area 712a. When a user performs an operation to select any of the placement postures displayed in the candidate display area 712, the analysis module 290 specifies the selected placement posture. The display control unit 255 displays the evaluation index of the placement posture specified by the analysis module 290 in graphical form in the evaluation index display area 712a. When the user selects another placement posture, the display control unit 255 displays the evaluation index of that placement posture in the evaluation index display area 712a. As described above, since the display control unit 255 can display the evaluation index of each of the multiple placement postures calculated by the posture calculation unit 293 on the display unit 400, the user can obtain the evaluation index of each placement posture when multiple placement postures are presented.

[0125] The display format of the evaluation index does not have to be... Figure 9 The display control unit 255 can display on the display unit 400 either a graphical form or a numerical form, or a combination of graphical and numerical forms. Furthermore, the display control unit 255 can display on the display unit 400 each of an evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296 and an evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit 295. In this case, the analysis module 290 calculates the first evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296, and calculates the second evaluation index separately based on the amount of overlapping region estimated by the overlapping region estimation unit 295. The display control unit 255 can display only the first evaluation index on the display unit 400 in graphical or numerical form, or only the second evaluation index on the display unit 400 in graphical or numerical form, or display both the first and second evaluation indices on the display unit 400 in graphical or numerical form respectively.

[0126] The analysis module 290 includes a designation unit 297, which designates a candidate pose from multiple placement poses calculated by the pose calculation unit 293 based on an evaluation index. Specifically, the designation unit 297 acquires the evaluation index for each of the multiple placement poses. The designation unit 297 designates the placement pose with the highest evaluation index among the acquired multiple evaluation indices. Since the evaluation index is based on the amount of additional data estimated by the additional data estimation unit 296 and the amount of overlapping region estimated by the overlapping region estimation unit 295, the designation unit 297 designates a candidate pose based on the amount of additional data and the amount of overlapping region. When designating a candidate pose, the designation unit 297 can designate a candidate pose based on the shape of the overlapping region, the amount of overlapping region, and the amount of additional data estimated by the overlapping region estimation unit 295. For example, when the overlapping region includes an irregular shape, the evaluation index can be improved compared to a flat overlapping region because the alignment accuracy is higher than that of an overlapping region with a flat shape.

[0127] When the designated unit 297 identifies a candidate posture, the display control unit 255 displays the candidate posture designated by the designated unit 297 on the display unit 400. The candidate posture displayed by the display unit 400 is the workpiece placement posture recommended by the 3D scanner 1 to the user. Therefore, the user can confirm the appropriate placement posture by viewing the display unit 400.

[0128] exist Figure 6 In step SA6, the user determines whether the candidate poses displayed on the display unit 400 are the desired poses. Since the placement pose is specified based on an evaluation index, and due to the large amount of additional data and overlapping area, it is believed that accurate combined 3D data can be obtained by placing the workpiece in this pose. However, since the designation unit 297 only specifies the placement pose based on the evaluation index, it may not necessarily specify the user's highly relevant area (the area requiring high-precision 3D data). Therefore, in the 3D scanner 1 of this embodiment, multiple placement poses are presented to the user along with evaluation indices, and the user can select the placement pose with a relatively high evaluation index and best suited to their interests from the presented multiple placement poses. Specifically, the analysis module 290 includes a receiving unit 298 that accepts the user's selection or adjustment operation for the placement pose. For example, when the user performs a selection or adjustment operation from... Figure 9 When selecting a desired placement posture from the candidate display areas 712 displayed on the user interface screen 710, the receiving unit 298 accepts the selected operation input. The display control unit 255 displays the placement posture accepted by the receiving unit 298 in the candidate display area 712.

[0129] Further, in step SA7, the candidate posture displayed on the display unit 400 can be adjusted. Specifically, the accepting unit 298 accepts an operation input of the user for adjusting one of the candidate postures specified by the specifying unit 297. The user operates the operation unit 250 in the computer graphics space to adjust the positional relationship between the measurement unit 100 and the workpiece W. For example, the workpiece W in the candidate posture is moved horizontally, moved in the height direction, or rotated. When the user completes the adjustment of the candidate posture, the process proceeds to step SA8, and the posture calculating unit 293 specifies the placement posture for which the adjustment is completed, the overlap area estimating unit 295 estimates the amount of the overlap area between the three-dimensional data acquired by the data acquiring unit 291 in the placement posture for which the adjustment is completed and the first three-dimensional data, and the additional data amount estimating unit 296 estimates the amount of the additional data. The evaluation index is recalculated by the analysis module 290 based on the estimated amount of the overlap area and the estimated amount of the additional data, and the evaluation index is presented to the user. Thus, in the case where the user scans the region of high interest, the user can determine whether the posture is likely to be successful in the alignment with the first three-dimensional data. Note that the evaluation value calculation and the posture candidate suggestion in step SA4 and the selection of the posture candidate in step SA5 can be skipped, and the posture calculating unit 293 can specify the placement posture for which the user completes the adjustment of the candidate posture in step SA7.

[0130] In step SA9, the analysis module 290 generates a computer graphics workpiece (model) in the determined placement posture. The display control unit 255 superimposes the computer graphics workpiece generated by the analysis module 290 on the real-time image acquired by the data acquiring unit 291 and displays the superimposed image on the display unit 400. The computer graphics workpiece is semitransparent, but can also be opaque.

[0131] In step SA10, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the workpiece W is captured together with the mounting surface 142 by the light receiving unit 120, and a real-time image including the workpiece W is acquired by the data acquiring unit 291 and displayed on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W mounted on the mounting surface 142 overlaps the computer graphics workpiece.

[0132] In step SA11, the user determines whether the actual workpiece W installed on the installation surface 142 is installed so as to overlap the computer graphics workpiece. In a case where the actual workpiece W installed on the installation surface 142 cannot be installed so as to overlap the computer graphics workpiece, the processing proceeds to step SA5, and another placement attitude is selected. In a case where the actual workpiece W is installed so as to overlap the computer graphics workpiece, since the workpiece W is in the second placement attitude, the processing proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement attitude.

[0133] In step SA12, the light projection unit 110 of the measurement unit 100 irradiates the workpiece W placed in the second placement attitude with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained by the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0134] In step SA14, the alignment unit 290A included in the analysis module 290 aligns the first three-dimensional data (three-dimensional data of the workpiece placed in the first placement attitude) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. At the time of this alignment, the overlapping region extracted by the extraction unit 290B included in the analysis module 290 is used. The extraction unit 290B is a component that extracts the overlapping region between the three-dimensional data converted to the recommended placement attitude by the arithmetic unit 294 and the three-dimensional data placed in the second placement attitude. The extraction unit 290B can extract the overlapping region by using, for example, the normal vector of the three-dimensional data and the color information of the workpiece. That is, the alignment unit 290A can perform alignment based on the three-dimensional data included in the overlapping region extracted by the extraction unit 290B. In a case where alignment is performed by using the texture information of the workpiece W, the texture feature is estimated based on the brightness or color information or the local contrast or difference that constitutes the texture information. This texture feature can be used in combination with the shape feature by various operations such as addition and multiplication.

[0135] In the present embodiment, the evaluation index of the placement attitude is calculated so that the amount of the overlapping region and the amount of the additional data are large at the time of the next scan, and so that the alignment of the three-dimensional data by the alignment unit 290A is likely to be successful. Then, while the user is presented with the placement attitude whose evaluation index is high, the final placement attitude is determined while the user adjusts the placement attitude, and the placement attitude is superimposed and displayed on the real-time image so that the user can arrange the actual workpiece W in an attitude close to the placement attitude. Thus, the alignment of the placement attitude is performed as the initial position, and therefore the success rate of the alignment can be increased without imposing an additional burden on the user.

[0136] The alignment unit 290A can acquire a normal line of the three-dimensional data of the workpiece W placed in the second placement attitude. The alignment unit 290A can also perform alignment based on the orientation of the normal line of the three-dimensional data of the workpiece W placed in the second placement attitude, the relative positional relationship calculated by the arithmetic unit 294, the orientation of the normal line of the three-dimensional data of the workpiece placed in the first placement attitude, and the shape features extracted from the first three-dimensional data and the second three-dimensional data. That is, the alignment unit 290A first narrows down the range of the candidate corresponding points based on the angle between the orientation of the normal line obtained by rotating the normal line of the three-dimensional data of the workpiece W placed in the first placement attitude based on the relative positional relationship and the orientation of the normal line of the three-dimensional data of the workpiece placed in the second placement attitude. Then, based on the corresponding point candidates after the range is narrowed down, alignment can be performed by using the shape features extracted from the first three-dimensional data and the second three-dimensional data.

[0137] In addition, when the user places the workpiece W, a high-precision initial estimate value can also be obtained by pattern matching between the computer graphics workpiece and the actual workpiece W through computer graphics. This is because the relative position between the measurement unit 100 and the workpiece W can be obtained from the correspondence relationship between the computer graphics workpiece and the actual workpiece W in the real-time image through epipolar geometry.

[0138] The initial estimate value can be used to detect and extract a region where two point clouds overlap, detect an erroneous correspondence relationship of the corresponding point candidates, and detect an erroneous candidate from the position attitude candidates obtained from the corresponding point candidates. In the case where the alignment source point cloud P in the first three-dimensional data that has been scanned is aligned with the alignment destination point cloud Q that is newly scanned, the alignment unit 290A can specify the corresponding points of the points included in the first three-dimensional data from the second three-dimensional data, and adjust the position attitude of the first three-dimensional data and the second three-dimensional data based on the specified corresponding points.

[0139] In Figure 11In step SB1 of the flowchart shown, the arithmetic unit 294 acquires the alignment source point cloud P, and in step SB2, the arithmetic unit 294 acquires the alignment destination point cloud Q. In step SB3, the arithmetic unit 294 acquires the initial estimated pose. In step SB4, the arithmetic unit 294 performs a transformation for the alignment source by using the placement pose determined as described above. This transformation can be performed by multiplying by a rotation matrix and adding a translation component. Thus, the transformed point cloud P' exists in substantially the same position as the point cloud of the alignment destination (step SB5).

[0140] In step SB6, the extraction unit 290B can extract Qo by extracting a point cloud P'o in which the points of the point cloud Q exist in the vicinity after the transformation and applying the point cloud P'o in reverse. These point clouds are referred to as overlapping region point clouds. Note that in the case where normal vectors or color information are assigned to the point clouds, the accuracy of the overlapping region can be changed by using the presence or absence of a point cloud with a high degree of similarity in the vicinity as a condition for extraction of the overlapping region. Furthermore, the definition of "vicinity" varies according to the degree to which the user can arrange the actual workpiece W to be close to the workpiece in the computer graphics in the position pose. In addition, the range of "vicinity" can be determined according to the scanning range of the measurement unit 100 or the size of the workpiece W, and the range of "vicinity" can be changed by the user.

[0141] The alignment unit 290A performs alignment using only the overlapping region extracted by the extraction unit 290B. Thus, the occurrence of erroneous correspondence of points present in the non-overlapping region can be eliminated.

[0142] However, the extraction of the overlapping region by the extraction unit 290B requires a computational cost. This is because, for example, when extracting the point cloud P'o, the point Q closest to each point of the point cloud P' is searched for positioning, and assuming that the number of points P is M and the number of points Q is N, the computational cost O is O(M*N).

[0143] Thus, the occurrence of erroneous correspondence can be suppressed by hiding surface removal as an alternative to the overlapping region. This is because the above-described initial estimate is a value obtained by shape analysis to obtain an overlapping region suitable for alignment. Each of the point clouds of P' and Q is projected by a virtual camera based on the initial estimate, and the hidden surface removal based on the depth buffer generally used in computer graphics (only extracting a point whose depth position viewed from the camera is the minimum value) can eliminate data of a back surface as erroneous correspondence.

[0144] Further, simple hidden surface removal using normal vectors can be performed as a simpler method. The hidden surface removal can be performed by extracting only points directed in a direction facing the measurement unit 100 from each of the point clouds P' and Q in the camera coordinate system. In the normal vector [nx, ny, nz] in the camera coordinate system, only the normal vector in which nz is negative is extracted. Thus, the calculation cost can be reduced.

[0145] Figure 12 is a schematic view related to the hidden surface removal. A white arrow W1 extending in a direction orthogonal to each surface of the workpiece W arranged on the mounting surface 142 indicates a normal vector of each surface. Further, a thin arrow W2 represents a component of the normal vector parallel to the Z axis of the light receiving unit 120, and an arrow W3 represents a component of the normal vector perpendicular to the Z axis of the light receiving unit 120. In the arrow W1, a dotted line indicates a normal in which the Z axis component of the light receiving unit 120 is determined to be positive, and a surface corresponding to the dotted arrow W1 is determined to be a hidden surface.

[0146] In Figure 11 In the step SB8 shown, the extraction unit 290B extracts a shape feature and a brightness feature of each point from the first three-dimensional data corresponding to the first placement attitude and the second three-dimensional data corresponding to the second placement attitude acquired by the data acquisition unit 291.

[0147] In the step SB10, the feature amount acquired in the step SB9 is input to the extraction unit 290B to detect the corresponding point. In the absence of the initial estimated value, a candidate of the corresponding point is obtained by a feature difference between the first three-dimensional data and the second three-dimensional data, a correlation between the first three-dimensional data and the second three-dimensional data, and the like. At this time, a point initially present in the distance but having similar features in the shape and the brightness can be selected as the corresponding point. On the other hand, there is an initial estimated value, and thus only points in the vicinity and points in which the orientation of the normal vector is close can be the corresponding candidates, and a correlation value of the features can be weighted based on the distance between the points and the inner product of the normal vectors.

[0148] Assuming that the i-th point of the point cloud P of the alignment source is pi, the point obtained by converting pi to the initial estimated value is p'i, the j-th point of the point cloud Q of the alignment destination is qj, the feature vector thereof is F(*), the normal vector thereof is N(*), the evaluation value w_d(dist(p'i, qj))*wn(1-dot(N(p'i), N(qj))*CORR(F(p'i), FU(qj)) can be obtained. wd is a weight related to the distance, and wn is a weight related to the orientation of the normal. When this weight is a step function of [1, 0], only the proximity weight is selected, and when this weight is a monotonically decreasing function, a continuous weight can be applied. Points having a high evaluation value are set as the corresponding points, and thus, it is possible to eliminate false corresponding points having similar shapes and brightness distributions. A matrix C(pi, qj) representing the corresponding point candidates is obtained as the output of the corresponding point detection unit. It can be represented by a matrix in which the points determined as the corresponding points are 1, and the other points are 0.

[0149] In step SB11, a candidate pose is acquired from these corresponding candidate points. A method for obtaining the final placement pose of the workpiece W in steps SB12 to SB17 will be described. For example, a random sample consensus (RANSAC) method can be used as the method for obtaining the final placement pose of the workpiece W. Hereinafter, an example using RANSAC will be described. In the pose candidate calculation of RANSAC, a candidate is calculated from a plurality of randomly extracted corresponding points (step SB12). As a result, a candidate pose can be acquired (step SB13). In the case of using only the coordinates of the points, the corresponding relationship between three points is required, and in the case of using the normal and the coordinates of the points, the corresponding relationship between two points is required. Taking the corresponding relationship between three points as an example, a rotation matrix R and a translation vector t that minimize the squared distance between the corresponding p' and q are obtained by linear and nonlinear optimization (step SB14).

[0150] Sum_C(p_i,q_j)(q_j-p'_i*R+t)^2

[0151] The present embodiment is not limited to this candidate calculation method, and a candidate can be calculated by using any method. For example, a method using graph cuts is also applicable to the present embodiment.

[0152] It is determined whether the rotation matrix and the translation vector obtained in this way are close to the initial estimate value to detect whether a correct candidate is obtained. When the sum of squares of each element is used, the initial estimate value is t, and the candidate is t', the translation vector can be defined as dist(t, t') = (t_x - t'_x)^2 + (t_y - t'_y)^2 + (t_z - t'_z)^2. In addition, the rotation matrix can be obtained as Dist(R, R') = arccos[(trace(R^T * R') - 1) / 2].

[0153] A placement attitude in which the error of the translation component is within a predetermined size and the angular error is within a predetermined angle can be extracted as a correct candidate. In step SB16, a detected attitude is obtained from these candidates by the final attitude selection process, and a selected candidate is acquired (step SB17). This is obtained by a method for evaluation by a predetermined evaluation value. For example, by using the rotation matrix R and the translation component t representing the candidate attitude, a conversion is performed as P" = R * P' + t, and the number of points in the point cloud Q that satisfy dist(P" - Q) < threshold can be set as the evaluation value. However, the calculation of the evaluation value to be combined in this embodiment is not particularly limited. Since this evaluation value calculation can also be performed only in the overlapping region, it is possible to suppress the phenomenon in which the evaluation value is erroneously increased due to the presence of a corresponding point in its vicinity although some points do not overlap. In this way, the second placement attitude is specified, and the workpiece W placed at the specified second placement attitude is scanned by the measurement unit 100. Therefore, the second three-dimensional data having high-precision alignment with the first three-dimensional data and the large amount of additional data can be acquired.

[0154] When performing alignment between the first three-dimensional data and the second three-dimensional data, the alignment unit 290A can perform alignment based on shape features and image features of the three-dimensional data. That is, the analysis module 290 includes a shape feature extraction unit (first extraction unit) 290C that extracts shape features of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291, and an image feature extraction unit (second extraction unit) 290D. The shape feature extraction unit 290C includes a neural network including an input layer, a plurality of intermediate layers, and an output layer. The input layer of the neural network is a component that accepts input of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291. The plurality of intermediate layers of the neural network are components that extract shape features based on the input accepted by the input layer. The output layer of the neural network is a component that outputs the shape features extracted in the intermediate layers. On the other hand, the image feature extraction unit 290D is a component that extracts image features from texture information included in each of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0155] As described above, in the present embodiment, when alignment between the first three-dimensional data and the second three-dimensional data is performed, image features such as brightness and color are incorporated into the feature vector using deep learning. In the case where the feature vector of each point of the image or the point cloud is calculated, a "convolution" arithmetic operation of multiplying the coordinates and the color, which are information of the own point and the information of the peripheral point, by a coefficient and obtaining a sum or a maximum value thereof can be performed. For example, in a convolutional neural network (CNN) in an image, a large number of 3x3 filters having coefficients determined by pre-learning are provided, the output value of each filter is set, and the set output vector is set as the output vector of the point, and a result obtained by repeating the output vector with different filters several times is set as the feature vector.

[0156] In the case where the point cloud is included in the first three-dimensional data or the second three-dimensional data, since it cannot be guaranteed that the data exists at a regular interval different from the image, other methods can be used. For example, a method for applying a CNN three-dimensionally by sampling points in a voxel, multiplying the points by a coefficient when the points exist, and outputting 0 when the points do not exist, PointNet++ for applying the same coefficient to each point to obtain a maximum value of the output, and a method such as KPConv for performing an interpolation arithmetic operation with respect to a deviation of the position of the point cloud from a coefficient determined on a grid to obtain a coefficient and convolving the coefficient can be exemplified.

[0157] In the processing of the point cloud included in the first three-dimensional data or the second three-dimensional data, since how the points are arranged is unknown, the amount of calculation becomes enormous and the processing time becomes long. In the technical field of the three-dimensional scanner 1 in this example, when the processing time becomes long, since the work is delayed and not performed, it is necessary to set the processing time without a problem in actual use. Therefore, the number of points that can be processed is naturally limited, and as an example, an upper limit of the number of key points can be set to several thousand points, and about several ten thousand points in the entire point cloud. Since the number of point clouds that can be scanned at once by the measurement unit 100 is several million points, the points used in deep learning need to be thinned to about 1%.

[0158] In the case where the image information is most simply input to the input layer of the neural network, the image information (brightness and color) can be input in addition to the coordinates and the normal vector, but in this case, the number of data input to the input layer is too small, and the information useful as the image is lost. Therefore, there is a case where the performance is not improved.

[0159] Hereinafter, the present embodiment will be described with reference to Figure 13The flowchart shown describes the process of extracting shape features and image features. In step SCI, the analysis module 290 acquires a point cloud of the first three-dimensional data or the second three-dimensional data (input point cloud). In steps SC2 and SC3, a rough thinning process and a dense thinning process are performed, respectively. That is, as shown in Figure 5 As shown, the analysis module 290 includes a resolution conversion unit 290E that converts the resolution of the first three-dimensional data and the resolution of the second three-dimensional data acquired by the data acquisition unit 291. The resolution conversion unit 290E performs a first conversion process (rough thinning process) that converts the first three-dimensional data and the second three-dimensional data into three-dimensional data having a first resolution lower than the resolutions acquired by the data acquisition unit 291. That is, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into third three-dimensional data as three-dimensional data having the first resolution, and converts the second three-dimensional data acquired by the data acquisition unit 291 into fourth three-dimensional data as three-dimensional data having the first resolution. In the first conversion process by the resolution conversion unit 290E, since the resolutions of the three-dimensional data acquired by the data acquisition unit 291 are reduced to low-resolution three-dimensional data, the number of points of the three-dimensional data is smaller than the number of points at the time of acquisition by the data acquisition unit 291.

[0160] The first conversion process of the resolution conversion unit 290E corresponds to step SC2. The third three-dimensional data (third point cloud) and the fourth three-dimensional data (fourth point cloud) as the shape feature extraction point cloud (Q) are generated by step SC2, and the shape feature extraction unit 290C acquires the shape feature extraction point cloud (Q) in step SC4. The shape feature extraction unit 290C extracts shape features from shape information included in each of the first three-dimensional data having the first resolution and the second three-dimensional data having the first resolution converted by the resolution conversion unit 290E.

[0161] The resolution conversion unit 290E performs a second conversion process (dense sparsification process) of converting the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291 into three-dimensional data having a second resolution lower than the resolution acquired by the data acquisition unit 291 and higher than the first resolution. That is, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into fifth three-dimensional data as three-dimensional data having the second resolution, and converts the second three-dimensional data acquired by the data acquisition unit 291 into sixth three-dimensional data as three-dimensional data having the second resolution. In the second conversion process by the resolution conversion unit 290E, since the resolution of the three-dimensional data acquired by the data acquisition unit 291 is lowered to low-resolution three-dimensional data, the number of points of the three-dimensional data is smaller than the number of points at the time of acquisition by the data acquisition unit 291, but since the resolution is higher than the resolution of the first conversion process, the number of points is larger than the number of the first conversion process. The second conversion process of the resolution conversion unit 290E corresponds to step SC3.

[0162] The fifth three-dimensional data (fifth point cloud) and the sixth three-dimensional data (sixth point cloud) (Q') generated as the image feature extraction point cloud by step SC3 are acquired by the image feature extraction unit 290D in step SC5. The image feature extraction unit 290D extracts an image feature from the texture information included in each of the first three-dimensional data and the second three-dimensional data having the second resolution converted by the resolution conversion unit 290E.

[0163] Note that the above-described step SC3 can be skipped, and the image feature extraction unit 290D can generate the image feature extraction point cloud by using the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0164] In the first conversion process and the second conversion process, for example, random sparsification for randomly selecting a predetermined number of data, voxel sparsification for calculating a voxel grid and reducing a plurality of point clouds entering a voxel to only one average value, or the like can be used. Here, the point cloud before sparsification (point cloud before the second conversion process) can be used as a point cloud for image feature extraction.

[0165] In step SC6, the shape feature extraction unit 290C extracts a key point from the first three-dimensional data and the second three-dimensional data of low resolution. For example, when extracting a key point, there are a method for randomly selecting and extracting a key point, a method for extracting a feature point as a key point by deep learning or rule-based point cloud analysis, or the like, and any method can be used. The key point extracted by the shape feature extraction unit 290C is acquired by the shape feature extraction unit 290C (step SC7).

[0166] In step SC8, the analysis module 290 acquires the shape feature extraction point cloud (Q) acquired in step SC4 and the key point acquired in step SC7, and the analysis module 290 samples points around the key point. At this time, the points for shape feature extraction are sampled based on the shape feature extraction point cloud (Q) acquired in step SC4.

[0167] In step SC12, which proceeds from step SC5, the analysis module 290 acquires the image feature extraction point cloud (Q') acquired in step SC5 and the key point acquired in step SC7, and the analysis module 290 samples points around the key point. The points for image feature calculation are sampled based on the image feature extraction point cloud (Q') acquired in step SC5. In addition, since the number of points of the image feature extraction point cloud (Q') is large, the vicinity range to be sampled can be narrowed compared to the shape feature extraction point cloud (Q).

[0168] In step SC9, which proceeds from step SC8, the shape feature is input to the shape feature extraction unit 290C. For example, the shape feature vector can be calculated only from the vicinity of the point sampled for the shape feature. The method for calculating the shape feature vector is not particularly limited, and for example, a method for projecting image information of a point cloud according to the normal of a key point to obtain a patch image and calculating a feature vector by a calculation such as CNN or scale-invariant feature transform (SIFT), a method for creating a histogram based on RGB or luminance information of a sampled point and using the histogram as a feature vector, or the like can be used. In step SC10, the shape feature extraction unit 290C calculates the shape feature vector of the key point.

[0169] On the other hand, in step SC13, which proceeds from step SC12, the image feature is input to the image feature extraction unit 290D. Here, the feature vector of the image is calculated. The feature vector of the key point is calculated by using all the points around the key point sampled from the image feature extraction point cloud (Q'). In step SC14, the image feature extraction unit 290D calculates the image feature vector of the key point, and in step SC15, the image feature vector calculated by the image feature extraction unit 290D is acquired.

[0170] That is, the image feature extraction unit 290D extracts a key point to be used for calculating a shape feature from the first three-dimensional data and the second three-dimensional data at low resolution. The image feature extraction unit 290D can specify a region corresponding to the key point extracted from the first three-dimensional data and the second three-dimensional data before conversion by the resolution conversion unit 290E. The image feature extraction unit 290D extracts an image feature from the specified corresponding region.

[0171] Note that when the eigenvector of the image is calculated, the image feature extraction unit 290D can specify a region corresponding to the key point extracted by the shape feature extraction unit 290C from the first three-dimensional data and the second three-dimensional data, and extract the image feature from the region corresponding to the key point specified by the image feature extraction unit 290D.

[0172] In the calculation of the shape eigenvector in step SC11, a three-dimensional to six-dimensional vector having X, Y, and Z coordinates and normal information Nx, Ny, and Nz if necessary can be input. As described above, the shape feature extraction unit 290C extracts a key point from the low-resolution first three-dimensional data and the second three-dimensional data as a partial region for calculating a shape feature, and extracts a shape feature for the extracted key point.

[0173] In step SC16, a candidate pair is obtained by comparing the shape eigenvector, the image eigenvector, or the shape eigenvector including the image feature with the key points of the three-dimensional data of the alignment source and the three-dimensional data of the alignment destination. The method for calculating the candidate pair is not particularly limited, and examples thereof include a method for obtaining a distance of the eigenvectors and making a pair with a small distance, and a method for making a pair with a high output value of a deep learning module that outputs a degree of correspondence. A squared Euclidean distance (L2 distance) or a cosine similarity is used as the distance. Furthermore, at this time, in order to compare the shape eigenvector and the image eigenvector, a method for simply obtaining a pair by using a vector obtained by combining the two vectors as input, a sum, a maximum value, a minimum value, and the like of distances calculated from the shape eigenvector and the image eigenvector can be used. Since the points of the candidate pair are used, a position relationship (rotation and translation) between the point clouds can be obtained by RANSAC or a deep learning-based method.

[0174] Figure 14 A case in which a vector to which an image feature is added is input to the shape feature extraction unit in the case where the shape feature and the image feature are extracted is shown. The processing proceeds to step SC9' in step SC8 in Figure 13 , and the processing proceeds to step SC13' in step SC12 in Figure 14 , and the processing proceeds to step SC9' in step SC8 in Figure 13 , and the processing proceeds to step SC13' in step SC12 in Figure 14 . Steps SC13' to SC15' are the same as steps SC13 to SC15 in Figure 13 . In step SC10', the image eigenvector is input to the shape feature extraction unit 290C. In this case, the output shape eigenvector is a shape eigenvector that already includes the image feature (step SC11'). Step SC16' is the same as step SC16 in Figure 13 .

[0175] The alignment unit 290A performs a process of combining the global alignment and the local alignment as a process of aligning the point cloud of the first three-dimensional data and the point cloud of the second three-dimensional data with high precision. In this process, the global alignment is performed by the low-resolution point cloud, and the local alignment is performed by using the position for conversion. For example, the local alignment is performed by using the point cloud data with the first resolution (low resolution) for shape feature extraction in the global alignment, using the point cloud with a third resolution higher than the first resolution for image feature extraction, and using the point cloud data with a second resolution higher than both the first resolution and the third resolution as input. Note that the point cloud data for the local alignment can be the point cloud data with the second resolution, or can be the point cloud data with the same resolution as that of the point cloud generated by receiving the light-receiving signal in the point cloud acquisition unit 263a.

[0176] Specifically, the alignment unit 290A acquires the shape feature extracted by the shape feature extraction unit 290C and the image feature extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A can perform the global alignment based on the shape feature extracted by the shape feature extraction unit 290C and the image feature extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A performs a first alignment process of calculating a low-precision alignment parameter indicating the relative position attitude of the second three-dimensional data with respect to the first three-dimensional data.

[0177] When the low-precision alignment parameter is calculated, steps SD1 to SD7 are performed as shown in Figure 15 Steps SD1 to SD5 are the same as steps SC1 to SC5 shown in Figure 13 In step SD6 shown in Figure 15 , the shape feature extraction point cloud acquired in step SD4 and the image feature extraction point cloud acquired in step SD5 are input to the alignment unit 290A. The alignment unit 290A performs the global alignment of the input point clouds and calculates the low-precision alignment parameter.

[0178] After the first alignment process, the alignment unit 290A acquires the low-precision alignment parameter calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data. The alignment unit 290A can perform the local alignment based on the low-precision alignment parameter, the first three-dimensional data, and the second three-dimensional data. In this case, the alignment unit 290A performs a second alignment process of calculating a high-precision alignment parameter indicating the relative position attitude of the second three-dimensional data with respect to the first three-dimensional data.

[0179] When the high-precision alignment parameter is calculated, steps SD8 to SD14 are performed as shown in Figure 16As shown, in step SE1, the analysis module 290 acquires an input point cloud of the first three-dimensional data. In step SE2, the analysis module 290 acquires an input point cloud of the second three-dimensional data. In step SE3, the analysis module 290 acquires a low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. Figure 15 The process of the flowchart shown calculates the low-precision alignment parameter.

[0180] The analysis module 290 includes a coordinate conversion unit 290F that performs coordinate conversion of the first three-dimensional data based on the low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. In step SE4, the coordinate conversion unit 290F performs coordinate conversion of the input point cloud of the first three-dimensional data based on the low-precision alignment parameter. In step SE5, a point cloud after the coordinate conversion is acquired. Figure 15 The process of the flowchart shown calculates the low-precision alignment parameter.

[0180] The analysis module 290 includes a coordinate conversion unit 290F that performs coordinate conversion of the first three-dimensional data based on the low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. In step SE4, the coordinate conversion unit 290F performs coordinate conversion of the input point cloud of the first three-dimensional data based on the low-precision alignment parameter. In step SE5, a point cloud after the coordinate conversion is acquired.

[0181] In step SE6, the input point cloud of the second three-dimensional data acquired in step SE2 and the point cloud after the coordinate conversion acquired in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point clouds and calculates a high-precision alignment parameter. In step SE7, the high-precision alignment parameter is acquired. As described above, in the second alignment process, the alignment unit 290A acquires the first three-dimensional data and the second three-dimensional data of which coordinate conversion is performed by the coordinate conversion unit 290F, and performs local alignment based on the first three-dimensional data and the second three-dimensional data.

[0182] In step SE6, the input point cloud of the second three-dimensional data acquired in step SE2 and the point cloud after the coordinate conversion acquired in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point clouds and calculates a high-precision alignment parameter. In step SE7, the high-precision alignment parameter is acquired. As described above, in the second alignment process, the alignment unit 290A acquires the first three-dimensional data and the second three-dimensional data of which coordinate conversion is performed by the coordinate conversion unit 290F, and performs local alignment based on the first three-dimensional data and the second three-dimensional data. Figure 6 After the alignment process in step SA14 shown above, a combination process is performed. That is, as shown in FIG. 8, the analysis module 290 includes a combination unit 290G. The combination unit 290G is a component that combines the first three-dimensional data of the workpiece W placed in the first placement attitude and the second three-dimensional data of the workpiece W placed in the second placement attitude, which are aligned by the alignment unit 290A, to generate combined three-dimensional data. Figure 5 As shown, the analysis module 290 includes a combination unit 290G. The combination unit 290G is a component that combines the first three-dimensional data of the workpiece W placed in the first placement attitude and the second three-dimensional data of the workpiece W placed in the second placement attitude, which are aligned by the alignment unit 290A, to generate combined three-dimensional data. The combination unit 290G can also acquire the first mesh data as the first three-dimensional data, and acquire the second mesh data as the second three-dimensional data. In this case, the combination unit 290G combines the first mesh data and the second mesh data to generate combined mesh data as the combined three-dimensional data.

[0183] In addition, Figure 12The removal of the hidden surface shown is performed before generating the combined three-dimensional data, and is used for the alignment by the alignment unit 290A. That is, the alignment unit 290A can perform the alignment by using the three-dimensional data from which the hidden surface is removed. Then, the combined unit 290G generates the combined three-dimensional data based on the alignment performed by using the three-dimensional data from which the hidden surface is removed. Here, the generation of the combined three-dimensional data is performed before the hidden surface removal, that is, using the three-dimensional data including the hidden surface.

[0184] The combined unit 290G can acquire the shape information of the workpiece W based on the first light-reception signal output from the light-reception unit 120, and can acquire the texture information of the workpiece W based on the second light-reception signal output from the light-reception unit 120. In a case where the shape information of the workpiece W and the texture information of the workpiece W are acquired, the combined unit 290G generates three-dimensional data including the shape information and the texture information of the workpiece W.

[0185] In addition, in a case where the data acquisition unit 291 acquires the combined three-dimensional data obtained by combining the first three-dimensional data and the second three-dimensional data, the attitude calculation unit 293 can calculate the placement attitude different from the first placement attitude based on the combined three-dimensional data acquired by the data acquisition unit 291. As described above, in addition to the already generated combined three-dimensional data, the placement attitude at the time of performing the scanning can be presented to the user. In this case, the additional data amount estimation unit 296 estimates the additional data amount to be added to the combined three-dimensional data. In addition, the overlapping region estimation unit 295 estimates the overlapping region of the combined three-dimensional data. Thus, the evaluation index based on the additional data amount and the overlapping region amount can be presented to the user at the time of performing the scanning in the additional placement attitude.

[0186] When the combination processing is completed, the processing proceeds to Figure 6 The step SA15 shown. In the step SA15, the user determines whether there is a portion that needs to be scanned in the workpiece W. When there is a portion that needs to be scanned on the workpiece W, the processing proceeds to the step SA2, and the workpiece W is placed on the mounting surface 142 in the third placement attitude and scanned by the measurement unit 100. By repeating this, all necessary scanning portions can be scanned. In a case where NO is determined in the step SA15, the full-surface scan model is acquired in the step SA16. Note that the full-surface scan model is not necessary, and any model can be used as long as the user can scan the necessary portion.

[0187] Next, the reference Figure 17The flowchart describes an example of a scanning process in the case where there is a reference model such as CAD data or measured three-dimensional data of a workpiece W. In step SG1, the read unit 292 of the analysis module 290 reads the CAD model (reference model) from the storage device 240, and acquires the CAD model in step SG2. In step SG3, the evaluation value calculation and pose candidate suggestion are performed in a manner similar to step SA4. Figure 6 The evaluation value calculation and pose candidate suggestion are performed in a manner similar to step SA4. In step SG3, since there is CAD data, the pose calculation unit 293 analyzes the CAD data read by the read unit 292. The pose calculation unit 293 estimates the amount of data acquired (additional data amount) by analyzing the CAD data, and calculates the recommended placement pose on the basis of the estimated amount of data. In addition, the pose calculation unit 293 analyzes the CAD data read by the read unit 292, and estimates the overlapping region between the three-dimensional data and the CAD data acquired in the case where placement is performed in the recommended placement pose. The pose calculation unit 293 calculates the recommended placement pose on the basis of the estimated overlapping region.

[0188] In step SG4, for example, the user selects the candidate pose on the user interface screen 700 as shown in Figure 8 In step SG5, the user determines whether the candidate pose displayed on the display unit 400 is the desired pose. In the case where the pose is not the desired pose, the process proceeds to step SG6, and the placement pose of the workpiece W is adjusted according to computer graphics. In step SG7, the overlapping region estimation unit 295 estimates the amount of the overlapping region in the adjusted placement pose, and the additional data amount estimation unit 296 estimates the additional data amount. The analysis module 290 recalculates the evaluation index on the basis of the estimated amount of the overlapping region and the estimated amount of additional data, and presents the evaluation index to the user.

[0189] In the case where it is determined in step SG5 that the pose is the desired pose, the process proceeds to step SG8, and the analysis module 290 generates a computer graphic workpiece (model) in the determined placement pose. The display control unit 255 superimposes the computer graphic workpiece generated by the analysis module 290 on the real-time image acquired by the data acquisition unit 291, and displays the superimposed image on the display unit 400.

[0190] In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires and displays a real-time image on the display unit 400. The user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the computer graphic workpiece.

[0191] In step SG10, the user determines whether the actual workpiece W is installed to overlap the computer graphics workpiece. In a case where the actual workpiece W cannot be installed to overlap the computer graphics workpiece, the processing proceeds to step SG4, and another placement attitude is selected. In a case where the actual workpiece W is installed to overlap the computer graphics workpiece, since the workpiece W is in the first placement attitude, the processing proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement attitude. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the first three-dimensional data and the CAD data as a reference model. In step SG14, the analysis module 290 acquires the aligned CAD model.

[0192] In step SG15, evaluation value calculation and attitude candidate suggestion are performed. In step SG16, the user selects a candidate attitude, and then the processing proceeds to step SG17. In a case where the attitude is not a desired attitude, after adjustment in step SG18, the evaluation value is updated in step SG19, and then the processing proceeds to step SG17. In a case where the attitude is a desired attitude, the processing proceeds to step SG20, and the display control unit 255 superimposes the computer graphics workpiece generated by the analysis module 290 on the real-time image acquired by the data acquisition unit 291, and displays the superimposed image on the display unit 400. Here, the computer graphics workpiece can be the first three-dimensional data acquired in step SG12 or the reference model. In addition, the first three-dimensional data and the reference model can be displayed on the display unit 400 at the same time, or can be switched and displayed.

[0193] In step SG21, the user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the computer graphics workpiece.

[0194] In step SG22, the user determines whether the actual workpiece W is installed to overlap the workpiece of the computer graphics. In a case where the actual workpiece W cannot be installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG16, and another placement attitude is selected. In a case where the actual workpiece W is installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement attitude (second placement attitude). In step SG24, the alignment unit 290A performs alignment with the acquired second three-dimensional data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the three-dimensional model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a portion that needs to be scanned. When there is a portion that needs to be scanned, the processing proceeds to step SG12. When there is no scan portion, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model on which the alignment processing is performed by the alignment unit 290A in step SG26. Further, the full-surface scan model and the CAD model can be displayed on the display unit 400 in an aligned state.

[0195] In the present embodiment, as shown in Figure 19 the alignment unit 290A performs alignment between the first three-dimensional data A and the second three-dimensional data B based on a first alignment parameter (first positional relationship) that is a positional relationship between the first three-dimensional data A and the second three-dimensional data B, and the combination unit 290G combines the first three-dimensional data (first mesh data) A and the second three-dimensional data (second mesh data) B acquired by the data acquisition unit 291 to generate combined three-dimensional data AB. Thereafter, when the third three-dimensional data C is acquired by the data acquisition unit 291, the alignment unit 290A performs alignment between the combined three-dimensional data AB obtained by combining the first three-dimensional data A and the second three-dimensional data B and the third three-dimensional data C based on a second alignment parameter (second positional relationship) that is a positional relationship between the combined three-dimensional data AB and the third three-dimensional data C, and the combination unit 290G combines the combined three-dimensional data AB and the third three-dimensional data (third mesh data) C acquired by the data acquisition unit 291 to generate combined three-dimensional data ABC. At this time, for example, when the editing unit 290H accepts an input for editing a position or a shape of at least one mesh data among the first mesh data, the second mesh data, and the third mesh data, the storage device 240 stores the first mesh data, the second mesh data, and the third mesh data. The editing of the shape of the mesh data includes, for example, removal of a portion of the point cloud.

[0196] Further, when the data acquisition unit 291 acquires the fourth three-dimensional data D, the alignment unit 290A performs alignment between the combined three-dimensional data ABC and the fourth three-dimensional data D based on a third alignment parameter (third positional relationship) that is a positional relationship between the combined three-dimensional data ABC and the fourth three-dimensional data D, and the combination unit 290G combines the combined three-dimensional data ABC and the fourth three-dimensional data (fourth mesh data) D acquired by the data acquisition unit 291 to generate combined three-dimensional data ABCD. When the editing unit 290H accepts an input for editing a position or a shape of the fourth mesh data, the storage device 240 stores the fourth mesh data.

[0197] As described above, the user can obtain three-dimensional data of the entire workpiece W by sequentially combining two pieces of three-dimensional data. Sequential combination is performed, and thus, the processing load of mesh data conversion from point clouds becomes constant, and the processing of combining mesh data for which sparsification processing is performed also becomes constant. For example, the processing load can be reduced compared to processing of collectively converting all original point clouds constituting the combined three-dimensional data ABCD. Note that the combined three-dimensional data ABCD is not limited to a method of sequentially adding the fourth three-dimensional data to the combined three-dimensional data ABC, and the combined three-dimensional data ABCD can be generated by combining the first three-dimensional data, the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data based on positional relationships between the pieces of three-dimensional data.

[0198] The display control unit 255 displays the combined three-dimensional data generated by the combination unit 290G on the display unit 400, and displays the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit 400. That is, the display control unit 255 generates a user interface screen 800 as illustrated in Figure 20 is displayed on the display unit 400. The first display region 801 in which the first three-dimensional data (first mesh data) is displayed, the second display region 802 in which the second three-dimensional data (second mesh data) is displayed, and the third display region 803 in which the combined three-dimensional data (combined mesh data) is displayed are provided on the user interface screen 800. Since the first display region 801 and the second display region 802 are distinguished from each other, the first mesh data and the second mesh data before being reorganized by the combination unit 290G can be displayed in a recognizable manner on the display unit 400. In the first display region 801, the second display region 802, and the third display region 803, coordinate systems of X, Y, and Z are respectively displayed.

[0199] In the present embodiment, the combining process of the first three-dimensional data and the second three-dimensional data by the combining unit 290G can be edited. That is, the analysis module 290 includes an editing unit 290H that edits the combining process of the first three-dimensional data and the second three-dimensional data by the combining unit 290G. The editing unit 290H accepts an input for editing a position or a shape of at least one of the first three-dimensional data and the second three-dimensional data. The editing unit 290H can edit the position of the first three-dimensional data in any one of the X direction, the Y direction, and the Z direction. The second three-dimensional shape data can be similarly edited.

[0200] The editing unit 290H accepts an input of an edit of the first three-dimensional data or an edit of the second three-dimensional data, and edits the first three-dimensional data or the second three-dimensional data based on the accepted input. Then, the combining unit 290G recombines the first three-dimensional data and the second three-dimensional data based on the input accepted by the editing unit 290H. At this time, the alignment unit 290A can perform alignment between the first three-dimensional data and the second three-dimensional data based on the first alignment parameter that is a positional relationship between the first three-dimensional data A and the second three-dimensional data B. Thus, it is not necessary for the user to newly accept an alignment designation, and convenience can be improved. In addition, when an input of editing the third three-dimensional data and the fourth three-dimensional data is similarly accepted, the editing unit 290H edits the third three-dimensional data and the fourth three-dimensional data based on the accepted input. Then, the combining unit 290G updates and regenerates the combined three-dimensional data based on the input accepted by the editing unit 290H.

[0201] For example, when the first three-dimensional data and the second three-dimensional data are combined, the first three-dimensional data and the second three-dimensional data having different initial positions are aligned on the background of the display process of the user interface screen 810, as an example, as illustrated in Figure 21 , and then combined by the combining unit 290G. The combining result display button 811, the additional shape display button 812, and the original shape display button 813 are provided on the user interface screen 810.

[0202] When the user operates the combining result display button 811, the display control unit 255 displays, in the display area 814 of the user interface screen 810, the combined three-dimensional data that is a combination result at a time point at which the combination of the first three-dimensional data and the second three-dimensional data is completed. The user can confirm the quality of the combination result by viewing the combined three-dimensional data. When there is a defective portion, it is also possible to designate the occurrence portion and a factor of the occurrence. Upon designation, when the user operates the additional shape display button 812 on the user interface screen 810 as illustrated in Figure 22 , the display control unit 255 displays the second three-dimensional data in the display area 814 of the user interface screen 810, and when the user operates the original shape display button 813 as illustrated in Figure 21When the original shape display button 813 on the user interface screen 810 is indicated, the display control unit 255 displays the first three-dimensional data in a display area 814 of the user interface screen 810. As described above, since the first three-dimensional data and the second three-dimensional data can be individually displayed by the user's switching operation, in a case where a problem is found in the combination result of the combined three-dimensional data, the user can select any one of the first three-dimensional data or the second three-dimensional data, and partially cut out or repair the three-dimensional data while confirming the selected three-dimensional data. Note that the combined three-dimensional data, the first three-dimensional data, and the second three-dimensional data can be displayed on one user interface screen.

[0203] In Figure 23 An edit button 815 is provided on the user interface screen 810. When the user operates the edit button 815, the display control unit 255 generates and displays a user interface screen 820 for data editing as Figure 20 indicated, and accepts an instruction to edit the three-dimensional data by the edit unit 290H. On the user interface screen 820, a display area 821 in which an instruction of partial cut-out, repair, or the like is accepted, and three-dimensional data to be edited by the edit unit 290H is displayed based on the instruction, a process display area 822 in which an editing process is displayed, and a setting area 823 in which a designation method of an editing area, a selection method, an editing method, or the like can be set are displayed. When editing such as partial cut-out or repair of the three-dimensional data is performed on the user interface screen 820, the edited three-dimensional data is stored in the storage device 240.

[0204] In a case where the edited three-dimensional data is, for example, the second three-dimensional data, alignment between the edited second three-dimensional data and the first three-dimensional data is performed again by the alignment unit 290A. At this time, the alignment unit 290A can perform alignment between the edited second three-dimensional data and the first three-dimensional data based on the first alignment parameter. Then, the combination unit 290G generates the combined three-dimensional data by combining the edited second three-dimensional data and the first three-dimensional data. The combined three-dimensional data generated in this way is displayed in the display area 814.

[0205] In addition, in a case where the quality of the combined three-dimensional data is poor due to editing of the second three-dimensional data, the user issues a reacquisition instruction to reacquire the second three-dimensional data acquired by the data acquisition unit 291. When the user operates Figure 20When the reacquiring button 816 of the user interface screen 810 is indicated, the accepting unit 298 accepts a reacquiring instruction to reacquire the second three-dimensional data. In this case, the data acquiring unit 291 acquires fifth mesh data (mesh data for update) based on the reacquiring instruction accepted by the accepting unit 298, and acquires the first mesh data from the storage device 240, and the aligning unit 290A performs alignment between the first mesh data and the fifth mesh data. The fifth mesh data is acquired to update the second mesh data. Thus, the aligning unit 290A can perform alignment between the mesh data for update (i.e., the fifth mesh data) acquired as a substitute for the second mesh data and the first mesh data based on the first alignment parameter. When the operation Figure 19 When the aligning button 817 of the user interface screen 810 is indicated, alignment is started.

[0206] When the accepting unit 298 accepts the reacquiring instruction, the display control unit 255 can also display a recommended placement attitude based on the first three-dimensional data on the display unit 400. The recommended placement attitude is an attitude calculated by the attitude calculating unit 293 as described above. The recommended placement attitude is displayed on the display unit 400, and thus, it is possible to grasp the placement attitude with a large amount of additional data.

[0207] Then, the combining unit 290G combines the first mesh data and the fifth mesh data aligned by the aligning unit 290A, and updates the combined three-dimensional data. The updated combined three-dimensional data is displayed in the display area 814. As described above, the first alignment parameter as the alignment parameter of the first mesh data and the second mesh data can be used to combine the first mesh data and the fifth mesh data aligned by the aligning unit 290A. That is, the first alignment parameter as the alignment parameter of the first three-dimensional data A and the second three-dimensional data B is associated with the combined three-dimensional data AB, and when the reacquiring instruction to reacquire the second three-dimensional data is accepted, it is possible to perform alignment between the first three-dimensional data A and the newly acquired three-dimensional data B' instead of the second three-dimensional data B by using the first alignment parameter associated with the combined three-dimensional data AB to generate a new combined three-dimensional data AB'. In a case where the combined three-dimensional data is updated, the combining unit 290G can discard the combined three-dimensional data before the update. Further, when the first mesh data and the third mesh data aligned by the aligning unit 290A are combined, the combining unit 290G can discard the second mesh data, which is mesh data before the update. That is, since mesh data that does not constitute the combined three-dimensional data is unnecessary data, it is possible to suppress wasteful occupation of a storage area by discarding the mesh data. When the unnecessary data is discarded, confirmation can be performed by the user.

[0208] Furthermore, alignment unit 290A performs alignment between the combined 3D data AB' obtained by combining the first and fifth mesh data and the third mesh data. At this time, alignment unit 290A can perform alignment between the combined mesh data AB' obtained by combining the first and fifth mesh data and the third mesh data based on a second alignment parameter. Then, combination unit 290G combines the combined mesh data AB' aligned by alignment unit 290A and the third mesh data to generate combined mesh data AB'C. The generated combined mesh data AB'C is displayed in display area 814.

[0209] On the other hand, since the first, fifth, and third 3D data constituting the combined 3D data are automatically stored as necessary data in the storage device 240, at least one of the first, fifth, and third 3D data can be read later. For noise or misalignment that is ignored during repeated combination processing, the individual scan results can be automatically stored, and the combination processing can be performed again. For example, as... Figure 24 As shown, in the case of generating combined 3D data AB, then generating combined 3D data ABC, and then generating combined 3D data ABCD, and in the case where the third 3D data C is combined with the fourth 3D data D without noticing noise contamination, the step can be recovered from the middle by recombining the third 3D data C with the combined 3D data AB.

[0210] As described above, the combined 3D data may include information indicating the combination order of the 3D data, and this information can be stored in the storage device 240 in a state associated with the combined 3D data. The combination unit 290G combines the first grid data and the third grid data aligned by the alignment unit 290A based on the information indicating the combination order corresponding to the second 3D data.

[0211] (Alignment of CAD data)

[0212] When the actual workpiece W is arranged to match the CAD data displayed on the display unit 400, the user needs to move the workpiece W on the rotating stage 143. However, the present invention is not limited to this, and the CAD data can be moved to match the actual workpiece W. That is, it may be difficult for the user to arrange the actual workpiece W to match the CAD data while viewing the display unit 400, because the arrangement needs to be performed simultaneously while viewing both the display unit 400 and the rotating stage 143. Furthermore, since the camera capturing the actual workpiece W and the user's line of sight are facing each other, the user performs the operation while viewing a mirror image, which leads to difficulties. Therefore, to improve user convenience, a function (alignment function) can be installed that enables the actual workpiece and CAD data to be aligned by the user moving the CAD data while viewing the display unit 400 without moving the actual workpiece W on the 3D scanner 1.

[0213] In the following text, reference will be made to Figure 25 The flowchart shown illustrates the details of the alignment function. In the following description, alignment is referred to as overlay alignment, and CAD data is referred to as "virtual objects." In step S100, the controller 200 reads the center position of the rotating platform 143 and the CAD data. In step S101, the controller 200 calculates a virtual ground based on the center position of the rotating platform 143 read in step S100, displays the virtual ground on the display unit 400, and displays virtual objects on the display unit 400 based on the CAD data.

[0214] In step S102, the controller 200 determines whether the mouse button of the operation unit 250 is pressed near the display position of the virtual object. If the mouse button is not pressed near the display position of the virtual object, the overlay alignment is terminated. On the other hand, if the mouse button is pressed near the display position of the virtual object, the process proceeds to step S103, and the controller 200 performs virtual object rotation processing by dragging the mouse.

[0215] Reference Figure 24The following flowchart describes the virtual object rotation process. In step S200, the controller 200 determines whether the rotation button of the operation unit 250 is pressed. If the controller 200 determines that the rotation button of the operation unit 250 is pressed, in step S201, the controller 200 determines whether the movement direction of the mouse of the operation unit 250 is close to horizontal. If the movement direction of the mouse is not close to horizontal, the process proceeds to step S202. On the other hand, if the movement direction of the mouse is close to horizontal, the process proceeds to step S203, and the controller 200 fixes the rotation axis of the virtual object to an axis perpendicular to the virtual ground. In step S204, the controller 200 extracts the horizontal component of the mouse movement. In step S205, the controller 200 calculates the posture of rotation based on the horizontal component extracted around the fixed axis. In step S206, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S207, the display of the virtual object is updated. In step S208, the controller 200 determines whether the mouse button is released. If the mouse button is not released, the process returns to step S201.

[0216] In step S202, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to vertical. If the mouse movement direction is close to vertical, the process proceeds to step S209, and the controller 200 fixes the rotation axis of the virtual object to the right and left axes in the line of sight. In step S210, the controller 200 extracts the vertical component of the mouse movement. In step S211, the controller 200 calculates the posture of rotation based on the vertical component extracted around the fixed axis, and the process proceeds to step S206.

[0217] If it is determined in step S202 that the mouse movement direction is not close to vertical, the process proceeds to step S212, and the amount of mouse movement is extracted. In step S213, the orientation for rotation in any direction based on the amount of mouse movement is calculated, and the process proceeds to step S206.

[0218] If no result is found in step S200, the process proceeds to step S214, and the mouse movement amount is extracted. In step S215, the orientation for rotation in any direction based on the mouse movement amount is calculated, and the process proceeds to step S206.

[0219] The process then proceeded to... Figure 26In step S104, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S105, the controller 200 extracts surfaces of the virtual object that are close to the virtual ground. In step S106, the controller 200 calculates the grounding degree between the surface of the virtual object and the virtual ground. In step S107, the controller 200 determines whether there are extracted surfaces for which the grounding degree has not been calculated. If there are extracted surfaces for which the grounding degree has not been calculated, the next surface is selected in step S108, and the process proceeds to step S106.

[0220] If no extraction surface has been found whose grounding degree has not been calculated, the process proceeds to step S109, and the controller 200 selects the surface with the highest grounding degree. In step S110, the controller 200 calculates the orientation of the selected surface and the virtual ground ground. In step S111, the display of the virtual object is updated to the calculated orientation.

[0221] Figure 27 The flowchart illustrates the process of adjusting the positional relationship between a virtual object and the virtual ground. In step S300, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S301, the controller 200 determines whether the virtual object is buried in the virtual ground. If it is determined to be yes in step S301, the process proceeds to step S302, pushing the virtual object upwards above the virtual ground, and then proceeds to step S303. If it is determined not to be yes in step S301, the process proceeds to step S303. In step S303, the controller 200 determines whether the virtual object floats off the virtual ground. If it is determined to be yes in step S303, the process proceeds to step S304, and the virtual object is placed on the virtual ground.

[0222] Figure 28 The flowchart illustrates the stage surface detection process. In step S400, the controller 200 measures and acquires the three-dimensional shape of the rotating stage 143. In step S401, the controller 200 obtains the position of the stage surface (mounting surface 142) from the three-dimensional shape of the rotating stage 143. In step S402, the controller 200 calculates the height information of the stage surface relative to the camera position based on the acquired planar position. In step S403, the calculated stage surface height information is stored in a storage device 240 or the like.

[0223] In step S404, the controller 200 determines whether to save the dedicated chart. The dedicated chart may be, for example, a calibration plate. If it is determined not to save in step S404, the process proceeds to step S405, and the center position of the stage is calculated based on the irregular shape on the stage surface having known design values. If it is determined to save in step S404, the process proceeds to step S406, and the dedicated chart is rotated and measured from multiple directions to calculate the center position of the stage. In step S407, the center position of the stage is stored in a storage device 240 or the like.

[0224] Figure 24 This is a flowchart illustrating an example of the alignment function processing when performing rotation and movement of virtual objects. Steps S500, S501, and S502 are respectively related to... Figure 24 Steps S100, S101, and S102 are the same. Furthermore, steps S505 to S512 are respectively... Figure 28 Steps S104 to S111 are the same.

[0225] In step S503, the virtual object is rotated and moved by dragging with the mouse. Figure 28 This is a flowchart illustrating an example of the processing when rotating and moving a virtual object. In step S600, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to horizontal. If the mouse movement direction is not close to horizontal, the process proceeds to step S601. On the other hand, if the mouse movement direction is close to horizontal, the process proceeds to step S602, and the controller 200 fixes the rotation axis of the virtual object to an axis perpendicular to the virtual ground. In step S603, the controller 200 extracts the horizontal component of the mouse movement. In step S604, the controller 200 calculates the posture of rotation based on the horizontal component extracted around the fixed axis. In step S605, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S606, the display of the virtual object is updated. In step S607, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S603.

[0226] In step S601, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to vertical. If the mouse movement direction is close to vertical, the process proceeds to step S608, and the controller 200 fixes the rotation axis of the virtual object to the right and left axes in the line of sight. In step S609, the controller 200 extracts the vertical component of the mouse movement. In step S610, the controller 200 calculates the rotation posture based on the vertical component extracted around the fixed axis, and the process proceeds to step S611. In step S611, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S612, the display of the virtual object is updated. In step S613, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S609.

[0227] If the mouse movement direction is not close to vertical, the process proceeds to step S614, and the controller 200 extracts the amount of mouse movement. In step S615, the controller 200 calculates the rotational posture in any direction based on the amount of mouse movement, and the process proceeds to step S616. In step S616, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S617, the display of the virtual object is updated. In step S618, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S614.

[0228] exist Figure 30 In step S513, the controller 200 calculates the distance between the virtual object and the virtual tilting platform. In step S514, surfaces close to the virtual tilting platform are extracted from the surfaces of the virtual object. In step S515, the grounding degree between the surface of the virtual object and the virtual tilting platform is calculated. In step S516, it is determined whether there are extracted surfaces whose grounding degree has not been calculated. If there are extracted surfaces that have not been calculated, the process proceeds to step S517, the next surface is selected, and the process proceeds to step S515. When there are no extracted surfaces that have not been calculated, the process proceeds to step S518, and the surface with the highest grounding degree is selected. In step S519, the orientation of the selected surface grounding with the virtual tilting platform is calculated. In step S520, the positional relationship between the virtual object, the virtual ground, and the virtual tilting platform is adjusted.

[0229] Figure 25 This is a flowchart illustrating another example of the processing in the case of performing rotation and movement of a virtual object. Steps S700 to S705 are respectively related to... Figure 25 Steps S200 to S205 are the same. Furthermore, steps S707 to S715 are respectively... Figure 31Steps S207 to S215 are the same. In step S706, the controller 200 adjusts the positional relationship between the virtual object, the virtual ground, and the virtual tilting platform. The virtual tilting platform virtually refers to the tilting platform set on the rotating stage 143. The tilting platform set on the rotating stage 143 is configured such that, for example, the tilt angle relative to the horizontal plane can be changed in various ways, and the workpiece W can be tilted by mounting the workpiece W on the tilting platform.

[0230] Figure 6 This is a flowchart illustrating an example of the process when adjusting the positional relationship between a virtual object, a virtual ground, and a virtual tilting platform. In step S800, the controller 200 calculates the distance between the virtual object, the virtual ground, and the virtual tilting platform. In step S801, the controller 200 determines whether the virtual object is buried in the virtual ground or the virtual tilting platform. If it is determined to be yes in step S801, the process proceeds to step S802, pushing the virtual object upwards above the virtual ground or the virtual tilting platform, and then proceeds to step S803. If it is determined not to be yes in step S801, the process proceeds to step S803. In step S803, the controller 200 determines whether the virtual object is floating from the virtual ground or the virtual tilting platform. If it is determined to be yes in step S803, the process proceeds to step S804, and the virtual object is grounded to the virtual ground or the virtual tilting platform.

[0231] As described above, the same determinations as those for the rotating stage 143 are performed in the space where the virtual object exists, and physical constraints are introduced to ground the virtual object to the rotating stage 143. Therefore, since rotation and translation are limited to the same degrees of freedom as the actual workpiece W, alignment by the user becomes easy.

[0232] Furthermore, although unstable postures may occur during mouse dragging on the operation unit 250, a stable posture is calculated at the moment the mouse is released, and this posture is automatically corrected to one where the grounding degree between the rotating stage 143 and the virtual object increases. Since the actual possible postures of the workpiece W also exist with a finite number of degrees of freedom, alignment becomes easier.

[0233] Furthermore, rotation operations performed by dragging the mouse through the operation unit 250 are restricted, allowing rotation to occur only relative to one of the workpiece's roll, pitch, and yaw axes at a time. Therefore, virtual objects on the rotary stage 143 can rotate while maintaining the rotary stage 143 in a grounded state, and alignment becomes easier.

[0234] Furthermore, when the rotation of the rotating stage 143 and the display of the virtual object are interlocked, it may be difficult to capture the positional relationship in the depth direction when capturing images from a fixed camera, as one of the difficulties in alignment. On the other hand, since the rotation of the rotating stage 143 and the display state of the virtual object are interlocked, when capturing objects on the rotating stage 143 from different angles, the positional relationship with the virtual object can be captured, and alignment can be performed using the captured images.

[0235] Furthermore, in the case of a structure that allows the rotating stage 143 to tilt as described above, tilt information, etc., can be interlocked with the alignment function. For example, the application provides a section for inputting tilt information (tilt angle information) of the rotating stage 143, and the controller 200 acquires this information. Therefore, the virtual object can be interlocked with the virtual space, and the virtual object can take into account the tilt angle of the rotating stage 143 to adopt a posture.

[0236] In addition, without Figure 17 In the case of the reference model shown, the processing can proceed to step SA5 without calculating the evaluation value and the recommended placement posture calculated in SA4. In this case, in step SA5, for example, the posture calculation unit 293 calculates a posture as the recommended placement posture, which is obtained by rotating a portion of the scanned model of the workpiece W placed in the first placement posture, acquired by the data acquisition unit 291, around a predetermined axis (such as a rotation axis horizontal with respect to the mounting surface 140) by a constant rotation angle. Then, the display control unit 255 can display the recommended placement posture calculated by the posture calculation unit 293 on the display unit 400. Here, for example, the posture calculation unit 293 can calculate the recommended placement posture by estimating the rotation axis horizontal with respect to the mounting surface 140 and rotating the workpiece W around the rotation axis by a specific rotation angle, such as 60 degrees or 90 degrees. In step SA6, the user determines whether the posture displayed on the display unit 400 is the desired posture. If the posture is not the desired posture, the processing proceeds to step SA7, and the placement posture of the workpiece W is adjusted on the computer graphics. In step SA8, the attitude calculation unit 293 can calculate and update the evaluation value of the adjusted attitude, but can skip to step SA6.

[0237] If the orientation is determined to be the desired orientation in step SA6, the process proceeds to step SA9, and the analysis module 290 generates a computer-generated workpiece (model) with the determined placement orientation. In step SA10, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires a real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the computer-generated workpiece.

[0238] In step SA11, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SA5, and the placement posture is adjusted. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the second placement posture, the process proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement posture. In step SA12, the light projection unit 110 of the measurement unit 100 illuminates the workpiece W placed in the second placement posture with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained through the processing in step SA12 is acquired as the second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0239] In step SA14, the alignment unit 290A, included in the analysis module 290, aligns the first three-dimensional data (three-dimensional data of the workpiece placed in a first placement posture) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. During this alignment, the overlapping area extracted by the extraction unit 290B, included in the analysis module 290, is used.

[0240] In addition, for the existence of such Figure 32In the example of scanning processing using a reference model of the CAD data or measured 3D data of the workpiece W, the processing can proceed to step SG4 without the posture calculation unit 293 calculating the evaluation value and the recommended placement posture in step SG3. In this case, in step SG4, the display control unit 255 displays the CAD data or measured 3D data on the display unit 400. The posture of the CAD data or measured 3D data at this time can be, for example, the posture of the CAD data or measured 3D data relative to the scanner, determined by matching the coordinate system of the CAD data or measured 3D data with the scanner coordinate system. In step SG4, the user can adjust the posture of the CAD data or measured 3D data based on this posture. In step SG5, the user determines whether the candidate posture displayed on the display unit 400 is the desired posture. If the posture is not the desired posture, the processing proceeds to step SG6, and the placement posture of the workpiece W is adjusted according to the computer graphics. In step SG7, the posture calculation unit 293 calculates and updates the evaluation value for the adjusted posture, but this process can be skipped, and the processing can proceed to step SG5.

[0241] If the orientation is determined to be the desired orientation in step SG5, the process proceeds to step SG8, ​​and the analysis module 290 generates a computer-generated graphic workpiece (model) with the determined placement orientation. The display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires the real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the computer-generated graphic workpiece.

[0242] In step SG10, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG4, and the placement posture is adjusted. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the first placement posture, the process proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement posture. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the measured first three-dimensional data and the CAD data used as a reference model. In step SG14, the analysis module 290 acquires the aligned CAD model or the measured three-dimensional data.

[0243] In step SG15, a candidate pose is suggested, and processing proceeds to step SG16. For example, for a candidate pose, the pose of the CAD data or measured 3D data can be changed according to a predetermined rule regarding the desired pose determined in step SG5. As an example of the prescribed rule, there is a case where the CAD data or measured 3D data is rotated by 60 degrees relative to the desired pose determined in step SG5, with one axis of the scanner coordinate system serving as the axis of rotation. Alternatively, for example, the user can select a rotation angle range within 90 degrees. In step SG16, the user selects a candidate pose, and processing proceeds to step SG17. If the pose is not the desired pose, after adjustment in step SG18, the evaluation value is updated in step SG19, and processing proceeds to step SG17. Step SG19 can be skipped.

[0244] If the pose is the desired pose, the processing proceeds to step SG20, and the display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. Here, the computer-generated graphic workpiece can be the first 3D data or reference model acquired in step SG12. Furthermore, the first 3D data and the reference model can be displayed simultaneously on the display unit 400, or they can be switched and displayed separately.

[0245] In step SG21, while viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the workpiece in the computer graphics.

[0246] In step SG22, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG16, and another placement orientation is selected. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement orientation (second placement orientation). In step SG24, the alignment unit 290A performs alignment using the acquired second 3D data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the 3D model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a part that needs to be scanned. If there is a part that needs to be scanned, the process proceeds to step SG12. If there is no part to be scanned, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model or the measured 3D data, wherein the alignment processing is performed by the alignment unit 290A in step SG26. In addition, full-surface scan models and CAD models or measured 3D data can be displayed on display unit 400 in an aligned state.

[0247] In this embodiment, CAD data or measured 3D data is used as a reference model to obtain the combined 3D data of workpiece W. However, even after obtaining the combined 3D data of workpiece W without a reference model, alignment with the CAD data or measured 3D data can still be performed. The reading unit 292 reads the 3D data and CAD data of workpiece W from the storage unit 240. Subsequently, the resolution conversion unit 290E, read by the reading unit 292, converts at least the 3D data of workpiece W into 3D data with a first resolution lower than the resolution obtained by the data acquisition unit 291. Then, the shape feature extraction unit 290C extracts the partial regions (key points) to be used for calculating shape features from the 3D data and CAD data at the first resolution. Subsequently, the analysis module 290 samples the periphery of the partial regions, and the shape feature extraction unit 290C calculates shape feature vectors from the vicinity of the points sampled by the analysis module 290. Subsequently, the analysis module 290 compares the shape feature vectors of the 3D data obtained by the shape feature extraction unit 290C with the shape feature vectors of the CAD data, and the alignment unit 290A aligns the 3D data with the CAD data based on the comparison of the shape feature vectors. The shape feature vectors of candidate pairs can be extracted by comparing their shape feature vectors, and the positional relationships (rotation and translation) between point clouds can be obtained by using the points of the candidate pairs through RANSAC or deep learning-based methods.

[0248] Furthermore, after the alignment between the combined 3D data and CAD data is completed, the analysis module 290 can automatically perform a comparison between the CAD data and the 3D data. Figure 33A An example of the analysis is shown, in which the dimensional differences between the scanned data and CAD data of workpiece W are calculated by comparing their three-dimensional shapes, and a color map is displayed by assigning colors corresponding to the degree of difference. The analysis module 290 calculates the shape difference between the three-dimensional data of workpiece W obtained by the data acquisition unit 291 and the CAD data for each grid based on the alignment result of the alignment unit 290A and the CAD data read by the reading unit 292, and assigns color information corresponding to the degree of difference to each grid. Then, the display control unit 255 displays the color map on the display unit 400, where a color is assigned to each grid based on the color information of at least one of the three-dimensional data and the CAD data. Additionally, after alignment by the alignment unit 290A, the analysis module 290 can compare the three-dimensional data with the CAD data, and the display control unit 255 can display the color map on the display unit 400 based on an instruction to assign analysis settings or start comparison analysis received by the receiving unit 298. In this case, the user can assign detailed settings for the comparison.

[0249] in addition, Figure 33B An example of performing cross-sectional measurements on three-dimensional data of workpiece W is shown. The user specifies the surface on which the cross-sectional measurement will be performed on the three-dimensional data of workpiece W, and indicates the type and location of the analysis tool to be performed on the cross-section. The analysis tool may be, for example, a measurement such as the distance between two points and the angle formed by surfaces. The analysis module 290 accepts the instruction and performs analysis on the specified surface of the three-dimensional data of workpiece W based on the instruction.

[0250] ​ An example of performing cross-sectional measurements on CAD data is shown. The user specifies the surface on which the cross-sectional measurement is performed on the CAD data and indicates the type and assignment location of the analysis tool to be performed on the cross-section. The analysis module 290 accepts the instructions and performs analysis on the indicated surface of the CAD data based on the instructions. The analysis module 290 can compare the results obtained by performing cross-sectional measurements on the three-dimensional data with the results obtained by performing cross-sectional measurements on the CAD data, and the display control unit 255 displays the comparison results on the display unit 400.

[0251] Furthermore, cross-sectional measurements can be performed on the data where the 3D data and CAD data of workpiece W are aligned. Alignment unit 290A aligns the 3D data and CAD data of workpiece W to obtain the data where the 3D data and CAD data of workpiece W are aligned. The user specifies the surface on which the cross-sectional measurement will be performed on the data where the workpiece W and CAD data are aligned, and indicates the type and location of the analysis tool to be performed on the cross-section. Analysis tools may include, for example, measurements such as the distance between two points and the angle formed by surfaces. Analysis module 290 receives instructions and performs analysis on the indicated surface of the data where the 3D data and CAD data of workpiece W are aligned, based on the instructions. Analysis module 290 can perform cross-sectional measurements on the data where the 3D data and CAD data are aligned. For example, a comparison can be performed based on the dimensional differences between the CAD data and the scanned data. Display control unit 255 can display the comparison results on display unit 400.

[0252] The above embodiments are merely illustrative in all respects and should not be construed as limiting. Furthermore, all modifications and alterations falling within the equivalent scope of the claims are within the scope of this invention.

[0253] As described above, the present invention can be used to generate multiple three-dimensional data of various workpieces.

Claims

1. A 3D scanner that generates combined 3D data of a workpiece by generating multiple 3D data points of a workpiece placed in different orientations and combining the multiple 3D data points, the 3D scanner comprising: A data acquisition unit acquires first three-dimensional data of the workpiece placed in a first placement posture; An attitude calculation unit calculates a placement attitude different from the first placement attitude based on the first three-dimensional data acquired by the data acquisition unit; An overlap region estimation unit estimates the overlap region between the three-dimensional data to be acquired by the data acquisition unit and the first three-dimensional data acquired by the data acquisition unit when the data is placed in the placement posture calculated by the posture calculation unit. as well as The display control unit displays an evaluation index on a display unit based on the amount of the overlapping region estimated by the overlapping region estimation unit.

2. The 3D scanner according to claim 1, wherein The attitude calculation unit calculates multiple placement attitudes different from the first placement attitude based on the first three-dimensional data acquired by the data acquisition unit, and The 3D scanner also includes: An additional data quantity estimation unit estimates the amount of additional data to be added to the first three-dimensional data by acquiring the three-dimensional data by the data acquisition unit in a state of being arranged in a placement posture for each of the plurality of placement postures calculated by the posture calculation unit.

3. The 3D scanner of claim 2, wherein the evaluation index is an index based on the amount of additional data estimated by the additional data quantity estimation unit and the amount of the overlapping region estimated by the overlapping region estimation unit.

4. The three-dimensional scanner according to claim 2 further includes: A designation unit, based on the evaluation index, designates a candidate pose from the plurality of placement poses calculated by the pose calculation unit. The display control unit displays the candidate pose specified by the designated unit on the display unit.

5. The 3D scanner of claim 2, wherein the additional data quantity estimation unit moves the first 3D data to each placement pose and estimates the additional data quantity based on the orientation of the normal vector after the movement.

6. The 3D scanner of claim 5, wherein the additional data quantity estimation unit estimates the 3D data quantity facing the imaging unit based on the inner product of the orientation of the normal vector after the movement from the predetermined viewpoint and the line-of-sight direction of the imaging unit of the 3D scanner, and estimates that the additional data quantity increases as the 3D data quantity facing the imaging unit increases.

7. The 3D scanner of claim 1, wherein the display control unit displays the evaluation index in digital or graphical form on the display unit.

8. The 3D scanner of claim 3, wherein the display control unit displays on the display unit each of an evaluation index based on the amount of additional data estimated by the additional data amount estimation unit and an evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit.

9. The 3D scanner of claim 2, wherein the display control unit displays on the display unit an evaluation index of each of the plurality of placement postures calculated by the posture calculation unit for each placement posture.

10. The three-dimensional scanner of claim 1, wherein the overlapping region estimation unit estimates the degree of feature based on the distribution of points in the overlapping region.

11. The three-dimensional scanner according to claim 10, further comprising: A designated unit, which designates a candidate posture from the plurality of placement postures calculated by the posture calculation unit based on the evaluation index; as well as An additional data volume estimation unit estimates the amount of additional data to be added to the first three-dimensional data by acquiring the three-dimensional data from the data acquisition unit for each of the plurality of placement postures calculated by the posture calculation unit, while the data is arranged in a placement posture. The designated unit designates a candidate pose based on the additional data amount estimated by the additional data amount estimation unit, the amount of the overlapping region estimated by the overlapping region estimation unit, and the feature degree estimated by the overlapping region estimation unit.

12. The 3D scanner of claim 1, wherein the attitude calculation unit calculates candidate attitudes based on the size of the contact area with the mounting surface in each of the plurality of placement attitudes.

13. The 3D scanner of claim 4 further includes a receiving unit that receives operational input for adjusting the candidate pose specified by the designating unit.

14. The three-dimensional scanner according to claim 2, wherein The data acquisition unit acquires combined three-dimensional data, in which the first three-dimensional data of the workpiece placed in the first placement posture and the second three-dimensional data of the workpiece placed in a second placement posture different from the first placement posture are combined. The attitude calculation unit calculates a placement attitude different from the first placement attitude based on the combined 3D data, and The additional data estimation unit estimates the amount of additional data to be added to the combined three-dimensional data by acquiring three-dimensional data from the data acquisition unit under the placement posture calculated by the posture calculation unit.

15. The three-dimensional scanner according to claim 1, further comprising: The stage control unit controls the rotation of the stage on which the workpiece is mounted. The attitude calculation unit calculates multiple placement attitudes by virtually rotating the first three-dimensional data around the rotation axis of the stage.

16. The three-dimensional scanner according to claim 2, further comprising: An evaluation unit calculates an evaluation index for each of the plurality of placement postures based on the amount of additional data and the amount of overlapping area calculated by the posture calculation unit for each of the plurality of placement postures, and calculates a comprehensive evaluation index based on each evaluation index.

17. A three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating multiple three-dimensional data of a workpiece placed in different orientations and combining the multiple three-dimensional data, the three-dimensional measurement method comprising: Acquire the first three-dimensional data of the workpiece placed in a first placement posture; Calculate a placement posture that is different from the first placement posture based on the acquired first three-dimensional data; Estimate the overlap area between the three-dimensional data acquired by the data acquisition unit and the first three-dimensional data acquired by the data acquisition unit when the data is placed in the calculated placement posture; as well as The evaluation index based on the estimated amount of overlapping area is displayed on the display unit.

18. A storage medium for storing a three-dimensional measurement program, the three-dimensional measurement program being used to cause a computer to execute a three-dimensional measurement method, the three-dimensional measurement method being used to generate combined three-dimensional data of the workpiece by generating multiple three-dimensional data of a workpiece placed in different placement postures and combining the multiple three-dimensional data. The three-dimensional measurement method includes: Acquire the first three-dimensional data of the workpiece placed in the first placement posture. Based on the acquired first three-dimensional data, a placement posture different from the first placement posture is calculated. Estimate the overlap area between the three-dimensional data acquired by the data acquisition unit and the first three-dimensional data acquired by the data acquisition unit when the data is placed in the calculated placement posture, and The evaluation index based on the estimated amount of overlapping area is displayed on the display unit.

Images