Three-dimensional scanner, three-dimensional measurement method, and storage medium storing three-dimensional measurement program

Abstract

The invention provides a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program. Multiple pieces of three-dimensional data can be combined with high precision. The three-dimensional scanner includes: an alignment unit that performs alignment between first three-dimensional data and second three-dimensional data; a combining unit that combines the first three-dimensional data and the second three-dimensional data aligned by the aligning unit to generate combined three-dimensional data; a display control unit that displays the combined three-dimensional data generated by the combining unit on a display unit, and displays the first three-dimensional data and the second three-dimensional data on the display unit in a recognizable manner; and an editing unit that edits the combined processing of the first three-dimensional data and the second three-dimensional data by the combining unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program. BACKGROUND

[0002] For example, JP 2024-051797 A discloses a three-dimensional scanner that scans a workpiece to generate three-dimensional data.

[0003] The three-dimensional scanner of JP 2024-051797 A combines a plurality of pieces of three-dimensional data acquired in different orders to generate combined three-dimensional data of the workpiece.

[0004] As disclosed in JP 2024-051797 A, in order to obtain three-dimensional data of one workpiece, it is necessary to acquire scan data for each posture and perform a combination process. In the related art, it is necessary to acquire a large amount of scan data in advance, combine a plurality of pieces of scan data at a time to generate three-dimensional data, or separately combine a plurality of pieces of scan data, and separately three-dimensionally combine scan data of the front side of the workpiece that has been stored and scan data of the back side of the workpiece that has been stored, and there are many correction work and re-measurement work due to quality deterioration caused by mixing of scan noise. As described above, three-dimensional combination requires specialized techniques and takes time. SUMMARY

[0005] The present disclosure has been made in view of this point, and an object thereof is to easily combine a plurality of pieces of high-precision three-dimensional data.

[0006] To achieve the above object, according to one embodiment of the present disclosure, a three-dimensional scanner can be premised which generates combined three-dimensional data of a workpiece by generating three-dimensional data of the workpiece placed in different placement attitudes and combining the plurality of three-dimensional data. The three-dimensional scanner includes a data acquisition unit which acquires first three-dimensional data which is three-dimensional data of the workpiece placed in a first placement attitude and second three-dimensional data which is three-dimensional data of the workpiece placed in a second placement attitude, an alignment unit which performs alignment between the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit based on first positional relationship which is a positional relationship between the first three-dimensional data and the second three-dimensional data, a combination unit which combines the first three-dimensional data and the second three-dimensional data aligned by the alignment unit to generate the combined three-dimensional data, a display control unit which displays the combined three-dimensional data generated by the combination unit on a display unit and displays the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit, and an editing unit which edits the combination processing of the first three-dimensional data and the second three-dimensional data by the combination unit. The editing unit accepts input for editing a position or a shape of at least one of the first three-dimensional data and the second three-dimensional data, and the combination unit recombines the first three-dimensional data and the second three-dimensional data based on the input accepted by the editing unit.

[0007] According to this configuration, since the combined three-dimensional data generated by combining the first three-dimensional data and the second three-dimensional data is displayed on the display unit, the user can confirm the combined three-dimensional data. As a result of the confirmation, the quality of the combined three-dimensional data can be poor. In this case, since the first three-dimensional data and the second three-dimensional data are displayed on the display unit in a recognizable manner, it is possible to edit a position or a shape of at least one of the first three-dimensional data and the second three-dimensional data. Since the first three-dimensional data and the second three-dimensional data are recombined after the editing, it is possible to confirm the quality again. As described above, since it is possible to bidirectionally perform confirmation and editing of the three-dimensional data before the combination and confirmation of the three-dimensional data after the combination, it is possible to easily combine a plurality of high-precision three-dimensional data.

[0008] According to another embodiment of the present disclosure, a three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating a plurality of pieces of three-dimensional data of the workpiece placed in different placement attitudes and combining the plurality of pieces of three-dimensional data can be premised. The three-dimensional measurement method includes: acquiring first three-dimensional data that is three-dimensional data of the workpiece placed in a first placement attitude and second three-dimensional data that is three-dimensional data of the workpiece placed in a second placement attitude; performing alignment between the acquired first three-dimensional data and the second three-dimensional data based on a first positional relationship that is a positional relationship between the first three-dimensional data and the second three-dimensional data; combining the aligned first three-dimensional data and the second three-dimensional data to generate combined three-dimensional data; displaying the generated combined three-dimensional data on a display unit and displaying the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit; accepting an input for editing a position or a shape of at least one piece of three-dimensional data among the first three-dimensional data and the second three-dimensional data displayed on the display unit; and recombining the first three-dimensional data and the second three-dimensional data based on the accepted input.

[0009] According to still another embodiment of the present disclosure, a storage medium storing a three-dimensional measurement program for causing a computer to execute a three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating a plurality of pieces of three-dimensional data of the workpiece placed in different placement attitudes and combining the plurality of pieces of three-dimensional data can be provided.

[0010] As described above, since editing of a position or a shape of three-dimensional data before combination and confirmation of the three-dimensional data after combination can be performed bidirectionally, a plurality of pieces of high-precision three-dimensional data can be easily combined. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a diagram showing an overall configuration of a three-dimensional scanner according to an embodiment of the present application;

[0012] Figure 2 is a block diagram of the three-dimensional scanner;

[0013] Figure 3 is a side view of a measurement unit and a base;

[0014] Figure 4 is a block diagram of the measurement unit;

[0015] Figure 5 is a diagram showing a configuration example of a module;

[0016] Figure 6 is a flowchart showing an example of a scanning process in the case where there is no CAD data of a workpiece;

[0017] Figure 7FIG. 1 is a diagram showing an example of a user interface screen displayed at the start of measurement;

[0018] Figure 8 FIG. 2 is a diagram showing an example of a user interface screen displayed when presenting posture candidates;

[0019] Figure 9 FIG. 3 is a diagram showing an example of a user interface screen displayed when accepting selection of a posture candidate;

[0020] Figure 10 FIG. 4 is a diagram showing an example of a user interface screen in which a model of the determined placement posture is superimposed and displayed on a real-time image;

[0021] Figure 11 FIG. 5 is a flowchart showing an example of an alignment process;

[0022] Figure 12 FIG. 6 is a schematic view related to hidden surface removal;

[0023] Figure 13 FIG. 7 is a flowchart showing an example of a process of extracting shape features and image features;

[0024] Figure 14 FIG. 8 is a flowchart showing an example of an extraction process in a case where a vector in which an image feature is added is input to a shape feature extraction unit;

[0025] Figure 15 FIG. 9 is a flowchart showing an example of a process of calculating low-precision alignment parameters;

[0026] Figure 16 FIG. 10 is a flowchart showing an example of a process of calculating high-precision alignment parameters;

[0027] Figure 17 FIG. 11 is a flowchart showing an example of a scanning process in a case where CAD data of a workpiece is present;

[0028] Figure 18 FIG. 12 is a diagram showing a user interface screen for displaying first three-dimensional data, second three-dimensional data, and combined three-dimensional data;

[0029] Figure 19 FIG. 13 is a diagram showing a workflow of generating combined three-dimensional data;

[0030] Figure 20 FIG. 14 is a diagram showing a confirmation screen of combined three-dimensional data;

[0031] Figure 21 FIG. 15 is a diagram showing a confirmation screen of second three-dimensional data;

[0032] Figure 22 FIG. 16 is a diagram showing a confirmation screen of first three-dimensional data;

[0033] Figure 23 is a diagram showing a user interface screen for data editing;

[0034] Figure 24 is a flowchart showing an example of the processing of the alignment function;

[0035] Figure 25 is a flowchart showing an example of the virtual object rotation processing by mouse drag;

[0036] Figure 26 is a flowchart showing an example of the processing of adjusting the positional relationship between the virtual object and the virtual ground;

[0037] Figure 27 is a flowchart showing an example of the stage surface detection processing;

[0038] Figure 28 is a flowchart showing an example of the processing of the alignment function in the case where rotation and movement of the virtual object are performed;

[0039] Figure 29 is a flowchart showing an example of the processing in the case where rotation and movement of the virtual object are performed;

[0040] Figure 30 is a flowchart showing another example of the processing in the case where rotation and movement of the virtual object are performed;

[0041] Figure 31 is a flowchart showing an example of the processing in the case where the positional relationship between the virtual object, the virtual ground, and the virtual tilt platform is adjusted;

[0042] Figure 32 is a color map based on the dimensional difference between the scan data and the CAD data of the workpiece;

[0043] Figure 33A is a diagram showing an example of cross-sectional measurement of the scan data of the workpiece; and

[0044] Figure 33B is a diagram showing an example of performing cross-sectional measurement on the CAD data of the workpiece. DETAILED DESCRIPTION

[0045] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings. Note that the following description of the preferred embodiments is merely illustrative in nature and is in no way intended to limit the application, its application, or its uses.

[0046] Figure 1is a diagram showing the overall configuration of a three-dimensional scanner 1 according to an embodiment of the present application. The three-dimensional scanner 1 is a device capable of acquiring three-dimensional data by measuring the shape of a workpiece (measurement object) W, converting the three-dimensional data into mesh data of the workpiece W, and outputting the mesh data. The three-dimensional scanner 1 can also convert the mesh data of the workpiece W into CAD data and output the CAD data, or convert the mesh data into surface data and output the surface data.

[0047] In the following description, when the shape of the workpiece W is measured, in a process of acquiring coordinate information of a front surface of the workpiece W, the workpiece W is irradiated with measurement light of a predetermined pattern, and the coordinate information is acquired by using a signal obtained from reflected light reflected by the front surface of the workpiece W. For example, a measurement method using triangulation using a fringe projection image obtained from the reflected light by projecting measurement light onto the workpiece W using structured illumination as the measurement light of the predetermined pattern can be used. However, in the present application, the principle and configuration for acquiring the coordinate information of the workpiece W are not limited thereto, and other methods can also be applied.

[0048] The three-dimensional scanner 1 includes a measurement unit 100 that measures the shape of the workpiece W, a base 600 on which the workpiece W can be mounted, a controller 200, a light source unit 300, a display unit 400, and the like. The controller 200 can be incorporated in the measurement unit 100, the light source unit 300 can be incorporated in the measurement unit 100, or the display unit 400 can be incorporated in the measurement unit 100. In addition, the controller 200 and the light source unit 300 can be integrated, or the controller 200 and the display unit 400 can be integrated.

[0049] The three-dimensional scanner 1 performs structured illumination on the workpiece W by the light source unit 300, captures a fringe projection image to generate a depth image having coordinate information, and can measure the three-dimensional size and shape of the workpiece W based on the depth image. Measurement using such fringe projection has an advantage that the measurement time can be shortened, because the three-dimensional measurement can be performed without moving the workpiece W or an optical system such as a lens in the Z direction (height direction).

[0050] Figure 2 A block diagram of the three-dimensional scanner 1 according to an embodiment of the present application is shown. As shown in this diagram, the measurement unit 100 includes a pattern light projection unit (first light projection unit) 110 that projects pattern light for measurement onto the workpiece W, a light receiving unit 120, a measurement control unit 150, and an illumination light output unit 130. The light projection unit 110 is a component that irradiates the workpiece W mounted on a mounting unit 140 described later with measurement light of a predetermined pattern. The mounting of the workpiece W on the mounting unit 140 is the same as the placement of the workpiece W on the mounting unit 140.

[0051] The light-receiving unit 120 is fixed in a posture inclined with respect to a mounting surface 142 of a later-described rotary stage 143. The light-receiving unit 120 receives measurement light emitted by the light-projecting unit 110 and reflected by the workpiece W. When receiving the measurement light as reflected light from the workpiece W, the light-receiving unit 120 generates and outputs a first light-receiving signal for measurement indicating the amount of the received measurement light. The light-receiving unit 120 can generate an observation image for observing the entire shape of the workpiece W by capturing the workpiece W mounted on the mounting unit 140. In this example, the illumination light output unit 130 is provided, but the workpiece W can be irradiated with uniform light from the light-projecting unit 110. In this case, the light-projecting unit 110 is a member that irradiates the workpiece W with the measurement light and the uniform light at different timings. The light-receiving unit 120 can also receive the uniform light emitted from the light-projecting unit 110 and output a second light-receiving signal for texture acquisition. For example, uniform light having the same wavelength as the measurement light can be emitted from the measurement light source, and a light-receiving signal including uniaxial color information can be output. Note that although not shown, a calibrated first camera and a second camera can also be prepared, the shape is acquired by the first camera, and the texture information is acquired by the second camera. The texture information includes color information and brightness information of the workpiece W.

[0052] The light-receiving unit 120 according to the present embodiment includes a high-magnification light-receiving unit and a low-magnification light-receiving unit. The high-magnification light-receiving unit is a member capable of capturing the workpiece W in a magnified manner as compared with the low-magnification light-receiving unit. On the other hand, the low-magnification light-receiving unit is a light-receiving unit having a wider field of view range than the high-magnification light-receiving unit.

[0053] The base 600 includes a bottom plate 602, the mounting unit 140, and a movement control unit (stage control unit) 144. The mounting unit 140 is supported on the bottom plate 602 of the base 600. The movement control unit 144 is a member that controls movement and rotation operation of the rotary stage 143 on which the workpiece W is mounted. The movement control unit 144 can be provided on the controller 200 side in addition to being provided on the base 600 side.

[0054] The light source unit 300 is connected to the measurement unit 100. The light source unit 300 is a member that generates measurement light and supplies the measurement light to the measurement unit 100. The controller 200 is a member that controls the measurement unit 100 and the like. The display unit 400 is connected to the controller 200 and is configured to display an image generated by the measurement unit 100 and perform necessary settings, inputs, selections, and the like.

[0055] The mounting unit 140 includes the rotary stage 143 having a top surface on which the mounting surface 142 on which the workpiece W is mounted is formed. As Figure 4As shown, two directions orthogonal to each other on the mounting surface 142 of the rotary stage 143 are defined as the X direction and the Y direction, and are indicated by arrows X and Y, respectively. A direction orthogonal to the mounting surface 142 of the mounting unit 140 is defined as the Z direction, and is indicated by an arrow Z. A direction of rotation about an axis parallel to the Z direction is defined as the Θ direction, and is indicated by an arrow Θ.

[0056] The mounting unit 140 includes the rotary stage 143 that rotates the mounting surface 142 about an axis extending in the Z direction, and the translation stage 141 that moves the mounting surface 142 in the horizontal directions (the X direction and the Y direction). The translation stage 141 includes an X-direction moving mechanism and a Y-direction moving mechanism. The rotary stage 143 has a Θ-direction rotating mechanism. The mounting unit 140 can include a workpiece holding member (jig or the like) that holds the workpiece W on the mounting surface 142. Further, the mounting unit 140 can include a tilt stage having a mechanism that can rotate about an axis parallel to the mounting surface 142.

[0057] The movement control unit 144 controls the rotational movement of the rotary stage 143 and the translation of the translation stage 141 in accordance with the measurement conditions set by the measurement condition setting unit 261 described later. In addition, the movement control unit 144 controls the movement action of the mounting unit 140 by the mounting movement unit based on the measurement region set by the measurement condition setting unit 261 described later.

[0058] The controller 200 includes a central processing unit (CPU) 210, a read only memory (ROM) 220, a work memory 230, a storage device (storage unit) 240, an operation unit 250, and the like. For example, a personal computer (PC) or the like can be used as the controller 200.

[0059] In Figure 4 A configuration of the measurement unit 100 is shown in a block diagram. The measurement unit 100 includes the light projection unit 110, the light reception unit 120, the illumination light output unit 130, the measurement control unit 150, and a main body case 101 that houses these units. The light projection unit 110 includes a measurement light source 111, a pattern generation unit 112, and a plurality of lenses 113, 114, and 115. The light reception unit 120 includes a camera 121 and a plurality of lenses 122 and 123. In a case where measurement is performed at different magnifications by providing a plurality of light reception units, a light reception unit 120a including the camera 121 for low magnification and the lens for low magnification, and a light reception unit 120b including the camera 121 for high magnification and the lens for high magnification can be installed. Note that the present application is not limited to this configuration, and the magnification can be changed by switching between a plurality of lenses for one camera 121, or the magnification can be changed by providing a zoom lens for one camera 121.

[0060] The light projection units 110 are obliquely arranged above the mounting unit 140. In the example shown, the light projection units 110 are arranged so that the center axes of the light projection units 110 and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140. In this way, the light projection units 110 and the light receiving unit 120 are arranged so that the center axes of the light projection units 110 and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 can include a plurality of light projection units 110. Here, a first measurement light projection unit 110A (right side in FIG. 1) capable of irradiating the workpiece W with first measurement light ML1 from a first direction and a second measurement light projection unit 110B (left side in FIG. 1) capable of irradiating the workpiece W with second measurement light ML2 from a second direction different from the first direction are provided. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 can include a plurality of light projection units 110. Here, a first measurement light projection unit 110A (right side in FIG. 1) capable of irradiating the workpiece W with first measurement light ML1 from a first direction and a second measurement light projection unit 110B (left side in FIG. 1) capable of irradiating the workpiece W with second measurement light ML2 from a second direction different from the first direction are provided. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 can include a plurality of light projection units 110. Here, a first measurement light projection unit 110A (right side in FIG. 1) capable of irradiating the workpiece W with first measurement light ML1 from a first direction and a second measurement light projection unit 110B (left side in FIG. 1) capable of irradiating the workpiece W with second measurement light ML2 from a second direction different from the first direction are provided.

[0061] Each of the first measurement light projection unit 110A and the second measurement light projection unit 110B includes a first measurement light source and a second measurement light source as the measurement light source 111. The measurement light source 111 is, for example, a halogen lamp that emits white light. The measurement light source 111 can be a light source that emits monochromatic light, such as a blue light emitting diode (LED) or another light source that emits blue light, such as an organic EL. Light emitted from the measurement light source 111 (hereinafter referred to as "measurement light") is appropriately condensed by a lens 113 and then incident on a pattern generation unit 112.

[0062] The relative positional relationship between the light receiving unit 120, the light projection units 110A and 110B, and the mounting unit 140 is determined so that the center axes of the light projection units 110A and 110B and the center axis of the light receiving unit 120 intersect each other at the position at which the workpiece W is arranged on the mounting unit 140, at which the depth of field of the light projection units 110 and the light receiving unit 120 is appropriate. In addition, since the center of the rotation axis in the θ direction coincides with the center axis of the light receiving unit 120, when the mounting unit 140 is rotated in the θ direction, the workpiece W does not deviate from the field of view and rotates around the rotation axis in the field of view.

[0063] The pattern generating unit 112 reflects the light emitted from the measurement light source 111 to project the measurement light onto the workpiece W. The measurement light incident on the pattern generating unit 112 is converted into a preset pattern and a preset intensity (brightness) and emitted. The measurement light emitted by the pattern generating unit 112 is converted into light having an observable and measurable field of view larger than the light receiving unit 120 by the plurality of lenses 114 and 115, and then the workpiece W on the mounting unit 140 is irradiated with the converted light.

[0064] The pattern generating unit 112 is a member capable of switching between a light projection state in which the measurement light is projected onto the workpiece W and a non-light projection state in which the measurement light is not projected onto the workpiece W. For example, for such a pattern generating unit 112, a digital micromirror device (DMD) or the like can be used. The pattern generating unit 112 using the DMD can be controlled by the measurement control unit 150 to be capable of switching between a reflection state in which the measurement light is reflected on the optical path to become the light projection state and a light shielding state in which the measurement light is shielded to become the non-light projection state.

[0065] Note that, in the above example, an example in which the DMD is used for the pattern generating unit 112 has been described, but the pattern generating unit 112 is not limited to the DMD in the present application, and other members can also be used. For example, a liquid crystal on silicon (LCOS) can be used as the pattern generating unit 112. Alternatively, the amount of transmitted measurement light can be adjusted by using a transmissive member instead of a reflective member. In this case, the pattern generating unit 112 is provided on the optical path of the measurement light to switch between a light projection state in which the measurement light is transmitted and a light shielding state in which the measurement light is shielded. For example, a liquid crystal display (LCD) can be used as the pattern generating unit 112. Alternatively, the pattern generating unit 112 can be formed by using a projection method of a plurality of line LEDs, a projection method using a plurality of optical paths, an optical scanner method including a laser and a galvanometer mirror, an accordion fringe interferometry (AFI) method using interference fringes generated by superimposing beams split by a beam splitter, a projection method using an actual grating and a moving mechanism including a piezoelectric stage, a high-resolution encoder, or the like.

[0066] The light receiving unit 120 is provided above the mounting unit 140. The measurement light reflected upward from the mounting unit 140 by the workpiece W is collected and captured by the plurality of lenses 122 and 123 of the light receiving unit 120, and then received by the camera 121.

[0067] The camera 121 is, for example, a charge-coupled device (CCD) camera including an imaging element 121a. The imaging element 121a is, for example, a monochrome charge-coupled device (CCD). The imaging element 121a can be another imaging element, such as a complementary metal-oxide semiconductor (CMOS) image sensor. In a color imaging element, since each pixel needs to receive light corresponding to red, green, and blue colors, the measurement resolution is lower than that of a monochrome imaging element. Since a color filter needs to be provided in each pixel, the sensitivity is reduced. Therefore, in the present embodiment, a color image is acquired by employing a monochrome CCD as the imaging element and causing the illumination light output unit 130 described later to emit illuminations corresponding to each RGB color in a time-division manner to take an image. With this configuration, it is possible to acquire a color image of the measurement object without reducing the measurement accuracy. The illumination light output unit 130 is an example of a second light projection unit that irradiates the workpiece W with illumination light. The illumination light can be uniform light.

[0068] Note that a color imaging element can be used as the imaging element 121a. In this case, although the measurement accuracy and the sensitivity are lower than those of a monochrome imaging element, it is not necessary to emit illuminations corresponding to each RGB color from the illumination light output unit 130 in a time-division manner, and it is possible to acquire a color image by emitting only white light, so the illumination optical system can be simply formed. An analog electric signal corresponding to the amount of received light (hereinafter referred to as a "light reception signal") is output from each pixel of the imaging element 121a to the measurement control unit 150.

[0069] An analog / digital converter (A / D converter) and a first-in first-out (FIFO) memory (both not shown) are installed on the measurement control unit 150. The light reception signal output from the camera 121 is sampled at a constant sampling period, and converted into a digital signal by the A / D converter of the measurement control unit 150 under the control of the light source unit 300. The digital signal output from the A / D converter is sequentially accumulated in the FIFO memory. The digital signal accumulated in the FIFO memory is sequentially transferred to the controller 200 as pixel data.

[0070] The operation unit 250 of the controller 200 can include, for example, a keyboard, a pointing device, and the like. For example, a mouse, a joystick, and the like are used as the pointing device.

[0071] The ROM 220 of the controller 200 stores a system program or the like. The work memory 230 of the controller 200 includes, for example, a random access memory (RAM), and is used to process various types of data. The storage device 240 includes a solid state drive, a hard disk drive, or the like. The storage device 240 stores a reverse engineering program. In addition, the storage device 240 is used to store various types of data, such as pixel data (image data), setting information, and a measurement condition given from the measurement control unit 150. The measurement condition includes, for example, various settings set by the scanning module 260, such as a setting of the light projection unit 110 (pattern frequency or pattern type) and a type of the light receiving unit 120 (low magnification light receiving unit or high magnification light receiving unit), when the shape of the workpiece W is measured, which will be described later. Furthermore, the storage device 240 can also store brightness information, coordinate information, and attribute information of each pixel constituting a measurement image.

[0072] The CPU 210 is a control circuit or a control element that processes a given signal or data, performs various arithmetic operations, and outputs an arithmetic operation result. In the present specification, the CPU refers to an element or a circuit that performs arithmetic operations, and is not limited to a processor (such as a CPU, MPU, GPU, or TPU) for a general-purpose PC regardless of the name, and is used in the sense of including a processor (such as an FPGA, ASIC, or LSI), a microcomputer, or a chipset (such as a SoC).

[0073] The CPU 210 generates image data based on pixel data given from the measurement control unit 150. In addition, the CPU 210 performs various types of processing on the generated image data by using the work memory 230. For example, the CPU 210 generates measurement data indicating a three-dimensional shape of the workpiece W included in the field of view of the light receiving unit 120 at a certain position of the mounting unit 140 based on a light receiving signal output from the light receiving unit 120. The measurement data is an image itself acquired by the light receiving unit 120, and, for example, in a case where the shape of the workpiece W is measured by a phase shift method, a plurality of images constitute one piece of measurement data. Note that the measurement data can be point cloud data that is a set of points having three-dimensional position information, and the measurement data of the workpiece W can be acquired from the point cloud data. The point cloud data is data represented by an aggregation of a plurality of points having three-dimensional coordinates.

[0074] The movement control unit 144 determines whether to perform only the rotation operation of the rotary stage 143 or to perform both the rotation operation of the rotary stage 143 and the translation operation of the translation stage 141 based on the measurement data of at least a part of the workpiece W. As a result, the imaging range is automatically determined without the user's awareness in accordance with the outer shape of the workpiece W, and thus, the three-dimensional measurement becomes easy. Note that the movement control unit 144 can control the rotary stage 143 to rotate in a state where the movement in the XY direction of the translation stage 141 is stopped after the translation stage 141 moves in the XY direction, and thus, the shape around the workpiece W can also be acquired. Note that scanning can also be performed by relatively moving and rotating the workpiece W with respect to the measurement unit 100 in a state where the measurement unit 100 is fixed.

[0075] The display unit 400 is a member for displaying the fringe projection image acquired by the measurement unit 100, the depth image generated based on the fringe projection image, the texture image captured by the measurement unit 100, various user interface screens, and the like. The display unit 400 includes, for example, an LCD panel or an organic electroluminescence (EL) panel. Further, a touch panel is used for the display unit 400, and thus, it can also be used as the operation unit 250. Further, the display unit 400 can also display the image generated by the light receiving unit 120.

[0076] The light source unit 300 includes a control board 310 and an observation illumination light source 320. A CPU (not shown) is mounted on the control board 310. The CPU of the control board 310 controls the light projection unit 110, the light receiving unit 120, and the measurement control unit 150 based on a command from the CPU 210 of the controller 200. Note that this configuration is an example, and other configurations can be used. For example, the control board can be omitted by controlling the light projection unit 110 and the light receiving unit 120 by the measurement control unit 150 or by the controller 200. Alternatively, a power supply circuit for driving the measurement unit 100 can be provided in the light source unit 300.

[0077] The observation illumination light source 320 includes, for example, LEDs that emit three colors of red, green, and blue light. The brightness of the light emitted from each LED is controlled, and thus, any color of light can be generated from the observation illumination light source 320. The illumination light IL generated from the observation illumination light source 320 is output from the illumination light output unit 130 of the measurement unit 100 through a light guide member (light guide). Note that, as the observation illumination light source, other light sources such as a semiconductor laser (LD), a halogen lamp, and an HID can be appropriately used in addition to the LED. Specifically, in a case where an element capable of color shooting is used as the imaging element, a white light source can be used as the observation illumination light source.

[0078] The illumination light IL output from the illumination light output unit 130 irradiates the workpiece W with red light, green light, and blue light in a time-division manner. As a result, a color texture image can be obtained by combining texture images respectively captured by these RGB lights, and the texture image is displayed on the display unit 400.

[0079] A three-dimensional measurement program and an application for realizing the functions of the three-dimensional scanner 1 by the controller 200 are installed on the controller 200. Therefore, the three-dimensional measurement method according to the present application can be executed by using the three-dimensional scanner 1. The three-dimensional measurement method is a method for measuring the three-dimensional shape of the workpiece W, and is executed by a computer included in the controller 200. The three-dimensional measurement program for causing the computer to execute the three-dimensional measurement method can be recorded in the storage medium 1000. The storage medium 1000 can be, for example, an optical disc such as a CD-ROM or a DVD-ROM, or can be a semiconductor memory such as a memory card.

[0080] In the controller 200 in which the three-dimensional measurement program and the application are installed, the CPU 210, the ROM 220, the work memory 230, the storage device 240, and the like constitute Figure 5 The scanning module 260, the conversion module 270, the integration module 280, and the analysis module 290 illustrated in the drawing are provided in the controller 200. In this embodiment, the scanning module 260, the conversion module 270, the integration module 280, and the analysis module 290 are divided into four modules, but any two or more of the modules 260, 270, 280, and 290 can be integrated to constitute one module. In addition, a part of each of the modules 260, 270, 280, and 290 can be incorporated into another module. That is, Figure 5 The configuration example illustrated in the drawing is an example, and is not limited to Figure 5 The configuration example illustrated in the drawing is an example, and is not limited to

[0081] The scanning module 260 is a component that acquires image data of the workpiece W by measuring the shape of the workpiece W and creates mesh data of the workpiece W based on the image data. The conversion module 270 is a component that converts the mesh data created by the scanning module 260 into CAD data. The CAD data is three-dimensional shape information constituted by analysis surfaces and free-form surfaces, and includes surface data, solid data, data for design, and the like. The surface data is data of a shape surface including a free-form surface and an analysis curved surface, such as side surface data of a cylinder and plane data.

[0082] The integration module 280 is a component that transmits signals and data from the scanning module 260 to the conversion module 270 and the analysis module 290 and transmits signals and data from the conversion module 270 to the scanning module 260. In this example, the module can perform a plurality of arithmetic processes in one unit, and can also be referred to as, for example, a functional unit, a functional block, or the like.

[0083] The scanning module 260 includes, for example, a measurement condition setting unit 261, a scanning control unit 262, a point cloud acquisition unit 263a, a mesh data generation unit 263b, a scanning output unit 264, and the like. The measurement condition setting unit 261 is a means for setting a measurement condition of a shape of a workpiece. The scanning control unit 262 is a means for controlling the measurement unit 100 to generate image data in accordance with the measurement condition set by the measurement condition setting unit 261 and acquire measurement data of the workpiece W based on the generated image data.

[0084] The point cloud acquisition unit 263a is a means for acquiring point cloud data of the workpiece W based on the image data of the workpiece W acquired by the scanning control unit 262. The mesh data generation unit 263b is a means for acquiring the point cloud data acquired by the point cloud acquisition unit 263a, processing the acquired point cloud data, and converting the data into mesh data.

[0085] The scanning output unit 264 is a means for outputting the mesh data created by the mesh data generation unit 263b and additional data to the conversion module 270. The additional data is, for example, data including at least one of a measurement condition and data calculated from the measurement data of the workpiece W.

[0086] The scanning module 260 controls the measurement unit 100, generates a condition for measuring a shape of the workpiece W (a measurement device model, a measurement magnification, a resolution, and the like) and raw data (for example, image data) at the time of measurement, and three-dimensional data. The three-dimensional data is mesh data including a plurality of polygons, and can also be referred to as polygon data. A polygon is data including information specifying a plurality of points and information indicating a polygon surface formed by connecting the points, and can include, for example, information specifying three points and information indicating a triangular surface formed by connecting the three points. The mesh data and the polygon data can also be defined as data represented by aggregation of a plurality of polygons.

[0087] In the conversion module 270, the mesh data is converted into CAD data, and a conversion process is determined based on the measurement condition and the raw data. Specifically, the conversion module 270 includes, for example, a data input unit 271, a processing parameter determination unit 272, a CAD conversion unit 273, a CAD output unit 274, and the like. The data input unit 271 is a means for accepting the mesh data and the additional data output from the scanning output unit 264. The processing parameter determination unit 272 is a means for determining a processing parameter at the time of converting the mesh data into CAD data in accordance with the additional data accepted by the data input unit 271. The CAD conversion unit 273 is a means for converting the mesh data into CAD data in accordance with the processing parameter determined by the processing parameter determination unit 272. The CAD output unit 274 is a means for outputting the CAD data converted by the CAD conversion unit 273.

[0088] The analysis module 290 of the three-dimensional scanner 1 is a module that generates combined three-dimensional data of the workpiece W by generating a plurality of pieces of three-dimensional data of the workpiece W placed in different placement attitudes and combining the plurality of pieces of three-dimensional data. The analysis module 290 includes, for example, a data acquisition unit 291 that acquires three-dimensional data of the workpiece W mounted on the rotary stage 143. The user can mount the workpiece W on the rotary stage 143 in any attitude. For example, in a case where the three-dimensional shapes of the front surface and the back surface of the workpiece W are acquired, the three-dimensional data can be acquired by mounting the workpiece W on the rotary stage 143 in a placement attitude in which the front surface of the workpiece W faces upward so as to acquire the three-dimensional data, and then mounting the workpiece W on the rotary stage 143 in a placement attitude in which the back surface of the workpiece W faces upward so as to acquire the three-dimensional data. In addition, in a case where the three-dimensional shape of the side surface of the workpiece W is acquired, the three-dimensional data can be acquired by mounting the workpiece W on the rotary stage 143 in a placement attitude in which the side surface of the workpiece W faces upward. For example, the placement attitude in which the front surface of the workpiece W faces upward can be set as a first placement attitude, and the placement attitude in which the back surface of the workpiece W faces upward can be set as a second placement attitude. In addition, the placement attitude in which the side surface of the workpiece W faces upward can be a third placement attitude. The definition of the placement attitude is an example, and the placement attitudes can be different from each other. For example, the first placement attitude, the second placement attitude, and the third placement attitude can be defined in accordance with the shape of the workpiece W, the range in which it is desired to acquire the three-dimensional data, or the like. In addition, a fourth placement attitude and a fifth placement attitude can be defined, and the number of placement attitudes is not particularly limited.

[0089] The data acquisition unit 291 acquires first three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the first placement attitude, and second three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the second placement attitude. Similarly, the data acquisition unit 291 also acquires third three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the third placement attitude, fourth three-dimensional data that is three-dimensional data of the workpiece W placed on the rotary stage 143 in the fourth placement attitude, and the like.

[0090] The data acquisition unit 291 acquires three-dimensional data measured by the scanning module 260. The three-dimensional data acquired by the data acquisition unit 291 includes shape information and texture information of the workpiece W, and the data acquisition unit 291 acquires shape data with texture. Therefore, the data acquisition unit 291 acquires first three-dimensional data that is three-dimensional data including shape information and texture information of the workpiece W placed in the first placement attitude, and second three-dimensional data that is three-dimensional data including shape information and texture information of the workpiece W placed in the second placement attitude.

[0091] The data acquisition unit 291 receives the light-reception signal generated by the light-reception unit 120 of the measurement unit 100, generates a real-time image of the work W based on the received light-reception signal, and acquires the generated real-time image.

[0092] The mesh data generation unit 263b generates first mesh data that is mesh data of the work W placed in the first placement attitude and second mesh data that is mesh data of the work W placed in the second placement attitude. Similarly, the mesh data generation unit 263b can also generate third mesh data that is mesh data of the work W placed in the third placement attitude and fourth mesh data that is mesh data of the work W placed in the fourth placement attitude.

[0093] In a case where the mesh data generation unit 263b generates mesh data, the data acquisition unit acquires the first mesh data and the second mesh data generated by the mesh data generation unit 263b as first three-dimensional data and second three-dimensional data. Similarly, the third mesh data and the fourth mesh data can also be acquired.

[0094] For example, the three-dimensional data of the work W placed in the first placement attitude (first three-dimensional data) can be stored in the storage device 240. In this case, the read unit 292 included in the analysis module 290 reads the first three-dimensional data stored in the storage device 240. Similarly, the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data can be stored in the storage device 240. In this case, the read unit 292 reads the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data from the storage device 240.

[0095] In a case where the CAD data of the work W exists, the CAD data of the work W can be stored in the storage device 240. In this case, the read unit 292 reads the CAD data stored in the storage device 240 from the storage device 240.

[0096] Hereinafter, a scanning process in a case where the CAD data of the work W does not exist and a scanning process in a case where the CAD data of the work W exists will be described. Figure 6 An example of the scanning process in a case where the CAD data of the work W does not exist will be described. Before or after the start of this flow and before proceeding to Step SA1, the work W is mounted on the rotary stage 143 in the first placement attitude. In Step SA1, the scanning of the work W is started. In Step SA2, the measurement unit 100 scans the work W placed in the first placement attitude. For example, in a case where the work W is placed so that the front surface of the work W faces upward, since the back surface of the work W cannot be scanned, the scanning in Step SA2 is referred to as "one-side scanning". Figure 7A user interface screen 700 displayed at the start of measurement is shown. The user interface screen 700 is generated by the controller 200 and displayed on the display unit 400.

[0097] A real-time image display region 701 and a model display region 702 are provided on the user interface screen 700. In the real-time image display region 701, a real-time image generated by the data acquisition unit 291 is displayed. In the real-time image, the rotary stage 143 and the workpiece W mounted on the rotary stage 143 are displayed. Further, the measurement unit 100 can also be referred to as a scan head, and the scan head includes the light projection unit 110, the light reception unit 120, the illumination light output unit 130, and the measurement control unit 150. The display control unit 255 can also display the scan head and the workpiece W on the display unit 400. This display is effective, for example, in a case where the workpiece W is placed on any stage and the workpiece W is scanned from different angles by moving the scan head side.

[0098] In step SA2, the light projection unit 110 of the measurement unit 100 irradiates the workpiece W placed in the first placement attitude with measurement light. The light reception unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light reception signal output from the light reception unit 120 is received by the point cloud acquisition unit 263a, and first point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the first point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired first point cloud data, and converts the first point cloud data into first mesh data. Note that here, the processing of the point cloud data is the sparsification of the point cloud, the removal of point clouds outside the measurement region, the removal of noise point clouds, and the like. In step SA3, the first mesh data obtained by the processing in step SA2 is acquired as first three-dimensional data. The model of the workpiece W based on the first three-dimensional data is displayed in the model display region 702 of the user interface screen 700 shown. The user interface screen 700 is generated by the display control unit 255 included in the controller 200 and displayed on the display unit 400. Figure 7 The model display region 702 of the user interface screen 700 shown. The user interface screen 700 is generated by the display control unit 255 included in the controller 200 and displayed on the display unit 400.

[0099] In step SA4, the three-dimensional scanner 1 calculates an evaluation value and proposes a candidate attitude. The evaluation value in step SA4 is an example of an evaluation index that will be described later. When the user operates the attitude suggestion button 702a in the model display region 702, the controller 200 detects the operation. Then, as Figure 8 The display control unit 255 generates a candidate display region 703 that displays the next scannable attitude candidate and displays the generated candidate display region on the user interface screen 700. The user can select the attitude candidate on the user interface screen 700 in step SA5.

[0100] When the three-dimensional scanner 1 proposes a candidate posture, the candidate posture is calculated before the evaluation value is calculated. That is, as shown in Figure 5 The posture calculation unit 293 is a component that calculates a placement posture different from the first placement posture based on the first three-dimensional data acquired by the data acquisition unit 291, and specifically, calculates a recommended placement posture (hereinafter also simply referred to as "placement posture") different from the first placement posture based on the three-dimensional data read by the reading unit 292 from the work memory 230 or the storage device 240. The display control unit 255 superimposes and displays the recommended placement posture calculated by the posture calculation unit 293 on the real-time image acquired by the data acquisition unit 291.

[0101] When a placement posture different from the first placement posture is calculated, the posture calculation unit 293 first specifies the first placement posture of the workpiece W based on the first three-dimensional data. The first placement posture is specified, so the posture calculation unit 293 can calculate a placement posture different from the first placement posture. The distance between the measurement unit 100 and the workpiece can be set based on the focal length of the lens of the measurement unit 100. In the present embodiment, since the measurement unit 100 includes the rotary stage 143, for example, the posture calculation unit 293 can calculate a plurality of placement postures by virtually rotating the first three-dimensional data around the rotation axis of the rotary stage 143. When the first three-dimensional data is rotated, the first three-dimensional data can not be rotated once, and can be rotated at a rotation angle smaller than 360°.

[0102] Further, in the case where the workpiece W is large, there is also a case where scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times. Whether the measurement unit is moved with respect to the workpiece a plurality of times to perform scanning can be determined by the analysis module 290 based on whether the first three-dimensional data exceeds the measurable range (length, width, and height) of the measurement unit 100. In the case where the first three-dimensional data exceeds the measurable range of the measurement unit 100, it can be determined that scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times, and conversely, in the case where the first three-dimensional data is within the measurable range of the measurement unit 100, it can be determined that scanning is performed without employing a plurality of times of translating the measurement unit 100.

[0103] In the case where scanning is performed by translating the measurement unit 100 with respect to the workpiece W a plurality of times, the measurement range of the measurement unit 100 overlaps at a certain level or higher, and the measurement unit 100 is virtually arranged in a plurality of directions based on the center point of the translation so as to include the workpiece to the greatest extent. Note that, although a method for virtually moving the measurement unit 100 has been described, the present application is not limited thereto, and the workpiece can be moved and rotated.

[0104] The analysis module 290 includes an arithmetic unit 294. The arithmetic unit 294 calculates a relative position attitude of the recommended placement attitude with respect to the first placement attitude. In a case where the attitude calculation unit 293 calculates a plurality of placement attitudes, the arithmetic unit 294 calculates a relative position attitude with respect to the first placement attitude for each of the plurality of placement attitudes. For example, the arithmetic unit 294 calculates a conversion expression for converting a relative positional relationship between the first placement attitude and the recommended placement attitude. The arithmetic unit 294 applies the conversion expression to the three-dimensional data of the first placement attitude to convert the first placement attitude into the recommended placement attitude.

[0105] Since the placement attitude is different from the first placement attitude, the attitude calculation unit 293 can also calculate the candidate attitude based on the size of the contact area with the mounting surface 142 at each of the plurality of placement attitudes. When the contact area with the mounting surface 142 is too small, it can be difficult to mount the workpiece W on the mounting surface 142. However, when the placement attitude is calculated, it is possible to stabilize the workpiece W when the workpiece W is mounted on the mounting surface 142 by calculating a placement attitude that can ensure a predetermined or more contact area with the mounting surface 142. Although Figure 8 An example in which four placement attitudes are calculated is shown, but the number of placement attitudes calculated by the attitude calculation unit 293 is not limited to four, and can be any number of one or two or more.

[0106] Figure 9 A user interface screen 710 displayed when the selection of the candidate attitude is accepted is shown, which is generated by the display control unit 255 and displayed on the display unit 400. A model display area 711 in which a model of the workpiece W based on the first three-dimensional data is displayed and a candidate display area 712 are provided on the user interface screen 710. In the candidate display area 712, four placement attitudes calculated by the attitude calculation unit 293 are displayed. The placement attitudes are displayed in the candidate display area 712, so it is possible to present the next scannable placement attitude to the user.

[0107] An evaluation index display area 712a in which an evaluation index indicating whether the placement attitude of the workpiece presented to the user is a placement attitude suitable for the next scan is provided in the candidate display area 712. The evaluation index display area 712a is also generated by the display control unit 255 and displayed on the display unit 400.

[0108] The evaluation index is based on the amount of additional data to be added to the first 3D data by performing the next scan and the amount of overlap between the 3D data acquired by performing the next scan and the first 3D data. That is, the analysis module 290 includes an overlap region estimation unit 295 and an additional data volume estimation unit 296. The overlap region estimation unit 295 estimates the overlap region between the 3D data to be acquired by the data acquisition unit 291 and the first 3D data acquired by the data acquisition unit 291, in a placement state calculated by the attitude calculation unit 293. The overlap region estimation unit 295 can also estimate the degree of feature based on the distribution of points in the overlap region.

[0109] An example of a method for estimating the overlapping region using the overlapping region estimation unit 295 will be described. Here, in the presence of 3D CAD data for the workpiece W, the outermost surface when projecting the 3D CAD data relative to the outermost surface when projecting the first 3D data acquired by the data acquisition unit 291 can be further estimated as the overlapping region. However, as... Figure 6 The flowchart shown illustrates that, in the absence of 3D CAD data for workpiece W, since estimation based on 3D CAD data cannot be performed, the front surface of the existing first 3D data is used as the overlapping region. When workpiece W has a complex shape, there is a possibility that a portion of the front surface of the existing first 3D data is hidden by the second additional scan and subsequent additional scans. When 3D CAD data exists, the effect of hiding a portion of the front surface of the existing first 3D data can be considered when estimating the overlapping region; however, this effect is not considered when 3D CAD data is unavailable. However, when the workpiece has a shape close to a convex polygon, the effect of hiding a portion of the front surface of the existing first 3D data is absent, and substantially good results can be obtained. Therefore, the quality of the overlapping region can be determined by evaluating the area (amount of overlapping region) and the degree of feature. The degree of feature is obtained by analyzing the distribution of points in the overlapping region. Specifically, the deviation of coordinates or the normal of a point can be used as an evaluation value.

[0110] Furthermore, it is necessary to consider whether the workpiece W can be placed on the mounting surface 142 of the rotating stage 143 in the placement posture calculated by the posture calculation unit 293 during actual measurement. When the measuring unit 100 moves relative to the workpiece W, the analysis module 290 can perform conflict determination between the three-dimensional shapes of the measuring unit 100 and the base 600 and the three-dimensional CAD data or first three-dimensional data of the workpiece in computer graphics space.

[0111] Further, of course, since the workpiece W cannot be placed below the mounting surface on which the measurement unit 100 is mounted, the analysis module 290 performs a collision determination with the mounting surface (Z coordinate < 0). Further, since it is difficult to arrange the measurement unit 100 at an angle from directly above the workpiece or at a low angle with respect to the workpiece, the analysis module 290 can evaluate the "arrangement easiness" depending on the arrangement angle of the measurement unit 100.

[0112] In the case where the workpiece W is moved and rotated, the analysis module 290 also determines the easiness of placement of the workpiece W on the mounting surface 142. When the three-dimensional data is approximated by a convex polygon, the analysis module 290 calculates the area of the three-dimensional data or the bottom surface in the placement attitude calculated by the attitude calculation unit 293, and determines that the larger the calculated area of the bottom surface, the more stable the placement on the mounting surface 142 can be performed. Further, in the case where the three-dimensional CAD data of the workpiece W exists, the analysis module 290 acquires the coordinates of the center of gravity and the center point of the bottom surface, and calculates the distance between the center of gravity and the center point of the bottom surface. The analysis module 290 can determine the arrangement easiness of the workpiece W based on the distance between the center of gravity point and the center point of the bottom surface.

[0113] Next, the additional data amount will be described. The additional data amount estimation unit 296 is a component that estimates the additional data amount to be added to the first three-dimensional data by acquiring the three-dimensional data by the data acquisition unit 291 in a state where arrangement is performed in one arrangement attitude, for each of the plurality of arrangement attitudes calculated by the attitude calculation unit 293. The additional data amount estimation unit 296 can move the first three-dimensional data so as to have each placement attitude, and estimate the additional data amount based on the orientation of the normal vector after the movement, and for example, can estimate the additional data amount based on the positional relationship between the normal vector after the movement from a predetermined viewpoint and the line-of-sight direction of the measurement unit 100. For example, the inner product of the normal vector after the movement from the predetermined viewpoint and the line-of-sight direction of the measurement unit 100 is calculated, and thus the three-dimensional data facing the measurement unit 100 is specified, and the larger the amount of three-dimensional data facing the measurement unit 100, the larger the amount of additional data is estimated.

[0114] An example of the method for estimating the additional data amount by the additional data amount estimation unit 296 will be described. Assuming that scanning is performed with the attitude candidate calculated by the attitude calculation unit 293, the additional data amount is obtained by estimating how much additional data can be acquired with respect to the first three-dimensional data that has already been acquired (this can also be referred to as a "useable footage ratio"). Here, in the case where the three-dimensional CAD data of the workpiece W exists, the points that exist in the three-dimensional CAD data but do not exist in the first three-dimensional data become additional data, and the amount of additional data increases as the number of points increases.

[0115] However, as shown in the flowchart Figure 6 In the absence of three-dimensional CAD data of the workpiece W, since estimation based on three-dimensional CAD data cannot be performed, the point cloud included in the first three-dimensional data is analyzed, and the number of points of which the back surface of the scanned surface is visible is set as the additional data. This is because the back surface is not visible in the fully scanned object, and thus the portion of the back surface that is visible can be determined as the unscanned portion.

[0116] The back surface of the workpiece W can be detected by detecting that the normal of the point included in the first three-dimensional data points in the opposite direction of the measurement unit 100. With this detection process, the amount of additional data can be estimated in the absence of three-dimensional CAD data of the workpiece W. Note that the analysis module 290 can calculate the percentage of the area in which the remaining scanning needs to be performed or the scanned area of the workpiece W based on the area of the already scanned area and the amount of additional data when scanning in each direction. The analysis module 290 can also automatically determine that the scanning is complete if the scanned area exceeds a certain percentage. Either of the order of estimating the overlapping area and estimating the amount of additional data can be in the front.

[0117] After estimating the overlapping area and the amount of additional data for each placement attitude as described above, the evaluation unit 299 included in the analysis module 290 calculates an evaluation index. When calculating the evaluation index, the evaluation index can be calculated based on, for example, the following expression. Evaluation index = (area of overlapping area) x (amount of additional data) x (dispersion of points in overlapping area) x (ease of arrangement)

[0118] As described above, the evaluation unit 299 calculates the evaluation index in each of the plurality of placement attitudes based on the amount of additional data and the amount of overlapping area in each of the plurality of placement attitudes calculated by the attitude calculation unit 293, and calculates a comprehensive evaluation index based on each evaluation index. Note that in the case of using texture information, integration, average, maximum, median, or the like of a texture feature amount based on the local contrast or differential value of the texture information can be incorporated.

[0119] The display control unit 255 displays the calculated evaluation index in numerical form or graphical form on the display unit 400. Figure 9The example illustrates the graphical display of evaluation indices in the evaluation index display area 712a. When a user performs an operation to select any of the placement postures displayed in the candidate display area 712, the analysis module 290 specifies the selected placement posture. The display control unit 255 displays the evaluation index of the placement posture specified by the analysis module 290 in graphical form in the evaluation index display area 712a. When the user selects another placement posture, the display control unit 255 displays the evaluation index of that placement posture in the evaluation index display area 712a. As described above, since the display control unit 255 can display the evaluation index of each of the multiple placement postures calculated by the posture calculation unit 293 on the display unit 400, the user can obtain the evaluation index of each placement posture when multiple placement postures are presented.

[0120] The display format of the evaluation index does not have to be... Figure 9 The display control unit 255 can display on the display unit 400 either a graphical form or a numerical form, or a combination of graphical and numerical forms. Furthermore, the display control unit 255 can display on the display unit 400 each of an evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296 and an evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit 295. In this case, the analysis module 290 calculates the first evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296, and calculates the second evaluation index separately based on the amount of overlapping region estimated by the overlapping region estimation unit 295. The display control unit 255 can display only the first evaluation index on the display unit 400 in graphical or numerical form, or only the second evaluation index on the display unit 400 in graphical or numerical form, or display both the first and second evaluation indices on the display unit 400 in graphical or numerical form respectively.

[0121] The analysis module 290 includes a designation unit 297, which designates a candidate pose from multiple placement poses calculated by the pose calculation unit 293 based on an evaluation index. Specifically, the designation unit 297 acquires the evaluation index for each of the multiple placement poses. The designation unit 297 designates the placement pose with the highest evaluation index among the acquired multiple evaluation indices. Since the evaluation index is based on the amount of additional data estimated by the additional data estimation unit 296 and the amount of overlapping region estimated by the overlapping region estimation unit 295, the designation unit 297 designates a candidate pose based on the amount of additional data and the amount of overlapping region. When designating a candidate pose, the designation unit 297 can designate a candidate pose based on the shape of the overlapping region, the amount of overlapping region, and the amount of additional data estimated by the overlapping region estimation unit 295. For example, when the overlapping region includes an irregular shape, the evaluation index can be improved compared to a flat overlapping region because the alignment accuracy is higher than that of an overlapping region with a flat shape.

[0122] When the designated unit 297 identifies a candidate posture, the display control unit 255 displays the candidate posture designated by the designated unit 297 on the display unit 400. The candidate posture displayed by the display unit 400 is the workpiece placement posture recommended by the 3D scanner 1 to the user. Therefore, the user can confirm the appropriate placement posture by viewing the display unit 400.

[0123] exist Figure 6 In step SA6, the user determines whether the candidate poses displayed on the display unit 400 are the desired poses. Since the placement pose is specified based on an evaluation index, and due to the large amount of additional data and overlapping area, it is believed that accurate combined 3D data can be obtained by placing the workpiece in this pose. However, since the designation unit 297 only specifies the placement pose based on the evaluation index, it may not necessarily specify the user's highly relevant area (the area requiring high-precision 3D data). Therefore, in the 3D scanner 1 of this embodiment, multiple placement poses are presented to the user along with evaluation indices, and the user can select the placement pose with a relatively high evaluation index and best suited to their interests from the presented multiple placement poses. Specifically, the analysis module 290 includes a receiving unit 298 that accepts the user's selection or adjustment operation for the placement pose. For example, when the user performs a selection or adjustment operation from... Figure 9 When selecting a desired placement posture from the candidate display areas 712 displayed on the user interface screen 710, the receiving unit 298 accepts the selected operation input. The display control unit 255 displays the placement posture accepted by the receiving unit 298 in the candidate display area 712.

[0124] Further, in step SA7, the candidate posture displayed on the display unit 400 can be adjusted. Specifically, the accepting unit 298 accepts an operation input of the user for adjusting one of the candidate postures specified by the specifying unit 297. The user operates the operation unit 250 in the computer graphics space to adjust the positional relationship between the measurement unit 100 and the workpiece W. For example, the workpiece W in the candidate posture is moved horizontally, moved in the height direction, or rotated. When the user completes the adjustment of the candidate posture, the process proceeds to step SA8, and the posture calculating unit 293 specifies the placement posture for which the adjustment is completed, the overlap area estimating unit 295 estimates the amount of the overlap area between the three-dimensional data acquired by the data acquiring unit 291 in the placement posture for which the adjustment is completed and the first three-dimensional data, and the additional data amount estimating unit 296 estimates the amount of the additional data. The analysis module 290 recalculates the evaluation index based on the estimated amount of the overlap area and the estimated amount of the additional data, and presents the evaluation index to the user. Thus, in the case where the user scans the region of high interest, the user can determine whether the posture is likely to be successful in the alignment with the first three-dimensional data. Note that the evaluation value calculation and the posture candidate suggestion in step SA4 and the selection of the posture candidate in step SA5 can be skipped, and the posture calculating unit 293 can specify the placement posture for which the user completes the adjustment of the candidate posture in step SA7.

[0125] In step SA9, the analysis module 290 generates a computer graphics workpiece (model) in the determined placement posture. The display control unit 255 superimposes the computer graphics workpiece generated by the analysis module 290 on the real-time image acquired by the data acquiring unit 291, and displays the superimposed image on the display unit 400. The computer graphics workpiece is semitransparent, but can also be opaque.

[0126] In step SA10, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the workpiece W is captured together with the mounting surface 142 by the light receiving unit 120, and a real-time image including the workpiece W is acquired by the data acquiring unit 291 and displayed on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W mounted on the mounting surface 142 overlaps the computer graphics workpiece.

[0127] In step SA11, the user determines whether the actual workpiece W installed on the installation surface 142 is installed so as to overlap the computer graphics workpiece. In a case where the actual workpiece W installed on the installation surface 142 cannot be installed so as to overlap the computer graphics workpiece, the processing proceeds to step SA5, and another placement attitude is selected. In a case where the actual workpiece W is installed so as to overlap the computer graphics workpiece, since the workpiece W is in the second placement attitude, the processing proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement attitude.

[0128] In step SA12, the light projection unit 110 of the measurement unit 100 irradiates the workpiece W placed in the second placement attitude with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained by the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0129] In step SA14, the alignment unit 290A included in the analysis module 290 aligns the first three-dimensional data (three-dimensional data of the workpiece placed in the first placement attitude) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. At the time of this alignment, the overlapping region extracted by the extraction unit 290B included in the analysis module 290 is used. The extraction unit 290B is a component that extracts the overlapping region between the three-dimensional data converted to the recommended placement attitude by the arithmetic unit 294 and the three-dimensional data placed in the second placement attitude. The extraction unit 290B can extract the overlapping region by using, for example, the normal vector of the three-dimensional data and the color information of the workpiece. That is, the alignment unit 290A can perform alignment based on the three-dimensional data included in the overlapping region extracted by the extraction unit 290B. In a case where alignment is performed by using the texture information of the workpiece W, the texture feature is estimated based on the brightness or color information or the local contrast or difference that constitutes the texture information. This texture feature can be used in combination with the shape feature by various operations such as addition and multiplication.

[0130] In the present embodiment, the evaluation index of the placement attitude is calculated so that the amount of the overlapping region and the amount of the additional data are large at the time of the next scan, and so that the alignment of the three-dimensional data by the alignment unit 290A is likely to be successful. Then, while the user is presented with the placement attitude whose evaluation index is high, the final placement attitude is determined while the user adjusts the placement attitude, and the placement attitude is superimposed and displayed on the real-time image so that the user can arrange the actual workpiece W in an attitude close to the placement attitude. Thus, the alignment of the placement attitude is performed as the initial position, and therefore the success rate of the alignment can be increased without imposing an additional burden on the user.

[0131] The alignment unit 290A can acquire a normal line of the three-dimensional data of the workpiece W placed in the second placement attitude. The alignment unit 290A can also perform alignment based on the orientation of the normal line of the three-dimensional data of the workpiece W placed in the second placement attitude, the relative positional relationship calculated by the arithmetic unit 294, the orientation of the normal line of the three-dimensional data of the workpiece placed in the first placement attitude, and the shape features extracted from the first three-dimensional data and the second three-dimensional data. That is, the alignment unit 290A first narrows down the range of the candidate corresponding points based on the angle between the orientation of the normal line obtained by rotating the normal line of the three-dimensional data of the workpiece W placed in the first placement attitude based on the relative positional relationship and the orientation of the normal line of the three-dimensional data of the workpiece placed in the second placement attitude. Then, based on the corresponding point candidates after the range is narrowed down, alignment can be performed by using the shape features extracted from the first three-dimensional data and the second three-dimensional data.

[0132] In addition, when the user places the workpiece W, a high-precision initial estimate value can also be obtained by pattern matching between the computer graphics workpiece and the actual workpiece W through computer graphics. This is because the relative position between the measurement unit 100 and the workpiece W can be obtained from the correspondence relationship between the computer graphics workpiece and the actual workpiece W in the real-time image through epipolar geometry.

[0133] The initial estimate value can be used to detect and extract a region where two point clouds overlap, detect a false correspondence relationship of a corresponding point candidate, and detect a false candidate from a position attitude candidate obtained from the corresponding point candidate. In the case where the alignment source point cloud P in the first three-dimensional data that has been scanned is aligned with the alignment destination point cloud Q that is newly scanned, the alignment unit 290A can specify a corresponding point of a point included in the first three-dimensional data from the second three-dimensional data, and adjust the position attitude of the first three-dimensional data and the second three-dimensional data based on the specified corresponding point.

[0134] In Figure 11In step SB1 of the flowchart shown, the arithmetic unit 294 acquires the alignment source point cloud P, and in step SB2, the arithmetic unit 294 acquires the alignment destination point cloud Q. In step SB3, the arithmetic unit 294 acquires the initial estimated pose. In step SB4, the arithmetic unit 294 performs a transformation for the alignment source by using the placement pose determined as described above. This transformation can be performed by multiplying by a rotation matrix and adding a translation component. Thus, the transformed point cloud P' exists in substantially the same position as the point cloud of the alignment destination (step SB5).

[0135] In step SB6, the extraction unit 290B can extract Qo by extracting a point cloud P'o in which the points of the point cloud Q exist in the vicinity after the transformation and applying the point cloud P'o in reverse. These point clouds are referred to as overlapping region point clouds. Note that in the case where normal vectors or color information are assigned to the point clouds, the accuracy of the overlapping region can be changed by using the presence or absence of a point cloud with a high degree of similarity in the vicinity as a condition for extraction of the overlapping region. Furthermore, the definition of "vicinity" varies according to the degree to which the user can arrange the actual workpiece W to be close to the workpiece in the computer graphics in the position pose. In addition, the range of "vicinity" can be determined according to the scanning range of the measurement unit 100 or the size of the workpiece W, and the range of "vicinity" can be changed by the user.

[0136] The alignment unit 290A performs alignment using only the overlapping region extracted by the extraction unit 290B. Thus, the occurrence of erroneous correspondence of points present in the non-overlapping region can be eliminated.

[0137] However, the extraction of the overlapping region by the extraction unit 290B requires a computational cost. This is because, for example, when extracting the point cloud P'o, the point Q closest to each point of the point cloud P' is searched for positioning, and assuming that the number of points P is M and the number of points Q is N, the computational cost O is O(M*N).

[0138] Thus, the occurrence of erroneous correspondence can be suppressed by hiding surface removal as an alternative to the overlapping region. This is because the initial estimated value described above is a value obtained by shape analysis to obtain an overlapping region suitable for alignment. Each point cloud of P' and Q is projected by a virtual camera based on the initial estimated value, and the hidden surface removal based on the depth buffer generally used in computer graphics (only extracting a point whose depth position viewed from the camera is the minimum value) can eliminate data of a back surface as erroneous correspondence.

[0139] Further, simple hidden surface removal using normal vectors can be performed as a simpler method. The hidden surface removal can be performed by extracting only points directed in a direction facing the measurement unit 100 from each of the point clouds P' and Q in the camera coordinate system. In the normal vector [nx, ny, nz] in the camera coordinate system, only the normal vector in which nz is negative is extracted. Thus, the calculation cost can be reduced.

[0140] Figure 12 is a schematic view related to the hidden surface removal. A white arrow W1 extending in a direction orthogonal to each surface of the workpiece W arranged on the mounting surface 142 indicates a normal vector of each surface. Further, a thin arrow W2 represents a component of the normal vector parallel to the Z axis of the light receiving unit 120, and an arrow W3 represents a component of the normal vector perpendicular to the Z axis of the light receiving unit 120. In the arrow W1, a dotted line indicates a normal in which the Z axis component of the light receiving unit 120 is determined to be positive, and a surface corresponding to the dotted arrow W1 is determined to be a hidden surface.

[0141] In Figure 11 In the step SB8 shown, the extraction unit 290B extracts a shape feature and a brightness feature of each point from the first three-dimensional data corresponding to the first placement attitude and the second three-dimensional data corresponding to the second placement attitude acquired by the data acquisition unit 291.

[0142] In the step SB10, the feature amount acquired in the step SB9 is input to the extraction unit 290B to detect the corresponding point. In the absence of the initial estimated value, a candidate of the corresponding point is obtained by a feature difference between the first three-dimensional data and the second three-dimensional data, a correlation between the first three-dimensional data and the second three-dimensional data, and the like. At this time, a point initially present in the distance but having similar features in the shape and the brightness can be selected as the corresponding point. On the other hand, there is an initial estimated value, and thus only points in the vicinity and points in which the orientation of the normal vector is close can be the corresponding candidates, and a correlation value of the features can be weighted based on the distance between the points and the inner product of the normal vectors.

[0143] Assuming that the i-th point of the point cloud P of the alignment source is pi, the point obtained by converting pi to the initial estimated value is p'i, the j-th point of the point cloud Q of the alignment destination is qj, the feature vector thereof is F(*), the normal vector thereof is N(*), the evaluation value w_d(dist(p'i, qj))*wn(1-dot(N(p'i), N(qj))*CORR(F(p'i), FU(qj)) can be obtained. wd is a weight related to the distance, and wn is a weight related to the orientation of the normal. When this weight is a step function of [1, 0], only the proximity weight is selected, and when this weight is a monotonically decreasing function, a continuous weight can be applied. Points having a high evaluation value are set as the corresponding points, and thus, it is possible to eliminate false corresponding points having similar shapes and brightness distributions. A matrix C(pi, qj) representing the corresponding point candidates is obtained as the output of the corresponding point detection unit. It can be represented by a matrix in which the points determined as the corresponding points are 1, and the other points are 0.

[0144] In step SB11, a candidate pose is acquired from these corresponding candidate points. A method for obtaining the final placement pose of the workpiece W in steps SB12 to SB17 will be described. For example, a random sample consensus (RANSAC) method can be used as the method for obtaining the final placement pose of the workpiece W. Hereinafter, an example using RANSAC will be described. In the pose candidate calculation of RANSAC, a candidate is calculated from a plurality of randomly extracted corresponding points (step SB12). As a result, a candidate pose can be acquired (step SB13). In the case of using only the coordinates of the points, the corresponding relationship between three points is required, and in the case of using the normal and the coordinates of the points, the corresponding relationship between two points is required. Taking the corresponding relationship between three points as an example, a rotation matrix R and a translation vector t that minimize the squared distance between the corresponding p' and q are obtained by linear and nonlinear optimization (step SB14).

[0145] Sum_C(p_i,q_j)(q_j-p'_i*R+t)^2

[0146] The present embodiment is not limited to this candidate calculation method, and a candidate can be calculated by using any method. For example, a method using graph cuts is also applicable to the present embodiment.

[0147] It is determined whether the rotation matrix and the translation vector obtained in this way are close to the initial estimate value to detect whether a correct candidate is obtained. When the sum of squares of each element is used, the initial estimate value is t, and the candidate is t', the translation vector can be defined as dist(t, t') = (t_x - t'_x)^2 + (t_y - t'_y)^2 + (t_z - t'_z)^2. In addition, the rotation matrix can be obtained as Dist(R, R') = arccos[(trace(R^T * R') - 1) / 2].

[0148] A placement attitude in which the error of the translation component is within a predetermined size and the angular error is within a predetermined angle can be extracted as a correct candidate. In step SB16, a detected attitude is obtained from these candidates by the final attitude selection process, and a selected candidate is acquired (step SB17). This is obtained by a method for evaluation by a predetermined evaluation value. For example, by using the rotation matrix R and the translation component t representing the candidate attitude, a conversion is performed as P" = R * P' + t, and the number of points in the point cloud Q that satisfy dist(P" - Q) < threshold can be set as the evaluation value. However, the calculation of the evaluation value to be combined in this embodiment is not particularly limited. Since this evaluation value calculation can also be performed only in the overlapping region, it is possible to suppress the phenomenon in which the evaluation value is erroneously increased due to the presence of a corresponding point in its vicinity although some points do not overlap. In this way, the second placement attitude is specified, and the workpiece W placed at the specified second placement attitude is scanned by the measurement unit 100. Therefore, the second three-dimensional data having high-precision alignment with the first three-dimensional data and the large amount of additional data can be acquired.

[0149] When performing alignment between the first three-dimensional data and the second three-dimensional data, the alignment unit 290A can perform alignment based on shape features and image features of the three-dimensional data. That is, the analysis module 290 includes a shape feature extraction unit (first extraction unit) 290C that extracts shape features of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291, and an image feature extraction unit (second extraction unit) 290D. The shape feature extraction unit 290C includes a neural network including an input layer, a plurality of intermediate layers, and an output layer. The input layer of the neural network is a component that accepts input of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291. The plurality of intermediate layers of the neural network are components that extract shape features based on the input accepted by the input layer. The output layer of the neural network is a component that outputs the shape features extracted in the intermediate layers. On the other hand, the image feature extraction unit 290D is a component that extracts image features from texture information included in each of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0150] As described above, in the present embodiment, when alignment between the first three-dimensional data and the second three-dimensional data is performed, image features such as brightness and color are incorporated into the feature vector using deep learning. In the case where the feature vector of each point of the image or the point cloud is calculated, a "convolution" arithmetic operation of multiplying the coordinates and the color, which are information of the own point and the information of the peripheral point, by a coefficient and obtaining a sum or a maximum value thereof can be performed. For example, in a convolutional neural network (CNN) in an image, a large number of 3x3 filters having coefficients determined by pre-learning are provided, the output value of each filter is set, and the set output vector is set as the output vector of the point, and a result obtained by repeating the output vector with different filters several times is set as the feature vector.

[0151] In the case where the point cloud is included in the first three-dimensional data or the second three-dimensional data, since it cannot be guaranteed that the data exists at a regular interval different from the image, other methods can be used. For example, a method for applying a CNN three-dimensionally by sampling points in a voxel, multiplying the points by a coefficient when the points exist, and outputting 0 when the points do not exist, PointNet++ for applying the same coefficient to each point to obtain a maximum value of the output, and a method such as KPConv for performing an interpolation arithmetic operation with respect to a deviation of the position of the point cloud from a coefficient determined on a grid to obtain a coefficient and convolving the coefficient can be exemplified.

[0152] In the processing of the point cloud included in the first three-dimensional data or the second three-dimensional data, since how the points are arranged is unknown, the amount of calculation becomes enormous and the processing time becomes long. In the technical field of the three-dimensional scanner 1 in this example, when the processing time becomes long, since the work is delayed and not performed, it is necessary to set the processing time without a problem in actual use. Therefore, the number of points that can be processed is naturally limited, and as an example, an upper limit of the number of key points can be set to several thousand points, and about several ten thousand points in the entire point cloud. Since the number of point clouds that can be scanned at once by the measurement unit 100 is several million points, the points used in deep learning need to be thinned to about 1%.

[0153] In the case where the image information is most simply input to the input layer of the neural network, the image information (brightness and color) can be input in addition to the coordinates and the normal vector, but in this case, the number of data input to the input layer is too small, and the information useful as the image is lost. Therefore, there is a case where the performance is not improved.

[0154] Hereinafter, the present embodiment will be described with reference to Figure 13The flowchart shown describes the process of extracting shape features and image features. In step SCI, the analysis module 290 acquires a point cloud of the first three-dimensional data or the second three-dimensional data (input point cloud). In steps SC2 and SC3, a rough thinning process and a dense thinning process are performed, respectively. That is, as shown in Figure 5 As shown, the analysis module 290 includes a resolution conversion unit 290E that converts the resolution of the first three-dimensional data and the resolution of the second three-dimensional data acquired by the data acquisition unit 291. The resolution conversion unit 290E performs a first conversion process (rough thinning process) that converts the first three-dimensional data and the second three-dimensional data into three-dimensional data having a first resolution lower than the resolutions acquired by the data acquisition unit 291. That is, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into third three-dimensional data as three-dimensional data having the first resolution, and converts the second three-dimensional data acquired by the data acquisition unit 291 into fourth three-dimensional data as three-dimensional data having the first resolution. In the first conversion process by the resolution conversion unit 290E, since the resolutions of the three-dimensional data acquired by the data acquisition unit 291 are reduced to low-resolution three-dimensional data, the number of points of the three-dimensional data is smaller than the number of points at the time of acquisition by the data acquisition unit 291.

[0155] The first conversion process of the resolution conversion unit 290E corresponds to step SC2. The third three-dimensional data (third point cloud) and the fourth three-dimensional data (fourth point cloud) as the shape feature extraction point cloud (Q) are generated by step SC2, and the shape feature extraction unit 290C acquires the shape feature extraction point cloud (Q) in step SC4. The shape feature extraction unit 290C extracts shape features from shape information included in each of the first three-dimensional data having the first resolution and the second three-dimensional data having the first resolution converted by the resolution conversion unit 290E.

[0156] The resolution conversion unit 290E performs a second conversion process (dense sparsification process) of converting the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291 into three-dimensional data having a second resolution lower than the resolution acquired by the data acquisition unit 291 and higher than the first resolution. That is, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into fifth three-dimensional data as three-dimensional data having the second resolution, and converts the second three-dimensional data acquired by the data acquisition unit 291 into sixth three-dimensional data as three-dimensional data having the second resolution. In the second conversion process by the resolution conversion unit 290E, since the resolution of the three-dimensional data acquired by the data acquisition unit 291 is lowered to low-resolution three-dimensional data, the number of points of the three-dimensional data is smaller than the number of points at the time of acquisition by the data acquisition unit 291, but since the resolution is higher than the resolution of the first conversion process, the number of points is larger than the number of the first conversion process. The second conversion process of the resolution conversion unit 290E corresponds to step SC3.

[0157] The fifth three-dimensional data (fifth point cloud) and the sixth three-dimensional data (sixth point cloud) (Q’) generated as the image feature extraction point cloud by step SC3 are acquired by the image feature extraction unit 290D in step SC5. The image feature extraction unit 290D extracts an image feature from the texture information included in each of the first three-dimensional data and the second three-dimensional data having the second resolution converted by the resolution conversion unit 290E.

[0158] Note that the above-described step SC3 can be skipped, and the image feature extraction unit 290D can generate the image feature extraction point cloud by using the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0159] In the first conversion process and the second conversion process, for example, random sparsification for randomly selecting a predetermined number of data, voxel sparsification for calculating a voxel grid and reducing a plurality of point clouds entering a voxel to only one average value, or the like can be used. Here, the point cloud before sparsification (point cloud before the second conversion process) can be used as a point cloud for image feature extraction.

[0160] In step SC6, the shape feature extraction unit 290C extracts a key point from the first three-dimensional data and the second three-dimensional data of low resolution. For example, when extracting a key point, there are a method for randomly selecting and extracting a key point, a method for extracting a feature point as a key point by deep learning or rule-based point cloud analysis, or the like, and any method can be used. The key point extracted by the shape feature extraction unit 290C is acquired by the shape feature extraction unit 290C (step SC7).

[0161] In step SC8, the analysis module 290 acquires the shape feature extraction point cloud (Q) acquired in step SC4 and the key point acquired in step SC7, and the analysis module 290 samples points around the key point. At this time, the points for shape feature extraction are sampled based on the shape feature extraction point cloud (Q) acquired in step SC4.

[0162] In step SC12, which proceeds from step SC5, the analysis module 290 acquires the image feature extraction point cloud (Q') acquired in step SC5 and the key point acquired in step SC7, and the analysis module 290 samples points around the key point. The points for image feature calculation are sampled based on the image feature extraction point cloud (Q') acquired in step SC5. In addition, since the number of points of the image feature extraction point cloud (Q') is large, the vicinity range to be sampled can be narrowed compared to the shape feature extraction point cloud (Q).

[0163] In step SC9, which proceeds from step SC8, the shape feature is input to the shape feature extraction unit 290C. For example, the shape feature vector can be calculated only from the vicinity of the point sampled for the shape feature. The method for calculating the shape feature vector is not particularly limited, and for example, a method for projecting image information of a point cloud according to the normal of a key point to obtain a patch image and calculating a feature vector by a calculation such as CNN or scale-invariant feature transform (SIFT), a method for creating a histogram based on RGB or luminance information of a sampled point and using the histogram as a feature vector, or the like can be used. In step SC10, the shape feature extraction unit 290C calculates the shape feature vector of the key point.

[0164] On the other hand, in step SC13, which proceeds from step SC12, the image feature is input to the image feature extraction unit 290D. Here, the feature vector of the image is calculated. The feature vector of the key point is calculated by using all the points around the key point sampled from the image feature extraction point cloud (Q'). In step SC14, the image feature extraction unit 290D calculates the image feature vector of the key point, and in step SC15, the image feature vector calculated by the image feature extraction unit 290D is acquired.

[0165] That is, the image feature extraction unit 290D extracts a key point to be used for calculating a shape feature from the first three-dimensional data and the second three-dimensional data at low resolution. The image feature extraction unit 290D can specify a region corresponding to the key point extracted from the first three-dimensional data and the second three-dimensional data before conversion by the resolution conversion unit 290E. The image feature extraction unit 290D extracts an image feature from the specified corresponding region.

[0166] Note that when the eigenvector of the image is calculated, the image feature extraction unit 290D can specify a region corresponding to the key point extracted by the shape feature extraction unit 290C from the first three-dimensional data and the second three-dimensional data, and extract the image feature from the region corresponding to the key point specified by the image feature extraction unit 290D.

[0167] In the calculation of the shape eigenvector in step SC11, a three-dimensional to six-dimensional vector having X, Y, and Z coordinates and, if necessary, normal information Nx, Ny, and Nz can be input. As described above, the shape feature extraction unit 290C extracts a key point from the low-resolution first three-dimensional data and the second three-dimensional data as a partial region for calculating a shape feature, and extracts a shape feature for the extracted key point.

[0168] In step SC16, a candidate pair is obtained by comparing the shape eigenvector, the image eigenvector, or the shape eigenvector including the image feature with the key points of the three-dimensional data of the alignment source and the three-dimensional data of the alignment destination. The method for calculating the candidate pair is not particularly limited, and examples thereof include a method for obtaining a distance of the eigenvectors and making a pair with a small distance, and a method for making a pair with a high output value of a deep learning module that outputs a degree of correspondence. A squared Euclidean distance (L2 distance) or a cosine similarity is used as the distance. Furthermore, at this time, in order to compare the shape eigenvector and the image eigenvector, a method for simply obtaining a pair by using a vector obtained by combining the two vectors as input, a sum, a maximum value, a minimum value, and the like of distances calculated from the shape eigenvector and the image eigenvector can be used. Since the points of the candidate pair are used, a position relationship (rotation and translation) between the point clouds can be obtained by RANSAC or a deep learning-based method.

[0169] Figure 14 A case in which a vector to which an image feature is added is input to the shape feature extraction unit in the case where the shape feature and the image feature are extracted is shown. The processing proceeds to step SC9' in step SC8 in Figure 13 , and the processing proceeds to step SC13' in step SC12 in Figure 14 , and the processing proceeds to step SC13' in step SC12 in Figure 13 , and the processing proceeds to step SC13' in step SC12 in Figure 14 . Steps SC13' to SC15' are the same as steps SC13 to SC15 in Figure 13 . In step SC10', the image eigenvector is input to the shape feature extraction unit 290C. In this case, the output shape eigenvector is a shape eigenvector that already includes the image feature (step SC11'). Step SC16' is the same as step SC16 in Figure 13 .

[0170] The alignment unit 290A performs a process of combining the global alignment and the local alignment as a process of aligning the point cloud of the first three-dimensional data and the point cloud of the second three-dimensional data with high precision. In this process, the global alignment is performed by the low-resolution point cloud, and the local alignment is performed by using the position for conversion. For example, the local alignment is performed by using the point cloud data with the first resolution (low resolution) for shape feature extraction in the global alignment, using the point cloud with a third resolution higher than the first resolution for image feature extraction, and using the point cloud data with a second resolution higher than both the first resolution and the third resolution as input. Note that the point cloud data for the local alignment can be the point cloud data with the second resolution, or can be the point cloud data with the same resolution as that of the point cloud generated by receiving the light-receiving signal in the point cloud acquisition unit 263a.

[0171] Specifically, the alignment unit 290A acquires the shape feature extracted by the shape feature extraction unit 290C and the image feature extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A can perform the global alignment based on the shape feature extracted by the shape feature extraction unit 290C and the image feature extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A performs a first alignment process of calculating a low-precision alignment parameter indicating the relative position attitude of the second three-dimensional data with respect to the first three-dimensional data.

[0172] When the low-precision alignment parameter is calculated, steps SD1 to SD7 are performed as shown in Figure 15 Steps SD1 to SD5 are the same as steps SC1 to SC5 shown in Figure 13 In step SD6 shown in Figure 15 , the shape feature extraction point cloud acquired in step SD4 and the image feature extraction point cloud acquired in step SD5 are input to the alignment unit 290A. The alignment unit 290A performs the global alignment of the input point clouds and calculates the low-precision alignment parameter.

[0173] After the first alignment process, the alignment unit 290A acquires the low-precision alignment parameter calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data. The alignment unit 290A can perform the local alignment based on the low-precision alignment parameter, the first three-dimensional data, and the second three-dimensional data. In this case, the alignment unit 290A performs a second alignment process of calculating a high-precision alignment parameter indicating the relative position attitude of the second three-dimensional data with respect to the first three-dimensional data.

[0174] When the high-precision alignment parameter is calculated, steps SD8 to SD14 are performed as shown in Figure 16As shown, in step SE1, the analysis module 290 acquires an input point cloud of the first three-dimensional data. In step SE2, the analysis module 290 acquires an input point cloud of the second three-dimensional data. In step SE3, the analysis module 290 acquires a low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. Figure 15 The process of the flowchart shown calculates the low-precision alignment parameter.

[0175] The analysis module 290 includes a coordinate conversion unit 290F that performs coordinate conversion of the first three-dimensional data based on the low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. In step SE4, the coordinate conversion unit 290F performs coordinate conversion of the input point cloud of the first three-dimensional data based on the low-precision alignment parameter. In step SE5, a point cloud after the coordinate conversion is acquired. Figure 15 The process of the flowchart shown calculates the low-precision alignment parameter.

[0175] The analysis module 290 includes a coordinate conversion unit 290F that performs coordinate conversion of the first three-dimensional data based on the low-precision alignment parameter calculated by the process of the flowchart shown in FIG. 7. In step SE4, the coordinate conversion unit 290F performs coordinate conversion of the input point cloud of the first three-dimensional data based on the low-precision alignment parameter. In step SE5, a point cloud after the coordinate conversion is acquired.

[0176] In step SE6, the input point cloud of the second three-dimensional data acquired in step SE2 and the point cloud after the coordinate conversion acquired in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point clouds and calculates a high-precision alignment parameter. In step SE7, the high-precision alignment parameter is acquired. As described above, in the second alignment process, the alignment unit 290A acquires the first three-dimensional data and the second three-dimensional data of which coordinate conversion is performed by the coordinate conversion unit 290F, and performs local alignment based on the first three-dimensional data and the second three-dimensional data.

[0177] In step SE6, the input point cloud of the second three-dimensional data acquired in step SE2 and the point cloud after the coordinate conversion acquired in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point clouds and calculates a high-precision alignment parameter. In step SE7, the high-precision alignment parameter is acquired. As described above, in the second alignment process, the alignment unit 290A acquires the first three-dimensional data and the second three-dimensional data of which coordinate conversion is performed by the coordinate conversion unit 290F, and performs local alignment based on the first three-dimensional data and the second three-dimensional data. Figure 6 After the alignment process in step SA14 shown above, a combination process is performed. That is, as Figure 5 As shown, the analysis module 290 includes a combination unit 290G. The combination unit 290G is a component that combines the first three-dimensional data of the workpiece W placed in the first placement attitude and the second three-dimensional data of the workpiece W placed in the second placement attitude, which are aligned by the alignment unit 290A, to generate combined three-dimensional data. The combination unit 290G can also acquire the first mesh data as the first three-dimensional data, and acquire the second mesh data as the second three-dimensional data. In this case, the combination unit 290G combines the first mesh data and the second mesh data to generate combined mesh data as the combined three-dimensional data.

[0178] In addition, Figure 12The removal of the hidden surface shown is performed before generating the combined three-dimensional data, and is used for the alignment by the alignment unit 290A. That is, the alignment unit 290A can perform the alignment by using the three-dimensional data from which the hidden surface is removed. Then, the combined unit 290G generates the combined three-dimensional data based on the alignment performed by using the three-dimensional data from which the hidden surface is removed. Here, the generation of the combined three-dimensional data is performed before the hidden surface removal, that is, using the three-dimensional data including the hidden surface.

[0179] The combined unit 290G can acquire the shape information of the workpiece W based on the first light-reception signal output from the light-reception unit 120, and can acquire the texture information of the workpiece W based on the second light-reception signal output from the light-reception unit 120. In a case where the shape information of the workpiece W and the texture information of the workpiece W are acquired, the combined unit 290G generates three-dimensional data including the shape information and the texture information of the workpiece W.

[0180] In addition, in a case where the data acquisition unit 291 acquires the combined three-dimensional data obtained by combining the first three-dimensional data and the second three-dimensional data, the attitude calculation unit 293 can calculate the placement attitude different from the first placement attitude based on the combined three-dimensional data acquired by the data acquisition unit 291. As described above, in addition to the already generated combined three-dimensional data, the placement attitude at the time of performing the scanning can be presented to the user. In this case, the additional data amount estimation unit 296 estimates the additional data amount to be added to the combined three-dimensional data. In addition, the overlapping region estimation unit 295 estimates the overlapping region of the combined three-dimensional data. Thus, the evaluation index based on the additional data amount and the overlapping region amount can be presented to the user at the time of performing the scanning in the additional placement attitude.

[0181] When the combination processing is completed, the processing proceeds to Figure 6 The step SA15 shown. In the step SA15, the user determines whether there is a portion that needs to be scanned in the workpiece W. When there is a portion that needs to be scanned on the workpiece W, the processing proceeds to the step SA2, and the workpiece W is placed on the mounting surface 142 in the third placement attitude and scanned by the measurement unit 100. By repeating this, all necessary scanning portions can be scanned. In a case where NO is determined in the step SA15, the full-surface scan model is acquired in the step SA16. Note that the full-surface scan model is not necessary, and any model can be used as long as the user can scan the necessary portion.

[0182] Next, the reference Figure 17The flowchart describes an example of scanning processing in the presence of a reference model with CAD data or measured 3D data of a workpiece W. In step SG1, the reading unit 292 of the analysis module 290 reads the CAD model (reference model) from the storage device 240, and in step SG2, the CAD model is acquired. In step SG3, the scanning process is performed in a manner similar to... Figure 6 The evaluation value calculation and attitude candidate suggestion are performed in step SA4. In step SG3, since CAD data exists, the attitude calculation unit 293 analyzes the CAD data read by the reading unit 292. The attitude calculation unit 293 estimates the amount of data acquired (additional data amount) by analyzing the CAD data, and calculates the recommended placement attitude based on the estimated data amount. In addition, the attitude calculation unit 293 analyzes the CAD data read by the reading unit 292 and estimates the overlap area between the 3D data and the CAD data acquired when the device is placed in the recommended placement attitude. The attitude calculation unit 293 calculates the recommended placement attitude based on the estimated overlap area.

[0183] In step SG4, for example, the user selects a candidate pose on the user interface screen 700, such as... Figure 8 As shown. In step SG5, the user determines whether the candidate pose displayed on the display unit 400 is the desired pose. If the pose is not the desired pose, the process proceeds to step SG6, and the placement pose of the workpiece W is adjusted according to the computer graphics. In step SG7, the overlap region estimation unit 295 estimates the amount of overlap region in the adjusted placement pose, and the additional data quantity estimation unit 296 estimates the additional data quantity. The analysis module 290 recalculates the evaluation index based on the estimated overlap region and the estimated additional data quantity, and presents the evaluation index to the user.

[0184] If the orientation is determined to be the desired orientation in step SG5, the process proceeds to step SG8, ​​and the analysis module 290 generates a computer-generated graphic workpiece (model) with the determined placement orientation. The display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400.

[0185] In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W along with the mounting surface 142, and the data acquisition unit 291 acquires a real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until it overlaps with the workpiece in the computer graphics.

[0186] In step SG10, the user determines whether the actual workpiece W is installed to overlap the computer graphics workpiece. In the case where the actual workpiece W cannot be installed to overlap the computer graphics workpiece, the processing proceeds to step SG4, and another placement attitude is selected. In the case where the actual workpiece W is installed to overlap the computer graphics workpiece, since the workpiece W is in the first placement attitude, the processing proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement attitude. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the first three-dimensional data and the CAD data as a reference model. In step SG14, the analysis module 290 acquires the aligned CAD model.

[0187] In step SG15, evaluation value calculation and attitude candidate suggestion are performed. In step SG16, the user selects a candidate attitude, and then the processing proceeds to step SG17. In the case where the attitude is not a desired attitude, after adjustment in step SG18, the evaluation value is updated in step SG19, and then the processing proceeds to step SG17. In the case where the attitude is a desired attitude, the processing proceeds to step SG20, and the display control unit 255 superimposes the computer graphics workpiece generated by the analysis module 290 on the real-time image acquired by the data acquisition unit 291, and displays the superimposed image on the display unit 400. Here, the computer graphics workpiece can be the first three-dimensional data acquired in step SG12 or the reference model. In addition, the first three-dimensional data and the reference model can be displayed on the display unit 400 at the same time, or can be switched and displayed.

[0188] In step SG21, the user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the computer graphics workpiece.

[0189] In step SG22, the user determines whether the actual workpiece W is installed to overlap the workpiece of the computer graphics. In a case where the actual workpiece W cannot be installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG16, and another placement attitude is selected. In a case where the actual workpiece W is installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement attitude (second placement attitude). In step SG24, the alignment unit 290A performs alignment with the acquired second three-dimensional data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the three-dimensional model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a portion that needs to be scanned. When there is a portion that needs to be scanned, the processing proceeds to step SG12. When there is no scan portion, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model on which the alignment processing is performed by the alignment unit 290A in step SG26. Further, the full-surface scan model and the CAD model can be displayed on the display unit 400 in an aligned state.

[0190] In the present embodiment, as shown in Figure 19 the alignment unit 290A performs alignment between the first three-dimensional data A and the second three-dimensional data B based on a first alignment parameter (first positional relationship) that is a positional relationship between the first three-dimensional data A and the second three-dimensional data B, and the combination unit 290G combines the first three-dimensional data (first mesh data) A and the second three-dimensional data (second mesh data) B acquired by the data acquisition unit 291 to generate combined three-dimensional data AB. Thereafter, when the third three-dimensional data C is acquired by the data acquisition unit 291, the alignment unit 290A performs alignment between the combined three-dimensional data AB obtained by combining the first three-dimensional data A and the second three-dimensional data B and the third three-dimensional data C based on a second alignment parameter (second positional relationship) that is a positional relationship between the combined three-dimensional data AB and the third three-dimensional data C, and the combination unit 290G combines the combined three-dimensional data AB and the third three-dimensional data (third mesh data) C acquired by the data acquisition unit 291 to generate combined three-dimensional data ABC. At this time, for example, when the editing unit 290H accepts an input for editing a position or a shape of at least one mesh data among the first mesh data, the second mesh data, and the third mesh data, the storage device 240 stores the first mesh data, the second mesh data, and the third mesh data. The editing of the shape of the mesh data includes, for example, removal of a portion of the point cloud.

[0191] Further, when the data acquisition unit 291 acquires the fourth three-dimensional data D, the alignment unit 290A performs alignment between the combined three-dimensional data ABC and the fourth three-dimensional data D based on a third alignment parameter (third positional relationship) that is a positional relationship between the combined three-dimensional data ABC and the fourth three-dimensional data D, and the combination unit 290G combines the combined three-dimensional data ABC and the fourth three-dimensional data (fourth mesh data) D acquired by the data acquisition unit 291 to generate combined three-dimensional data ABCD. When the editing unit 290H accepts an input for editing a position or a shape of the fourth mesh data, the storage device 240 stores the fourth mesh data.

[0192] As described above, the user can obtain three-dimensional data of the entire workpiece W by sequentially combining two pieces of three-dimensional data. Sequential combination is performed, and thus, the processing load of mesh data conversion from point clouds becomes constant, and the processing of combining mesh data for which sparsification processing is performed also becomes constant. For example, the processing load can be reduced compared to processing of collectively converting all original point clouds constituting the combined three-dimensional data ABCD. Note that the combined three-dimensional data ABCD is not limited to a method of sequentially adding the fourth three-dimensional data to the combined three-dimensional data ABC, and the combined three-dimensional data ABCD can be generated by combining the first three-dimensional data, the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data based on positional relationships between the pieces of three-dimensional data.

[0193] The display control unit 255 displays the combined three-dimensional data generated by the combination unit 290G on the display unit 400, and displays the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit 400. That is, the display control unit 255 generates a user interface screen 800 as illustrated in Figure 20 The display control unit 255 displays the combined three-dimensional data generated by the combination unit 290G on the display unit 400, and displays the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit 400. That is, the display control unit 255 generates a user interface screen 800 as illustrated in

[0194] In the present embodiment, the combining process of the first three-dimensional data and the second three-dimensional data by the combining unit 290G can be edited. That is, the analysis module 290 includes an editing unit 290H that edits the combining process of the first three-dimensional data and the second three-dimensional data by the combining unit 290G. The editing unit 290H accepts an input for editing a position or a shape of at least one of the first three-dimensional data and the second three-dimensional data. The editing unit 290H can edit the position of the first three-dimensional data in any one of the X direction, the Y direction, and the Z direction. The second three-dimensional shape data can be similarly edited.

[0195] The editing unit 290H accepts an input of an edit of the first three-dimensional data or an edit of the second three-dimensional data, and edits the first three-dimensional data or the second three-dimensional data based on the accepted input. Then, the combining unit 290G recombines the first three-dimensional data and the second three-dimensional data based on the input accepted by the editing unit 290H. At this time, the alignment unit 290A can perform alignment between the first three-dimensional data and the second three-dimensional data based on the first alignment parameter that is a positional relationship between the first three-dimensional data A and the second three-dimensional data B. Thus, it is not necessary for the user to newly accept an alignment designation, and convenience can be improved. In addition, when an input of editing the third three-dimensional data and the fourth three-dimensional data is similarly accepted, the editing unit 290H edits the third three-dimensional data and the fourth three-dimensional data based on the accepted input. Then, the combining unit 290G updates and regenerates the combined three-dimensional data based on the input accepted by the editing unit 290H.

[0196] For example, when the first three-dimensional data and the second three-dimensional data are combined, the first three-dimensional data and the second three-dimensional data having different initial positions are aligned on the background of the display process of the user interface screen 810, as an example, as illustrated in Figure 21 , and then combined by the combining unit 290G. The combining result display button 811, the additional shape display button 812, and the original shape display button 813 are provided on the user interface screen 810.

[0197] When the user operates the combining result display button 811, the display control unit 255 displays, in the display area 814 of the user interface screen 810, the combined three-dimensional data that is a combination result at a time point at which the combination of the first three-dimensional data and the second three-dimensional data is completed. The user can confirm the quality of the combination result by viewing the combined three-dimensional data. When there is a defective portion, it is also possible to designate the occurrence portion and a factor of the occurrence. Upon designation, when the user operates the additional shape display button 812 on the user interface screen 810 as illustrated in Figure 22 , the display control unit 255 displays the second three-dimensional data in the display area 814 of the user interface screen 810, and when the user operates the original shape display button 813 as illustrated in Figure 21When the original shape display button 813 on the user interface screen 810 is indicated, the display control unit 255 displays the first three-dimensional data in a display area 814 of the user interface screen 810. As described above, since the first three-dimensional data and the second three-dimensional data can be individually displayed by the user's switching operation, in a case where a problem is found in the combination result of the combined three-dimensional data, the user can select any one of the first three-dimensional data or the second three-dimensional data, and partially cut out or repair the three-dimensional data while confirming the selected three-dimensional data. Note that the combined three-dimensional data, the first three-dimensional data, and the second three-dimensional data can be displayed on one user interface screen.

[0198] In Figure 23 An edit button 815 is provided on the user interface screen 810. When the user operates the edit button 815, the display control unit 255 generates and displays a user interface screen 820 for data editing as Figure 20 indicated, and accepts an instruction to edit the three-dimensional data by the edit unit 290H. On the user interface screen 820, a display area 821 in which an instruction of partial cut-out, repair, or the like is accepted, and three-dimensional data to be edited by the edit unit 290H is displayed based on the instruction, a process display area 822 in which an editing process is displayed, and a setting area 823 in which a designation method of an editing area, a selection method, an editing method, or the like can be set are displayed. When editing such as partial cut-out or repair of the three-dimensional data is performed on the user interface screen 820, the edited three-dimensional data is stored in the storage device 240.

[0199] In a case where the edited three-dimensional data is, for example, the second three-dimensional data, alignment between the edited second three-dimensional data and the first three-dimensional data is performed again by the alignment unit 290A. At this time, the alignment unit 290A can perform alignment between the edited second three-dimensional data and the first three-dimensional data based on the first alignment parameter. Then, the combination unit 290G generates the combined three-dimensional data by combining the edited second three-dimensional data and the first three-dimensional data. The combined three-dimensional data generated in this way is displayed in the display area 814.

[0200] In addition, in a case where the quality of the combined three-dimensional data is poor due to editing of the second three-dimensional data, the user issues a reacquisition instruction to reacquire the second three-dimensional data acquired by the data acquisition unit 291. When the user operates Figure 20When the reacquiring button 816 of the user interface screen 810 is indicated, the accepting unit 298 accepts a reacquiring instruction to reacquire the second three-dimensional data. In this case, the data acquiring unit 291 acquires fifth mesh data (mesh data for update) based on the reacquiring instruction accepted by the accepting unit 298, and acquires the first mesh data from the storage device 240, and the aligning unit 290A performs alignment between the first mesh data and the fifth mesh data. The fifth mesh data is acquired to update the second mesh data. Thus, the aligning unit 290A can perform alignment between the mesh data for update (i.e., the fifth mesh data) acquired as a substitute for the second mesh data and the first mesh data based on the first alignment parameter. When the operation Figure 19 When the aligning button 817 of the user interface screen 810 is indicated, alignment is started.

[0201] When the accepting unit 298 accepts the reacquiring instruction, the display control unit 255 can also display a recommended placement attitude based on the first three-dimensional data on the display unit 400. The recommended placement attitude is an attitude calculated by the attitude calculating unit 293 as described above. The recommended placement attitude is displayed on the display unit 400, and thus, it is possible to grasp the placement attitude with a large amount of additional data.

[0202] Then, the combining unit 290G combines the first mesh data and the fifth mesh data aligned by the aligning unit 290A, and updates the combined three-dimensional data. The updated combined three-dimensional data is displayed in the display area 814. As described above, the first alignment parameter as the alignment parameter of the first mesh data and the second mesh data can be used to combine the first mesh data and the fifth mesh data aligned by the aligning unit 290A. That is, the first alignment parameter as the alignment parameter of the first three-dimensional data A and the second three-dimensional data B is associated with the combined three-dimensional data AB, and when the reacquiring instruction to reacquire the second three-dimensional data is accepted, it is possible to perform alignment between the first three-dimensional data A and the newly acquired three-dimensional data B' instead of the second three-dimensional data B by using the first alignment parameter associated with the combined three-dimensional data AB to generate a new combined three-dimensional data AB'. In a case where the combined three-dimensional data is updated, the combining unit 290G can discard the combined three-dimensional data before the update. Further, when the first mesh data and the third mesh data aligned by the aligning unit 290A are combined, the combining unit 290G can discard the second mesh data, which is mesh data before the update. That is, since mesh data that does not constitute the combined three-dimensional data is unnecessary data, it is possible to suppress wasteful occupation of a storage area by discarding the mesh data. When the unnecessary data is discarded, confirmation can be performed by the user.

[0203] Further, the alignment unit 290A performs alignment between the combined three-dimensional data AB' obtained by combining the first mesh data and the fifth mesh data and the third mesh data. At this time, the alignment unit 290A can perform alignment between the combined mesh data AB' obtained by combining the first mesh data and the fifth mesh data and the third mesh data on the basis of the second alignment parameter. Then, the combining unit 290G combines the combined mesh data AB' aligned by the alignment unit 290A and the third mesh data to generate combined mesh data ABC. The generated combined mesh data ABC is displayed in the display area 814.

[0204] On the other hand, since the first three-dimensional data, the fifth three-dimensional data, and the third three-dimensional data constituting the combined three-dimensional data are automatically stored in the storage device 240 as necessary data, at least one of the first three-dimensional data, the fifth three-dimensional data, and the third three-dimensional data can be read later. For noise or misalignment that is overlooked at the time of repeated combination processing, it is possible to perform the combination processing again by automatically storing the individual scan results. For example, as shown in Figure 24 In the case where the combined three-dimensional data ABC is generated after the combined three-dimensional data AB is generated and then the combined three-dimensional data ABCD is generated, and in the case where the third three-dimensional data C is combined even with the fourth three-dimensional data D without noticing noise contamination, it is possible to recover this step from the middle by recombining the third three-dimensional data C with the combined three-dimensional data AB.

[0205] As described above, the combined three-dimensional data can include information indicating a combination order of three-dimensional data, and the information indicating the combination order of three-dimensional data can be stored in the storage device 240 in association with the combined three-dimensional data. The combining unit 290G combines the first mesh data and the third mesh data aligned by the alignment unit 290A on the basis of the information indicating the combination order corresponding to the second three-dimensional data.

[0206] (CAD data alignment)

[0207] In a case where the actual workpiece W is arranged to match the CAD data displayed on the display unit 400, the user needs to move the workpiece W on the rotary stage 143, but the present application is not limited thereto, and the CAD data can be moved to match the actual workpiece W. That is, it can be difficult for the user to arrange the actual workpiece W to match the CAD data while viewing the display unit 400, because the arrangement needs to be performed while viewing both the display unit 400 and the rotary stage 143. Further, since the camera that captures the actual workpiece W and the user's line of sight face each other, the user performs the operation while viewing the mirror image, which causes difficulty. Therefore, in order to improve the convenience of the user, a function (alignment function) that enables the actual workpiece and the CAD data to be aligned by the user moving the CAD data while viewing the display unit 400 without moving the actual workpiece W on the three-dimensional scanner 1 can be installed.

[0208] Hereinafter, details of the alignment function will be described with reference to the flowchart shown in Figure 25 In the following description, the alignment is referred to as overlay alignment, and the CAD data is referred to as a "virtual object". In step S100, the controller 200 reads the center position of the rotary stage 143 and the CAD data. In step S101, the controller 200 calculates a virtual ground based on the center position of the rotary stage 143 read in step S100, displays the virtual ground on the display unit 400, and displays the virtual object on the display unit 400 based on the CAD data.

[0209] In step S102, the controller 200 determines whether a mouse button of the operation unit 250 is pressed near the display position of the virtual object. In a case where the mouse button is not pressed near the display position of the virtual object, the overlay alignment is terminated. On the other hand, in a case where the mouse button is pressed near the display position of the virtual object, the process proceeds to step S103, and the controller 200 performs a virtual object rotation process by mouse drag.

[0210] Hereinafter, details of the alignment function will be described with reference to the flowchart shown in Figure 24the flowchart of FIG. 8. In step S200, the controller 200 determines whether the rotation button of the operation unit 250 is pressed. In the case where it is determined that the rotation button of the operation unit 250 is pressed, the processing proceeds to step S201. In step S201, the controller 200 determines whether the moving direction of the mouse of the operation unit 250 is close to horizontal. In the case where the moving direction of the mouse is not close to horizontal, the processing proceeds to step S202. On the other hand, in the case where the moving direction of the mouse is close to horizontal, the processing proceeds to step S203, and the controller 200 fixes the rotation axis of the virtual object to an axis perpendicular to the virtual ground. In step S204, the controller 200 extracts the horizontal component of the moving amount of the mouse. In step S205, the controller 200 calculates the posture rotated according to the horizontal component around the fixed axis. In step S206, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S207, the display of the virtual object is updated. In step S208, the controller 200 determines whether the mouse button is released. In the case where the mouse button is not released, the processing returns to step S201.

[0211] In step S202, the controller 200 determines whether the moving direction of the mouse of the operation unit 250 is close to vertical. In the case where the moving direction of the mouse is close to vertical, the processing proceeds to step S209, and the controller 200 fixes the rotation axis of the virtual object to the right axis and the left axis in the line-of-sight direction. In step S210, the controller 200 extracts the vertical component of the moving amount of the mouse. In step S211, the controller 200 calculates the posture rotated according to the vertical component around the fixed axis, and the processing proceeds to step S206.

[0212] In the case where it is determined in step S202 that the moving direction of the mouse is not close to vertical, the processing proceeds to step S212, and the moving amount of the mouse is extracted. In step S213, the posture rotated in any direction according to the moving amount of the mouse is calculated, and the processing proceeds to step S206.

[0213] In the case where it is determined in step S200 that No, the processing proceeds to step S214, and the moving amount of the mouse is extracted. In step S215, the posture rotated in any direction according to the moving amount of the mouse is calculated, and the processing proceeds to step S206.

[0214] Thereafter, the processing proceeds to Figure 26The step S104 is shown, and the controller 200 calculates the distance between the virtual object and the virtual ground. In step S105, the controller 200 extracts a surface of the virtual object that is close to the virtual ground among the surfaces of the virtual object. In step S106, the controller 200 calculates the degree of grounding between the surface of the virtual object and the virtual ground. In step S107, the controller 200 determines whether there is an extracted surface for which the degree of grounding is not calculated. In the case where there is an extracted surface for which the degree of grounding is not calculated, a next surface is selected in step S108, and the processing proceeds to step S106.

[0215] In the case where there is no extracted surface for which the degree of grounding is not calculated, the processing proceeds to step S109, and the controller 200 selects a surface having the highest degree of grounding. In step S110, the controller 200 calculates a posture in which the selected surface and the virtual ground are grounded. In step S111, the display of the virtual object is updated to the calculated posture.

[0216] Figure 27 The flowchart of FIG. 3 shows a process of adjusting the positional relationship between the virtual object and the virtual ground. In step S300, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S301, the controller 200 determines whether the virtual object is buried in the virtual ground. In the case where the determination in step S301 is YES, the processing proceeds to step S302, the virtual object is pushed up above the virtual ground, and the processing proceeds to step S303. In the case where the determination in step S301 is NO, the processing proceeds to step S303. In step S303, the controller 200 determines whether the virtual object is floating from the virtual ground. In the case where the determination in step S303 is YES, the processing proceeds to step S304, and the virtual object is placed on the virtual ground.

[0217] Figure 28 The flowchart of FIG. 4 shows a stage surface detection process. In step S400, the three-dimensional shape of the rotary stage 143 is measured and acquired by the controller 200. In step S401, the controller 200 obtains the position of the stage surface (mounting surface 142) from the three-dimensional shape of the rotary stage 143. In step S402, the controller 200 calculates height information of the stage surface with respect to the camera position on the basis of the obtained plane position. In step S403, the calculated stage surface height information is stored in the storage device 240 or the like.

[0218] In step S404, the controller 200 determines whether or not to save a dedicated chart. The dedicated chart is, for example, a calibration plate. In the case where NO is determined in step S404, the processing proceeds to step S405, and the stage center position is calculated based on an irregular shape on the stage surface with a known design value. In the case where YES is determined in step S404, the processing proceeds to step S406, and the dedicated chart is rotated and measured from a plurality of directions to calculate the stage center position. In step S407, the stage center position is stored in the storage device 240 or the like.

[0219] Figure 24 is a flowchart showing an example of processing in a case where rotation and movement of a virtual object are performed. Steps S500, S501, and S502 are respectively the same as steps S100, S101, and S102 in Figure 24 . Further, steps S505 to S512 are respectively the same as steps S104 to S111 in Figure 28 .

[0220] In step S503, rotation and movement of the virtual object by mouse dragging are performed. Figure 28 is a flowchart showing an example of processing in a case where rotation and movement of a virtual object are performed. In step S600, the controller 200 judges whether or not a moving direction of a mouse of the operation unit 250 is close to horizontal. In the case where the moving direction of the mouse is not close to horizontal, the processing proceeds to step S601. On the other hand, in the case where the moving direction of the mouse is close to horizontal, the processing proceeds to step S602, and the controller 200 fixes an axis of rotation of the virtual object to an axis perpendicular to a virtual ground. In step S603, the controller 200 extracts a horizontal component of a mouse movement amount. In step S604, the controller 200 calculates a posture rotated according to the horizontal component extracted around the fixed axis. In step S605, the controller 200 adjusts a positional relationship between the virtual object and the virtual ground. In step S606, display of the virtual object is updated. In step S607, the controller 200 determines whether or not a mouse button is released. In the case where the mouse button is not released, the processing returns to step S603.

[0221] In step S601, the controller 200 determines whether the moving direction of the mouse of the operation unit 250 is close to vertical. In the case where the moving direction of the mouse is close to vertical, the process proceeds to step S608, and the controller 200 fixes the rotation axis of the virtual object to the right and left axis in the line-of-sight direction. In step S609, the controller 200 extracts the vertical component of the moving amount of the mouse. In step S610, the controller 200 calculates the posture rotated according to the vertical component extracted around the fixed axis, and the process proceeds to step S611. In step S611, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S612, the display of the virtual object is updated. In step S613, the controller 200 determines whether the mouse button is released. In the case where the mouse button is not released, the process returns to step S609.

[0222] In the case where the moving direction of the mouse is not close to vertical, the process proceeds to step S614, and the controller 200 extracts the moving amount of the mouse. In step S615, the controller 200 calculates the posture rotated in any direction according to the moving amount of the mouse, and the process proceeds to step S616. In step S616, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S617, the display of the virtual object is updated. In step S618, the controller 200 determines whether the mouse button is released. In the case where the mouse button is not released, the process returns to step S614.

[0223] In Figure 30 step S513, the controller 200 calculates the distance between the virtual object and the virtual inclined platform. In step S514, the surface close to the virtual inclined platform is extracted among the surfaces of the virtual object. In step S515, the degree of grounding between the surface of the virtual object and the virtual inclined platform is calculated. In step S516, it is determined whether there is an extracted surface whose degree of grounding is not calculated. In the case where there is an extracted surface whose degree of grounding is not calculated, the process proceeds to step S517, the next surface is selected, and the process proceeds to step S515. When there is no extracted surface whose degree of grounding is not calculated, the process proceeds to step S518, and the surface having the highest degree of grounding is selected. In step S519, the posture in which the selected surface is grounded to the virtual inclined platform is calculated. In step S520, the positional relationship among the virtual object, the virtual ground, and the virtual inclined platform is adjusted.

[0224] Figure 25 is a flowchart showing another example of the process in the case where the rotation and the movement of the virtual object are performed. Steps S700 to S705 are respectively the same as steps S200 to S205 in Figure 25 . Further, steps S707 to S715 are respectively the same as steps S307 to S315 in Figure 31The steps S207 to S215 in FIG. 7A are the same as those in steps S207 to S215 in FIG. 2B. In step S706, the controller 200 adjusts the positional relationship between the virtual object, the virtual ground, and the virtual tilt stage. The virtual tilt stage virtually indicates a tilt stage provided on the rotary stage 143. The tilt stage provided on the rotary stage 143 is configured so that, for example, the tilt angle with respect to the horizontal plane can be changed in a plurality of ways, and the workpiece W can be tilted by mounting the workpiece W on the tilt stage.

[0225] Figure 6 FIG. 8 is a flowchart showing an example of the process in the case where the positional relationship between the virtual object, the virtual ground, and the virtual tilt stage is adjusted. In step S800, the controller 200 calculates the distance between the virtual object, the virtual ground, and the virtual tilt stage. In step S801, the controller 200 determines whether the virtual object is buried in the virtual ground or the virtual tilt stage. In the case where the determination in step S801 is YES, the process proceeds to step S802, the virtual object is pushed up above the virtual ground or the virtual tilt stage, and then the process proceeds to step S803. In the case where the determination in step S801 is NO, the process proceeds to step S803. In step S803, the controller 200 determines whether the virtual object is floating from the virtual ground or the virtual tilt stage. In the case where the determination in step S803 is YES, the process proceeds to step S804, and the virtual object is grounded to the virtual ground or the virtual tilt stage.

[0226] As described above, the same determination as that for the rotary stage 143 is performed in the space where the virtual object exists, and the physical constraint condition that the virtual object is grounded to the rotary stage 143 is introduced. Therefore, since the rotation and the translation are limited to the same degrees of freedom as the actual workpiece W, the alignment becomes easy for the user.

[0227] In addition, although an unstable posture can occur during the mouse drag of the operation unit 250, the stable posture is calculated at the time when the mouse is released, and the posture is automatically corrected to a posture in which the degree of grounding between the rotary stage 143 and the virtual object becomes high. Since the actual possible postures of the workpiece W also exist with limited degrees of freedom, the alignment becomes easy.

[0228] Further, the rotation operation by the mouse drag of the operation unit 250 is limited so that the rotation is performed with respect to only one axis of the roll, the pitch, and the yaw of the workpiece at the same time. Therefore, the virtual object on the rotary stage 143 can be rotated while maintaining the grounded state of the rotary stage 143, and the alignment becomes easier.

[0229] Furthermore, when the rotation of the rotating stage 143 and the display of the virtual object are interlocked, it may be difficult to capture the positional relationship in the depth direction when capturing images from a fixed camera, as one of the difficulties in alignment. On the other hand, since the rotation of the rotating stage 143 and the display state of the virtual object are interlocked, when capturing objects on the rotating stage 143 from different angles, the positional relationship with the virtual object can be captured, and alignment can be performed using the captured images.

[0230] Furthermore, in the case of a structure that allows the rotating stage 143 to tilt as described above, tilt information, etc., can be interlocked with the alignment function. For example, the application provides a section for inputting tilt information (tilt angle information) of the rotating stage 143, and the controller 200 acquires this information. Therefore, the virtual object can be interlocked with the virtual space, and the virtual object can take into account the tilt angle of the rotating stage 143 to adopt a posture.

[0231] In addition, without Figure 17 In the case of the reference model shown, the processing can proceed to step SA5 without calculating the evaluation value and the recommended placement posture calculated in SA4. In this case, in step SA5, for example, the posture calculation unit 293 calculates a posture as the recommended placement posture, which is obtained by rotating a portion of the scanned model of the workpiece W placed in the first placement posture, acquired by the data acquisition unit 291, around a predetermined axis (such as a rotation axis horizontal with respect to the mounting surface 140) by a constant rotation angle. Then, the display control unit 255 can display the recommended placement posture calculated by the posture calculation unit 293 on the display unit 400. Here, for example, the posture calculation unit 293 can calculate the recommended placement posture by estimating the rotation axis horizontal with respect to the mounting surface 140 and rotating the workpiece W around the rotation axis by a specific rotation angle, such as 60 degrees or 90 degrees. In step SA6, the user determines whether the posture displayed on the display unit 400 is the desired posture. If the posture is not the desired posture, the processing proceeds to step SA7, and the placement posture of the workpiece W is adjusted on the computer graphics. In step SA8, the attitude calculation unit 293 can calculate and update the evaluation value of the adjusted attitude, but can skip to step SA6.

[0232] When it is determined in step SA6 that the posture is the desired posture, the processing proceeds to step SA9, and the analysis module 290 generates a computer graphic of the workpiece (model) in the determined placement posture. In step SA10, the user installs the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires and displays a real-time image on the display unit 400. The user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the computer graphic of the workpiece.

[0233] In step SA11, the user determines whether the actual workpiece W is installed to overlap the computer graphic of the workpiece. In a case where the actual workpiece W cannot be installed to overlap the computer graphic of the workpiece, the processing proceeds to step SA5, and the placement posture is adjusted. In a case where the actual workpiece W is installed to overlap the computer graphic of the workpiece, since the workpiece W is in the second placement posture, the processing proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement posture. In step SA12, the light projection unit 110 of the measurement unit 100 irradiates the workpiece W placed in the second placement posture with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained by the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0234] In step SA14, the alignment unit 290A included in the analysis module 290 aligns the first three-dimensional data (three-dimensional data of the workpiece placed in the first placement posture) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. At the time of this alignment, the overlapping region extracted by the extraction unit 290B included in the analysis module 290 is used.

[0235] In addition, for example, in a case where there are a plurality of workpieces W, the first three-dimensional data and the second three-dimensional data are aligned for each workpiece W. Figure 32In the case of the example of the scanning process of the reference model of the CAD data or the measured three-dimensional data of the workpiece W shown, the process can proceed to step SG4 without the pose calculation unit 293 calculating the evaluation value and calculating the recommended placement pose in step SG3. In this case, in step SG4, the display control unit 255 displays the CAD data or the measured three-dimensional data on the display unit 400. The pose of the CAD data or the measured three-dimensional data at this time can be, for example, a pose of the CAD data or the measured three-dimensional data with respect to the scanner determined by matching a coordinate system of the CAD data or the measured three-dimensional data with a scanner coordinate system. In step SG4, the user can adjust the pose of the CAD data or the measured three-dimensional data based on the pose. In step SG5, the user determines whether the candidate pose displayed on the display unit 400 is a desired pose. In the case where the pose is not the desired pose, the process proceeds to step SG6, and the placement pose of the workpiece W is adjusted according to computer graphics. In step SG7, the pose calculation unit 293 calculates and updates the evaluation value with respect to the adjusted pose, but can skip the process, and the process can proceed to step SG5.

[0236] In the case where it is determined in step SG5 that the pose is the desired pose, the process proceeds to step SG8, and the analysis module 290 generates a computer graphics workpiece (model) in the determined placement pose. The display control unit 255 superimposes the computer graphics workpiece generated by the analysis module 290 on the real-time image acquired by the data acquisition unit 291, and displays the superimposed image on the display unit 400. In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires and displays a real-time image on the display unit 400. The user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the computer graphics workpiece.

[0237] In step SG10, the user determines whether the actual workpiece W is mounted to overlap the computer graphics workpiece. In the case where the actual workpiece W cannot be mounted to overlap the computer graphics workpiece, the process proceeds to step SG4, and the placement pose is adjusted. In the case where the actual workpiece W is mounted to overlap the computer graphics workpiece, since the workpiece W is in the first placement pose, the process proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement pose. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the measured first three-dimensional data and the CAD data as the reference model. In step SG14, the analysis module 290 acquires the aligned CAD model or the measured three-dimensional data.

[0238] In step SG15, a pose candidate is suggested, and the process proceeds to step SG16. For example, for the pose candidate, the pose of the CAD data or the measured three-dimensional data can be changed according to a predetermined rule with respect to the desired pose determined in step SG5. As an example of the prescribed rule, there is a case where the CAD data or the measured three-dimensional data is rotated by 60 degrees with respect to the desired pose determined in step SG5, with one axis of the scanner coordinate system as the rotation axis. In addition, for example, the user can select a rotation angle range of 90 degrees. In step SG16, the user selects the candidate pose, and then the process proceeds to step SG17. In the case where the pose is not the desired pose, after adjustment in step SG18, the evaluation value is updated in step SG19, and then the process proceeds to step SG17. Step SG19 can be skipped.

[0239] In the case where the pose is the desired pose, the process proceeds to step SG20, and the display control unit 255 superimposes the workpiece of the computer graphics generated by the analysis module 290 on the real-time image acquired by the data acquisition unit 291, and displays the superimposed image on the display unit 400. Here, the workpiece of the computer graphics can be the first three-dimensional data acquired in step SG12 or the reference model. In addition, the first three-dimensional data and the reference model can be displayed on the display unit 400 at the same time, or can be switched and displayed.

[0240] In step SG21, the user moves or rotates the actual workpiece W while viewing the real-time image on the display unit 400 until the actual workpiece W overlaps the workpiece of the computer graphics.

[0241] In step SG22, the user determines whether the actual workpiece W is installed to overlap the workpiece of the computer graphics. In a case where the actual workpiece W cannot be installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG16, and another placement attitude is selected. In a case where the actual workpiece W is installed to overlap the workpiece of the computer graphics, the processing proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement attitude (second placement attitude). In step SG24, the alignment unit 290A performs alignment using the acquired second three-dimensional data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the three-dimensional model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a portion that needs to be scanned. When there is a portion that needs to be scanned, the processing proceeds to step SG12. When there is no scan portion, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model or the measured three-dimensional data, in which the alignment processing is performed by the alignment unit 290A in step SG26. Further, the full-surface scan model and the CAD model or the measured three-dimensional data can be displayed on the display unit 400 in an aligned state.

[0242] In the present embodiment, the CAD data or the measured three-dimensional data is used as a reference model in order to acquire the combined three-dimensional data of the workpiece W. However, after the combined three-dimensional data of the workpiece W is acquired in a state where there is no reference model, alignment with the CAD data or the measured three-dimensional data can also be performed. The reading unit 292 reads the three-dimensional data of the workpiece W and the CAD data from the storage section 240. Subsequently, the resolution conversion unit 290E that reads the three-dimensional data of the workpiece W by the reading unit 292 converts at least the three-dimensional data of the workpiece W into three-dimensional data having a first resolution that is lower than the resolution acquired by the data acquisition unit 291. Subsequently, the shape feature extraction unit 290C extracts a partial region (key point) to be used for calculating a shape feature from the three-dimensional data of the first resolution and the CAD data. Subsequently, the analysis module 290 samples the periphery of the partial region, and the shape feature extraction unit 290C calculates a shape feature vector from the vicinity of the point sampled by the analysis module 290. Subsequently, the analysis module 290 compares the shape feature vector of the three-dimensional data obtained by the shape feature extraction unit 290C with the shape feature vector of the CAD data, and the alignment unit 290A aligns the three-dimensional data with the CAD data based on the comparison of the shape feature vectors. The shape feature vectors of the pair candidates can be extracted by comparing the shape feature vectors, and the positional relationship (rotation and translation) between the point clouds can be obtained by using the points of the pair candidates by the RANSAC or the deep learning-based method.

[0243] Further, after the alignment between the combined three-dimensional data and the CAD data is completed, the analysis module 290 can automatically perform a comparison between the CAD data and the three-dimensional data. Figure 33A An example of analysis is shown, in which a dimensional difference between the scan data and the CAD data of the workpiece W is calculated by comparing the three-dimensional shape of the scan data and the CAD data, and a color map is displayed by assigning a color corresponding to the degree of difference. The analysis module 290 calculates, for each mesh, a shape difference between the three-dimensional data of the workpiece W obtained by the data acquisition unit 291 and the CAD data based on the alignment result of the alignment unit 290A and the CAD data read by the reading unit 292, and assigns color information corresponding to the degree of difference to each mesh. Then, the display control unit 255 displays a color map on the display unit 400, in which a color is assigned to each mesh based on color information of at least one of the three-dimensional data and the CAD data. In addition, after alignment by the alignment unit 290A, the analysis module 290 can compare the three-dimensional data with the CAD data, and the display control unit 255 can display a color map on the display unit 400 based on an instruction to accept the analysis settings assigned by the accepting unit 298 or to start a comparison analysis. In this case, the user can assign detailed settings for the comparison.

[0244] In addition, Figure 33B An example of performing cross-sectional measurement on the three-dimensional data of the workpiece W is shown. The user specifies a surface on which cross-sectional measurement is to be performed on the three-dimensional data of the workpiece W, and indicates the type of analysis tool to be performed for the cross section and the assignment position. The analysis module 290 accepts the instruction and performs analysis on the indicated surface of the three-dimensional data of the workpiece W based on the instruction.

[0245] ​ An example of performing cross-sectional measurement on the CAD data is shown. The user specifies a surface on which cross-sectional measurement is to be performed on the CAD data, and indicates the type of analysis tool to be performed for the cross section and the assignment position. The analysis module 290 accepts the instruction and performs analysis on the indicated surface of the CAD data based on the instruction. The analysis module 290 can compare the result obtained by performing cross-sectional measurement on the three-dimensional data with the result obtained by performing cross-sectional measurement on the CAD data, and the display control unit 255 displays the comparison result on the display unit 400.

[0246] Further, cross-sectional measurement can be performed on the data in which the three-dimensional data of the workpiece W and the CAD data are aligned. The data in which the three-dimensional data of the workpiece W and the CAD data are aligned is obtained by the alignment unit 290A for alignment between the three-dimensional data of the workpiece W and the CAD data. The user specifies a surface on which the cross-sectional measurement is to be performed on the data in which the workpiece W and the CAD data are aligned, and indicates a type of analysis tool to be performed for the cross-section and an allocation position. The analysis tool is, for example, a measurement content such as a distance between two points and an angle formed by surfaces. The analysis module 290 accepts the instruction and performs the analysis on the indicated surface of the data in which the three-dimensional data of the workpiece W and the CAD data are aligned based on the instruction. The analysis module 290 can perform the cross-sectional measurement on the data in which the three-dimensional data and the CAD data are aligned. For example, a comparison can be performed based on a dimensional difference between the CAD data and the scan data. The display control unit 255 can display the comparison result on the display unit 400.

[0247] The above-described embodiments are merely examples in all aspects and should not be construed in a limiting manner. Furthermore, all modifications and changes falling within the scope of equivalents of the claims are within the scope of the present application.

[0248] As described above, the present application can be used to generate a plurality of pieces of three-dimensional data of various workpieces.

Claims

1. A 3D scanner that generates combined 3D data of a workpiece by generating multiple 3D data points of a workpiece placed in different orientations and combining the multiple 3D data points, the 3D scanner comprising: The data acquisition unit acquires first three-dimensional data as three-dimensional data of the workpiece placed in a first placement posture and second three-dimensional data as three-dimensional data of the workpiece placed in a second placement posture. An alignment unit performs alignment between the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit based on a first positional relationship that serves as the positional relationship between the first three-dimensional data and the second three-dimensional data. A combining unit that combines the first three-dimensional data and the second three-dimensional data aligned by the alignment unit to generate combined three-dimensional data; The display control unit displays the combined three-dimensional data generated by the combination unit on a display unit, and displays the first three-dimensional data and the second three-dimensional data on the display unit in a recognizable manner; as well as The editing unit edits the combined processing of the first three-dimensional data and the second three-dimensional data performed by the combining unit. The editing unit accepts input for editing the position or shape of at least one of the first and second three-dimensional data, and The combining unit recombines the first three-dimensional data and the second three-dimensional data based on the input received by the editing unit.

2. The 3D scanner according to claim 1 further includes: A point cloud acquisition unit acquires point cloud data of the workpiece; as well as A grid data generation unit processes the point cloud data acquired by the point cloud acquisition unit and converts the point cloud data into grid data. The data acquisition unit acquires first grid data and second grid data generated by the grid data generation unit, as the first three-dimensional data and the second three-dimensional data, and The combining unit combines the first grid data and the second grid data based on the first positional relationship to generate combined grid data.

3. The 3D scanner of claim 2, wherein the display control unit displays the first grid data and the second grid data on the display unit in a recognizable manner before they are recombined by the combining unit.

4. The 3D scanner according to claim 2 further includes a storage unit, the storage unit storing the first grid data and the second grid data generated by the grid data generation unit.

5. The three-dimensional scanner according to claim 4, wherein The point cloud acquisition unit acquires point cloud data of the workpiece placed in a third placement posture. The mesh data generation unit processes the point cloud data acquired by the point cloud acquisition unit, and converts the point cloud data of the workpiece placed in the third placement posture into third mesh data. The data acquisition unit acquires third mesh data, which is the three-dimensional data of the workpiece placed in the third placement posture, generated by the mesh data generation unit. The alignment unit performs alignment between the combined grid data (obtained by combining the first and second grid data) and the third grid data based on a second positional relationship, which is the positional relationship between the combined grid data and the third grid data. The combining unit combines the combined grid data and the third grid data to update the combined grid data. The editing unit accepts input for editing the position or shape of at least one of the first, second, and third grid data, and The storage unit stores the first grid data, the second grid data, and the third grid data generated by the grid data generation unit.

6. The three-dimensional scanner according to claim 4, wherein The point cloud acquisition unit acquires point cloud data of the workpiece placed in a third placement posture. The mesh data generation unit processes the point cloud data acquired by the point cloud acquisition unit, and converts the point cloud data of the workpiece placed in the third placement posture into third mesh data. The data acquisition unit acquires third mesh data, which is the three-dimensional data of the workpiece placed in the third placement posture, generated by the mesh data generation unit. The alignment unit performs alignment between the combined grid data (obtained by combining the first and second grid data) and the third grid data based on a second positional relationship, which is the positional relationship between the combined grid data and the third grid data. The combining unit combines the first grid data, the second grid data, and the third grid data to update the combined grid data. The editing unit accepts input for editing the position or shape of at least one of the first, second, and third grid data, and The storage unit stores the first grid data, the second grid data, and the third grid data generated by the grid data generation unit.

7. The three-dimensional scanner according to claim 5, further comprising: The receiving unit accepts a reacquisition instruction to reacquire the second three-dimensional data acquired by the data acquisition unit. The data acquisition unit acquires the fourth grid data based on the reacquisition instruction received by the receiving unit, and acquires the first grid data and the third grid data from the storage unit. The alignment unit performs alignment between the first grid data and the fourth grid data based on the first positional relationship. The combining unit combines the first grid data and the fourth grid data aligned by the alignment unit. The alignment unit further performs alignment between the first grid data and the fourth grid data combined by the combining unit and the second grid data, based on the second positional relationship. The combining unit also combines the first grid data, the fourth grid data, and the second grid data aligned by the alignment unit, and updates the combined grid data.

8. The 3D scanner of claim 7, wherein the combining unit discards the combined 3D data prior to the update.

9. The 3D scanner of claim 7, wherein when the first grid data and the fourth grid data aligned by the alignment unit are combined, the combining unit discards the second grid data.

10. The three-dimensional scanner according to claim 7, wherein The combined 3D data includes information indicating the combination order of the 3D data, and The combining unit combines the first grid data and the fourth grid data aligned by the alignment unit based on the information indicating the combining order corresponding to the second three-dimensional data.

11. The three-dimensional scanner according to claim 7, wherein, When the receiving unit accepts the reacquisition instruction, the display control unit displays a recommended placement posture based on the first three-dimensional data on the display unit.

12. A three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating multiple three-dimensional data of a workpiece placed in different orientations and combining the multiple three-dimensional data, the three-dimensional measurement method comprising: Acquire first three-dimensional data as three-dimensional data of the workpiece placed in a first placement posture and second three-dimensional data as three-dimensional data of the workpiece placed in a second placement posture; Based on the first positional relationship, which is the positional relationship between the first three-dimensional data and the second three-dimensional data, the alignment between the acquired first three-dimensional data and the second three-dimensional data is performed. Combine the aligned first and second 3D data to generate the combined 3D data; The generated combined three-dimensional data is displayed on the display unit, and the first three-dimensional data and the second three-dimensional data are displayed on the display unit in a recognizable manner; Accepts input for editing the position or shape of at least one of the first and second three-dimensional data displayed on the display unit; as well as The first 3D data and the second 3D data are recombined based on the received input.

13. A storage medium for storing a three-dimensional measurement program, the three-dimensional measurement program being used to cause a computer to execute a three-dimensional measurement method, the three-dimensional measurement method being used to generate combined three-dimensional data of the workpiece by generating multiple three-dimensional data of a workpiece placed in different placement postures and combining the multiple three-dimensional data, wherein the three-dimensional measurement method includes: Acquire first three-dimensional data as three-dimensional data of the workpiece placed in a first placement posture and second three-dimensional data as three-dimensional data of the workpiece placed in a second placement posture. Based on a first positional relationship that serves as the positional relationship between the first 3D data and the second 3D data, alignment is performed between the acquired first 3D data and the second 3D data. The first and second 3D data are combined and aligned to generate the combined 3D data. The generated combined 3D data is displayed on a display unit, and the first 3D data and the second 3D data are displayed on the display unit in a recognizable manner. Accepts input for editing the position or shape of at least one of the first and second three-dimensional data displayed on the display unit, and The first 3D data and the second 3D data are recombined based on the received input.

Images