A storage medium containing a three-dimensional scanner, a three-dimensional measurement method, and a three-dimensional measurement program.

Abstract

To enable high-precision alignment when aligning multiple 3D data sets. [Solution] The three-dimensional scanner 1 includes a shape feature extraction unit 290C which includes an input layer of a neural network that accepts input of first three-dimensional data and second three-dimensional data, a plurality of intermediate layers that extract shape features, and an output layer that outputs shape features; an image feature extraction unit 290D which extracts image features from the texture information of the first three-dimensional data and the second three-dimensional data; an alignment unit 290A which aligns the first three-dimensional data and the second three-dimensional data based on the shape features extracted by the shape feature extraction unit 290C and the image features extracted by the image feature extraction unit 290D; and a synthesis unit 290F which synthesizes the first three-dimensional data and the second three-dimensional data that have been aligned by the alignment unit 290A.

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program.

Background Art

[0002] For example, Patent Document 1 discloses a three-dimensional scanner that scans a workpiece to generate three-dimensional data. The three-dimensional scanner of Patent Document 1 synthesizes a plurality of three-dimensional data obtained in different sequences to generate composite three-dimensional data of the workpiece.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As disclosed in Patent Document 1, when synthesizing a plurality of three-dimensional data obtained in different sequences, alignment is performed after aligning the positions of the three-dimensional data. However, there are cases where alignment cannot be achieved only by aligning the three-dimensional data based on rules.

[0005] That is, there is a method of performing alignment of three-dimensional data by pattern matching. As pattern matching techniques, there are mainly those using three-dimensional data (point cloud) and those using images obtained by the camera of a three-dimensional scanner. Those using three-dimensional data require shape features of the workpiece, and those using image features require image features.

[0006] Generally, industrial products often have distinctive shape features, while simply processed metals, resins, etc., tend to have fewer distinctive image features. Therefore, traditionally, alignment has been performed using three-dimensional data.

[0007] Furthermore, when using image features for alignment of industrial products, a method is used in which markers are attached to the workpiece and scanned, the image coordinates of the markers are detected by image processing, and the relative positional relationship of the camera (a transformation matrix consisting of a rotation matrix and a translation vector) is calculated from the correspondence between the marker coordinates of multiple images using epipolar geometry. By adopting this method, high-precision alignment can be achieved even without initial estimations of the positional relationship.

[0008] On the other hand, analyzing the shape of point clouds is technically more difficult than analyzing images. In particular, aligning three-dimensional data without initial estimates is challenging.

[0009] In recent years, deep learning has enabled highly sophisticated discrimination of point cloud shape features, and there are increasing cases where alignment can be achieved without initial estimates. However, deep learning requires a massive amount of computation, and the number of point clouds that can be used for alignment on a realistic computer is limited to, for example, tens of thousands of points. Therefore, the accuracy of the resulting positional relationships is at best at the level of a few degrees or a few millimeters, and while it is possible to obtain an accuracy similar to that of the initial estimate, it is difficult to obtain high accuracy, such as tens of micrometers.

[0010] This disclosure is made in view of the above points, and its purpose is to enable high-precision alignment when aligning multiple three-dimensional data. [Means for solving the problem]

[0011] To achieve the above objective, one aspect of this disclosure may be based on a three-dimensional scanner that generates three-dimensional data of workpieces arranged in different orientations and synthesizes each of these three-dimensional data to generate composite three-dimensional data of a workpiece. The three-dimensional scanner includes a data acquisition unit that acquires first three-dimensional data which includes shape information and texture information of a workpiece arranged in a first orientation, and second three-dimensional data which includes shape information and texture information of a workpiece arranged in a second orientation; an input layer of a neural network that accepts the input of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit; a plurality of intermediate layers that extract shape features based on the input received by the input layer; and an output that outputs the shape features extracted by the intermediate layers. The system includes: a first extraction unit including layers; a second extraction unit that extracts image features from texture information contained in the first three-dimensional data and the second three-dimensional data respectively acquired by the data acquisition unit; an alignment unit that aligns the first three-dimensional data and the second three-dimensional data based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit; and a synthesis unit that combines the first three-dimensional data and the second three-dimensional data aligned by the alignment unit to generate the synthesized three-dimensional data.

[0012] In this configuration, the shape features of the first and second 3D data are extracted by a neural network, and image features are extracted from the texture information contained in each of the first and second 3D data. The alignment of the first and second 3D data is performed based on both the extracted shape features and image features, thereby improving the accuracy of the alignment.

[0013] Another aspect of this disclosure may be based on a three-dimensional measurement method that generates three-dimensional data of workpieces arranged in different orientations and then synthesizes each of these three-dimensional data to generate composite three-dimensional data of the workpiece. In the three-dimensional measurement method, first three-dimensional data, which is three-dimensional data including shape information and texture information of a workpiece placed in a first placement orientation, and second three-dimensional data, which is three-dimensional data including shape information and texture information of a workpiece placed in a second placement orientation, are acquired. The acquired first three-dimensional data and second three-dimensional data are input to the input layer of a neural network. Based on the first three-dimensional data and second three-dimensional data input to the input layer of the neural network, shape features are extracted in multiple intermediate layers of the neural network. The shape features extracted in the intermediate layers of the neural network are output from the output layer of the neural network. Image features are extracted from the texture information contained in each of the first three-dimensional data and second three-dimensional data. Based on the extracted shape features and image features, the first three-dimensional data and the second three-dimensional data are aligned. The aligned first three-dimensional data and the second three-dimensional data are combined to generate the combined three-dimensional data.

[0014] A further aspect of this disclosure may be a storage medium that stores a three-dimensional measurement program that causes a computer to execute a three-dimensional measurement method that generates three-dimensional data of workpieces arranged in different orientations and synthesizes each of these three-dimensional data to generate composite three-dimensional data of a workpiece.

[0015] Furthermore, this disclosure may also be based on the premise of a three-dimensional scanner that aligns a first point cloud data and a second point cloud data obtained by measuring a workpiece with the three-dimensional scanner. The three-dimensional scanner may include: a resolution conversion unit that generates a third point cloud data by reducing the resolution of the first point cloud data and a fourth point cloud data by reducing the resolution of the second point cloud data; a first extraction unit that extracts shape features from the third point cloud data and the fourth point cloud data generated by the resolution conversion unit; a first alignment unit that performs global alignment based on the shape features extracted by the first extraction unit and calculates low-precision alignment parameters that indicate the relative position and orientation of the second point cloud data with respect to the first point cloud data; a second alignment unit that performs local alignment based on the low-precision alignment parameters calculated by the first alignment unit, the first point cloud data and the second point cloud data and calculates high-precision alignment parameters that indicate the relative position and orientation of the second point cloud data with respect to the first point cloud data; and a synthesis unit that generates a composite point cloud by combining the first point cloud data and the second point cloud data based on the high-precision alignment parameters calculated by the second alignment unit.

[0016] The resolution conversion unit may generate a fifth point cloud data by reducing the resolution of the first point cloud data to a higher resolution than the third point cloud data, and may also generate a sixth point cloud data by reducing the resolution of the second point cloud data to a higher resolution than the fourth point cloud data.

[0017] In this case, the three-dimensional scanner can extract image features from each of the fifth point cloud data and the sixth point cloud data generated by the resolution conversion unit. The first alignment unit executes global alignment based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit, and can calculate low-precision alignment parameters indicating the relative position and orientation of the second point cloud data with respect to the first point cloud data. For example, the first extraction unit can be composed of a deep learning module or the like.

[0018] Further, the three-dimensional scanner may include a coordinate conversion unit that performs coordinate conversion of the first point cloud data based on the low-precision alignment parameters calculated by the first alignment unit. In this case, the second alignment unit may execute local alignment based on the first point cloud data and the second point cloud data for which coordinate conversion has been performed by the coordinate conversion unit.

Advantages of the Invention

[0019] As described above, when aligning a plurality of three-dimensional data, high-precision alignment can be performed.

Brief Description of the Drawings

[0020] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a three-dimensional scanner according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of the three-dimensional scanner. [Figure 3] FIG. 3 is a side view of the measurement unit and the pedestal unit. [Figure 4] FIG. 4 is a block diagram of the measurement unit. [Figure 5] FIG. 5 is a diagram showing a configuration example of the module. [Figure 6] FIG. 6 is a flowchart showing an example of a scan process when there is no CAD data of the workpiece. [Figure 7] FIG. 7 is a diagram showing an example of a user interface screen displayed at the start of measurement. [Figure 8] FIG. 8 is a diagram showing an example of a user interface screen displayed when presenting posture candidates. [Figure 9] FIG. 9 is a diagram showing an example of a user interface screen displayed when accepting selection of posture candidates. [Figure 10] FIG. 10 is a diagram showing an example of a user interface screen in which a model of the determined arrangement posture is superimposed on a live image. [Figure 11] FIG. 11 is a flowchart showing an example of the alignment process. [Figure 12] FIG. 12 is a schematic diagram regarding hidden surface removal. [Figure 13] FIG. 13 is a flowchart showing an example of the extraction process of shape features and image features. [Figure 14] FIG. 14 is a flowchart showing an example of the extraction process when a vector added with image features is input to the shape feature extraction unit. [Figure 15] FIG. 15 is a flowchart showing an example of the calculation process of low-precision alignment parameters. [Figure 16] FIG. 16 is a flowchart showing an example of the calculation process of high-precision alignment parameters. [Figure 17] FIG. 17 is a flowchart showing an example of the scan process when there is CAD data of the workpiece. [Figure 18] FIG. 18 is a diagram showing a user interface screen for displaying the first three-dimensional data, the second three-dimensional data, and the composite three-dimensional data. [Figure 19] FIG. 19 is a diagram showing the flow of the generation work of the composite three-dimensional data. [Figure 20] FIG. 20 is a diagram showing a confirmation screen of the composite three-dimensional data. [Figure 21] FIG. 21 is a diagram showing a confirmation screen of the second three-dimensional data. [Figure 22] FIG. 22 is a diagram showing a confirmation screen of the first three-dimensional data. [Figure 23]Figure 23 shows the user interface screen for data editing. [Figure 24] Figure 24 is a flowchart showing an example of the alignment function's processing. [Figure 25] Figure 25 is a flowchart showing an example of virtual object rotation processing using mouse dragging. [Figure 26] Figure 26 is a flowchart showing an example of the process for adjusting the positional relationship between a virtual object and the virtual ground. [Figure 27] Figure 27 is a flowchart showing an example of stage surface detection processing. [Figure 28] Figure 28 is a flowchart illustrating an example of the processing of the alignment function when a virtual object is rotated and moved. [Figure 29] Figure 29 is a flowchart illustrating an example of the process when a virtual object is rotated and translated. [Figure 30] Figure 30 is a flowchart showing another example of processing when a virtual object is rotated and translated. [Figure 31] Figure 31 is a flowchart showing an example of the process for adjusting the positional relationship between a virtual object, a virtual ground, and a virtual inclined platform. [Figure 32] Figure 32 is a color map based on the difference in dimensions between the scan data and CAD data of the workpiece. [Figure 33A] Figure 33A shows an example of performing cross-sectional measurements on scan data of a workpiece. [Figure 33B] Figure 33B shows an example of performing cross-sectional measurements on CAD data of a workpiece. [Modes for carrying out the invention]

[0021] Embodiments of the present invention will be described in detail below with reference to the drawings. The following description of preferred embodiments is essentially illustrative and is not intended to limit the present invention, its applications, or its uses.

[0022] Figure 1 shows the overall configuration of a three-dimensional scanner 1 according to an embodiment of the present invention. The three-dimensional scanner 1 is a device capable of acquiring three-dimensional data by measuring the shape of a workpiece (object to be measured) W, converting it into mesh data of the workpiece W, and outputting it. The three-dimensional scanner 1 can also convert the mesh data of the workpiece W into CAD data and output it, or convert the mesh data into surface data and output it.

[0023] In the following description, when measuring the shape of a workpiece W, coordinate information of the workpiece W surface is obtained by irradiating the workpiece W with a predetermined pattern of measurement light and using the signal obtained from the reflected light reflected from the surface of the workpiece W. For example, as the predetermined pattern of measurement light, a measurement method using triangulation with a fringe projection image obtained from the reflected light, which is projected onto the workpiece W, can be used. However, in this invention, the principle and configuration for obtaining the coordinate information of the workpiece W are not limited to this, and other methods can also be applied.

[0024] The three-dimensional scanner 1 includes a measuring unit 100 for measuring the shape of a workpiece W, a base unit 600 on which the workpiece W can be placed, a controller 200, a light source unit 300, and a display unit 400, etc. The controller 200 may be incorporated into the measuring unit 100, the light source unit 300 may be incorporated into the measuring unit 100, or the display unit 400 may be incorporated into the measuring unit 100. Furthermore, the controller 200 and the light source unit 300 may be integrated, or the controller 200 and the display unit 400 may be integrated.

[0025] The three-dimensional scanner 1 uses a light source unit 300 to provide structured illumination to the workpiece W, captures a fringe projection image, generates a depth image containing coordinate information, and can measure the three-dimensional dimensions and shape of the workpiece W based on this image. Measurement using such fringe projection has the advantage of shortening measurement time because it allows for three-dimensional measurement without moving the workpiece W or optical systems such as lenses in the Z direction (height direction).

[0026] Figure 2 shows a block diagram of a three-dimensional scanner 1 according to an embodiment of the present invention. As shown in this figure, the measurement unit 100 includes a pattern light projection unit (first projection unit) 110 that projects pattern light for measurement onto the workpiece W, a light receiving unit 120, a measurement control unit 150, and an illumination light output unit 130. The projection unit 110 is the part that irradiates the workpiece W, which is placed on the mounting unit 140 (described later), with measurement light having a predetermined pattern. Placing the workpiece W on the mounting unit 140 is the same as arranging the workpiece W on the mounting unit 140.

[0027] The light-receiving unit 120 is fixed in an inclined position with respect to the mounting surface 142 of the rotating stage 143, which will be described later. The light-receiving unit 120 receives the measurement light that is irradiated by the light-emitting unit 110 and reflected by the workpiece W. When the light-receiving unit 120 receives the measurement light as reflected light from the workpiece W, it generates and outputs a first measurement reception signal that represents the amount of measurement light received. The light-receiving unit 120 can generate an observation image for observing the overall shape of the workpiece W by imaging the workpiece W placed on the mounting unit 140. In this example, there is an illumination light output unit 130, but uniform light may be irradiated onto the workpiece W from the light-emitting unit 110. In this case, the light-emitting unit 110 is a component that irradiates the workpiece W with measurement light and uniform light at different timings. The light-receiving unit 120 can also receive the uniform light irradiated from the light-emitting unit 110 and output a second reception signal for texture acquisition. For example, it is also possible to irradiate the measurement light source with uniform light of the same wavelength as the measurement light and output a received signal containing uniaxial color information. Although not shown in the diagram, it is also possible to prepare a first camera and a second camera that are calibrated, acquire the shape with the first camera and acquire texture information with the second camera. The texture information includes color information and brightness information of the workpiece W.

[0028] The light-receiving unit 120 according to this embodiment includes a high-magnification light-receiving unit and a low-magnification light-receiving unit. The high-magnification light-receiving unit is the part of the workpiece W that can be imaged at a magnified view compared to the low-magnification light-receiving unit. On the other hand, the low-magnification light-receiving unit is a light-receiving unit with a wider field of view compared to the high-magnification light-receiving unit.

[0029] The base unit 600 comprises a base plate 602, a mounting unit 140, and a movement control unit (stage control unit) 144. The mounting unit 140 is supported on the base plate 602 of the base unit 600. The movement control unit 144 controls the movement and rotation of the rotating stage 143 on which the workpiece W is placed. The movement control unit 144 may be provided on the base unit 600 side or on the controller 200 side.

[0030] The light source unit 300 is connected to the measurement unit 100. The light source unit 300 is the part that generates measurement light and supplies it to the measurement unit 100. The controller 200 is the part that controls the measurement unit 100 and the like. The display unit 400 is connected to the controller 200 and is configured to display the image generated by the measurement unit 100, and to allow necessary settings, inputs, selections, etc.

[0031] The mounting section 140 has a rotating stage 143 with a mounting surface 142 on its upper surface on which the workpiece W is placed. As shown in Figure 4, two mutually orthogonal directions within the mounting surface 142 of the rotating stage 143 are defined as the X direction and the Y direction, and are indicated by arrows X and Y, respectively. The direction perpendicular to the mounting surface 142 of the mounting section 140 is defined as the Z direction, and is indicated by arrow Z. The direction of rotation around an axis parallel to the Z direction is defined as the θ direction, and is indicated by arrow θ.

[0032] The mounting section 140 includes a rotating stage 143 that rotates the mounting surface 142 around an axis extending in the Z direction, and a translational stage 141 that moves the mounting surface 142 horizontally (in the X and Y directions). The translational stage 141 has an X-direction movement mechanism and a Y-direction movement mechanism. The rotating stage 143 has a θ-direction rotation mechanism. The mounting section 140 may also include a workpiece holding member (such as a clamp) for holding the workpiece W on the mounting surface 142. Furthermore, the mounting section 140 may also include a tilt stage having a mechanism that can rotate around an axis parallel to the mounting surface 142.

[0033] The movement control unit 144 controls the rotational movement of the rotating stage 143 and the parallel movement of the translational stage 141 according to the measurement conditions set by the measurement condition setting unit 261, which will be described later. The movement control unit 144 also controls the movement of the mounting unit 140 by the mounting movement unit based on the measurement area set by the measurement condition setting unit 261, which will be described later.

[0034] The controller 200 includes a CPU (Central Processing Unit) 210, ROM (Read-Only Memory) 220, working memory 230, storage device (storage unit) 240, and an operation unit 250, etc. For example, a PC (Personal Computer) can be used as the controller 200.

[0035] The configuration of the measurement unit 100 is shown in the block diagram of Figure 4. The measurement unit 100 comprises a light-emitting unit 110, a light-receiving unit 120, an illumination light output unit 130, a measurement control unit 150, and a main body case 101 housing these components. The light-emitting unit 110 includes a measurement light source 111, a pattern generation unit 112, and a plurality of lenses 113, 114, and 115. The light-receiving unit 120 includes a camera 121 and a plurality of lenses 122 and 123. When performing measurements at different magnifications by providing multiple light-receiving units, a light-receiving unit 120a including a low-magnification camera 121 and low-magnification lenses, and a light-receiving unit 120b including a high-magnification camera 121 and high-magnification lenses may be installed. Note that the configuration is not limited to this one; the magnification may be varied by switching between multiple lenses for a single camera 121, or by providing a zoom lens for a single camera 121.

[0036] The light-emitting unit 110 is positioned diagonally above the mounting unit 140. In the example shown in Figure 4, the measurement unit 100 includes two light-emitting units 110, but the measurement unit 100 may include multiple light-emitting units 110. Here, a first measurement light-emitting unit 110A (right side in Figure 4) capable of irradiating the workpiece W with a first measurement light ML1 from a first direction, and a second measurement light-emitting unit 110B (left side in Figure 4) capable of irradiating the workpiece W with a second measurement light ML2 from a second direction different from the first direction are provided. The first measurement light-emitting unit 110A and the second measurement light-emitting unit 110B are arranged symmetrically with the optical axis of the light-receiving unit 120 as the center of symmetry. Although not shown, it is also possible to have three or more light-emitting units 110, or to move the light-emitting unit 110 and the mounting unit 140 relative to each other to project light onto the workpiece W in different directions, even while using a common light-emitting unit 110. In the above example, multiple light-emitting units 110 are provided and the light is received by a common light-receiving unit 120. However, conversely, multiple light-receiving units 120 may be provided to receive light from a common light-emitting unit 110. Furthermore, in this example, the irradiation angle of the illumination light emitted by the light-emitting unit 110 with respect to the Z direction is fixed, but this can also be made variable.

[0037] Each first measurement light projection unit 110A and second measurement light projection unit 110B is equipped with a first measurement light source and a second measurement light source, respectively, as measurement light sources 111. These measurement light sources 111 are, for example, halogen lamps that emit white light. The measurement light source 111 may also be a light source that emits monochromatic light, such as a blue LED (light-emitting diode) or organic EL that emits blue light. The light emitted from the measurement light source 111 (hereinafter referred to as "measurement light") is appropriately focused by the lens 113 and then incident on the pattern generation unit 112.

[0038] The relative positions of the light-receiving unit 120, light-emitting units 110A and 110B, and the light-receiving unit 120 are determined such that the central axes of the light-emitting units 110A and 110B intersect at a position where the arrangement of the workpiece W on the mounting unit 140 and the depth of field of the light-emitting and light-receiving units 110 and 120 are appropriate. Furthermore, since the center of the rotation axis in the θ direction coincides with the central axis of the light-receiving unit 120, when the mounting unit 140 rotates in the θ direction, the workpiece W rotates within the field of view around the rotation axis without moving out of the field of view.

[0039] The pattern generation unit 112 reflects the light emitted from the measurement light source 111 so that it projects measurement light onto the workpiece W. The measurement light incident on the pattern generation unit 112 is converted into a preset pattern and a preset intensity (brightness) and emitted. The measurement light emitted by the pattern generation unit 112 is converted by a plurality of lenses 114 and 115 into light with a diameter larger than the observation and measurement field of view of the light receiving unit 120, and then irradiated onto the workpiece W on the mounting unit 140.

[0040] The pattern generation unit 112 is a component that can switch between a projection state in which measurement light is projected onto the workpiece W and a non-projection state in which measurement light is not projected onto the workpiece W. For example, a DMD (Digital Micromirror Device) can be used for such a pattern generation unit 112. A pattern generation unit 112 using a DMD can be controlled by the measurement control unit 150 to switch between a reflection state in which the measurement light is reflected onto the optical path as the projection state and a light-shielding state in which the measurement light is blocked as the non-projection state.

[0041] In the above example, an example using a DMD for the pattern generation unit 112 was described, but the present invention is not limited to a DMD for the pattern generation unit 112, and other materials can be used. For example, an LCOS (Liquid Crystal on Silicon: reflective liquid crystal element) may be used as the pattern generation unit 112. Alternatively, a transmissive material may be used instead of a reflective material to adjust the amount of light transmitted for measurement. In this case, the pattern generation unit 112 is placed on the optical path of the measurement light, and a light projection state that transmits the measurement light and a light shielding state that blocks the measurement light are switched. For example, an LCD (liquid crystal display) can be used for such a pattern generation unit 112. Alternatively, the pattern generation unit 112 may be configured using a projection method using multiple line LEDs, a projection method using multiple optical paths, an optical scanner method composed of a laser and a galvanometer mirror, an AFI (Accordion fringe interferometry) method that uses interference fringes generated by superimposing beams divided by a beam splitter, or a projection method using a physical grid composed of a piezo stage and a high-resolution encoder and a moving mechanism.

[0042] The light-receiving unit 120 is positioned above the mounting unit 140. The measurement light reflected upward from the workpiece W towards the mounting unit 140 is collected and imaged by the multiple lenses 122 and 123 of the light-receiving unit 120, and then received by the camera 121.

[0043] Camera 121 is a CCD (charge-coupled device) camera, for example, including an image sensor 121a. Image sensor 121a is, for example, a monochrome CCD (charge-coupled device). Image sensor 121a may be other image sensors such as a CMOS (complementary metal-oxide-semiconductor) image sensor. Color image sensors require each pixel to correspond to the reception of red, green, and blue light, resulting in lower measurement resolution compared to monochrome image sensors, and sensitivity is reduced because each pixel requires a color filter. Therefore, in this embodiment, a monochrome CCD is used as the image sensor, and a color image is acquired by illuminating the workpiece W with illumination corresponding to RGB in a time-division manner using the illumination light output unit 130, which will be described later. With this configuration, a color image of the object to be measured can be acquired without reducing the measurement accuracy. Illumination light output unit 130 is an example of a second light projection unit that irradiates the workpiece W with illumination light. The illumination light can be uniform light.

[0044] A color image sensor may also be used as the image sensor 121a. In this case, although the measurement accuracy and sensitivity will be lower compared to a monochrome image sensor, it will no longer be necessary to irradiate the image sensor with illumination corresponding to RGB in a time-division manner from the illumination light output unit 130. A color image can be acquired simply by irradiating with white light, thus simplifying the illumination optical system. Each pixel of the image sensor 121a outputs an analog electrical signal corresponding to the amount of light received (hereinafter referred to as the "received light signal") to the measurement control unit 150.

[0045] The measurement control unit 150 is equipped with an A / D converter (analog-to-digital converter) and a FIFO (First In First Out) memory (not shown). The light received signal output from the camera 121 is sampled at a constant sampling period and converted into a digital signal by the A / D converter of the measurement control unit 150, based on control by the light source unit 300. The digital signals output from the A / D converter are sequentially stored in the FIFO memory. The digital signals stored in the FIFO memory are sequentially transferred to the controller 200 as pixel data.

[0046] The control unit 250 of the controller 200 may include, for example, a keyboard or a pointing device. Examples of pointing devices include a mouse or a joystick.

[0047] The ROM 220 of the controller 200 stores system programs and the like. The working memory 230 of the controller 200 consists of, for example, RAM (Random Access Memory) and is used for processing various data. The storage device 240 consists of a solid-state drive, a hard disk drive, etc. The storage device 240 stores a reverse engineering program. The storage device 240 is also used to store various data such as pixel data (image data), setting information, and measurement conditions provided by the measurement control unit 150. Measurement conditions include, for example, the settings of the light-emitting unit 110 (pattern frequency, pattern type) and the type of light-receiving unit 120 (low-magnification light-receiving unit, high-magnification light-receiving unit), which are set by the scanner module 260 described later when measuring the shape of the workpiece W. Furthermore, the storage device 240 can also store brightness information, coordinate information, and attribute information for each pixel that makes up the measurement image.

[0048] The CPU210 is a control circuit or control element that processes given signals and data, performs various calculations, and outputs the calculation results. In this specification, CPU refers to an element or circuit that performs calculations, and is used to mean not limited to processors such as CPUs, MPUs, GPUs, and TPUs for general-purpose PCs, regardless of their name, but also including processors such as FPGAs, ASICs, LSIs, microcontrollers, and chipsets such as SoCs.

[0049] The CPU 210 generates image data based on pixel data provided by the measurement control unit 150. The CPU 210 also performs various processing on the generated image data using the working memory 230. For example, based on the light-receiving signal output from the light-receiving unit 120, the CPU 210 generates measurement data representing the three-dimensional shape of the workpiece W contained within the field of view of the light-receiving unit 120 at a specific position on the mounting unit 140. The measurement data is the image itself acquired by the light-receiving unit 120. For example, when measuring the shape of the workpiece W using a phase-shift method, multiple images constitute one set of measurement data. The measurement data may also be point cloud data, which is a collection of points having three-dimensional position information. Measurement data of the workpiece W can be obtained using this point cloud data. Point cloud data is data represented by a collection of multiple points having three-dimensional coordinates.

[0050] The movement control unit 144 determines, based on measurement data of at least a portion of the workpiece W, whether to perform only the rotation of the rotating stage 143 or both the rotation of the rotating stage 143 and the translation of the translation stage 141. This facilitates three-dimensional measurement by automatically determining the imaging range according to the external shape of the workpiece W without the user having to be aware of it. The movement control unit 144 can also control the rotation of the rotating stage 143 after moving the translation stage 141 in the XY direction and then stopping the movement in the XY direction, thereby acquiring the shape around the workpiece W. Furthermore, scanning can also be performed by moving and rotating the workpiece W relative to the measuring unit 100 while the measuring unit 100 is fixed.

[0051] The display unit 400 is a component for displaying stripe projection images acquired by the measurement unit 100, depth images generated based on the stripe projection images, texture images captured by the measurement unit 100, various user interface screens, etc. The display unit 400 is composed of, for example, an LCD panel or an organic EL (electroluminescent) panel. Furthermore, by using a touch panel in the display unit 400, it can also be used in conjunction with the operation unit 250. The display unit 400 can also display images generated by the light receiving unit 120.

[0052] The light source unit 300 includes a control board 310 and an observation illumination light source 320. A CPU (not shown) is mounted on the control board 310. The CPU of the control board 310 controls the light-emitting unit 110, the light-receiving unit 120, and the measurement control unit 150 based on commands from the CPU 210 of the controller 200. Note that this configuration is just one example, and other configurations are possible. For example, the light-emitting unit 110 and the light-receiving unit 120 could be controlled by the measurement control unit 150, or the light-emitting unit 110 and the light-receiving unit 120 could be controlled by the controller 200, thus omitting the control board. Alternatively, a power supply circuit for driving the measurement unit 100 can be provided in this light source unit 300.

[0053] The observation illumination light source 320 includes, for example, three LEDs that emit red, green, and blue light. By controlling the brightness of the light emitted from each LED, any color of light can be generated from the observation illumination light source 320. The illumination light IL generated from the observation illumination light source 320 is output from the illumination light output unit 130 of the measurement unit 100 through a light guide member (light guide). In addition to LEDs, other light sources such as semiconductor lasers (LDs), halogen lights, and HIDs can also be used as appropriate for the observation illumination light source. In particular, if a color imaging sensor is used as the image sensor, a white light source can be used for the observation illumination light source.

[0054] The illumination light IL output from the illumination light output unit 130 illuminates the workpiece W by switching between red, green, and blue light in a time-division manner. This allows the texture images captured by these RGB lights to be combined to obtain a color texture image, which can then be displayed on the display unit 400.

[0055] A three-dimensional measurement program and applications for realizing the functions of the three-dimensional scanner 1 are installed on the controller 200. This allows the three-dimensional measurement method according to the present invention to be executed using the three-dimensional scanner 1. The three-dimensional measurement method is a method for measuring the three-dimensional shape of a workpiece W and is executed by a computer in the controller 200. The three-dimensional measurement program that causes the computer to execute the three-dimensional measurement method can be recorded on the storage medium 1000. The storage medium 1000 may be an optical disc such as a CD-ROM or DVD-ROM, or a semiconductor memory such as a memory card.

[0056] In the controller 200, on which the three-dimensional measurement program and application are installed, the CPU 210, ROM 220, working memory 230, storage device 240, etc., constitute the scanner module 260, conversion module 270, integration module 280, and analysis module 290 shown in Figure 5. In this embodiment, the system is divided into four modules: scanner module 260, conversion module 270, integration module 280, and analysis module 290. However, any two or more of these modules 260, 270, 280, and 290 may be integrated to form a single module. Furthermore, parts of each module 260, 270, 280, and 290 may be incorporated into other modules. In other words, the configuration example shown in Figure 5 is just one example and is not limited to the configuration example shown in Figure 5.

[0057] The scanner module 260 acquires image data of the workpiece W by measuring its shape and creates mesh data of the workpiece W based on that image data. The conversion module 270 converts the mesh data created by the scanner module 260 into CAD data. CAD data is three-dimensional shape information composed of analytical surfaces and freeform surfaces, and includes surface data, solid data, and data used for design. Surface data is data of shape surfaces composed of freeform surfaces and analytical surfaces, such as side data and planar data of a cylinder.

[0058] The integration module 280 is responsible for transmitting signals and data from the scanner module 260 to the conversion module 270 and the analysis module 290, and transmitting signals and data from the conversion module 270 to the scanner module 260. In this example, a module is a unit capable of executing multiple arithmetic processes, and can also be called a functional unit, functional block, etc.

[0059] The scanner module 260 includes, for example, a measurement condition setting unit 261, a scanner control unit 262, a point cloud acquisition unit 263a, a mesh data generation unit 263b, a scanner output unit 264, etc. The measurement condition setting unit 261 is the part that sets the measurement conditions for the shape of the workpiece. The scanner control unit 262 is the part that controls the measurement unit 100 according to the measurement conditions set in the measurement condition setting unit 261 to generate image data and acquires measurement data for the workpiece W based on the generated image data.

[0060] The point cloud acquisition unit 263a is responsible for acquiring point cloud data of the workpiece W based on the image data of the workpiece W acquired by the scanner control unit 262. The mesh data generation unit 263b is responsible for acquiring the point cloud data acquired by the point cloud acquisition unit 263a, processing the acquired point cloud data, and converting it into mesh data.

[0061] The scanner output unit 264 is the part that outputs the mesh data created by the mesh data generation unit 263b and additional data to the conversion module 270. The additional data is, for example, data that includes at least one of the measurement conditions and data calculated from the measurement data of the workpiece W.

[0062] The scanner module 260 controls the measurement unit 100 and generates three-dimensional data along with the various conditions under which the shape of the workpiece W was measured (measurement model, measurement magnification, resolution, etc.) and the raw data (e.g., image data) at the time of measurement. The three-dimensional data is mesh data containing multiple polygons and can also be called polygon data. A polygon is data composed of information that identifies multiple points and information that shows the polygonal surface formed by connecting those points. For example, it can consist of information that identifies three points and information that shows the triangular surface formed by connecting those three points. Mesh data and polygon data can also be defined as data represented by a collection of multiple polygons.

[0063] The conversion module 270 converts mesh data into CAD data and determines the conversion process based on measurement conditions and raw data. Specifically, the conversion module 270 includes, for example, a data input unit 271, a processing parameter determination unit 272, a CAD conversion unit 273, and a CAD output unit 274. The data input unit 271 is the part that receives mesh data output from the scanner output unit 264 and additional data. The processing parameter determination unit 272 is the part that determines the processing parameters for converting mesh data into CAD data according to the additional data received by the data input unit 271. The CAD conversion unit 273 is the part that converts mesh data into CAD data according to the processing parameters determined by the processing parameter determination unit 272. The CAD output unit 274 is the part that outputs the CAD data converted by the CAD conversion unit 273.

[0064] The analysis module 290 of the 3D scanner 1 is a module for generating 3D data of a workpiece W placed in different orientations and synthesizing the respective 3D data to generate composite 3D data of the workpiece W. The analysis module 290 has a data acquisition unit 291 that acquires, for example, 3D data of a workpiece W placed on a rotating stage 143. The rotating stage 143 is designed so that the user can place the workpiece W in any orientation. For example, to acquire the 3D shapes of the front and back sides of the workpiece W, the workpiece W can be placed on the rotating stage 143 with the front side facing upwards to acquire 3D data, and then the workpiece W can be placed on the rotating stage 143 with the back side facing upwards to acquire 3D data. In addition, to acquire the 3D shape of the side of the workpiece W, the workpiece W can be placed on the rotating stage 143 with the side facing upwards to acquire 3D data. For example, the orientation in which the front side of the workpiece W faces upwards can be defined as the first orientation, and the orientation in which the back side of the workpiece W faces upwards can be defined as the second orientation. Furthermore, the orientation in which the side of the workpiece W faces upwards can be defined as the third orientation. This definition of orientations is merely an example, and any two orientations can be different. For instance, the first, second, and third orientations can be defined according to the shape of the workpiece W and the range from which three-dimensional data is to be acquired. A fourth and fifth orientation may also be defined; the number of orientations is not particularly limited.

[0065] The data acquisition unit 291 acquires first three-dimensional data, which is the three-dimensional data of the workpiece W placed on the rotating stage 143 in a first position, and second three-dimensional data, which is the three-dimensional data of the workpiece W placed on the rotating stage 143 in a second position. Similarly, the data acquisition unit 291 also acquires third three-dimensional data, which is the three-dimensional data of the workpiece W placed on the rotating stage 143 in a third position, fourth three-dimensional data, which is the three-dimensional data of the workpiece W placed on the rotating stage 143 in a fourth position, and so on.

[0066] The data acquisition unit 291 acquires three-dimensional data measured by the scanner module 260. The three-dimensional data acquired by the data acquisition unit 291 includes shape information and texture information of the workpiece W, and the data acquisition unit 291 acquires textured shape data. Therefore, the data acquisition unit 291 acquires first three-dimensional data, which is three-dimensional data including shape information and texture information of the workpiece W placed in a first placement orientation, and second three-dimensional data, which is three-dimensional data including shape information and texture information of the workpiece W placed in a second placement orientation.

[0067] The data acquisition unit 291 receives the light-receiving signal generated by the light-receiving unit 120 of the measurement unit 100, generates a live image of the workpiece W based on the received light-receiving signal, and acquires the generated live image.

[0068] The mesh data generation unit 263b generates first mesh data, which is the mesh data of the workpiece W positioned in a first position, and second mesh data, which is the mesh data of the workpiece W positioned in a second position. Similarly, the mesh data generation unit 263b can also generate third mesh data, which is the mesh data of the workpiece W positioned in a third position, and fourth mesh data, which is the mesh data of the workpiece W positioned in a fourth position.

[0069] When the mesh data generation unit 263b is generating mesh data, the data acquisition unit acquires the first mesh data and the second mesh data generated by the mesh data generation unit 263b as the first three-dimensional data and the second three-dimensional data, respectively. Similarly, the third mesh data and the fourth mesh data can also be acquired.

[0070] For example, the three-dimensional data (first three-dimensional data) of a workpiece W positioned in a first orientation can be stored in the storage device 240. In this case, the reading unit 292 of the analysis module 290 reads the first three-dimensional data stored in the storage device 240. Similarly, the second, third, and fourth three-dimensional data can be stored in the storage device 240, and in this case, the reading unit 292 reads the second, third, and fourth three-dimensional data from the storage device 240, respectively.

[0071] If CAD data for workpiece W exists, the CAD data for workpiece W can also be stored in the storage device 240. In this case, the reading unit 292 reads the CAD data stored in the storage device 240 from the storage device 240.

[0072] The following describes the scanning process for both cases: when there is no CAD data for workpiece W and when there is CAD data for workpiece W. Figure 6 shows an example of the scanning process when there is no CAD data for workpiece W. Before or after the start of this flow, before proceeding to step SA1, workpiece W is placed on the rotating stage 143 in a first placement orientation. In step SA1, the scanning of workpiece W is started. In step SA2, the measuring unit 100 scans workpiece W, which is placed in the first placement orientation. For example, if workpiece W is placed with the front side facing upwards, the back side of workpiece W cannot be scanned, so the scan in step SA2 is called a "single-sided scan". Figure 7 shows the user interface screen 700 that is displayed when measurement starts. The user interface screen 700 is generated by the controller 200 and displayed on the display unit 400.

[0073] The user interface screen 700 is provided with a live image display area 701 and a model display area 702. The live image display area 701 displays a live image generated by the data acquisition unit 291. The live image displays the rotating stage 143 and the workpiece W placed on the rotating stage 143. The measurement unit 100 can also be called a scan head, and the scan head includes a light emitter 110, a light receiver 120, an illumination light output unit 130, and a measurement control unit 150. The display control unit 255 can also display the scan head and the workpiece W on the display unit 400. Displaying them in this way is effective, for example, when the workpiece W is placed on an arbitrary table and the scan head is moved to scan the workpiece W from different angles.

[0074] In step SA2, the light-emitting unit 110 of the measurement unit 100 irradiates measurement light onto the workpiece W, which is positioned in a first arrangement orientation. The light-receiving unit 120 of the measurement unit 100 receives the measurement light reflected from the workpiece W. The received signal output from the light-receiving unit 120 is received by the point cloud acquisition unit 263a to generate first point cloud data of the workpiece W. The mesh data generation unit 263b acquires the first point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired first point cloud data, and converts it into first mesh data. Here, processing of the point cloud data includes thinning of the point cloud, removal of point clouds outside the measurement area, and removal of noise point clouds. In step SA3, the first mesh data obtained by the processing in step SA2 is acquired as first three-dimensional data. The first three-dimensional data is stored in the working memory 230 or the storage device 240. The model of the workpiece W, based on the first three-dimensional data, is displayed together with the model of the rotating stage 143 in the model display area 702 of the user interface screen 700 shown in Figure 7. The user interface screen 700 is generated by the display control unit 255 of the controller 200 and displayed on the display unit 400.

[0075] In step SA4, the 3D scanner 1 calculates evaluation values ​​and proposes pose candidates. The evaluation values ​​in step SA4 are an example of an evaluation index, which will be described later. When the user operates the pose suggestion button 702a in the model display area 702, the controller 200 detects the operation. Then, as shown in Figure 8, the display control unit 255 generates a candidate display area 703 where the next scannable pose candidates are displayed and displays it on the user interface screen 700. The user can then select a pose candidate in step SA5 on this user interface screen 700.

[0076] When the 3D scanner 1 proposes a pose candidate, it calculates the pose candidate before calculating the evaluation value. That is, as shown in Figure 5, the analysis module 290 has a pose calculation unit 293. The pose calculation unit 293 is the part that calculates a placement pose different from the first placement pose based on the first 3D data acquired by the data acquisition unit 291. Specifically, it calculates a recommended placement pose (hereinafter also simply referred to as "placement pose") different from the first placement pose based on the 3D data read from the working memory 230 or storage device 240 by the reading unit 292. The display control unit 255 overlays the recommended placement pose calculated by the pose calculation unit 293 onto the live image acquired by the data acquisition unit 291.

[0077] When calculating a positioning position different from the first positioning position, the positioning calculation unit 293 first identifies the first positioning position of the workpiece W based on the first three-dimensional data. By identifying the first positioning position, the positioning calculation unit 293 can calculate positioning positions different from this first positioning position. The distance between the measuring unit 100 and the workpiece can be set based on the focal length of the lens of the measuring unit 100. In this embodiment, since the measuring unit 100 is equipped with a rotating stage 143, for example, the positioning calculation unit 293 can calculate multiple positioning positions by virtually rotating the first three-dimensional data around the rotation axis of the rotating stage 143. When rotating the first three-dimensional data, it is not necessary to rotate it a full rotation, and the rotation angle may be less than 360°.

[0078] Furthermore, if the workpiece W is large, there are cases where the measuring unit 100 needs to be moved parallel to the workpiece W multiple times for scanning. Whether or not this is the case can be determined by the analysis module 290 based on whether or not the first three-dimensional data exceeds the measurable range (length, width, and height) of the measuring unit 100. If the first three-dimensional data exceeds the measurable range of the measuring unit 100, it can be determined that the measuring unit 100 needs to be moved parallel to the workpiece W multiple times for scanning. Conversely, if the first three-dimensional data is within the measurable range of the measuring unit 100, it can be determined that the scanning is possible without moving the measuring unit 100 parallel to the workpiece W multiple times.

[0079] In cases where the measuring unit 100 is moved parallel to the workpiece W multiple times and scanned, the measuring unit 100 is virtually positioned in multiple directions based on the center point obtained by parallel movement so that the measurement range of the measuring unit 100 overlaps by a certain amount and includes the largest portion of the workpiece. Although the method of virtually moving the measuring unit 100 has been described, the method is not limited to this; the workpiece may also be moved or rotated.

[0080] The analysis module 290 has a calculation unit 294. The calculation unit 294 calculates the relative position and orientation of the recommended placement posture to the first placement posture. If multiple placement postures have been calculated by the posture calculation unit 293, the calculation unit 294 calculates the relative position and orientation of each of the multiple placement postures to the first placement posture. For example, the calculation unit 294 calculates a conversion formula to convert the relative positional relationship between the first placement posture and the recommended placement posture. The calculation unit 294 applies the conversion formula to the three-dimensional data of the first placement posture to convert the first placement posture to the recommended placement posture.

[0081] As an alternative to the first placement posture, the posture calculation unit 293 can also calculate candidate postures based on the size of the contact area with the mounting surface 142 in each of the multiple placement postures. If the contact area with the mounting surface 142 is too small, it may be difficult to place the workpiece W on the mounting surface 142. However, by calculating a placement posture in which a contact area with the mounting surface 142 of a predetermined size or larger can be secured, the workpiece W can be placed stably on the mounting surface 142. Figure 8 shows an example in which four placement postures have been calculated, but the number of placement postures calculated by the posture calculation unit 293 is not limited to four; it may be one, or any number of two or more.

[0082] Figure 9 shows the user interface screen 710 displayed when accepting a selection of posture candidates, which is generated by the display control unit 255 and displayed on the display unit 400. The user interface screen 710 includes a model display area 711 where a model of the workpiece W based on the first three-dimensional data is displayed, and a candidate display area 712. The candidate display area 712 displays four placement postures calculated by the posture calculation unit 293. By displaying the placement postures in the candidate display area 712, the next scannable placement posture can be presented to the user.

[0083] The candidate display area 712 is provided with an evaluation index display area 712a that displays an evaluation index indicating whether the workpiece placement orientation presented to the user is suitable for the next scan. The evaluation index display area 712a is also generated by the display control unit 255 and displayed on the display unit 400.

[0084] The evaluation index is based on the amount of additional data added to the first three-dimensional data by performing the next scan, and the amount of overlap between the three-dimensional data acquired by performing the next scan and the first three-dimensional data. In other words, the analysis module 290 includes an overlapping area estimation unit 295 and an additional data amount estimation unit 296. The overlapping area estimation unit 295 is the part that estimates the overlapping area between the three-dimensional data acquired by the data acquisition unit 291 in a state where it is positioned in the orientation calculated by the orientation calculation unit 293, and the first three-dimensional data acquired by the data acquisition unit 291. The overlapping area estimation unit 295 can also estimate the degree of feature based on the distribution of points in the overlapping area.

[0085] An example of the overlapping region estimation method by the overlapping region estimation unit 295 is described below. Here, if three-dimensional CAD data of the workpiece W exists, the overlapping region can be estimated by projecting the outermost surface when the first three-dimensional data acquired by the data acquisition unit 291 is projected onto the outermost surface when the three-dimensional CAD data is projected onto the outermost surface. However, as shown in the flowchart in Figure 6, if there is no three-dimensional CAD data for the workpiece W, estimation based on three-dimensional CAD data is not possible, so the surface of the existing first three-dimensional data is used as the overlapping region. If the workpiece W has a complex shape, additional scans from the second time onward may hide part of the surface of the existing first three-dimensional data. If three-dimensional CAD data exists, the overlapping region can be estimated while considering the effect of hiding part of the surface of the existing first three-dimensional data, but if there is no three-dimensional CAD data, this effect cannot be considered. However, if the workpiece has a shape close to a convex polygon, there is no effect of hiding part of the surface of the existing first three-dimensional data, and generally good results can be obtained. Therefore, the quality of the overlapping region can be determined by evaluating the area of ​​the overlapping region (amount of overlapping region) and the degree of feature. The degree of feature is obtained by analyzing the distribution of points in the overlapping part. Specifically, the coordinates of a point and the variance of its normal vector can be used as evaluation values.

[0086] Furthermore, during actual measurement, it is also necessary to consider whether the workpiece W can be placed on the mounting surface 142 of the rotating stage 143 in the positioning orientation calculated by the attitude calculation unit 293. If the measurement unit 100 moves relative to the workpiece W, the analysis module 290 can perform collision detection between the three-dimensional shapes of the measurement unit 100 and the base unit 600 and the three-dimensional CAD data or first three-dimensional data of the workpiece in computer graphics space.

[0087] Furthermore, since the workpiece W cannot be placed below the installation surface where the measurement unit 100 is installed, the analysis module 290 performs collision detection with the installation surface (Z coordinate < 0). Also, since it becomes difficult to position the measurement unit 100 at angles from directly above the workpiece or at low angles relative to the workpiece, the analysis module 290 can evaluate the "ease of placement" based on the placement angle of the measurement unit 100.

[0088] When moving or rotating the workpiece W, the analysis module 290 also determines how easy it is to position the workpiece W on the mounting surface 142. The analysis module 290 calculates the three-dimensional data of the positioning orientation calculated by the orientation calculation unit 293, or the area of ​​the base when that three-dimensional data is approximated by a convex polygon, and determines that the larger the calculated base area, the more stably the workpiece can be positioned on the mounting surface 142. In addition, if three-dimensional CAD data of the workpiece W is available, the analysis module 290 obtains the coordinates of the center of gravity and the center of the base, and calculates the distance between the center of gravity and the center of the base. Based on the distance between the center of gravity and the center of the base, the analysis module 290 can determine how easy it is to position the workpiece W.

[0089] Next, the amount of additional data will be explained. The additional data amount estimation unit 296 estimates the amount of additional data to be added to the first three-dimensional data by acquiring three-dimensional data by the data acquisition unit 291 in a state where the data is arranged in one of the multiple arrangement orientations calculated by the orientation calculation unit 293. The additional data amount estimation unit 296 can move the first three-dimensional data to each arrangement orientation and estimate the amount of additional data based on the direction of the normal vector after the move. For example, it can estimate the amount of additional data based on the positional relationship between the normal vector after the move from a predetermined viewpoint and the line of sight direction of the measurement unit 100. For example, by calculating the dot product of the normal vector after the move from a predetermined viewpoint and the line of sight direction of the measurement unit 100, it can identify three-dimensional data that are directly facing the measurement unit 100, and estimates that the more three-dimensional data that are directly facing the measurement unit 100 there are, the greater the amount of additional data.

[0090] An example of how the additional data amount estimation unit 296 estimates the amount of additional data will be explained. The additional data amount estimates how much additional data can be obtained (this can also be called "data yield") in addition to the first three-dimensional data already acquired, assuming that the scan is performed with the posture candidate calculated by the posture calculation unit 293. Here, if three-dimensional CAD data of the workpiece W exists, points that exist in the three-dimensional CAD data but not in the first three-dimensional data become additional data, and the more of these points there are, the larger the amount of additional data.

[0091] However, as shown in the flowchart in Figure 6, if there is no 3D CAD data for workpiece W, estimation based on 3D CAD data is not possible. Therefore, the point cloud included in the first 3D data is analyzed, and the number of points where the back side of the scanned surface is visible is used as additional data. This is because the back side is never visible in a completely scanned object, so the parts where the back side is visible can be determined to be unscanned parts.

[0092] The back surface of the workpiece W can be detected by detecting points in the first three-dimensional data whose normal vectors point in the opposite direction to the measurement unit 100. This detection process allows for the estimation of the amount of additional data to be added when three-dimensional CAD data for the workpiece W is unavailable. Furthermore, the analysis module 290 can calculate the percentage of the scanned area and the area requiring further scanning of the workpiece W based on the area of ​​the already scanned region and the amount of additional data obtained when scanning in each direction. The analysis module 290 can also automatically determine that scanning is complete when the scanned area exceeds a certain percentage. The order in which the overlapping area estimation and the amount of additional data to be added are performed does not matter.

[0093] After estimating the overlapping area and the amount of additional data for each configuration as described above, the evaluation unit 299 of the analysis module 290 calculates an evaluation index. The evaluation index can be calculated based on, for example, the following formula. Evaluation metric = (Area of ​​overlapping region) × (Amount of data added) × (Variance of points in the overlapping region) × (Ease of placement) Thus, the evaluation unit 299 calculates an evaluation index for each of the multiple placement postures based on the amount of additional data and the amount of overlapping region for each of the multiple placement postures calculated by the posture calculation unit 293, and calculates an overall evaluation index based on each evaluation index. When using texture information, the integration, average, maximum, and median of texture features based on the local contrast and derivative values ​​of the texture information may also be incorporated.

[0094] The calculated evaluation index is displayed on the display unit 400 by the display control unit 255 in numerical or graphical format. Figure 9 shows the evaluation index displayed in graphical format in the evaluation index display area 712a. When the user selects one of the placement postures displayed in the candidate display area 712, the analysis module 290 identifies the selected placement posture. The display control unit 255 displays the evaluation index for the placement posture identified by the analysis module 290 in graphical format in the evaluation index display area 712a. If the user selects a different placement posture, the display control unit 255 displays the evaluation index for that placement posture in the evaluation index display area 712a. In this way, the display control unit 255 can display the evaluation index for each of the multiple placement postures calculated by the posture calculation unit 293 on the display unit 400, so that the user can obtain the evaluation index for each placement posture when multiple placement postures are presented.

[0095] The display format of the evaluation index does not have to be the graph format shown in Figure 9; it may be a numerical format displaying numerical values, or a combination of graph and numerical formats. The display control unit 255 may also display on the display unit 400 an evaluation index based on the amount of additional data estimated by the additional data amount estimation unit 296, and an evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit 295. In this case, the analysis module 290 calculates a first evaluation index based on the amount of additional data estimated by the additional data amount estimation unit 296, and separately calculates a second evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit 295. The display control unit 255 may display only the first evaluation index on the display unit 400 in graph or numerical format, or only the second evaluation index on the display unit 400 in graph or numerical format, or display the first and second evaluation indicators separately on the display unit 400 in graph or numerical format.

[0096] The analysis module 290 includes a selection unit 297 that identifies a candidate posture from among multiple placement postures calculated by the posture calculation unit 293 based on evaluation indices. Specifically, the selection unit 297 acquires evaluation indices for each of the multiple placement postures. The selection unit 297 identifies the placement posture with the highest evaluation indices among the acquired multiple evaluation indices. Since the evaluation indices are based on the amount of additional data estimated by the additional data amount estimation unit 296 and the amount of overlapping area estimated by the overlapping area estimation unit 295, the selection unit 297 identifies a candidate posture based on the amount of additional data and the amount of overlapping area. When identifying a candidate posture, the selection unit 297 may also identify it based on the amount of additional data, the amount of overlapping area, and the shape of the overlapping area estimated by the overlapping area estimation unit 295. For example, if the overlapping area includes an uneven shape, the accuracy of alignment is higher than if the overlapping area is flat. Therefore, if the overlapping area includes a concave or convex shape, the evaluation indices can be increased compared to a flat overlapping area.

[0097] When a candidate posture is identified by the identification unit 297, the display control unit 255 displays the candidate posture identified by the identification unit 297 on the display unit 400. The candidate posture displayed on the display unit 400 is the workpiece placement posture recommended by the three-dimensional scanner 1 to the user. Therefore, the user can confirm the appropriate placement posture by looking at the display unit 400.

[0098] In step SA6 of Figure 6, the user determines whether the candidate posture displayed on the display unit 400 is the desired posture. Since the placement posture is identified based on evaluation indicators, it involves a large amount of additional data and overlapping areas, so it is believed that accurate composite three-dimensional data can be obtained by placing the workpiece in this posture. However, since the identification unit 297 identifies the placement posture based only on evaluation indicators, it does not necessarily identify the areas of high interest to the user (areas where high-precision three-dimensional data is desired). Therefore, in the three-dimensional scanner 1 of this embodiment, multiple placement postures are presented to the user along with evaluation indicators, and the user is allowed to select the placement posture from among the presented multiple placement postures that has a relatively high evaluation indicator and best suits the user's interests. Specifically, the analysis module 290 has a reception unit 298 that accepts the user's selection and adjustment operations for placement postures. For example, when the user selects a desired placement posture from among the placement postures displayed in the candidate display area 712 of the user interface screen 710 shown in Figure 9, the selection operation input is received by the reception unit 298. The display control unit 255 displays the arrangement orientation received by the reception unit 298 in the candidate display area 712.

[0099] In step SA7, the candidate posture displayed on the display unit 400 can also be adjusted. Specifically, the reception unit 298 receives user input to adjust a candidate posture identified by the identification unit 297. The user adjusts the positional relationship between the measurement unit 100 and the workpiece W by operating the operation unit 250 in the computer graphics space. For example, the workpiece W in the candidate posture can be moved horizontally, vertically, or rotated. Once the user has finished adjusting the candidate posture, the process proceeds to step SA8, where the posture calculation unit 293 identifies the adjusted placement posture, and the overlap area estimation unit 295 estimates the amount of overlap between the three-dimensional data acquired by the data acquisition unit 291 in the adjusted placement posture and the first three-dimensional data, while the additional data amount estimation unit 296 estimates the amount of additional data. Based on the estimated amount of overlap area and additional data amount, the analysis module 290 calculates new evaluation indicators and presents them to the user. This allows the user to determine whether the scanned area of ​​high interest to them is in a position that is likely to result in successful alignment with the first three-dimensional data. Alternatively, the calculation of evaluation values ​​and proposal of pose candidates in step SA4, and the selection of pose candidates in step SA5 may be skipped, and in step SA7, the pose calculation unit 293 may identify the placement pose after the user has finished adjusting the candidate poses.

[0100] In step SA9, the analysis module 290 generates a computer graphics workpiece (model) in the determined placement orientation. The display control unit 255 displays the computer graphics workpiece generated by the analysis module 290 on the display unit 400 by superimposing it with the live image acquired by the data acquisition unit 291. The computer graphics workpiece is semi-transparent, but may be opaque.

[0101] In step SA10, the user places the workpiece W in a temporary position on the mounting surface 142. The light receiving unit 120 then captures the workpiece W along with the mounting surface 142, and the live image including the workpiece W is acquired by the data acquisition unit 291 and displayed on the display unit 400. While viewing the live image on the display unit 400, the user moves or rotates the actual workpiece W on the mounting surface 142 until it overlaps with the computer graphics workpiece.

[0102] In step SA11, the user determines whether the actual workpiece W placed on the mounting surface 142 is positioned to overlap with the computer graphics workpiece. If the actual workpiece W on the mounting surface 142 cannot be positioned to overlap with the computer graphics workpiece, the user proceeds to step SA5 and selects a different placement position. If the actual workpiece W can be positioned to overlap with the computer graphics workpiece, the user proceeds to step SA12, where the workpiece W positioned in the second placement position is scanned by the measuring unit 100.

[0103] In step SA12, the light-emitting unit 110 of the measurement unit 100 irradiates measurement light onto the workpiece W, which is positioned in a second orientation. The light-receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The received signal output from the light-receiving unit 120 is received by the point cloud acquisition unit 263a to generate second point cloud data of the workpiece W. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts it into second mesh data. In step SA12, the second mesh data obtained by the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0104] In step SA14, the alignment unit 290A of the analysis module 290 aligns the first three-dimensional data (three-dimensional data of the workpiece placed in the first placement orientation) acquired by the data acquisition unit 291 with the second three-dimensional data based on the relative positional relationship calculated by the calculation unit 294. During this alignment, the overlapping region extracted by the extraction unit 290B of the analysis module 290 is used. The extraction unit 290B is the part that extracts the overlapping region between the three-dimensional data converted to the recommended placement orientation by the calculation unit 294 and the three-dimensional data placed in the second placement orientation. The extraction unit 290B can extract the overlapping region using, for example, the normal vector of the three-dimensional data or the color information of the workpiece. In other words, the alignment unit 290A can perform alignment based on the three-dimensional data included in the overlapping region extracted by the extraction unit 290B. When performing alignment using the texture information of the workpiece W, the texture features are estimated based on the luminance or color information that constitutes the texture information, or on local contrast or differential values. These texture features can be used in combination with shape features through various operations such as addition and multiplication.

[0105] In this embodiment, the evaluation index for placement posture is calculated so that the amount of overlapping area and the amount of additional data are large in the next scan, in order to increase the likelihood of successful three-dimensional data alignment by the alignment unit 290A. Then, while presenting the user with placement postures that have high evaluation indices, the user adjusts and determines the final placement posture. The placement posture is then superimposed on the live image, allowing the user to place the actual workpiece W in a posture close to that final posture. By performing alignment using that placement posture as the initial position, the success rate of alignment can be increased without any additional burden on the user.

[0106] The alignment unit 290A can obtain the normal vector of the three-dimensional data of the workpiece W positioned in the second positioning orientation. The alignment unit 290A can also perform alignment based on the orientation of the normal vector of the three-dimensional data of the workpiece W positioned in the second positioning orientation, the relative positional relationship calculated by the calculation unit 294, the orientation of the normal vector of the three-dimensional data of the workpiece positioned in the first positioning orientation, and the shape features extracted from the first and second three-dimensional data. That is, the alignment unit 290A first narrows down the candidate correspondence points based on the angle between the orientation of the normal vector obtained by rotating the normal vector of the three-dimensional data of the workpiece W positioned in the first positioning orientation based on the relative positional relationship, and the orientation of the normal vector of the three-dimensional data of the workpiece positioned in the second positioning orientation. Then, based on the narrowed-down candidate correspondence points, alignment can also be performed using the shape features extracted from the first and second three-dimensional data.

[0107] Furthermore, when the user places the workpiece W, an even more accurate initial estimate can be obtained by pattern matching between the computer graphics workpiece and the actual workpiece W. This is because the relative position between the measurement unit 100 and the workpiece W can be determined by epipolar geometry from the correspondence between the computer graphics workpiece and the actual workpiece W in the live image.

[0108] The initial estimates can be used to detect and extract areas where two point clouds overlap, to detect mismatches among candidate points, and to detect mismatches among position and orientation candidates obtained from the candidate points. When aligning a source point cloud P in a first three-dimensional data set that has already been scanned with a newly scanned target point cloud Q, the alignment unit 290A can identify corresponding points in the second three-dimensional data set for points included in the first three-dimensional data set, and adjust the position and orientation of the first and second three-dimensional data sets based on the identified corresponding points.

[0109] In step SB1 of the flowchart shown in Figure 11, the calculation unit 294 acquires the source point cloud P for alignment, and in step SB2, the calculation unit 294 acquires the target point cloud Q for alignment. In step SB3, the calculation unit 294 acquires the initial estimated orientation. In step SB4, the calculation unit 294 transforms the source point cloud using the arrangement orientation determined as described above. This transformation can be performed by multiplying rotation matrices and adding translation components. As a result, the transformed point cloud P' will be located at approximately the same position as the target point cloud (step SB5).

[0110] In step SB6, the extraction unit 290B extracts point cloud P'o from the converted point cloud P' where points from point cloud Q are located in its vicinity, and then applies this inversely to extract Qo. These are called overlapping region point clouds. If the point cloud is assigned normal vectors and color information, the accuracy of the overlapping region can be changed by using the existence of points with high similarity in the vicinity as a condition for extracting the overlapping region. Furthermore, the definition of "neighborhood" depends on how closely the user can position and orient the actual workpiece W to the computer graphics workpiece. Alternatively, the range of "neighborhood" may be determined from the scan range of the measurement unit 100 and the size of the workpiece W, and the user may be able to change the range of "neighborhood".

[0111] The alignment unit 290A uses only the overlapping regions extracted by the extraction unit 290B for alignment. This eliminates the occurrence of mismatches of points located in non-overlapping regions.

[0112] However, the extraction of overlapping regions by the extraction unit 290B incurs computational costs. For example, when extracting points from the point cloud P'o, the nearest point Q is searched for for each point in the point cloud P'. If M is the number of points in P and N is the number of points in Q, the computational cost O becomes O(M*N).

[0113] Therefore, the occurrence of mismatches can be suppressed by removing hidden surfaces as an alternative to overlapping regions. This is because the aforementioned initial estimates are values ​​obtained through shape analysis that result in overlapping regions suitable for alignment. Based on these initial estimates, the point clouds of P' and Q are projected using virtual cameras, and hidden surface removal based on a depth buffer commonly used in computer graphics (extracting only the points with the minimum depth position as seen from the camera) can be used to eliminate back-face data that would result in mismatches.

[0114] Furthermore, a simpler approach is to perform a simplified hidden surface removal using normal vectors. Hidden surface removal can be performed by extracting only the points in the point clouds P' and Q in the camera coordinate system that are facing the measurement unit 100. This involves extracting only the normal vectors [nx,ny,nz] in the camera coordinate system where the value of nz is negative. Therefore, computational costs can be reduced.

[0115] Figure 12 is a schematic diagram of hidden surface removal. The white arrows W1 extending in directions perpendicular to each face of the workpiece W placed on the mounting surface 142 indicate the normal vectors of each face. The thin arrow W2 indicates the component of the normal vector parallel to the Z-axis of the light-receiving unit 120, and the arrow W3 indicates the component of the normal vector perpendicular to the Z-axis of the light-receiving unit 120. Among the arrows W1, the dashed line indicates the normal vector for which the Z-axis component of the light-receiving unit 120 is positive, and the face corresponding to the dashed arrow W1 is determined to be a hidden surface.

[0116] In step SB8 shown in Figure 11, the extraction unit 290B extracts the shape features and brightness features of each point from the first three-dimensional data corresponding to the first arrangement orientation acquired by the data acquisition unit 291 and the second three-dimensional data corresponding to the second arrangement orientation.

[0117] In step SB10, the feature quantities obtained in step SB9 are input to the extraction unit 290B to detect potential point correspondences. If there are no initial estimates, candidate points are found by the difference in features between the first and second 3D data, or by the correlation between the first and second 3D data. However, in this case, points that are actually far apart but have similar shape and brightness features may be selected as point correspondences. In contrast, if initial estimates exist, only nearby points and points with similar normal vector directions can be selected as candidate correspondences, and the correlation values ​​of features can be weighted based on the distance between points and the dot product of their normal vectors.

[0118] Let pi be the i-th point in the source point cloud P, and p'i be the point pi transformed with the initial estimate. Let qj be the j-th point in the target point cloud Q. If F(*) is their feature vector and N(*) is their normal vector, then, Evaluation value w_d(dist(p'i,qj))*wn(1-dot(N(p'i),N(qj)))*CORR(F(p'i),FU(qj)) This can be calculated as follows: wd is the weight related to distance, and wn is the weight related to the direction of the normal. If this weight is set to a [1,0] step function, only close points will be selected, and if it is set to a monotonically decreasing function, continuous weights can be assigned. By selecting points with high evaluation values ​​as corresponding points, it is possible to eliminate false correspondence points that have similar shapes or brightness distributions. The output of this correspondence point detection unit is a matrix C(pi,qj) representing the candidate points for correspondence. It can be represented as a matrix where points judged to be corresponding are 1 and others are 0.

[0119] Step SB11 obtains pose candidates from these corresponding point candidates. Steps SB12 to SB17 describe the method for determining the final placement pose of workpiece W. For example, the RANSAC method can be used to determine the final placement pose of workpiece W. The following explanation uses an example with RANSAC. In RANSAC's pose candidate calculation, candidates are calculated from multiple randomly selected point correspondences (Step SB12). This allows us to obtain pose candidates (Step SB13). If only the coordinates of the points are used, a correspondence of three points is required, and if the normal and the coordinates of the points are used, a correspondence of two points is required. Taking a correspondence of three points as an example, the rotation matrix R and translation vector t that minimize the squared distance between corresponding p' and q are found by linear and nonlinear optimization (Step SB14).

[0120] Sum_C(p_i,q_j) (q_j - p'_i*R+t)^2 In this embodiment, the method for calculating candidates is not limited to this method, and candidates can be calculated using any method. For example, a method using graph cuts can also be applied to this embodiment.

[0121] The rotation matrix and translation vector obtained in this way are then used to determine if they are close to the initial estimates and to detect whether they are correct candidates. The translation vector is the sum of the squares of each element; if the initial estimate is t and the candidate is t', then... dist(t,t')=(t_x-t'_x)^2+(t_y-t'_y)^2+(t_z-t'_z)^2 It can be defined as follows. Furthermore, the rotation matrix is, Dist(R, R')= arccos[( trace(R^T*R')-1 ) / 2] It can be calculated as follows.

[0122] An arrangement posture in which the translational component error is within a predetermined dimension and the angular error is within a predetermined angle can be extracted as a correct candidate. In step SB16, the detected posture is obtained from these candidates by the final posture selection process, and the candidates after selection are acquired (step SB17). This is obtained by a method evaluated by a predetermined evaluation value. For example, assuming P’’ = R * P’ + t to represent the candidate posture with the rotation matrix R and the translational component t, the number of points of the point cloud Q for which dist(P’’ - Q) < threshold exists can be used as the evaluation value. However, the calculation of the evaluation value combined in this embodiment is not particularly limited. Since this evaluation value calculation can also be narrowed down to the overlapping region, it is possible to suppress the phenomenon that corresponding points happen to exist in the vicinity where they should not overlap and the evaluation value becomes erroneously high. In this way, the second arrangement posture is specified, and by scanning the workpiece W arranged in the specified second arrangement posture with the measurement unit 100, the accuracy of alignment with the first three-dimensional data is high, and the second three-dimensional data with a large additional data amount can be obtained.

[0123] When performing alignment between the first three-dimensional data and the second three-dimensional data, the alignment unit 290A can perform alignment based on the shape features and image features of the three-dimensional data. That is, the analysis module 290 includes a shape feature extraction unit (first extraction unit) 290C that extracts the shape features of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291, and an image feature extraction unit (second extraction unit) 290D. The shape feature extraction unit 290C has a neural network including an input layer, a plurality of intermediate layers, and an output layer. The input layer of the neural network is a part that receives the input of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291. The plurality of intermediate layers of the neural network are parts that extract shape features based on the input received by the input layer. The output layer of the neural network is a part that outputs the shape features extracted by the intermediate layer. On the other hand, the image feature extraction unit 290D is a part that extracts image features from the texture information included in each of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0124] In this embodiment, deep learning is used to align the first three-dimensional data with the second three-dimensional data, incorporating image features such as brightness and color into the feature vector. When calculating the feature vector for each point in an image or point cloud, a "convolution" operation can be performed by multiplying the coordinates and color of the point itself and the information of surrounding points by coefficients, and then taking the sum or maximum value of these coefficients. For example, in a Convolutional Neural Network (CNN) for images, there are many 3x3 filters with coefficients determined by prior training, and the output vector for that point is obtained by arranging the output values ​​of each filter. The feature vector is obtained by repeating this process many times with different filters.

[0125] In the case of point clouds, such as those included in the first or second 3D data, unlike images, there is no guarantee that data exists at regular intervals, so other methods can be used. For example, one method is to apply a CNN in 3D that samples points into voxels, multiplies by a coefficient if a point exists, and outputs 0 if a point does not exist. Another method is PointNet++, which multiplies each point by the same coefficient and takes the maximum value of the output. Finally, there is KPConv, which calculates coefficients by interpolating the difference between the point cloud position and a coefficient determined on a grid, and then convolves the data.

[0126] Processing point clouds, such as those contained in the first and second 3D data, requires a massive amount of computation and long processing times because the arrangement of the points is unknown. In the field of 3D scanner technology, as in this example, long processing times would halt work, so the processing time must be within a practically usable range. Therefore, the number of points that can be processed is naturally limited. For example, the number of keypoints may be limited to a few thousand, and the total number of points in the point cloud may be limited to tens of thousands. Since the measurement unit 100 can scan millions of points in a single scan, the points used for deep learning need to be thinned to about 1% of that.

[0127] In the simplest way to input image information into a neural network's input layer, you only need to input coordinates, normal vectors, and image information (luminance and color). However, in this case, the amount of data input to the input layer is too small, resulting in the loss of useful image information. Therefore, this may not lead to performance improvement.

[0128] The following describes the extraction process of shape features and image features based on the flowchart shown in Figure 13. In step SC1, the analysis module 290 acquires point clouds (input point clouds) of the first three-dimensional data and the second three-dimensional data. In steps SC2 and SC3, coarse thinning and dense thinning are performed, respectively. That is, as shown in Figure 5, the analysis module 290 has a resolution conversion unit 290E that converts the resolution of the first three-dimensional data acquired by the data acquisition unit 291 to the resolution of the second three-dimensional data. The resolution conversion unit 290E performs a first conversion process (coarse thinning process) that converts the first three-dimensional data and the second three-dimensional data into three-dimensional data with a first resolution lower than the resolution acquired by the data acquisition unit 291. In other words, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into third three-dimensional data, which is three-dimensional data with a first resolution, and also converts the second three-dimensional data acquired by the data acquisition unit 291 into fourth three-dimensional data, which is three-dimensional data with a first resolution. In the first conversion process by the resolution conversion unit 290E, the three-dimensional data acquired by the data acquisition unit 291 is reduced in resolution to become low-resolution three-dimensional data, so the number of points in the three-dimensional data becomes less than the number of points acquired by the data acquisition unit 291.

[0129] The first conversion process by the resolution conversion unit 290E corresponds to step SC2. Step SC2 generates a third three-dimensional data (third point cloud) and a fourth three-dimensional data (fourth point cloud), which are point clouds for shape feature extraction (Q). In step SC4, the shape feature extraction unit 290C acquires the point clouds for shape feature extraction (Q). The shape feature extraction unit 290C extracts shape features from the shape information contained in the first three-dimensional data and the second three-dimensional data of the first resolution, which have been converted by the resolution conversion unit 290E.

[0130] The resolution conversion unit 290E performs a second conversion process (dense thinning) to convert the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291 into three-dimensional data with a resolution lower than that acquired by the data acquisition unit 291 and a resolution higher than that of the first three-dimensional data. In other words, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into fifth three-dimensional data, which is three-dimensional data with the second resolution, and also converts the second three-dimensional data acquired by the data acquisition unit 291 into sixth three-dimensional data, which is three-dimensional data with the second resolution. In the second conversion process by the resolution conversion unit 290E, the three-dimensional data acquired by the data acquisition unit 291 is reduced in resolution to become low-resolution three-dimensional data, so the number of points in the three-dimensional data is less than the number of points acquired by the data acquisition unit 291, but the resolution is higher than that of the first conversion process, so the number of points is greater than that of the first conversion process. The second conversion process performed by the resolution conversion unit 290E corresponds to step SC3.

[0131] Step SC3 generates a fifth three-dimensional data set (fifth point cloud) and a sixth three-dimensional data set (sixth point cloud) (Q'), which are point clouds for image feature extraction. In step SC5, the image feature extraction unit 290D acquires the point cloud (Q'). The image feature extraction unit 290D extracts image features from the texture information contained in the first three-dimensional data set and the second three-dimensional data set of the second resolution, which have been converted by the resolution conversion unit 290E.

[0132] Alternatively, step SC3 described above may be skipped, and the image feature extraction unit 290D may generate a point cloud for image feature extraction using the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0133] In the first and second transformation processes, for example, random decimation, which randomly selects a predetermined number of data points, and voxel decimation, which calculates a voxel grid and reduces the number of points within a voxel to just one average value, can be used. Here, the point cloud before decimation (the point cloud before the second transformation process) may be used as the point cloud for image feature extraction.

[0134] In step SC6, the shape feature extraction unit 290C extracts keypoints from the low-resolution first three-dimensional data and the second three-dimensional data. Any method can be used for keypoint extraction, such as random selection or extraction of characteristic points using deep learning or rule-based point cloud analysis. The keypoints extracted by the shape feature extraction unit 290C are acquired by the shape feature extraction unit 290C (step SC7).

[0135] In step SC8, the analysis module 290 acquires the point cloud (Q) for shape feature extraction obtained in step SC4 and the keypoints obtained in step SC7, and the analysis module 290 samples points around the keypoints. At this time, the points for shape feature extraction are sampled based on the point cloud (Q) for shape feature extraction obtained in step SC4.

[0136] In step SC12, which follows step SC5, the analysis module 290 acquires the point cloud for image feature extraction (Q') obtained in step SC5 and the keypoints obtained in step SC7, and the analysis module 290 samples points around the keypoints. Points for image feature calculation are sampled based on the point cloud for image feature extraction (Q') obtained in step SC5. Also, since the number of points in the point cloud for image feature extraction (Q') is large, the neighborhood range for sampling may be narrower compared to the point cloud for shape feature extraction (Q).

[0137] In step SC9, which proceeds after step SC8, the shape features are input to the shape feature extraction unit 290C. For example, a shape feature vector can be calculated only from the vicinity of the points sampled for shape features. The method for calculating the shape feature vector is not particularly limited, but for example, it is possible to project the image information of the point cloud according to the normal of the keypoint to obtain a patch image and calculate the feature vector using calculations such as CNN or SIFT (Scale-Invariant Feature Transform), or to create a histogram based on the RGB and brightness information of the sampled points and use that histogram as the feature vector. In step SC10, the shape feature extraction unit 290C calculates the shape feature vector of the keypoint.

[0138] Meanwhile, in step SC13, which proceeds after step SC12, image features are input to the image feature extraction unit 290D. Here, the image feature vectors are calculated. All points around the keypoints sampled from the point cloud for image feature extraction (Q') are used to calculate the feature vector for each keypoint. In step SC14, the image feature extraction unit 290D calculates the image feature vectors of the keypoints, and in step SC15, the image feature vectors calculated by the image feature extraction unit 290D are obtained.

[0139] Specifically, the image feature extraction unit 290D extracts key points used to calculate shape features from the low-resolution first three-dimensional data and the second three-dimensional data. The image feature extraction unit 290D can identify the region corresponding to each extracted key point from the first three-dimensional data and the second three-dimensional data before they are converted by the resolution conversion unit 290E. The image feature extraction unit 290D then extracts image features from the identified corresponding regions.

[0140] When calculating the image feature vector, the image feature extraction unit 290D may identify the region corresponding to each keypoint extracted by the shape feature extraction unit 290C from the first three-dimensional data and the second three-dimensional data, and image features may be extracted from the region corresponding to the keypoint identified by the image feature extraction unit 290D.

[0141] In step SC11, the calculation of the shape feature vector requires inputting a 3- to 6-dimensional vector with X, Y, Z coordinates and, if necessary, normal information Nx, Ny, Nz. In this way, the shape feature extraction unit 290C extracts keypoints as sub-regions used for calculating shape features from the low-resolution first three-dimensional data and the second three-dimensional data, and extracts shape features for each extracted keypoint.

[0142] In step SC16, the shape feature vector and the image feature vector, or the shape feature vector containing the image feature, are compared at keypoints in the source 3D data and the target 3D data to obtain pair candidates. The method for calculating pair candidates is not particularly limited, but for example, one method is to calculate the distance between the feature vectors and select those with the smallest distance, or to select those with high output values ​​from a deep learning module that outputs correspondence. For distance, the mean square (L2 distance) or cosine similarity can be used. In this case, when comparing the shape feature vector and the image feature vector, one method is to simply input a vector that combines both vectors to find pairs, or the sum, maximum, or minimum values ​​of the distances calculated for the shape feature vector and the image feature vector respectively can be used. Using the points of the pair candidates, it is possible to determine the positional relationship (rotation and translation) between the point clouds using RANSAC or a deep learning-based method.

[0143] Figure 14 shows the case where a vector with added image features is input to the shape feature extraction unit during the extraction of shape features and image features. The process proceeds from step SC8 in Figure 13 to step SC9' in Figure 14, and then from step SC12 in Figure 13 to step SC13' in Figure 14. Steps SC13' to SC15' are the same as steps SC13 to SC15 in Figure 13. In step SC10', the image feature vector is input to the shape feature extraction unit 290C. In this case, the output shape feature vector is a shape feature vector that already includes image features (step SC11'). Step SC16' is the same as step SC16 in Figure 13.

[0144] The alignment unit 290A performs a combined global alignment and local alignment process to accurately align the point cloud of the first three-dimensional data with the point cloud of the second three-dimensional data. In this process, global alignment is performed using a low-resolution point cloud, and local alignment is performed using a high-resolution point cloud converted from that position. For example, point cloud data with a first resolution (low resolution) is used for shape feature extraction in global alignment, and point cloud data with a third resolution higher than the first resolution is used for image feature extraction. Local alignment is performed using point cloud data with a second resolution higher than both the first and second resolutions as input. Note that the point cloud data used for local alignment may be point cloud data with a second resolution, or it may be point cloud data with the same resolution as the point cloud generated by receiving the light signal in the point cloud acquisition unit 263a.

[0145] Specifically, the alignment unit 290A acquires shape features extracted by the shape feature extraction unit 290C and image features extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A can perform global alignment based on the shape features extracted by the shape feature extraction unit 290C and the image features extracted by the image feature extraction unit 290D. In this case, the alignment unit 290A performs a first alignment process to calculate low-precision alignment parameters that indicate the relative position and orientation of the second three-dimensional data to the first three-dimensional data.

[0146] When calculating the low-precision alignment parameters, steps SD1 to SD7 are performed as shown in Figure 15. Steps SD1 to SD5 are the same as steps SC1 to SC5 shown in Figure 13. In step SD6 shown in Figure 15, the point cloud for shape feature extraction acquired in step SD4 and the point cloud for image feature extraction acquired in step SD5 are input to the alignment unit 290A. The alignment unit 290A performs global alignment of the input point clouds and calculates the low-precision alignment parameters.

[0147] After the first alignment process, the alignment unit 290A acquires the low-precision alignment parameters calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data. Based on the low-precision alignment parameters, the first three-dimensional data, and the second three-dimensional data, the alignment unit 290A can perform local alignment. In this case, the alignment unit 290A performs a second alignment process to calculate high-precision alignment parameters that indicate the relative position and orientation of the second three-dimensional data with respect to the first three-dimensional data.

[0148] When calculating high-precision alignment parameters, as shown in Figure 16, in step SE1, the analysis module 290 acquires the input point cloud of the first three-dimensional data. In step SE2, the analysis module 290 acquires the input point cloud of the second three-dimensional data. In step SE3, the analysis module 290 acquires the low-precision alignment parameters calculated by the flowchart shown in Figure 15.

[0149] The analysis module 290 has a coordinate transformation unit 290F that performs coordinate transformation of the first three-dimensional data based on low-precision alignment parameters calculated by the flowchart shown in Figure 15. In step SE4, the coordinate transformation unit 290F performs coordinate transformation of the input point cloud of the first three-dimensional data based on the low-precision alignment parameters. In step SE5, the point cloud after coordinate transformation is acquired.

[0150] In step SE6, the input point cloud of the second three-dimensional data acquired in step SE2 and the point cloud after coordinate transformation acquired in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point clouds and calculates high-precision alignment parameters. In step SE7, the high-precision alignment parameters are obtained. In this way, in the second alignment process, the alignment unit 290A acquires the first three-dimensional data and the second three-dimensional data that have undergone coordinate transformation by the coordinate transformation unit 290F, and performs local alignment based on the first three-dimensional data and the second three-dimensional data.

[0151] As described above, after the alignment process of step SA14 shown in Figure 6 is performed, the synthesis process is executed. That is, as shown in Figure 5, the analysis module 290 has a synthesis unit 290G. The synthesis unit 290G is the part that synthesizes the first three-dimensional data of the workpiece W positioned in a first position aligned by the alignment unit 290A and the second three-dimensional data of the workpiece W positioned in a second position to generate synthesized three-dimensional data. The synthesis unit 290G can also acquire the first mesh data as the first three-dimensional data and the second mesh data as the second three-dimensional data. In this case, the synthesis unit 290G synthesizes the first mesh data and the second mesh data to generate synthesized mesh data as synthesized three-dimensional data.

[0152] Furthermore, the removal of hidden surfaces shown in Figure 12 is performed before generating the composite three-dimensional data and is used for alignment by the alignment unit 290A. That is, the alignment unit 290A can perform alignment using the three-dimensional data from which the hidden surfaces have been removed. Then, the compositing unit 290G generates the composite three-dimensional data based on the alignment performed using the three-dimensional data from which the hidden surfaces have been removed. Here, the generation of the composite three-dimensional data is performed before the removal of hidden surfaces, i.e., using the three-dimensional data that includes the hidden surfaces.

[0153] The synthesis unit 290G can acquire shape information of the workpiece W based on a first light-receiving signal output by the light-receiving unit 120, and can also acquire texture information of the workpiece W based on a second light-receiving signal output by the light-receiving unit 120. When the synthesis unit 290G has acquired shape information and texture information of the workpiece W, it generates three-dimensional data including the shape information and texture information of the workpiece W.

[0154] Furthermore, when the data acquisition unit 291 acquires composite three-dimensional data obtained by combining the first three-dimensional data and the second three-dimensional data, the posture calculation unit 293 can calculate a different placement posture from the first placement posture based on the composite three-dimensional data acquired by the data acquisition unit 291. In this way, the user can be presented with a placement posture to be added to the already generated composite three-dimensional data during scanning. In this case, the additional data amount estimation unit 296 estimates the amount of additional data to be added to the composite three-dimensional data. The overlapping area estimation unit 295 also estimates the overlapping area to the composite three-dimensional data. This allows the user to be presented with an evaluation index based on the amount of additional data and overlapping area when scanning with the additional placement posture.

[0155] Once the synthesis process is complete, the process proceeds to step SA15 shown in Figure 6. In step SA15, the user determines whether or not there are areas on the workpiece W that need to be scanned. If there are areas on the workpiece W that need to be scanned, the process proceeds to step SA2, where the workpiece W is placed on the mounting surface 142 in the third placement orientation and scanned by the measuring unit 100. By repeating this process, all necessary scanning areas can be scanned. If the result in step SA15 is NO, a fully scanned model is obtained in step SA16. Note that a fully scanned model is not mandatory; any model in which the user's required areas have been scanned is acceptable.

[0156] Next, an example of the scanning process when there is a reference model such as CAD data or measured 3D data of the workpiece W will be explained based on the flowchart in Figure 17. In step SG1, the reading unit 292 of the analysis module 290 reads the CAD model (reference model) from the storage device 240, and the CAD model is acquired in step SG2. In step SG3, evaluation value calculation and posture candidate proposal are performed using the same approach as in step SA4 in Figure 6. In step SG3, since CAD data is available, the posture calculation unit 293 analyzes the CAD data read by the reading unit 292. By analyzing the CAD data, the posture calculation unit 293 estimates the amount of data to be acquired (amount of additional data) and calculates the recommended placement posture based on the estimated amount of data. The posture calculation unit 293 also analyzes the CAD data read by the reading unit 292 and estimates the overlapping area between the 3D data and the CAD data that would be acquired when placed in the recommended placement posture. The posture calculation unit 293 calculates the recommended placement posture based on the estimated overlapping area.

[0157] In step SG4, the user selects a candidate posture on a user interface screen 700, such as shown in Figure 8. In step SG5, the user determines whether the candidate posture displayed on the display unit 400 is the desired posture. If it is not the desired posture, the process proceeds to step SG6, where the placement posture of the workpiece W is adjusted using computer graphics. In step SG7, the overlapping area estimation unit 295 estimates the amount of overlapping area of ​​the adjusted placement posture, and the additional data amount estimation unit 296 estimates the amount of additional data. Based on the estimated overlapping area amount and additional data amount, the analysis module 290 calculates new evaluation indicators and presents them to the user.

[0158] If the desired orientation is determined in step SG5, the process proceeds to step SG8, ​​where the analysis module 290 generates a computer graphics workpiece (model) for the determined placement orientation. The display control unit 255 displays the computer graphics workpiece generated by the analysis module 290 on the display unit 400 by superimposing it with the live image acquired by the data acquisition unit 291.

[0159] In step SG9, the user places the workpiece W in a temporary position on the mounting surface 142. The light receiving unit 120 then captures the workpiece W along with the mounting surface 142, and the live image is acquired by the data acquisition unit 291 and displayed on the display unit 400. While viewing the live image on the display unit 400, the user moves or rotates the actual workpiece W until it overlaps with the computer graphics workpiece.

[0160] In step SG10, the user determines whether the actual workpiece W is placed so as to overlap with the computer graphics workpiece. If the actual workpiece W cannot be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG4 and selects a different placement orientation. If the actual workpiece W can be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG11, where the measurement unit 100 scans the workpiece W in the first placement orientation. In step SG12, the first three-dimensional data is acquired. In step SG13, the alignment unit 290A aligns the first three-dimensional data with the CAD data, which is the reference model. In step SG14, the analysis module 290 acquires the aligned CAD model.

[0161] Step SG15 performs evaluation value calculation and proposes pose candidates. In step SG16, the user selects a pose candidate, and then proceeds to step SG17. If the pose is not the desired one, adjustments are made in step SG18, the evaluation value is updated in step SG19, and then the process proceeds to step SG17. If the pose is the desired one, the process proceeds to step SG20, where the display control unit 255 displays the computer graphics work generated by the analysis module 290 on the display unit 400, superimposed on the live image acquired by the data acquisition unit 291. Here, the computer graphics work may be the first three-dimensional data or reference model acquired in step SG12. Alternatively, the first three-dimensional data or reference model may be displayed simultaneously on the display unit 400, or they may be displayed separately.

[0162] In step SG21, the user moves or rotates the actual workpiece W while viewing the live image on the display unit 400 until the actual workpiece W overlaps with the computer graphics workpiece.

[0163] In step SG22, the user determines whether the actual workpiece W is placed so as to overlap with the computer graphics workpiece. If the actual workpiece W cannot be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG16 and selects a different placement orientation. If the actual workpiece W can be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG23 and scans the workpiece W in that placement orientation (second placement orientation) using the measurement unit 100. In step SG24, the alignment unit 290A aligns the acquired second three-dimensional data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs the alignment process of the three-dimensional model, and the synthesis unit 290G performs the synthesis process. In step SG27, the user determines whether there are any areas that need to be scanned. If there are areas that need to be scanned, the user proceeds to step SG12. If there are no areas that need to be scanned, the user acquires a full scan model in step SG28. The full-surface scan model acquired in step SG28 may be output together with the CAD model, which has undergone alignment processing by the alignment unit 290A in step SG26. Furthermore, the full-surface scan model and the CAD model may be displayed on the display unit 400 in an aligned state.

[0164] In this embodiment, as schematically shown in Figure 19, the alignment unit 290A aligns the first three-dimensional data A and the second three-dimensional data B based on a first alignment parameter (first positional relationship), which is the positional relationship between the first three-dimensional data A and the second three-dimensional data B. At the same time, the synthesis unit 290G synthesizes the first three-dimensional data (first mesh data) A and the second three-dimensional data (second mesh data) B acquired by the data acquisition unit 291 to generate synthesized three-dimensional data AB. Subsequently, when the data acquisition unit 291 acquires the third three-dimensional data C, the alignment unit 290A aligns the composite three-dimensional data AB, which is a composite of the first three-dimensional data A and the second three-dimensional data B, with the third three-dimensional data C based on the second alignment parameter (second positional relationship), which is the positional relationship between the composite three-dimensional data AB and the third three-dimensional data C. At the same time, the compositing unit 290G combines the composite three-dimensional data AB with the third three-dimensional data (third mesh data) C acquired by the data acquisition unit 291 to generate composite three-dimensional data ABC. At this time, when the editing unit 290H receives input to edit the position or shape of at least one of the mesh data, such as the first mesh data, the second mesh data, and the third mesh data, the storage device 240 saves the first mesh data, the second mesh data, and the third mesh data. Editing the shape of the mesh data includes, for example, the removal of some point clouds.

[0165] Furthermore, when the data acquisition unit 291 acquires the fourth three-dimensional data D, the alignment unit 290A aligns the composite three-dimensional data ABC with the fourth three-dimensional data D based on the third alignment parameter (third positional relationship), which is the positional relationship between the composite three-dimensional data ABC and the fourth three-dimensional data D. At the same time, the synthesis unit 290G synthesizes the composite three-dimensional data ABC with the fourth three-dimensional data (fourth mesh data) D acquired by the data acquisition unit 291 to generate composite three-dimensional data ABCD. When the editing unit 290H receives input to edit the position or shape of the fourth mesh data, the storage device 240 saves the fourth mesh data.

[0166] In this way, by sequentially combining two sets of three-dimensional data, the user can obtain three-dimensional data for the entire work W. By performing sequential combination, the processing load for converting from point cloud to mesh data becomes constant, and the processing load for joining mesh data that has undergone decimation also becomes constant. For example, this reduces the processing load compared to converting all the original point clouds that make up the combined three-dimensional data ABCD at once. Note that the combined three-dimensional data ABCD is not limited to the method of sequentially adding the fourth three-dimensional data to the combined three-dimensional data ABC, but may also be generated by combining the first three-dimensional data, the second three-dimensional data, the third three-dimensional data, and the fourth three-dimensional data based on the positional relationships between each of the three-dimensional data.

[0167] The display control unit 255 displays the composite three-dimensional data generated by the synthesis unit 290G on the display unit 400, and also displays the first three-dimensional data and the second three-dimensional data on the display unit 400 in a distinguishable manner. That is, the display control unit 255 generates a user interface screen 800 as shown in Figure 18 and displays it on the display unit 400. The user interface screen 800 is provided with a first display area 801 on which the first three-dimensional data (first mesh data) is displayed, a second display area 802 on which the second three-dimensional data (second mesh data) is displayed, and a third display area 803 on which the composite three-dimensional data (composite mesh data) is displayed. Since the first display area 801 and the second display area 802 are distinguished, the first mesh data and the second mesh data before resynthesis by the synthesis unit 290G can be displayed on the display unit 400 in a distinguishable manner. The first display area 801, the second display area 802, and the third display area 803 each display the X, Y, and Z coordinate systems, respectively.

[0168] In this embodiment, the synthesis process of the first three-dimensional data and the second three-dimensional data performed by the synthesis unit 290G can be edited. That is, the analysis module 290 has an editing unit 290H that edits the synthesis process of the first three-dimensional data and the second three-dimensional data performed by the synthesis unit 290G. The editing unit 290H accepts input to edit the position or shape of at least one of the three-dimensional data, either the first three-dimensional data or the second three-dimensional data. The editing unit 290H can edit the position of the first three-dimensional data in the X, Y, or Z directions. The second three-dimensional shape data can also be edited in the same way.

[0169] The editing unit 290H receives input for editing the first or second 3D data, and edits the first or second 3D data based on the received input. Then, the synthesis unit 290G re-synthesizes the first and second 3D data based on the input received by the editing unit 290H. At this time, the alignment unit 290A can align the first and second 3D data based on the first alignment parameter, which is the positional relationship between the first 3D data A and the second 3D data B. This eliminates the need to accept alignment specifications from the user, improving convenience. Similarly, when the editing unit 290H receives editing input for the third and fourth 3D data, it edits the third and fourth 3D data based on the received input. Then, the synthesis unit 290G updates and regenerates the synthesized three-dimensional data based on the input received by the editing unit 290H.

[0170] For example, when combining the first three-dimensional data and the second three-dimensional data, the first and second three-dimensional data, which have different initial positions, are aligned in the background of the display processing of the user interface screen 810, as shown as an example in Figure 20, and then the combining unit 290G performs the combining. The user interface screen 810 is provided with a button 811 to display the combining result, a button 812 to display the additional shape, and a button 813 to display the original shape.

[0171] When the user operates the synthesis result display button 811, once the synthesis of the first three-dimensional data and the second three-dimensional data is complete, the display control unit 255 displays the synthesized three-dimensional data, which is the synthesis result, in the display area 814 of the user interface screen 810. By viewing the synthesized three-dimensional data, the user can check the quality of the synthesis result. If there are any defects, the location and cause of the defects can also be identified. When identifying defects, as shown in Figure 21, if the user operates the additional shape display button 812 on the user interface screen 810, the display control unit 255 displays the second three-dimensional data in the display area 814 of the user interface screen 810. Also, as shown in Figure 22, if the user operates the original shape display button 813 on the user interface screen 810, the display control unit 255 displays the first three-dimensional data in the display area 814 of the user interface screen 810. In this way, the first and second 3D data can be displayed individually by the user's switching operation. Therefore, if a problem is found in the composite result of the composite 3D data, the user can select either the first or second 3D data and, while checking the selected 3D data, perform actions such as partial removal or repair of the 3D data. Alternatively, the composite 3D data, the first 3D data, and the second 3D data may be displayed on a single user interface screen.

[0172] The user interface screen 810 shown in Figure 21 is provided with an edit button 815. When the user operates the edit button 815, the display control unit 255 generates and displays a user interface screen 820 for data editing, as shown in Figure 23, and accepts editing instructions for three-dimensional data from the editing unit 290H. The user interface screen 820 is provided with a display area 821 that accepts instructions such as partial excision or repair and displays the three-dimensional data to be edited by the editing unit 290H based on those instructions, a procedure display area 822 that displays the editing procedure, and a setting area 823 that allows setting the method for specifying and selecting the editing area, the editing method, etc. When editing such as partial excision or repair of the three-dimensional data is performed on the user interface screen 820, the edited three-dimensional data is stored in the storage device 240.

[0173] If the edited 3D data is, for example, the second 3D data, the alignment unit 290A performs alignment between the edited second 3D data and the first 3D data again. At this time, the alignment unit 290A can align the edited second 3D data and the first 3D data based on the first alignment parameter. Then, the synthesis unit 290G synthesizes the edited second 3D data and the first 3D data to generate synthesized 3D data. The synthesized 3D data thus generated is displayed in the display area 814.

[0174] Furthermore, if the quality of the composite three-dimensional data is poor even after editing the second three-dimensional data, the user issues a re-acquisition instruction to re-acquire the second three-dimensional data acquired by the data acquisition unit 291. When the user operates the retake button 816 on the user interface screen 810 shown in Figure 20, the reception unit 298 receives a re-acquisition instruction to re-acquire the second three-dimensional data. In this case, the data acquisition unit 291 acquires the fifth mesh data (update mesh data) based on the re-acquisition instruction received by the reception unit 298, and also acquires the first mesh data from the storage device 240, and the alignment unit 290A aligns the first mesh data with the fifth mesh data. The fifth mesh data is acquired to update the second mesh data. Therefore, the alignment unit 290A can align the update mesh data acquired as a replacement for the second mesh data, i.e., the fifth mesh data, with the first mesh data based on the first alignment parameter. Alignment begins when the alignment button 817 on the user interface screen 810 shown in Figure 20 is operated.

[0175] When the reception unit 298 receives a reacquisition instruction, the display control unit 255 can also display the recommended placement orientation based on the first three-dimensional data on the display unit 400. The recommended placement orientation is the orientation calculated by the orientation calculation unit 293 as described above. By displaying the recommended placement orientation on the display unit 400, placement orientations with a large amount of additional data can be identified.

[0176] The synthesis unit 290G then synthesizes the first mesh data and the fifth mesh data aligned by the alignment unit 290A and updates the synthesized three-dimensional data. The updated synthesized three-dimensional data is displayed in the display area 814. As described above, the synthesis of the first mesh data and the fifth mesh data aligned by the alignment unit 290A may also be performed using the first alignment parameter, which is the alignment parameter between the first mesh data and the second mesh data. That is, the synthesized three-dimensional data AB is associated with the first alignment parameter, which is the alignment parameter between the first three-dimensional data A and the second three-dimensional data B. When a reacquisition instruction is received to reacquire the second three-dimensional data, the alignment of the first three-dimensional data A and the newly acquired three-dimensional data B' in place of the second three-dimensional data B may be performed using the first alignment parameter that was associated with the synthesized three-dimensional data AB, thereby generating new synthesized three-dimensional data AB'. When the composite 3D data is updated, the compositing unit 290G may discard the composite 3D data before the update. Also, when compositing the first mesh data and the third mesh data aligned by the alignment unit 290A, the compositing unit 290G may discard the second mesh data, which is the mesh data before the update. In other words, since mesh data that does not constitute the composite 3D data is unnecessary data, discarding it can prevent unnecessary occupancy of memory space. The system may also require user confirmation before discarding unnecessary data.

[0177] Furthermore, the alignment unit 290A performs alignment between the composite three-dimensional data AB' obtained by combining the first mesh data and the fifth mesh data and the third mesh data. At this time, the alignment unit 290A can perform alignment between the composite mesh data AB' obtained by combining the first mesh data and the fifth mesh data and the third mesh data based on the second alignment parameter. Then, the compositing unit 290G combines the composite mesh data AB' aligned by the alignment unit 290A with the third mesh data to generate composite mesh data AB'C. The generated composite mesh data AB'C is displayed in the display area 814.

[0178] On the other hand, the first, fifth, and third three-dimensional data that constitute the composite three-dimensional data are automatically stored in the storage device 240 as necessary data, so at least one of the first, fifth, and third three-dimensional data can be read out later. For noise or misalignment that may have been missed during repeated synthesis processing, the individual scan results are automatically saved, making it possible to re-execute only the synthesis process. For example, as shown in Figure 19, if composite three-dimensional data AB is generated, then composite three-dimensional data ABC is generated, and then composite three-dimensional data ABCD is generated, and if noise is introduced in the third three-dimensional data C and synthesis continues up to the fourth three-dimensional data D, the process can be restarted from the middle by recombining the third three-dimensional data C with the composite three-dimensional data AB.

[0179] Thus, the composite three-dimensional data may include information indicating the synthesis order of the three-dimensional data, and this information can be stored in the storage device 240 in association with the composite three-dimensional data. The synthesis unit 290G synthesizes the first mesh data and the third mesh data, which have been aligned by the alignment unit 290A, based on the information indicating the synthesis order corresponding to the second three-dimensional data.

[0180] (CAD data alignment) When positioning an actual workpiece W to match the CAD data displayed on the display unit 400, the user needs to move the workpiece W on the rotating stage 143. However, the present invention is not limited to this, and the user may also move the CAD data to match the actual workpiece W. In other words, positioning the actual workpiece W to match the CAD data while looking at the display unit 400 can be difficult because it requires looking at both the display unit 400 and the rotating stage 143. Furthermore, since the camera capturing the actual workpiece W is opposite the user's line of sight, the user is operating while looking at a mirrored image, which creates difficulties. Therefore, in order to improve user convenience, the 3D scanner 1 can be equipped with a function (alignment function) that allows the user to align the CAD data while looking at the display unit 400 without moving the actual workpiece W.

[0181] The details of the alignment function will be explained below based on the flowchart shown in Figure 24. In the following explanation, the alignment will be referred to as overlay alignment, and the CAD data will be referred to as a "virtual object". In step S100, the controller 200 reads the center position of the rotating stage 143 and the CAD data. In step S101, the controller 200 calculates the virtual ground based on the center position of the rotating stage 143 read in step S100 and displays it on the display unit 400, and also displays the virtual object on the display unit 400 based on the CAD data.

[0182] In step S102, the controller 200 determines whether or not the mouse button on the operation unit 250 was pressed near the display position of the virtual object. If the mouse button was not pressed near the display position of the virtual object, the overlay alignment is terminated. However, if the mouse button was pressed near the display position of the virtual object, the process proceeds to step S103, where the controller 200 performs virtual object rotation by mouse dragging.

[0183] The virtual object rotation process will be explained based on the flowchart in Figure 25. In step S200, the controller 200 determines whether or not the rotation button on the operation unit 250 has been pressed. If it is determined that the rotation button on the operation unit 250 has been pressed, the controller 200 determines in step S201 whether or not the direction of movement of the mouse on the operation unit 250 is close to horizontal. If the direction of movement of the mouse is not close to horizontal, the process proceeds to step S202. On the other hand, if the direction of movement of the mouse is close to horizontal, the process proceeds to step S203, where the controller 200 fixes the rotation axis of the virtual object with an axis perpendicular to the virtual ground. In step S204, the controller 200 extracts the horizontal component of the mouse movement. In step S205, the controller 200 calculates the orientation rotated according to the extracted horizontal component around the fixed axis. In step S206, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S207, the display of the virtual object is updated. In step S208, the controller 200 determines whether the mouse button has been released or not. If the mouse button has not been released, the process returns to step S201.

[0184] In step S202, the controller 200 determines whether the direction of movement of the mouse on the operating unit 250 is close to vertical. If the direction of movement of the mouse is close to vertical, the process proceeds to step S209, where the controller 200 fixes the rotation axis of the virtual object on the left-right axis in the line of sight direction. In step S210, the controller 200 extracts the vertical component of the mouse movement. In step S211, the controller 200 calculates the posture rotated according to the extracted vertical component around the fixed axis, and proceeds to step S206.

[0185] If it is determined in step S202 that the direction of mouse movement is not nearly vertical, the process proceeds to step S212 to extract the amount of mouse movement. In step S213, the posture is calculated by rotating the mouse in an arbitrary direction according to the amount of mouse movement, and the process proceeds to step S206.

[0186] If the result in step S200 is NO, the process proceeds to step S214 to extract the mouse movement amount. In step S215, the posture is calculated by rotating the mouse in an arbitrary direction according to the mouse movement amount, and the process proceeds to step S206.

[0187] Next, the process proceeds to step S104 shown in Figure 24, where the controller 200 calculates the distance between the virtual object and the virtual ground. In step S105, the controller 200 extracts the faces of the virtual object that are closest to the virtual ground. In step S106, the controller 200 calculates the degree of contact between the faces of the virtual object and the virtual ground. In step S107, the controller 200 determines whether there are any extracted faces for which the degree of contact has not been calculated. If there are extracted faces for which the degree of contact has not been calculated, the next face is selected in step S108 and the process proceeds to step S106.

[0188] If there are no extracted surfaces for which the degree of contact has not been calculated, the process proceeds to step S109, where the controller 200 selects the surface with the highest degree of contact. In step S110, the controller 200 calculates the orientation in which the selected surface and the virtual ground are in contact. In step S111, the display of the virtual object is updated to reflect the calculated orientation.

[0189] The flowchart in Figure 26 shows the process of adjusting the positional relationship between a virtual object and the virtual ground. In step S300, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S301, the controller 200 determines whether the virtual object is embedded in the virtual ground or not. If the result in step S301 is YES, the process proceeds to step S302, where the virtual object is pushed onto the virtual ground and then proceeds to step S303. If the result in step S301 is NO, the process proceeds to step S303. In step S303, the controller 200 determines whether the virtual object is floating above the virtual ground or not. If the result in step S303 is YES, the process proceeds to step S304, where the virtual object is grounded on the virtual ground.

[0190] The flowchart in Figure 27 shows the stage surface detection process. In step S400, the controller 200 measures and acquires the three-dimensional shape of the rotating stage 143. In step S401, the controller 200 determines the position of the stage surface (mounting surface 142) from the three-dimensional shape of the rotating stage 143. In step S402, the controller 200 calculates the height information of the stage surface relative to the camera position based on the obtained planar position. In step S403, the calculated stage surface height information is stored in the storage device 240 or the like.

[0191] In step S404, the controller 200 determines whether or not a dedicated chart is available. A dedicated chart is, for example, a calibration board. If the result in step S404 is NO, the process proceeds to step S405, where the stage center position is calculated based on the known uneven shape of the stage surface. If the result in step S404 is YES, the process proceeds to step S406, where the dedicated chart is rotated and measured from multiple directions to calculate the stage center position. In step S407, the stage center position is stored in the memory device 240 or the like.

[0192] Figure 28 is a flowchart illustrating an example of the processing of the alignment function when a virtual object is rotated and moved. Steps S500, S501, and S502 are the same as steps S100, S101, and S102 in Figure 24, respectively. Also, steps S505 to S512 are the same as steps S104 to S111 in Figure 24, respectively.

[0193] In step S503, the virtual object is rotated and moved by mouse dragging. Figure 28 is a flowchart showing an example of the process when the virtual object is rotated and moved. In step S600, the controller 200 determines whether the direction of movement of the mouse on the operation unit 250 is close to horizontal. If the direction of movement of the mouse is not close to horizontal, the process proceeds to step S601. On the other hand, if the direction of movement of the mouse is close to horizontal, the process proceeds to step S602, where the controller 200 fixes the rotation axis of the virtual object with an axis perpendicular to the virtual ground. In step S603, the controller 200 extracts the horizontal component of the mouse movement. In step S604, the controller 200 calculates the orientation rotated according to the extracted horizontal component around the fixed axis. In step S605, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S606, the display of the virtual object is updated. In step S607, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S603.

[0194] In step S601, the controller 200 determines whether the mouse movement direction of the operation unit 250 is close to vertical. If the mouse movement direction is close to vertical, the process proceeds to step S608, where the controller 200 fixes the rotation axis of the virtual object on the left-right axis in the line of sight direction. In step S609, the controller 200 extracts the vertical component of the mouse movement. In step S610, the controller 200 calculates the orientation rotated according to the extracted vertical component around the fixed axis and proceeds to step S611. In step S611, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S612, the display of the virtual object is updated. In step S613, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S609.

[0195] If the mouse movement direction is not nearly vertical, the process proceeds to step S614, where the controller 200 extracts the mouse movement amount. In step S615, the controller 200 calculates the orientation rotated in an arbitrary direction according to the mouse movement amount, and proceeds to step S616. In step S616, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S617, the display of the virtual object is updated. In step S618, the controller 200 determines whether the mouse button has been released or not. If the mouse button has not been released, the process returns to step S614.

[0196] In step S513 of Figure 28, the controller 200 calculates the distance between the virtual object and the virtual inclined platform. In step S514, the faces of the virtual object that are closest to the virtual inclined platform are extracted. In step S515, the degree of contact between the faces of the virtual object and the virtual inclined platform is calculated. In step S516, it is determined whether there are any extracted faces for which the degree of contact has not been calculated. If there are extracted faces for which the degree of contact has not been calculated, the process proceeds to step S517, where the next face is selected and the process proceeds to step S515. If there are no extracted faces for which the degree of contact has not been calculated, the process proceeds to step S518, where the face with the highest degree of contact is selected. In step S519, the orientation in which the selected face and the virtual inclined platform are in contact is calculated. In step S520, the positional relationship between the virtual object, the virtual ground, and the virtual inclined platform is adjusted.

[0197] Figure 30 is a flowchart showing another example of the process when a virtual object is rotated and moved. Steps S700 to S705 are the same as steps S200 to S205 in Figure 25, respectively. Also, steps S707 to S715 are the same as steps S207 to S215 in Figure 25, respectively. In step S706, the controller 200 adjusts the positional relationship between the virtual object, the virtual ground, and the virtual inclined platform. The virtual inclined platform is a virtual representation of the inclined platform provided on the rotating stage 143. The inclined platform provided on the rotating stage 143 is configured so that, for example, the inclination angle with respect to the horizontal plane can be changed in multiple ways, and by placing the workpiece W on the inclined platform, the workpiece W can be made to be inclined.

[0198] Figure 31 is a flowchart showing an example of the process for adjusting the positional relationship between a virtual object, a virtual ground, and a virtual inclined platform. In step S800, the virtual object, virtual ground, and virtual inclined platform are adjusted. The controller 200 calculates the distance to the platform. In step S801, the controller 200 determines whether the virtual object is embedded in the virtual ground or virtual inclined platform. If the result in step S801 is YES, the process proceeds to step S802, where the virtual object is pushed onto the virtual ground or virtual inclined platform before proceeding to step S803. If the result in step S801 is NO, the process proceeds to step S803. In step S803, the controller 200 determines whether the virtual object is floating above the virtual ground or virtual inclined platform. If the result in step S803 is YES, the process proceeds to step S804, where the virtual object is grounded on the virtual ground or virtual inclined platform.

[0199] As described above, the same collision detection as the rotating stage 143 is performed in the space where the virtual object exists, and physical constraints are introduced such that the virtual object is in contact with the rotating stage 143. As a result, the rotation and translation are limited to the same degree of freedom as the actual workpiece W, making it easier for the user to align the virtual object.

[0200] Furthermore, while the mouse is being dragged on the control unit 250, an unstable posture may occur. However, when the mouse is released, a stable posture is calculated and automatically corrected to a posture in which the degree of contact between the rotation stage 143 and the virtual object is increased. Since the actual posture that the workpiece W can take also exists within its limited degrees of freedom, positioning becomes easier.

[0201] Furthermore, the rotation operation performed by dragging the mouse on the control unit 250 is restricted to rotating only one axis at a time relative to the workpiece: roll, pitch, or yaw. This allows the virtual object on the rotating stage 143 to be rotated while maintaining the ground contact state of the rotating stage 143, making positioning easier.

[0202] Furthermore, when linking the rotation of the rotating stage 143 with the display of the virtual object, one difficulty in alignment is that it can be difficult to grasp the positional relationship in the depth direction when imaging from a fixed camera. In contrast, by linking the rotation of the rotating stage 143 with the display state of the virtual object, it is possible to grasp the positional relationship between the object on the rotating stage 143 and the virtual object when the object is photographed from different angles, and use this to aid in alignment.

[0203] Furthermore, if the rotating stage 143 has a structure that allows it to be tilted, as described above, the tilt information can be linked with the alignment function. For example, by providing a section in the application to input tilt information (tilt angle information) of the rotating stage 143, and having the controller 200 acquire this information, it can be linked to the virtual space, and the virtual object can be positioned considering the tilt angle of the rotating stage 143.

[0204] Furthermore, in the case of a scan where there is no reference model as shown in Figure 6, the process may proceed to step SA5 without calculating the evaluation value and the recommended placement posture in SA4. In that case, in step SA5, the posture calculation unit 293 calculates the recommended placement posture by rotating the partial scan model of the workpiece W, which is placed in a first placement posture acquired by the data acquisition unit 291, by a certain rotation angle around a predetermined axis, such as a rotation axis horizontal to the mounting surface 140. The display control unit 255 can then display the recommended placement posture calculated by the posture calculation unit 293 on the display unit 400. Here, the posture calculation unit 293 may, for example, estimate a rotation axis horizontal to the mounting surface 140 for the workpiece W and calculate the recommended placement posture by rotating it by a certain rotation angle, such as 60 or 90 degrees, around this rotation axis. In step SA6, the user determines whether the posture displayed on the display unit 400 is the desired posture. If it is not the desired posture, the process proceeds to step SA7, where the placement posture of the workpiece W is adjusted using computer graphics. In step SA8, the posture calculation unit 293 can calculate and update evaluation values ​​for the adjusted posture, but it is also possible to skip this step and proceed to step SA6.

[0205] If the desired orientation is determined in step SA6, the process proceeds to step SA9, where the analysis module 290 generates a computer graphics workpiece (model) in the determined placement orientation. In step SA10, the user places the workpiece W in a temporary position on the mounting surface 142. The light receiving unit 120 then captures the workpiece W together with the mounting surface 142, and the live image is acquired by the data acquisition unit 291 and displayed on the display unit 400. While viewing the live image on the display unit 400, the user moves or rotates the actual workpiece W until it overlaps with the computer graphics workpiece.

[0206] In step SA11, the user determines whether the actual workpiece W is placed so as to overlap with the computer graphics workpiece. If the actual workpiece W cannot be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SA5 to adjust the placement orientation. If the actual workpiece W is placed so as to overlap with the computer graphics workpiece, the user proceeds to step SA12, where the measurement unit 100 scans the workpiece W in the second placement orientation. In step SA12, the light-emitting unit 110 of the measurement unit 100 irradiates the workpiece W in the second placement orientation with measurement light. The light-receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The received signal output from the light-receiving unit 120 is received by the point cloud acquisition unit 263a to generate second point cloud data of the workpiece W. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts it into second mesh data. In step SA12, the second mesh data obtained by the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0207] In step SA14, the alignment unit 290A of the analysis module 290 aligns the first three-dimensional data (three-dimensional data of the workpiece positioned in the first position) acquired by the data acquisition unit 291 and the second three-dimensional data based on the relative positional relationship calculated by the calculation unit 294. During this alignment, the overlapping region extracted by the extraction unit 290B of the analysis module 290 is used.

[0208] Furthermore, in the example of scanning when there is a reference model such as CAD data or measured 3D data of the workpiece W shown in Figure 17, the process may proceed to step SG4 without the attitude calculation unit 293 calculating the evaluation value and the recommended placement attitude in step SG3. In that case, in step SG4, the display control unit 255 displays the CAD data or measured 3D data on the display unit 400. The attitude of the CAD data or measured 3D data at this time may be the attitude of the CAD data or measured 3D data relative to the scanner, for example, by matching the coordinate system of the CAD data or measured 3D data with the scanner coordinate system. In step SG4, the user can adjust the attitude of the CAD data or measured 3D data based on this attitude. In step SG5, the user determines whether the candidate attitude displayed on the display unit 400 is the desired attitude. If it is not the desired attitude, the process proceeds to step SG6, where the placement attitude of the workpiece W is adjusted on the computer graphics. In step SG7, the attitude calculation unit 293 calculates and updates the evaluation value for the adjusted attitude, but this can be skipped and the process proceeds to step SG5.

[0209] If the desired orientation is determined in step SG5, the process proceeds to step SG8, ​​where the analysis module 290 generates a computer graphics workpiece (model) in the determined placement orientation. The display control unit 255 displays the computer graphics workpiece generated by the analysis module 290 on the display unit 400, superimposed on the live image acquired by the data acquisition unit 291. In step SG9, the user places the workpiece W in a temporary position on the mounting surface 142. The light receiving unit 120 then captures the workpiece W together with the mounting surface 142, and the live image is acquired by the data acquisition unit 291 and displayed on the display unit 400. While viewing the live image on the display unit 400, the user moves or rotates the actual workpiece W until it overlaps with the computer graphics workpiece.

[0210] In step SG10, the user determines whether the actual workpiece W is placed so as to overlap with the computer graphics workpiece. If the actual workpiece W cannot be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG4 to adjust the placement orientation. If the actual workpiece W is placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG11, where the measurement unit 100 scans the workpiece W in the first placement orientation. In step SG12, the first three-dimensional data is acquired. In step SG13, the alignment unit 290A aligns the first three-dimensional data with the reference model, which is either CAD data or measured data. In step SG14, the analysis module 290 acquires the aligned CAD model or measured three-dimensional data.

[0211] Step SG15 proposes a candidate posture and proceeds to Step SG16. For example, regarding the candidate posture, the posture of the CAD data or measured 3D data may be modified according to predetermined rules relative to the desired posture determined in Step SG5. One example of a predetermined rule is to rotate the CAD data or measured 3D data by 60 degrees relative to the desired posture determined in Step SG5, using one axis of the scanner coordinate system as the axis of rotation. Alternatively, the user may select a rotation angle range, for example, within 90 degrees. In Step SG16, the user selects a candidate posture, and then proceeds to Step SG17. If the posture is not the desired one, adjustments are made in Step SG18, the evaluation value is updated in Step SG19, and then proceeds to Step SG17. Step SG19 may be skipped.

[0212] If the desired posture is achieved, the process proceeds to step SG20, where the display control unit 255 displays the computer graphics work generated by the analysis module 290 on the display unit 400, superimposed on the live image acquired by the data acquisition unit 291. Here, the computer graphics work may be the first three-dimensional data or reference model acquired in step SG12. Alternatively, the first three-dimensional data or reference model may be displayed simultaneously on the display unit 400, or they may be displayed separately.

[0213] In step SG21, the user moves or rotates the actual workpiece W while viewing the live image on the display unit 400 until the actual workpiece W overlaps with the computer graphics workpiece.

[0214] In step SG22, the user determines whether the actual workpiece W is placed so as to overlap with the computer graphics workpiece. If the actual workpiece W cannot be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG16 and selects a different placement orientation. If the actual workpiece W can be placed so as to overlap with the computer graphics workpiece, the user proceeds to step SG23 and scans the workpiece W in that placement orientation (second placement orientation) using the measurement unit 100. In step SG24, the alignment unit 290A aligns the acquired second three-dimensional data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs the alignment process of the three-dimensional model, and the synthesis unit 290G performs the synthesis process. In step SG27, the user determines whether there are any areas that need to be scanned. If there are areas that need to be scanned, the user proceeds to step SG12. If there are no areas that need to be scanned, the user acquires a full scan model in step SG28. The full-surface scan model acquired in step SG28 may be output together with the CAD model or measured three-dimensional data that has undergone alignment processing by the alignment unit 290A in step SG26. Furthermore, the full-surface scan model and the CAD model or measured three-dimensional data may be displayed on the display unit 400 in an aligned state.

[0215] In this embodiment, CAD data or measured 3D data was used as a reference model to acquire composite 3D data of the workpiece W. However, it is also possible to acquire composite 3D data of the workpiece W without a reference model and then align it with the CAD data or measured 3D data. The reading unit 292 reads the 3D data and CAD data of the workpiece W from the storage unit 240. Next, the resolution conversion unit 290E reads the 3D data of the workpiece W from the reading unit 292 and converts it into 3D data of a first resolution lower than the resolution acquired by the data acquisition unit 291. Next, the shape feature extraction unit 290C extracts subregions (keypoints) used for calculating shape features from the 3D data and CAD data of the first resolution. Next, the analysis module 290 samples the area around the subregions, and the shape feature extraction unit 290C calculates shape feature vectors from the vicinity of the points sampled by the analysis module 290C. Next, the analysis module 290C compares the shape feature vectors of the three-dimensional data obtained by the shape feature extraction unit 290C with the shape feature vectors of the CAD data, and the alignment unit 290A aligns the three-dimensional data and the CAD data based on the comparison of the shape feature vectors. By comparing the shape feature vectors, candidate pair shape feature vectors can be extracted, and using the points of the candidate pairs, it becomes possible to determine the positional relationship (rotation and translation) between the point clouds using RANSAC or deep learning-based methods.

[0216] Furthermore, after the alignment of the composite 3D data and the CAD data is complete, the analysis module 290 can also automatically perform a comparison between the CAD data and the 3D data. Figure 32 shows an example of an analysis in which the scan data of the workpiece W is compared with the 3D shape of the CAD data, the difference in dimensions between the scan data and the CAD data is calculated, and a color map is displayed with colors assigned according to the degree of that difference. Based on the alignment results from the alignment unit 290A and the CAD data read out by the readout unit 292, the analysis module 290 calculates the difference in shape between the 3D data of the workpiece W obtained by the data acquisition unit 291 and the CAD data for each mesh, and assigns color information to each mesh according to the degree of that difference. Then, the display control unit 255 displays a color map on the display unit 400 with colors assigned to each mesh based on the color information for at least one of the 3D data and the CAD data. Furthermore, after alignment by the alignment unit 290A, the analysis module 290 may compare the three-dimensional data with the CAD data based on the reception unit 298 receiving an instruction to assign analysis settings or start comparative analysis, and the display control unit 255 may display a color map on the display unit 400. In this case, the user can assign detailed settings for the comparison.

[0217] Figure 33A shows an example of cross-sectional measurement performed on the three-dimensional data of workpiece W. The user specifies the surface to be measured in the three-dimensional data of workpiece W and instructs the type of analysis tool to be performed on that cross-section and its assigned position. The analysis tool refers to measurement content such as the distance between two points or the angle between two surfaces. The analysis module 290 receives these instructions and performs the analysis on the specified surface of the three-dimensional data of workpiece W based on the instructions.

[0218] Figure 33B shows an example of cross-sectional measurement performed on CAD data. The user specifies the surface to be measured in the CAD data and instructs the type of analysis tool to be performed on that cross-section and its assignment location. The analysis module 290 receives these instructions and performs the analysis on the specified surface of the CAD data based on the instructions. The analysis module 290 can compare the results of cross-sectional measurement performed on three-dimensional data with the results of cross-sectional measurement performed on CAD data, and the display control unit 255 displays the comparison results on the display unit 400.

[0219] Furthermore, cross-sectional measurements can be performed on data in which the three-dimensional data and CAD data of the workpiece W have been aligned. The data in which the three-dimensional data and CAD data of the workpiece W have been aligned is obtained by the alignment unit 290A aligns the three-dimensional data and CAD data of the workpiece W. The user specifies the surface to be measured in cross-section on the data in which the workpiece W and CAD data have been aligned, and instructs the type of analysis tool to be performed on that cross-section and the assigned position. The analysis tool is, for example, measurement content such as the distance between two points or the angle between two surfaces. The analysis module 290 receives this instruction and performs the analysis based on the instruction on the specified surface of the data in which the three-dimensional data and CAD data of the workpiece W have been aligned. The analysis module 290 can perform cross-sectional measurements on the data in which the three-dimensional data and CAD data have been aligned. For example, a comparison may be made based on the difference in dimensions between the CAD data and the scanned data. The display control unit 255 may display the comparison results on the display unit 400.

[0220] The embodiments described above are merely illustrative in all respects and should not be interpreted restrictively. Furthermore, any modifications or changes that fall within the equivalent scope of the claims are all within the scope of the present invention. [Industrial applicability]

[0221] As described above, the present invention can be used when generating three-dimensional data of various workpieces. [Explanation of symbols]

[0222] 1. Three-dimensional scanner 291 Data Acquisition Unit 292 Reading section 293 Posture calculation section 294 Arithmetic section 295 Overlapping region estimation unit 296 Additional Data Estimation Unit 297 Specific part 298 Reception Department 299 Evaluation Department 290A Alignment section 290B Extraction part 290C Shape Feature Extraction Unit (First Extraction Unit) 290D Image Feature Extraction Unit (Second Extraction Unit) 290E Resolution Conversion Unit 290F Coordinate Transformation Unit 290G synthesis section 290H Editorial Department

Claims

1. A three-dimensional scanner that generates three-dimensional data of workpieces placed in different orientations and then synthesizes this data to generate composite three-dimensional data of the workpiece, A data acquisition unit acquires: first three-dimensional data, which is three-dimensional data including shape information and texture information of a workpiece placed in a first placement orientation; and second three-dimensional data, which is three-dimensional data including shape information and texture information of a workpiece placed in a second placement orientation. A first extraction unit includes an input layer of a neural network that receives input of the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit, a plurality of intermediate layers that extract shape features based on the input received by the input layer, and an output layer that outputs the shape features extracted by the intermediate layers. A second extraction unit extracts image features from the texture information contained in the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit, An alignment unit that aligns the first three-dimensional data and the second three-dimensional data based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit, A three-dimensional scanner comprising a synthesis unit that synthesizes the first three-dimensional data, which has been aligned by the alignment unit, and the second three-dimensional data, to generate the synthesized three-dimensional data.

2. In the three-dimensional scanner according to claim 1, The system further includes a resolution conversion unit that converts the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit into three-dimensional data with a first resolution lower than the resolution acquired by the data acquisition unit. The first extraction unit is a three-dimensional scanner that extracts shape features from shape information contained in the first three-dimensional data and the second three-dimensional data of the first resolution converted by the resolution conversion unit.

3. In the three-dimensional scanner according to claim 2, The resolution conversion unit converts the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit into three-dimensional data with a second resolution that is lower than the resolution acquired by the data acquisition unit and higher than the first resolution. The second extraction unit is a three-dimensional scanner that extracts image features from texture information contained in the first three-dimensional data and the second three-dimensional data of the second resolution converted by the resolution conversion unit.

4. In the three-dimensional scanner according to claim 2, The first extraction unit is a three-dimensional scanner that extracts subregions used for calculating shape features from the first three-dimensional data and the second three-dimensional data of the first resolution, and extracts shape features for each extracted subregion.

5. In the three-dimensional scanner according to claim 4, The second extraction unit identifies regions corresponding to each sub-region extracted by the first extraction unit from the first three-dimensional data and the second three-dimensional data before conversion by the resolution conversion unit, and extracts image features from the identified corresponding regions, thereby forming a three-dimensional scanner.

6. In the three-dimensional scanner according to claim 1, The alignment unit performs global alignment based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit, and performs a first alignment process that calculates low-precision alignment parameters indicating the relative position and orientation of the second three-dimensional data with respect to the first three-dimensional data, as a three-dimensional scanner.

7. In the three-dimensional scanner according to claim 6, The alignment unit is a three-dimensional scanner that performs local alignment based on the low-precision alignment parameters calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data, and performs a second alignment process that calculates high-precision alignment parameters indicating the relative position and orientation of the second three-dimensional data with respect to the first three-dimensional data.

8. In the three-dimensional scanner according to claim 7, A three-dimensional scanner further comprising a coordinate transformation unit that performs coordinate transformation of the first three-dimensional data based on low-precision alignment parameters calculated by the first alignment process.

9. In the three-dimensional scanner according to claim 8, The alignment unit is a three-dimensional scanner that, in the second alignment process, performs local alignment based on the first three-dimensional data and the second three-dimensional data, which have undergone coordinate transformation by the coordinate transformation unit.

10. In the three-dimensional scanner according to claim 1, The alignment unit is a three-dimensional scanner that identifies corresponding points in the second three-dimensional data for points included in the first three-dimensional data, and adjusts the positional orientation of the first three-dimensional data and the second three-dimensional data based on the identified corresponding points.

11. In the three-dimensional scanner according to claim 1, A light-emitting unit that irradiates the workpiece with measurement light and uniform light at different timings, The system further includes a light receiving unit that receives measurement light emitted by the light emitting unit and reflected by the workpiece, and outputs a first light receiving signal for measurement, and a light receiving unit that receives uniform light emitted by the light emitting unit and reflected by the workpiece, and outputs a second light receiving signal for texture acquisition. The data acquisition unit is a three-dimensional scanner that acquires three-dimensional data, including shape information and texture information of a workpiece, as the first three-dimensional data and the second three-dimensional data, based on the first and second light-receiving signals output by the light-receiving unit.

12. In the three-dimensional scanner according to claim 1, A first light-emitting unit that irradiates a measurement pattern light onto the workpiece, A second light-emitting unit that illuminates the workpiece with illumination light, The system further includes a light receiving unit that receives pattern light emitted by the first light-emitting unit and reflected by the workpiece, and outputs a first light-receiving signal, and a light-receiving unit that receives illumination light emitted by the second light-emitting unit and reflected by the workpiece, and outputs a second light-receiving signal. The data acquisition unit is a three-dimensional scanner that acquires three-dimensional data, including shape information and texture information of a workpiece, as the first three-dimensional data and the second three-dimensional data, based on the first and second light-receiving signals output by the light-receiving unit.

13. A three-dimensional measurement method that generates three-dimensional data of workpieces positioned in different orientations and then synthesizes this data to generate composite three-dimensional data of the workpiece, The system obtains a first three-dimensional data set containing shape information and texture information of a workpiece positioned in a first position, and a second three-dimensional data set containing shape information and texture information of a workpiece positioned in a second position. The acquired first three-dimensional data and the second three-dimensional data are input to the input layer of the neural network. Based on the first three-dimensional data and the second three-dimensional data input to the input layer of the neural network, shape features are extracted in a plurality of intermediate layers of the neural network. The shape features extracted in the intermediate layer of the neural network are output from the output layer of the neural network. Image features are extracted from the texture information contained in the first three-dimensional data and the second three-dimensional data, Based on the extracted shape features and image features, the first three-dimensional data and the second three-dimensional data are aligned. A three-dimensional measurement method that generates the combined three-dimensional data by combining the first three-dimensional data, which has been aligned, and the second three-dimensional data.

14. A storage medium containing a three-dimensional measurement program that causes a computer to execute a three-dimensional measurement method that generates three-dimensional data of workpieces positioned in different orientations and then synthesizes this data to generate composite three-dimensional data of the workpiece. The aforementioned three-dimensional measurement method is The system obtains a first three-dimensional data set containing shape information and texture information of a workpiece positioned in a first position, and a second three-dimensional data set containing shape information and texture information of a workpiece positioned in a second position. The acquired first three-dimensional data and the second three-dimensional data are input to the input layer of the neural network. Based on the first three-dimensional data and the second three-dimensional data input to the input layer of the neural network, shape features are extracted in a plurality of intermediate layers of the neural network. The shape features extracted in the intermediate layer of the neural network are output from the output layer of the neural network. Image features are extracted from the texture information contained in the first three-dimensional data and the second three-dimensional data, Based on the extracted shape features and image features, the first three-dimensional data and the second three-dimensional data are aligned. This method involves combining the first three-dimensional data, which has been aligned, with the second three-dimensional data to generate the combined three-dimensional data. A storage medium containing a three-dimensional measurement program that causes the computer to execute the aforementioned three-dimensional measurement method.

Images