Three-dimensional scanner, three-dimensional measurement method, and storage medium storing three-dimensional measurement program

Abstract

The invention provides a three-dimensional scanner, a three-dimensional measurement method, and a storage medium storing a three-dimensional measurement program. When alignment among a plurality of pieces of three-dimensional data is performed, high-precision alignment is performed. A three-dimensional scanner includes: a shape feature extraction unit including an input layer, a plurality of intermediate layers, and an output layer of a neural network, in which the input layer accepts input of first three-dimensional data and second three-dimensional data, the plurality of intermediate layers extract shape features, and the output layer outputs the shape features; an image feature extraction unit that extracts image features from texture information included in each of the first three-dimensional data and the second three-dimensional data; an alignment unit that performs alignment between the first three-dimensional data and the second three-dimensional data based on the shape feature extracted by the shape feature extraction unit and the image feature extracted by the image feature extraction unit; and a combining unit that combines the first three-dimensional data and the second three-dimensional data aligned by the aligning unit.

Description

Technical Field

[0001] This disclosure relates to a 3D scanner, a 3D measurement method, and a storage medium for storing 3D measurement programs. Background Technology

[0002] For example, JP2024-051797A discloses a 3D scanner for scanning workpieces to generate 3D data.

[0003] The 3D scanner in JP2024-051797A combines multiple 3D data acquired in different orders to generate combined 3D data of a workpiece.

[0004] As disclosed in JP2024-051797A, when combining multiple 3D data acquired in different orders, the multiple 3D data are combined after alignment. However, there are cases where alignment cannot be performed solely by aligning multiple 3D data based on rules.

[0005] That is, there exist methods for performing alignment between multiple 3D data sets through pattern matching, and as pattern matching methods, there are methods that primarily use 3D data (point clouds) and methods that use images acquired by a camera from a 3D scanner. When using 3D data, the workpiece requires shape features, and when using image features, image features are required.

[0006] Industrial products typically possess shape characteristics, while products obtained by simply processing metals, resins, etc., often have fewer image features. Therefore, in existing technologies, alignment is performed using three-dimensional data.

[0007] Additionally, when performing alignment using image features from industrial products, there is a method for pasting markers onto the workpiece to perform scanning, detecting the image coordinates of the markers through image processing, and calculating the relative positional relationships of the cameras (including transformation matrices of rotation and translation vectors) using epipolar geometry based on the correspondence between marker coordinates across multiple images. By employing this method, high-precision alignment can be performed even without initial estimates of the positional relationships.

[0008] On the other hand, shape analysis of point clouds is technically more difficult than that of images. In particular, it is difficult to align multiple 3D data without initial estimates.

[0009] On the other hand, in recent years, deep learning has been able to highly distinguish the shape features of point clouds, and there are more cases where alignment can be achieved without initial estimates. However, deep learning is computationally intensive, and the number of point clouds that can be aligned in real-world computing is limited to, for example, point clouds containing tens of thousands of points. Therefore, the accuracy of the obtained positional relationships is only at the level of a few degrees and millimeters, and it is difficult to obtain high accuracy of, for example, around tens of micrometers, although the accuracy of the initial estimates can be obtained. Summary of the Invention

[0010] This disclosure has been made in view of this, and the purpose of this disclosure is to achieve high-precision alignment when performing alignment between multiple sets of 3D data.

[0011] To achieve the above objectives, according to one embodiment of this disclosure, a three-dimensional scanner can be used as a prerequisite: the three-dimensional scanner generates combined three-dimensional data of the workpiece by generating three-dimensional data of workpieces placed in different orientations and combining multiple three-dimensional data. The 3D scanner includes: a data acquisition unit that acquires first 3D data and second 3D data, wherein the first 3D data is 3D data including shape and texture information of a workpiece placed in a first placement posture, and the second 3D data is 3D data including shape and texture information of a workpiece placed in a second placement posture; a first extraction unit that includes an input layer, multiple intermediate layers, and an output layer of a neural network, wherein the input layer of the neural network receives input from the first 3D data and the second 3D data acquired by the data acquisition unit, the multiple intermediate layers extract shape features based on the input received by the input layer, and the output layer outputs the shape features extracted in the intermediate layers; a second extraction unit that extracts image features from the texture information included in each of the first 3D data and the second 3D data acquired by the data acquisition unit; an alignment unit that performs alignment between the first 3D data and the second 3D data based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit; and a combination unit that combines the first 3D data and the second 3D data aligned by the alignment unit to generate combined 3D data.

[0012] According to this configuration, the shape features of the first and second 3D data are extracted by a neural network, and the image features are extracted from the texture information included in each of the first and second 3D data. Since the alignment between the first and second 3D data is performed based on both the extracted shape features and image features, the accuracy of the alignment is improved.

[0013] According to another embodiment of this disclosure, a three-dimensional measurement method can be used as a basis for generating combined three-dimensional data of a workpiece by generating multiple three-dimensional data of a workpiece placed in different placement postures and combining the multiple three-dimensional data. The three-dimensional measurement method may include: acquiring first three-dimensional data and second three-dimensional data, wherein the first three-dimensional data is three-dimensional data including shape and texture information of a workpiece placed in a first placement posture, and the second three-dimensional data is three-dimensional data including shape and texture information of a workpiece placed in a second placement posture; inputting the acquired first and second three-dimensional data into the input layer of a neural network; extracting shape features from multiple intermediate layers of the neural network based on the first and second three-dimensional data input to the input layer of the neural network; outputting the shape features extracted from the intermediate layers of the neural network from the output layer of the neural network; extracting image features from the texture information included in each of the first and second three-dimensional data; performing alignment between the first and second three-dimensional data based on the extracted shape features and image features; and combining the aligned first and second three-dimensional data to generate combined three-dimensional data.

[0014] According to another embodiment of this disclosure, it may be a storage medium storing a three-dimensional measurement program for causing a computer to execute a three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating multiple three-dimensional data of a workpiece placed in different orientations and combining the multiple three-dimensional data.

[0015] Furthermore, according to another embodiment of this disclosure, a 3D scanner can be used as a premise for performing alignment between first point cloud data and second point cloud data obtained by measuring a workpiece using a 3D scanner. The 3D scanner may include a resolution conversion unit that generates third point cloud data obtained by reducing the resolution of the first point cloud data, and generates fourth point cloud data obtained by reducing the resolution of the second point cloud data; a first extraction unit that extracts shape features from each of the third and fourth point cloud data generated by the resolution conversion unit; a first alignment unit that performs global alignment based on the shape features extracted by the first extraction unit and calculates low-precision alignment parameters indicating the relative position and orientation of the second point cloud data relative to the first point cloud data; a second alignment unit that performs global alignment based on the low-precision alignment parameters calculated by the first alignment unit, the first point cloud data, and the second point cloud data, and calculates high-precision alignment parameters indicating the relative position and orientation of the second point cloud data relative to the first point cloud data; and a combination unit that generates a combined point cloud obtained by combining the first and second point cloud data based on the high-precision alignment parameters calculated by the second alignment unit.

[0016] The resolution conversion unit can generate a fifth point cloud data, in which the resolution of the first point cloud data is reduced to a resolution higher than that of the third point cloud data, and can generate a sixth point cloud data, in which the resolution of the second point cloud data is reduced to a resolution higher than that of the fourth point cloud data.

[0017] In this configuration, the 3D scanner can extract image features from each of the fifth and sixth point cloud data generated by the resolution conversion unit. The first alignment unit can perform global alignment based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit, and calculate low-precision alignment parameters indicating the relative position and pose of the second point cloud data relative to the first point cloud data. For example, the first extraction unit may include a deep learning module, etc.

[0018] Furthermore, the 3D scanner may include a coordinate transformation unit that performs coordinate transformation of the first point cloud data based on low-precision alignment parameters calculated by the first alignment unit. In this case, the second alignment unit may perform local alignment based on the first and second point cloud data to which the coordinate transformation unit has performed coordinate transformation.

[0019] As mentioned above, high-precision alignment can be performed when aligning multiple 3D data sets. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the overall configuration of a three-dimensional scanner according to an embodiment of the present invention;

[0021] Figure 2 This is a block diagram of a 3D scanner;

[0022] Figure 3 This is a side view of the measuring unit and the base;

[0023] Figure 4 This is a block diagram of the measurement unit;

[0024] Figure 5 This is a diagram showing an example of module configuration;

[0025] Figure 6 This is a flowchart illustrating an example of scan processing without CAD data for the workpiece;

[0026] Figure 7 This is an example diagram showing the user interface screen displayed at the start of a measurement;

[0027] Figure 8 This is an example diagram showing the user interface screen displayed when pose candidates are presented;

[0028] Figure 9This is a diagram showing an example of the user interface screen displayed when a selection of a pose candidate is accepted;

[0029] Figure 10 This is an example of a user interface screen showing a model with the determined placement posture superimposed and displayed on a real-time image;

[0030] Figure 11 This is a flowchart illustrating an example of alignment processing;

[0031] Figure 12 This is a schematic diagram related to the removal of hidden surfaces;

[0032] Figure 13 This is a flowchart illustrating an example of the process for extracting shape features and image features;

[0033] Figure 14 This is a flowchart illustrating an example of the extraction process when a vector with added image features is input into a shape feature extraction unit;

[0034] Figure 15 This is a flowchart illustrating an example of the process for calculating low-precision alignment parameters;

[0035] Figure 16 This is a flowchart illustrating an example of the process for calculating high-precision alignment parameters;

[0036] Figure 17 This is a flowchart illustrating an example of scan processing in the presence of CAD data for a workpiece;

[0037] Figure 18 This is a diagram showing a user interface screen for displaying first three-dimensional data, second three-dimensional data, and combined three-dimensional data;

[0038] Figure 19 This is a diagram illustrating the workflow for generating combined 3D data;

[0039] Figure 20 This is a screenshot showing the confirmation screen of the combined 3D data;

[0040] Figure 21 This is a diagram showing the confirmation screen for the second and third-dimensional data;

[0041] Figure 22 This is a screenshot showing the confirmation of the first three-dimensional data;

[0042] Figure 23 This is a diagram showing the user interface screen used for data editing;

[0043] Figure 24 This is a flowchart illustrating an example of the alignment function's processing;

[0044] Figure 25 This is a flowchart illustrating an example of rotating a virtual object by dragging it with the mouse;

[0045] Figure 26 This is a flowchart illustrating an example of the process for adjusting the positional relationship between a virtual object and a virtual ground surface;

[0046] Figure 27 This is a flowchart illustrating an example of stage surface inspection processing;

[0047] Figure 28 This is a flowchart illustrating an example of the alignment function processing when performing rotation and movement of virtual objects;

[0048] Figure 29 This is a flowchart illustrating an example of the processing when performing rotation and movement of a virtual object;

[0049] Figure 30 This is a flowchart illustrating another example of the processing in the case of performing rotation and movement of virtual objects;

[0050] Figure 31 This is a flowchart illustrating an example of the process when adjusting the positional relationship between virtual objects, virtual ground, and virtual tilting platforms;

[0051] Figure 32 It is a color map based on the dimensional differences between the workpiece's scanned data and CAD data;

[0052] Figure 33A This is a diagram illustrating an example of cross-sectional measurement based on scanned data of a workpiece; and

[0053] Figure 33B This is a diagram illustrating an example of performing a cross-sectional measurement on the CAD data of a workpiece. Detailed Implementation

[0054] In the following, embodiments of the invention will be described in detail with reference to the accompanying drawings. Note that the following description of preferred embodiments is merely exemplary in nature and is not intended to limit the invention, its uses, or its applications.

[0055] Figure 1 This is a diagram illustrating the overall configuration of a 3D scanner 1 according to an embodiment of the present invention. The 3D scanner 1 is a device capable of acquiring 3D data by measuring the shape of a workpiece (measurement object) W, converting the 3D data into mesh data of the workpiece W, and outputting the mesh data. The 3D scanner 1 can also convert the mesh data of the workpiece W into CAD data and output CAD data, or convert the mesh data into surface data and output surface data.

[0056] In the following description, when measuring the shape of a workpiece W, during the process of acquiring the coordinate information of the front surface of the workpiece W, the workpiece W is illuminated with a measurement light of a predetermined pattern, and the coordinate information is acquired by using the signal obtained from the reflected light reflected from the front surface of the workpiece W. For example, a measurement method utilizing triangulation can be used, which uses a striped projection image obtained from the reflected light by projecting measurement light onto the workpiece W using a measurement light of a predetermined pattern with structured illumination. However, in this invention, the principle and configuration for acquiring the coordinate information of the workpiece W are not limited thereto, and other methods may be applied.

[0057] The 3D scanner 1 includes a measurement unit 100 for measuring the shape of a workpiece W, a base 600 for mounting the workpiece W, a controller 200, a light source unit 300, and a display unit 400. The controller 200, the light source unit 300, or the display unit 400 can be integrated into the measurement unit 100. Alternatively, the controller 200 and the light source unit 300 can be integrated, or the controller 200 and the display unit 400 can be integrated.

[0058] The 3D scanner 1 performs structured illumination on the workpiece W through the light source unit 300, captures a fringe projection image to generate a depth image with coordinate information, and can measure the 3D dimensions and shape of the workpiece W based on the depth image. Measurement using this fringe projection method has the advantage of reducing measurement time because 3D measurement can be performed without moving the workpiece W or optical systems such as lenses in the Z-direction (height direction).

[0059] Figure 2 A block diagram of a three-dimensional scanner 1 according to an embodiment of the present invention is shown. As shown in the figure, the measurement unit 100 includes a pattern light projection unit (first light projection unit) 110 that projects pattern light for measurement onto a workpiece W, a light receiving unit 120, a measurement control unit 150, and an illumination light output unit 130. The light projection unit 110 illuminates the components of the workpiece W mounted on a mounting unit 140, which will be described later, with measurement light of a predetermined pattern. The mounting of the workpiece W on the mounting unit 140 is the same as the placement of the workpiece W on the mounting unit 140.

[0060] The light receiving unit 120 is fixed in an inclined orientation relative to the mounting surface 142 of the rotating stage 143 described later. The light receiving unit 120 receives measurement light emitted by the light projection unit 110 and reflected by the workpiece W. When the measurement light, as reflected light from the workpiece W, is received, the light receiving unit 120 generates and outputs a first light receiving signal indicating the amount of measurement light received for measurement. The light receiving unit 120 can generate an observation image of the entire shape of the workpiece W by capturing the workpiece W mounted on the mounting unit 140. In this example, an illumination light output unit 130 is provided, but the workpiece W can be illuminated with uniform light from the light projection unit 110. In this case, the light projection unit 110 is a component that illuminates the workpiece W with measurement light and uniform light at different timings. The light receiving unit 120 can also receive uniform light emitted from the light projection unit 110 and output a second light receiving signal for texture acquisition. For example, uniform light with the same wavelength as the measurement light can be emitted from the measurement light source, and a light receiving signal including uniaxial color information can be output. Note that, although not shown, a calibrated first and second camera can also be prepared, with the first camera acquiring the shape and the second camera acquiring the texture information. The texture information includes the color and brightness information of the workpiece W.

[0061] The light receiving unit 120 according to this embodiment includes a high-magnification light receiving unit and a low-magnification light receiving unit. The high-magnification light receiving unit is a component that can capture the workpiece W in a magnified manner compared to the low-magnification light receiving unit. On the other hand, the low-magnification light receiving unit is a light receiving unit that has a wider field of view than the high-magnification light receiving unit.

[0062] The base 600 includes a base plate 602, a mounting unit 140, and a movement control unit (stage control unit) 144. The mounting unit 140 is supported on the base plate 602 of the base 600. The movement control unit 144 is a component that controls the movement and rotation of the rotating stage 143 on which the workpiece W is mounted. In addition to being located on the base 600 side, the movement control unit 144 can also be located on the controller 200 side.

[0063] The light source unit 300 is connected to the measurement unit 100. The light source unit 300 is a component that generates measurement light and provides the measurement light to the measurement unit 100. The controller 200 is a component that controls the measurement unit 100, etc. The display unit 400 is connected to the controller 200 and is configured to display the image generated by the measurement unit 100 and perform necessary settings, inputs, selections, etc.

[0064] The mounting unit 140 includes a rotating stage 143 with a top surface on which a mounting surface 142 is formed for mounting the workpiece W. Figure 4As shown, two directions orthogonal to each other on the mounting surface 142 of the rotating stage 143 are defined as the X direction and the Y direction, and are indicated by arrows X and Y, respectively. The direction orthogonal to the mounting surface 142 of the mounting unit 140 is defined as the Z direction, and is indicated by arrow Z. The direction of rotation about an axis parallel to the Z direction is defined as the θ direction, and is indicated by arrow θ.

[0065] The mounting unit 140 includes a rotary stage 143 for rotating the mounting surface 142 about an axis extending in the Z direction, and a translational stage 141 for moving the mounting surface 142 in horizontal directions (X and Y directions). The translational stage 141 includes X-direction movement mechanisms and Y-direction movement mechanisms. The rotary stage 143 has a θ-direction rotation mechanism. The mounting unit 140 may include a workpiece holding member (clamp, etc.) for holding the workpiece W on the mounting surface 142. Furthermore, the mounting unit 140 may include a tilting platform having a mechanism capable of rotating about an axis parallel to the mounting surface 142.

[0066] The motion control unit 144 controls the rotation of the rotating stage 143 and the translation of the translation stage 141 according to the measurement conditions set by the measurement condition setting unit 261 (described later). Furthermore, the motion control unit 144 controls the movement of the mounting unit 140 by installing a motion unit based on the measurement area set by the measurement condition setting unit 261 (described later).

[0067] The controller 200 includes a central processing unit (CPU) 210, a read-only memory (ROM) 220, a working memory 230, a storage device (storage unit) 240, an operation unit 250, etc. For example, a personal computer (PC) can be used as the controller 200.

[0068] exist Figure 4 The block diagram illustrates the configuration of the measurement unit 100. The measurement unit 100 includes a light projection unit 110, a light receiving unit 120, an illumination light output unit 130, a measurement control unit 150, and a main housing 101 housing these units. The light projection unit 110 includes a measurement light source 111, a pattern generation unit 112, and multiple lenses 113, 114, and 115. The light receiving unit 120 includes a camera 121 and multiple lenses 122 and 123. When performing measurements at different magnifications by providing multiple light receiving units, a light receiving unit 120a comprising a camera 121 and lenses for low magnification, and a light receiving unit 120b comprising a camera 121 and lenses for high magnification, can be installed. Note that the invention is not limited to this configuration, and the magnification can be varied by switching between multiple lenses of a single camera 121, or by providing a zoom lens to a single camera 121.

[0069] The light projection unit 110 is arranged at an angle above the mounting unit 140. Figure 4 In the example shown, the measurement unit 100 includes two light projection units 110, but the measurement unit 100 may include multiple light projection units 110. Here, a first measurement light projection unit 110A is provided, capable of illuminating the workpiece W from a first direction with a first measurement light ML1. Figure 4 (right side of the image) and a second measuring light projection unit 110B capable of illuminating the workpiece W from a second direction different from the first direction with a second measuring light ML2. Figure 4 (Left side of the image). The first measuring light projection unit 110A and the second measuring light projection unit 110B are arranged symmetrically with respect to the optical axis of the light receiving unit 120. Note that although not shown, three or more light projection units 110 may be included, or the light projection units 110 and the mounting unit 140 may be moved relative to each other to project light onto the workpiece W in different illumination directions while using a common light projection unit 110. Furthermore, in the example above, multiple light projection units 110 are prepared and light is received by a common light receiving unit 120; however, conversely, multiple light receiving units 120 may be prepared for a common light projection unit 110, and light may be received by multiple light receiving units. Additionally, in this example, the illumination angle of the light projected by the light projection unit 110 relative to the Z direction is fixed, but this can be variable.

[0070] Each of the first measurement light projection units 110A and the second measurement light projection unit 110B includes a first measurement light source and a second measurement light source as a measurement light source 111. The measurement light source 111 is, for example, a halogen lamp that emits white light. The measurement light source 111 can also be a light source that emits monochromatic light, such as a blue light-emitting diode (LED) or another light source such as an organic EL that emits blue light. The light emitted from the measurement light source 111 (hereinafter referred to as "measurement light") is appropriately focused by a lens 113 and then incident on the pattern generation unit 112.

[0071] The relative positional relationship between the light receiving unit 120, the light projection units 110A and 110B, and the mounting unit 140 is determined such that the central axes of the light projection units 110A and 110B intersect with the central axis of the light receiving unit 120 at the position where the workpiece W is arranged on the mounting unit 140, where the depth of field of the light projection units 110 and the light receiving unit 120 is appropriate. Furthermore, since the center of the rotation axis in the θ direction coincides with the central axis of the light receiving unit 120, when the mounting unit 140 rotates in the θ direction, the workpiece W does not deviate from the field of view and rotates around the rotation axis within the field of view.

[0072] The pattern generation unit 112 reflects light emitted from the measurement light source 111 to project the measurement light onto the workpiece W. The measurement light incident on the pattern generation unit 112 is converted into a preset pattern and a preset intensity (brightness) and then emitted. The measurement light emitted by the pattern generation unit 112 is converted by multiple lenses 114 and 115 into light with a diameter larger than that of the light receiving unit 120, allowing for an observable and measurable field of view. The converted light is then used to illuminate the workpiece W on the mounting unit 140.

[0073] The pattern generation unit 112 is a component capable of switching between a light projection state where the measuring light is projected onto the workpiece W and a non-light projection state where the measuring light is not projected onto the workpiece W. For example, a digital micromirror device (DMD) or the like can be used for such a pattern generation unit 112. The pattern generation unit 112 using a DMD can be controlled by the measurement control unit 150 to switch between a reflection state and a light shielding state. In the reflection state, the measuring light is reflected along the optical path, resulting in a light projection state. In the light shielding state, the measuring light is shielded, resulting in a non-light projection state.

[0074] Note that in the above example, an example in which a DMD is used for the pattern generation unit 112 has been described; however, the pattern generation unit 112 is not limited to the DMD of this invention, and other components may also be used. For example, a liquid crystal on silicon (LCOS) can be used as the pattern generation unit 112. Alternatively, the amount of transmitted measurement light can be adjusted by using a transmissive component instead of a reflective component. In this case, the pattern generation unit 112 is arranged in the optical path of the measurement light to switch between a light projection state of transmitting the measurement light and a light shielding state of shielding the measurement light. For example, a liquid crystal display (LCD) can be used as the pattern generation unit 112. Alternatively, the pattern generation unit 112 can be formed by a projection method using multiple line LEDs, a projection method using multiple optical paths, an optical scanner method including a laser and a galvanometer mirror, an accordion fringe interferometry (AFI) method using interference fringes generated by superimposing beams separated by a beam splitter, a projection method using an actual grating and a moving mechanism including a piezoelectric stage, a high-resolution encoder, etc.

[0075] The light receiving unit 120 is positioned above the mounting unit 140. The measuring light reflected upward from the mounting unit 140 by the workpiece W is collected and captured by multiple lenses 122 and 123 of the light receiving unit 120, and then received by the camera 121.

[0076] Camera 121 is, for example, a charge-coupled device (CCD) camera including an imaging element 121a. Imaging element 121a is, for example, a monochrome CCD. Imaging element 121a can also be another imaging element, such as a complementary metal-oxide-semiconductor (CMOS) image sensor. In color imaging elements, since each pixel needs to receive light corresponding to red, green, and blue, the measurement resolution is lower than that of monochrome imaging elements. Sensitivity is reduced because a color filter needs to be provided in each pixel. Therefore, in this embodiment, a color image is acquired by using a monochrome CCD as the imaging element and emitting illumination corresponding to each RGB color in a time-division manner through the illumination light output unit 130 (described later), thereby capturing an image. Using this configuration, a color image of the measured object can be acquired without reducing measurement accuracy. The illumination light output unit 130 is an example of a second light projection unit that illuminates the workpiece W with illumination light. The illumination light can be uniform light.

[0077] Note that a color imaging element can be used as imaging element 121a. In this case, although the measurement accuracy and sensitivity are lower than those of a monochrome imaging element, it is not necessary to emit illumination corresponding to each RGB color from the illumination light output unit 130 in a time-division manner, and a color image can be acquired simply by emitting white light, thus allowing for a simple illumination optical system. An analog electrical signal corresponding to the amount of received light (hereinafter referred to as the "light receiving signal") is output from each pixel of the imaging element 121a to the measurement control unit 150.

[0078] An analog-to-digital converter (A / D converter) and a first-in-first-out (FIFO) memory (neither shown) are mounted on the measurement control unit 150. The light received signal output from the camera 121 is sampled at a constant sampling period and converted into a digital signal by the A / D converter of the measurement control unit 150 under the control of the light source unit 300. The digital signal output from the A / D converter is sequentially accumulated in the FIFO memory. The digital signal accumulated in the FIFO memory is sequentially transmitted to the controller 200 as pixel data.

[0079] The operating unit 250 of the controller 200 may include, for example, a keyboard, a pointing device, etc. For example, a mouse, a joystick, etc. are used as pointing devices.

[0080] The ROM 220 of the controller 200 stores system programs, etc. The working memory 230 of the controller 200 includes, for example, random access memory (RAM) and is used to process various types of data. The storage device 240 includes a solid-state drive, hard disk drive, etc. The storage device 240 stores reverse engineering programs. Additionally, the storage device 240 is used to store various types of data, such as pixel data (image data), setting information, and measurement conditions given from the measurement control unit 150. Measurement conditions include, for example, various settings set by the scanning module 260 when measuring the shape of the workpiece W, as will be described later, such as the settings of the light projection unit 110 (pattern frequency or pattern type) and the type of the light receiving unit 120 (low-magnification light receiving unit or high-magnification light receiving unit). Furthermore, the storage device 240 can also store brightness information, coordinate information, and attribute information for each pixel constituting the measurement image.

[0081] CPU 210 is a control circuit or control element that processes given signals or data, performs various arithmetic operations, and outputs the results of arithmetic operations. In this specification, CPU refers to an element or circuit that performs arithmetic operations, and is not limited to processors used in general-purpose PCs (such as CPU, MPU, GPU, or TPU) regardless of the name, and is used in the sense of including processors (such as FPGA, ASIC, or LSI), microcomputers, or chipsets (such as SoC).

[0082] The CPU 210 generates image data based on pixel data provided by the measurement control unit 150. Furthermore, the CPU 210 performs various types of processing on the generated image data using the working memory 230. For example, based on the light receiving signal output from the light receiving unit 120, the CPU 210 generates measurement data representing the three-dimensional shape of the workpiece W included in the field of view of the light receiving unit 120 at a specific location on the mounting unit 140. The measurement data is the image itself acquired by the light receiving unit 120, and, for example, in the case of measuring the shape of the workpiece W using a phase-shifting method, multiple images constitute a single measurement data set. Note that the measurement data can be point cloud data, which is a set of points with three-dimensional positional information, and the measurement data of the workpiece W can be obtained from the point cloud data. The point cloud data is data represented by the aggregation of multiple points with three-dimensional coordinates.

[0083] The motion control unit 144 determines, based on measurement data of at least a portion of the workpiece W, whether to perform only the rotation operation of the rotary stage 143, or both the rotation operation of the rotary stage 143 and the translation operation of the translation stage 141. As a result, the imaging range is automatically determined according to the external shape of the workpiece W without user intervention, thus facilitating 3D measurement. Note that after the translation stage 141 has moved in the XY direction, the motion control unit 144 can control the rotation of the rotary stage 143 even when the movement in the XY direction has stopped, thus also acquiring the shape around the workpiece W. Note that scanning can also be performed by moving and rotating the workpiece W relative to the measurement unit 100 while the measurement unit 100 is fixed.

[0084] Display unit 400 is a component used to display a stripe projection image acquired by measurement unit 100, a depth image generated based on the stripe projection image, a texture image captured by measurement unit 100, various user interface screens, etc. Display unit 400 includes, for example, an LCD panel or an organic electroluminescent (EL) panel. Furthermore, a touch panel is used in display unit 400, therefore, it can also be used as operation unit 250. In addition, display unit 400 can also display images generated by light receiving unit 120.

[0085] The light source unit 300 includes a control board 310 and an observation illumination source 320. A CPU (not shown) is mounted on the control board 310. The CPU of the control board 310 controls the light projection unit 110, the light receiving unit 120, and the measurement control unit 150 based on commands from the CPU 210 of the controller 200. Note that this configuration is an example, and other configurations can be used. For example, the control board can be omitted by having the light projection unit 110 and the light receiving unit 120 controlled by the measurement control unit 150 or by the controller 200. Alternatively, a power supply circuit for driving the measurement unit 100 can be provided in the light source unit 300.

[0086] The observation illumination source 320 includes LEDs emitting, for example, three colors: red, green, and blue. The brightness of the light emitted from each LED is controlled, thus allowing light of any color to be generated from the observation illumination source 320. The illumination light IL generated from the observation illumination source 320 is output from the illumination light output unit 130 of the measurement unit 100 via a light guide member (light guide). Note that, in addition to LEDs, other light sources such as semiconductor lasers (LDs), halogen lamps, and HIDs can be appropriately used as the observation illumination source. Specifically, when using an imager capable of color imaging, a white light source can be used as the observation illumination source.

[0087] The illumination light IL output from the illumination light output unit 130 illuminates the workpiece W in a time-division manner using red, green, and blue light. As a result, a color texture image can be obtained by combining the texture images captured by these RGB lights respectively, and the texture image is displayed on the display unit 400.

[0088] A three-dimensional measurement program and application for implementing the functions of the three-dimensional scanner 1 via the controller 200 are installed on the controller 200. Therefore, the three-dimensional measurement method according to the present invention can be performed using the three-dimensional scanner 1. The three-dimensional measurement method is a method for measuring the three-dimensional shape of a workpiece W and is performed by a computer included in the controller 200. The three-dimensional measurement program for enabling the computer to perform the three-dimensional measurement method can be recorded in a storage medium 1000. The storage medium 1000 can be, for example, an optical disc, such as a CD-ROM or DVD-ROM, or it can be a semiconductor memory, such as a memory card.

[0089] In the controller 200, which is equipped with 3D measurement programs and applications, components such as CPU 210, ROM 220, working memory 230, and storage device 240 are included. Figure 5 The diagram shows a scanning module 260, a conversion module 270, an integration module 280, and an analysis module 290. In this embodiment, the scanning module 260, conversion module 270, integration module 280, and analysis module 290 are divided into four modules, but any two or more of modules 260, 270, 280, and 290 can be integrated to form a single module. Furthermore, a portion of each of modules 260, 270, 280, and 290 can be incorporated into another module. That is, Figure 5 The configuration example shown is an example and is not limited to this. Figure 5 The configuration example shown.

[0090] The scanning module 260 is a component that acquires image data of the workpiece W by measuring its shape and creates mesh data of the workpiece W based on the image data. The conversion module 270 is a component that converts the mesh data created by the scanning module 260 into CAD data. CAD data consists of three-dimensional shape information composed of analyzed curved surfaces and freeform surfaces, and includes surface data, solid data, and data used for design. Surface data includes data on the shape surfaces of freeform surfaces and analyzed curved surfaces, such as the side surface data and planar data of a cylinder.

[0091] Integration module 280 is a component that transmits signals and data from scanning module 260 to conversion module 270 and analysis module 290, and also transmits signals and data from conversion module 270 to scanning module 260. In this example, the module can perform multiple arithmetic processes in one unit, and can also be referred to as, for example, a functional unit, functional block, etc.

[0092] The scanning module 260 includes, for example, a measurement condition setting unit 261, a scanning control unit 262, a point cloud acquisition unit 263a, a mesh data generation unit 263b, and a scanning output unit 264. The measurement condition setting unit 261 is a component for setting measurement conditions for the shape of the workpiece. The scanning control unit 262 is a component that controls the measurement unit 100 to generate image data according to the measurement conditions set by the measurement condition setting unit 261 and acquires measurement data of the workpiece W based on the generated image data.

[0093] The point cloud acquisition unit 263a is a component that acquires point cloud data of workpiece W based on image data of workpiece W acquired by the scanning control unit 262. The mesh data generation unit 263b is a component that acquires the point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired point cloud data, and converts the data into mesh data.

[0094] The scan output unit 264 is a component that outputs the mesh data and additional data created by the mesh data generation unit 263b to the conversion module 270. The additional data includes, for example, at least one of the following: measurement conditions and data calculated from the measurement data of the workpiece W.

[0095] The scanning module 260 controls the measuring unit 100 to generate conditions for measuring the shape of the workpiece W (measuring equipment model, magnification, resolution, etc.) and raw data during measurement (e.g., image data) as well as three-dimensional data. The three-dimensional data is mesh data comprising multiple polygons and can also be referred to as polygon data. A polygon is data that includes information specifying multiple points and information indicating the polygonal surface formed by connecting these points, and may include, for example, information specifying three points and information indicating the triangular surface formed by connecting these three points. Mesh data and polygon data can also be defined as data represented by an aggregation of multiple polygons.

[0096] In conversion module 270, mesh data is converted into CAD data, and the conversion process is determined based on measurement conditions and raw data. Specifically, conversion module 270 includes, for example, a data input unit 271, a processing parameter determination unit 272, a CAD conversion unit 273, and a CAD output unit 274. Data input unit 271 is a component that receives mesh data and additional data output from scan output unit 264. Processing parameter determination unit 272 is a component that determines the processing parameters for converting mesh data into CAD data based on the additional data received by data input unit 271. CAD conversion unit 273 is a component that converts mesh data into CAD data based on the processing parameters determined by processing parameter determination unit 272. CAD output unit 274 is a component that outputs the CAD data converted by CAD conversion unit 273.

[0097] The analysis module 290 of the 3D scanner 1 generates combined 3D data of the workpiece W by generating multiple 3D data points of the workpiece W placed in different orientations and combining these multiple 3D data points. The analysis module 290 includes, for example, a data acquisition unit 291 that acquires the 3D data of the workpiece W mounted on the rotating stage 143. The user can mount the workpiece W on the rotating stage 143 in any orientation. For example, when acquiring the 3D shape of the front and back surfaces of the workpiece W, the 3D data can be acquired by mounting the workpiece W on the rotating stage 143 with the front surface facing up for 3D data acquisition, and then mounting the workpiece W on the rotating stage 143 with the back surface facing up for 3D data acquisition. Similarly, when acquiring the 3D shape of the side surfaces of the workpiece W, the 3D data can be acquired by mounting the workpiece W on the rotating stage 143 with the side surfaces facing up. For example, a placement posture with the front face up can be set as the first placement posture, and a placement posture with the back face up can be set as the second placement posture. Additionally, a placement posture with the side face up can be a third placement posture. The definitions of placement postures are examples, and the placement postures can be different from each other. For example, the first, second, and third placement postures can be defined based on the shape of the workpiece W, the desired range of 3D data acquisition, etc. Furthermore, a fourth and fifth placement posture can be defined, and there is no particular limitation on the number of placement postures.

[0098] The data acquisition unit 291 acquires first three-dimensional data as three-dimensional data of a workpiece W placed on the rotating stage 143 in a first placement posture, and second three-dimensional data as three-dimensional data of a workpiece W placed on the rotating stage 143 in a second placement posture. Similarly, the data acquisition unit 291 also acquires third three-dimensional data as three-dimensional data of a workpiece W placed on the rotating stage 143 in a third placement posture, fourth three-dimensional data as three-dimensional data of a workpiece W placed on the rotating stage 143 in a fourth placement posture, and so on.

[0099] The data acquisition unit 291 acquires the three-dimensional data measured by the scanning module 260. The three-dimensional data acquired by the data acquisition unit 291 includes the shape information and texture information of the workpiece W, and the data acquisition unit 291 acquires the shape data with texture. Therefore, the data acquisition unit 291 acquires first three-dimensional data and second three-dimensional data. The first three-dimensional data is three-dimensional data including the shape information and texture information of the workpiece W placed in a first placement posture, and the second three-dimensional data is three-dimensional data including the shape information and texture information of the workpiece W placed in a second placement posture.

[0100] The data acquisition unit 291 receives the light receiving signal generated by the light receiving unit 120 of the measurement unit 100, generates a real-time image of the workpiece W based on the received light receiving signal, and acquires the generated real-time image.

[0101] The mesh data generation unit 263b generates first mesh data as mesh data for a workpiece W placed in a first placement posture and second mesh data as mesh data for a workpiece W placed in a second placement posture. Similarly, the mesh data generation unit 263b can also generate third mesh data as mesh data for a workpiece W placed in a third placement posture and fourth mesh data as mesh data for a workpiece W placed in a fourth placement posture.

[0102] When the grid data generation unit 263b generates grid data, the data acquisition unit acquires the first grid data and the second grid data generated by the grid data generation unit 263b as the first three-dimensional data and the second three-dimensional data. Similarly, the third grid data and the fourth grid data can also be acquired.

[0103] For example, the three-dimensional data (first three-dimensional data) of a workpiece W placed in a first placement posture can be stored in the storage device 240. In this case, the reading unit 292 included in the analysis module 290 reads the first three-dimensional data stored in the storage device 240. Similarly, second, third, and fourth three-dimensional data can be stored in the storage device 240. In this case, the reading unit 292 reads the second, third, and fourth three-dimensional data from the storage device 240.

[0104] If CAD data for workpiece W exists, the CAD data for workpiece W can be stored in storage device 240. In this case, reading unit 292 reads the CAD data stored in storage device 240.

[0105] The following sections will describe the scanning process in the absence of CAD data for workpiece W and the scanning process in the presence of CAD data for workpiece W. Figure 6 An example of scanning processing without CAD data for workpiece W is shown. Before or after the start of this process and before proceeding to step SA1, workpiece W is mounted on the rotating stage 143 in a first placement orientation. In step SA1, scanning of workpiece W begins. In step SA2, the measuring unit 100 scans workpiece W placed in the first placement orientation. For example, if workpiece W is placed with its front side facing up, the scan in step SA2 is referred to as a "one-sided scan" because the back side of workpiece W cannot be scanned. Figure 7The user interface screen 700 displayed at the start of the measurement is shown. The user interface screen 700 is generated by the controller 200 and displayed on the display unit 400.

[0106] A real-time image display area 701 and a model display area 702 are provided on the user interface screen 700. The real-time image display area 701 displays a real-time image generated by the data acquisition unit 291. The real-time image displays a rotating stage 143 and a workpiece W mounted on the rotating stage 143. Furthermore, the measurement unit 100 can also be referred to as a scanning head, and the scanning head includes a light projection unit 110, a light receiving unit 120, an illumination light output unit 130, and a measurement control unit 150. The display control unit 255 can also display the scanning head and the workpiece W on the display unit 400. This display is effective, for example, when the workpiece W is placed on any platform and scanned from different angles by moving the scanning head side.

[0107] In step SA2, the light projection unit 110 of the measurement unit 100 illuminates the workpiece W, which is placed in a first placement posture, with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and first point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the first point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired first point cloud data, and converts the first point cloud data into first mesh data. Note that here, the processing of the point cloud data includes point cloud sparsification, removal of point clouds outside the measurement area, and removal of noisy point clouds. In step SA3, the first mesh data obtained through the processing in step SA2 is acquired as first three-dimensional data. The first three-dimensional data is stored in the working memory 230 or the storage device 240. The model of the workpiece W based on the first three-dimensional data is displayed together with the model of the rotating stage 143. Figure 7 The user interface screen 700 shown is displayed in the model display area 702. The user interface screen 700 is generated by the display control unit 255 included in the controller 200 and displayed on the display unit 400.

[0108] In step SA4, the 3D scanner 1 calculates evaluation values ​​and proposes candidate poses. The evaluation values ​​in step SA4 are examples of evaluation indices, which will be described later. When the user operates the pose suggestion button 702a in the model display area 702, the controller 200 detects the operation. Then, as... Figure 8 As shown, the display control unit 255 generates a candidate display area 703 for displaying the next scannable attitude candidate, and displays the generated candidate display area on the user interface screen 700. The user can select an attitude candidate on the user interface screen 700 in step SA5.

[0109] When the 3D scanner 1 proposes a candidate pose, the candidate pose is calculated before the evaluation value is calculated. That is, as... Figure 5 As shown, the analysis module 290 includes an attitude calculation unit 293. The attitude calculation unit 293 calculates a placement attitude different from the first placement attitude based on the first three-dimensional data acquired by the data acquisition unit 291, and specifically, calculates a recommended placement attitude (hereinafter also simply referred to as "placement attitude") different from the first placement attitude based on the three-dimensional data read from the working memory 230 or storage device 240 by the reading unit 292. The display control unit 255 overlays and displays the recommended placement attitude calculated by the attitude calculation unit 293 on the real-time image acquired by the data acquisition unit 291.

[0110] When a placement posture different from the first placement posture is calculated, the posture calculation unit 293 first specifies the first placement posture of the workpiece W based on the first three-dimensional data. Since the first placement posture is specified, the posture calculation unit 293 can calculate placement postures different from the first placement posture. The distance between the measuring unit 100 and the workpiece can be set based on the focal length of the lens of the measuring unit 100. In this embodiment, since the measuring unit 100 includes a rotating stage 143, for example, the posture calculation unit 293 can calculate multiple placement postures by virtually rotating the first three-dimensional data about the rotation axis of the rotating stage 143. When the first three-dimensional data is rotated, the first three-dimensional data may not be rotated once, and may be rotated by a rotation angle of less than 360°.

[0111] Furthermore, when the workpiece W is large, there is a possibility that scanning may be performed by translating the measuring unit 100 multiple times relative to the workpiece W. Whether the measuring unit moves multiple times in parallel with the workpiece to perform scanning can be determined by the analysis module 290 based on whether the first three-dimensional data exceeds the measurable range (length, width, and height) of the measuring unit 100. If the first three-dimensional data exceeds the measurable range of the measuring unit 100, it can be determined that scanning is performed by translating the measuring unit 100 multiple times relative to the workpiece W; conversely, if the first three-dimensional data is within the measurable range of the measuring unit 100, it can be determined that scanning is performed without translating the measuring unit 100 multiple times.

[0112] When scanning is performed by translating the measuring unit 100 multiple times relative to the workpiece W, the measurement ranges of the measuring unit 100 overlap at a certain level or higher, and the measuring unit 100 is virtually arranged in multiple directions based on the center point of the translation, thereby maximizing the inclusion of the workpiece. Note that although a method for virtually moving the measuring unit 100 has been described, the invention is not limited thereto, and the workpiece can be moved and rotated.

[0113] Analysis module 290 includes an arithmetic unit 294. The arithmetic unit 294 calculates the relative positional pose of the recommended placement pose relative to the first placement pose. When the pose calculation unit 293 calculates multiple placement poses, the arithmetic unit 294 calculates the relative positional pose relative to the first placement pose for each of the multiple placement poses. For example, the arithmetic unit 294 calculates a transformation expression for converting the relative positional relationship between the first placement pose and the recommended placement pose. The arithmetic unit 294 applies the transformation expression to the 3D data of the first placement pose to convert the first placement pose into the recommended placement pose.

[0114] Since the placement posture differs from the first placement posture, the posture calculation unit 293 can also calculate candidate postures based on the size of the contact area with the mounting surface 142 in each of the multiple placement postures. When the contact area with the mounting surface 142 is too small, it may be difficult to mount the workpiece W onto the mounting surface 142. However, when calculating the placement posture, a placement posture that ensures a predetermined or greater contact area with the mounting surface 142 can be calculated, thus stabilizing the workpiece W when it is mounted on the mounting surface 142. Although Figure 8 An example of calculating four placement postures is shown, but the number of placement postures calculated by posture calculation unit 293 is not limited to four, and can be any number of one, two or more.

[0115] Figure 9 A user interface screen 710, generated by the display control unit 255 and displayed on the display unit 400, is shown when a candidate pose selection is accepted. The user interface screen 710 provides a model display area 711 and a candidate display area 712, in which a model of the workpiece W based on first 3D data is displayed. In the candidate display area 712, four placement poses calculated by the pose calculation unit 293 are displayed. Since the placement poses are displayed in the candidate display area 712, the next scannable placement pose can be presented to the user.

[0116] An evaluation index display area 712a is provided in the candidate display area 712, where the evaluation index indicates whether the workpiece placement posture presented to the user is suitable for the next scan. The evaluation index display area 712a is also generated by the display control unit 255 and displayed on the display unit 400.

[0117] The evaluation index is based on the amount of additional data to be added to the first 3D data by performing the next scan and the amount of overlap between the 3D data acquired by performing the next scan and the first 3D data. That is, the analysis module 290 includes an overlap region estimation unit 295 and an additional data volume estimation unit 296. The overlap region estimation unit 295 estimates the overlap region between the 3D data to be acquired by the data acquisition unit 291 and the first 3D data acquired by the data acquisition unit 291, in a placement state calculated by the attitude calculation unit 293. The overlap region estimation unit 295 can also estimate the degree of feature based on the distribution of points in the overlap region.

[0118] An example of a method for estimating the overlapping region using the overlapping region estimation unit 295 will be described. Here, in the presence of 3D CAD data for the workpiece W, the outermost surface when projecting the 3D CAD data relative to the outermost surface when projecting the first 3D data acquired by the data acquisition unit 291 can be further estimated as the overlapping region. However, as... Figure 6 The flowchart shown illustrates that, in the absence of 3D CAD data for workpiece W, since estimation based on 3D CAD data cannot be performed, the front surface of the existing first 3D data is used as the overlapping region. When workpiece W has a complex shape, there is a possibility that a portion of the front surface of the existing first 3D data is hidden by the second additional scan and subsequent additional scans. When 3D CAD data exists, the effect of hiding a portion of the front surface of the existing first 3D data can be considered when estimating the overlapping region; however, this effect is not considered when 3D CAD data is unavailable. However, when the workpiece has a shape close to a convex polygon, the effect of hiding a portion of the front surface of the existing first 3D data is absent, and substantially good results can be obtained. Therefore, the quality of the overlapping region can be determined by evaluating the area (amount of overlapping region) and the degree of feature. The degree of feature is obtained by analyzing the distribution of points in the overlapping region. Specifically, the deviation of coordinates or the normal of a point can be used as an evaluation value.

[0119] Furthermore, it is necessary to consider whether the workpiece W can be placed on the mounting surface 142 of the rotating stage 143 in the placement posture calculated by the posture calculation unit 293 during actual measurement. When the measuring unit 100 moves relative to the workpiece W, the analysis module 290 can perform conflict determination between the three-dimensional shapes of the measuring unit 100 and the base 600 and the three-dimensional CAD data or first three-dimensional data of the workpiece in computer graphics space.

[0120] Additionally, since the workpiece W cannot be placed below the mounting surface on which the measuring unit 100 is mounted, the analysis module 290 performs a conflict determination with the mounting surface (Z coordinate < 0). Furthermore, since it is difficult to arrange the measuring unit 100 at a certain angle directly above the workpiece or at a low angle relative to the workpiece, the analysis module 290 can evaluate the "arrangement ease" based on the arrangement angle of the measuring unit 100.

[0121] In the case of workpiece W moving and rotating, analysis module 290 also determines the ease of placing workpiece W on mounting surface 142. When the three-dimensional data is approximated by a convex polygon, analysis module 290 calculates the three-dimensional data or the area of ​​the bottom surface under the placement posture calculated by posture calculation unit 293, and determines that the larger the calculated area of ​​the bottom surface, the more stable the placement on mounting surface 142 can be performed. Furthermore, if three-dimensional CAD data of workpiece W exists, analysis module 290 obtains the coordinates of the center of gravity and the center point of the bottom surface, and calculates the distance between the center of gravity and the center point of the bottom surface. Analysis module 290 can determine the ease of placement of workpiece W based on the distance between the center of gravity and the center point of the bottom surface.

[0122] Next, the amount of additional data will be described. The additional data estimation unit 296 is a component that estimates the amount of additional data to be added to the first three-dimensional data for each of the multiple arrangement postures calculated by the posture calculation unit 293, by acquiring three-dimensional data from the data acquisition unit 291 while the arrangement is in one posture. The additional data estimation unit 296 can move the first three-dimensional data to have each placement posture and estimate the amount of additional data based on the orientation of the normal vector after the movement, and for example, it can estimate the amount of additional data based on the positional relationship between the normal vector after movement from a predetermined viewpoint and the viewing direction of the measurement unit 100. For example, the inner product of the normal vector after movement from the predetermined viewpoint and the viewing direction of the measurement unit 100 is calculated, and thus the three-dimensional data facing the measurement unit 100 is specified, and it is estimated that the greater the amount of three-dimensional data facing the measurement unit 100, the greater the amount of additional data.

[0123] An example of a method for estimating the amount of additional data by the additional data quantity estimation unit 296 will be described. Assuming a scan is performed using attitude candidates calculated by the attitude calculation unit 293, the amount of additional data (which may also be referred to as the "useful footage ratio") is obtained by estimating how much additional data can be acquired for the already acquired first 3D data. Here, in the case of 3D CAD data of the workpiece W, points that exist in the 3D CAD data but not in the first 3D data become additional data, and the amount of additional data increases with the number of points.

[0124] However, as Figure 6 The flowchart shown illustrates that, in the absence of 3D CAD data for workpiece W, since estimation based on 3D CAD data cannot be performed, the point cloud included in the first 3D data is analyzed, and the number of points visible on the back side of the scanned surface is set as additional data. This is because the back surface is invisible in a fully scanned object, therefore the visible portion of the back surface can be identified as the unscanned portion.

[0125] The rear surface of the workpiece W can be detected by detecting that the normals of points included in the first 3D data point in the opposite direction to the measurement unit 100. Using this detection process, the amount of additional data can be estimated even without 3D CAD data of the workpiece W. Note that the analysis module 290 can calculate the area requiring further scanning or the percentage of the already scanned area of ​​the workpiece W based on the area of ​​the already scanned region and the amount of additional data scanned in each direction. The analysis module 290 can also automatically determine that scanning is complete when the scanned area exceeds a certain percentage. Either the estimation of the overlapping area or the estimation of the additional data can be prioritized.

[0126] After estimating the overlapping area and additional data volume for each placement orientation as described above, the evaluation unit 299 included in the analysis module 290 calculates an evaluation index. When calculating the evaluation index, it can be based on, for example, the following expression: Evaluation Index = (Area of ​​Overlapping Area) × (Amount of Additional Data) × (Dispersion of Points in the Overlapping Area) × (Ease of Placement)

[0127] As described above, the evaluation unit 299 calculates an evaluation index for each of the multiple placement postures based on the amount of additional data and the amount of overlapping regions for each of the multiple placement postures calculated by the posture calculation unit 293, and calculates a comprehensive evaluation index based on each evaluation index. Note that when using texture information, integrals, averages, maximum values, medians, etc. of texture feature quantities based on local contrast or difference values ​​of texture information can be incorporated.

[0128] The display control unit 255 displays the calculated evaluation index in digital or graphical form on the display unit 400. Figure 9The example illustrates the graphical display of evaluation indices in the evaluation index display area 712a. When a user performs an operation to select any of the placement postures displayed in the candidate display area 712, the analysis module 290 specifies the selected placement posture. The display control unit 255 displays the evaluation index of the placement posture specified by the analysis module 290 in graphical form in the evaluation index display area 712a. When the user selects another placement posture, the display control unit 255 displays the evaluation index of that placement posture in the evaluation index display area 712a. As described above, since the display control unit 255 can display the evaluation index of each of the multiple placement postures calculated by the posture calculation unit 293 on the display unit 400, the user can obtain the evaluation index of each placement posture when multiple placement postures are presented.

[0129] The display format of the evaluation index does not have to be... Figure 9 The display control unit 255 can display on the display unit 400 either a graphical form or a numerical form, or a combination of graphical and numerical forms. Furthermore, the display control unit 255 can display on the display unit 400 each of an evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296 and an evaluation index based on the amount of overlapping region estimated by the overlapping region estimation unit 295. In this case, the analysis module 290 calculates the first evaluation index based on the amount of additional data estimated by the additional data quantity estimation unit 296, and calculates the second evaluation index separately based on the amount of overlapping region estimated by the overlapping region estimation unit 295. The display control unit 255 can display only the first evaluation index on the display unit 400 in graphical or numerical form, or only the second evaluation index on the display unit 400 in graphical or numerical form, or display both the first and second evaluation indices on the display unit 400 in graphical or numerical form respectively.

[0130] The analysis module 290 includes a designation unit 297, which designates a candidate pose from multiple placement poses calculated by the pose calculation unit 293 based on an evaluation index. Specifically, the designation unit 297 acquires the evaluation index for each of the multiple placement poses. The designation unit 297 designates the placement pose with the highest evaluation index among the acquired multiple evaluation indices. Since the evaluation index is based on the amount of additional data estimated by the additional data estimation unit 296 and the amount of overlapping region estimated by the overlapping region estimation unit 295, the designation unit 297 designates a candidate pose based on the amount of additional data and the amount of overlapping region. When designating a candidate pose, the designation unit 297 can designate a candidate pose based on the shape of the overlapping region, the amount of overlapping region, and the amount of additional data estimated by the overlapping region estimation unit 295. For example, when the overlapping region includes an irregular shape, the evaluation index can be improved compared to a flat overlapping region because the alignment accuracy is higher than that of an overlapping region with a flat shape.

[0131] When the designated unit 297 identifies a candidate posture, the display control unit 255 displays the candidate posture designated by the designated unit 297 on the display unit 400. The candidate posture displayed by the display unit 400 is the workpiece placement posture recommended by the 3D scanner 1 to the user. Therefore, the user can confirm the appropriate placement posture by viewing the display unit 400.

[0132] exist Figure 6 In step SA6, the user determines whether the candidate poses displayed on the display unit 400 are the desired poses. Since the placement pose is specified based on an evaluation index, and due to the large amount of additional data and overlapping area, it is believed that accurate combined 3D data can be obtained by placing the workpiece in this pose. However, since the designation unit 297 only specifies the placement pose based on the evaluation index, it may not necessarily specify the user's highly relevant area (the area requiring high-precision 3D data). Therefore, in the 3D scanner 1 of this embodiment, multiple placement poses are presented to the user along with evaluation indices, and the user can select the placement pose with a relatively high evaluation index and best suited to their interests from the presented multiple placement poses. Specifically, the analysis module 290 includes a receiving unit 298 that accepts the user's selection or adjustment operation for the placement pose. For example, when the user performs a selection or adjustment operation from... Figure 9 When selecting a desired placement posture from the candidate display areas 712 displayed on the user interface screen 710, the receiving unit 298 accepts the selected operation input. The display control unit 255 displays the placement posture accepted by the receiving unit 298 in the candidate display area 712.

[0133] Furthermore, in step SA7, the candidate pose displayed on the display unit 400 can be adjusted. Specifically, the receiving unit 298 accepts user input for adjusting a candidate pose specified by the designating unit 297. The user operates the operating unit 250 in computer graphics space to adjust the positional relationship between the measuring unit 100 and the workpiece W. For example, the workpiece W in the candidate pose is moved horizontally, moved in the height direction, or rotated. When the user completes the adjustment of the candidate pose, the process proceeds to step SA8, and the pose calculation unit 293 specifies the adjusted placement pose, the overlap region estimation unit 295 estimates the amount of overlap between the three-dimensional data acquired by the data acquisition unit 291 and the first three-dimensional data under the adjusted placement pose, and the additional data quantity estimation unit 296 estimates the additional data quantity. The analysis module 290 recalculates the evaluation index based on the estimated overlap region and the estimated additional data quantity, and presents the evaluation index to the user. Therefore, in the case of scanning the user's region of interest at a certain height, the user can determine whether the pose is likely to be successfully aligned with the first three-dimensional data. Note that the evaluation value calculation and attitude candidate suggestion in step SA4 and the attitude candidate selection in step SA5 can be skipped, and the attitude calculation unit 293 can specify the placement attitude for which the user adjusts the candidate attitude in step SA7.

[0134] In step SA9, the analysis module 290 generates a computer-generated graphic workpiece (model) with a determined placement orientation. The display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. The computer-generated graphic workpiece can be semi-transparent, but it can also be opaque.

[0135] In step SA10, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W along with the mounting surface 142, and the data acquisition unit 291 acquires a real-time image including the workpiece W and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W mounted on the mounting surface 142 overlaps with the workpiece in the computer graphics.

[0136] In step SA11, the user determines whether the actual workpiece W mounted on the mounting surface 142 is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W mounted on the mounting surface 142 cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SA5, and another placement orientation is selected. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the second placement orientation, the process proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement orientation.

[0137] In step SA12, the light projection unit 110 of the measurement unit 100 illuminates the workpiece W, which is positioned in a second placement posture, with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained through the processing in step SA12 is acquired as second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0138] In step SA14, the alignment unit 290A, included in the analysis module 290, aligns the first three-dimensional data (three-dimensional data of a workpiece placed in a first placement posture) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. During this alignment, the overlapping region extracted by the extraction unit 290B, included in the analysis module 290, is used. The extraction unit 290B is a component that extracts the overlapping region between the three-dimensional data converted by the arithmetic unit 294 to a recommended placement posture and the three-dimensional data placed in a second placement posture. The extraction unit 290B can extract the overlapping region using, for example, the normal vector of the three-dimensional data and the color information of the workpiece. That is, the alignment unit 290A can perform alignment based on the three-dimensional data included in the overlapping region extracted by the extraction unit 290B. When alignment is performed using the texture information of the workpiece W, texture features are estimated based on the brightness or color information or local contrast or difference constituting the texture information. These texture features can be used in combination with shape features through various operations such as addition and multiplication.

[0139] In this embodiment, an evaluation index for the placement posture is calculated, ensuring a large overlap area and a large amount of additional data during the next scan, thus increasing the likelihood of successful alignment of the 3D data performed by the alignment unit 290A. Then, while presenting a placement posture with a high evaluation index to the user, the final placement posture is determined simultaneously with the user's adjustments. This placement posture is then superimposed and displayed on the real-time image, allowing the user to arrange the actual workpiece W in a posture close to the original placement posture. Therefore, alignment with the placement posture is performed as the initial position, thereby increasing the alignment success rate without imposing additional burden on the user.

[0140] Alignment unit 290A can acquire the normals of the three-dimensional data of a workpiece W placed in a second placement posture. Alignment unit 290A can also perform alignment based on the orientation of the normals of the three-dimensional data of the workpiece W placed in the second placement posture, the relative positional relationship calculated by arithmetic unit 294, the orientation of the normals of the three-dimensional data of the workpiece placed in the first placement posture, and shape features extracted from the first and second three-dimensional data. Specifically, alignment unit 290A first narrows down the range of candidate corresponding points based on the angle between the orientation of the normals obtained by rotating the normals of the three-dimensional data of the workpiece W placed in the first placement posture based on the relative positional relationship and the orientation of the normals of the three-dimensional data of the workpiece placed in the second placement posture. Then, based on the narrowed range of candidate corresponding points, alignment can be performed using shape features extracted from the first and second three-dimensional data.

[0141] Furthermore, when the user places the workpiece W, a high-precision initial estimate can be obtained through pattern matching between the computer-generated workpiece graphic and the actual workpiece W. This is because the relative position between the measuring unit 100 and the workpiece W can be obtained through epipolar geometry from the correspondence between the computer-generated workpiece graphic and the actual workpiece W in the real-time image.

[0142] The initial estimate can be used to detect and extract overlapping regions between two point clouds, detect erroneous correspondences among corresponding point candidates, and detect erroneous candidates from the position and pose candidates obtained from the corresponding point candidates. When aligning the source point cloud P in the already scanned first 3D data with the newly scanned destination point cloud Q, the alignment unit 290A can specify corresponding points from the second 3D data that are included in the first 3D data, and adjust the position and pose of the first and second 3D data based on the specified corresponding points.

[0143] exist Figure 11In step SB1 of the flowchart shown, arithmetic unit 294 acquires the alignment source point cloud P. In step SB2, arithmetic unit 294 acquires the alignment destination point cloud Q. In step SB3, arithmetic unit 294 acquires the initial estimated pose. In step SB4, arithmetic unit 294 performs a transformation on the alignment source using the placement pose determined as described above. This transformation can be performed by multiplying with a rotation matrix and adding to a translation component. Therefore, the transformed point cloud P' exists at substantially the same position as the point cloud of the alignment destination (step SB5).

[0144] In step SB6, extraction unit 290B can extract Qo by extracting point cloud P'o, in which points of point cloud Q exist in the vicinity after transformation, and applying point cloud P'o in reverse. These point clouds are called overlapping region point clouds. Note that when normal vectors or color information are assigned to point clouds, the accuracy of overlapping region extraction can be changed by using the existence of nearby point clouds with high similarity as a condition for overlapping region extraction. Furthermore, the definition of "nearby" varies depending on how close the user can arrange the actual workpiece W to the workpiece in the computer graphics at its position and pose. In addition, the range of "nearby" can be determined based on the scanning range of measurement unit 100 or the size of workpiece W, and the range of "nearby" can be changed by the user.

[0145] Alignment unit 290A uses only the overlapping regions extracted by extraction unit 290B for alignment. Therefore, it can eliminate the occurrence of incorrect correspondences of points in non-overlapping regions.

[0146] However, the extraction unit 290B requires computational cost to extract overlapping regions. This is because, for example, when extracting point cloud P'o, it searches for the point Q closest to each point in point cloud P' for localization, and assuming the number of points P is M and the number of points Q is N, the computational cost O is O(M*N).

[0147] Therefore, erroneous correspondences can be suppressed by removing hidden surfaces as alternatives to overlapping regions. This is because the initial estimates described above are obtained through shape analysis to determine the values ​​of overlapping regions suitable for alignment. Each point cloud of P' and Q is projected by a virtual camera based on the initial estimates, and hidden surface removal (extracting only the points whose depth position is minimum as viewed from the camera) based on the depth buffer commonly used in computer graphics eliminates the data of the back surface as erroneous correspondences.

[0148] Furthermore, simple hidden surface removal using normal vectors can be performed as a more straightforward method. Hidden surface removal can be performed by extracting only the points pointing towards the measurement unit 100 from each of the point clouds P' and Q in the camera coordinate system. In the normal vectors [nx, ny, nz] in the camera coordinate system, only normal vectors with negative nz values ​​are extracted. Therefore, computational costs can be reduced.

[0149] Figure 12 This is a schematic diagram related to the removal of hidden surfaces. White arrows W1 extending in a direction orthogonal to each surface of the workpiece W arranged on the mounting surface 142 indicate the normal vector of each surface. Furthermore, thin arrows W2 represent the component of the normal vector parallel to the Z-axis of the light receiving unit 120, and arrow W3 represents the component of the normal vector perpendicular to the Z-axis of the light receiving unit 120. In arrow W1, the dashed line indicates that the Z-axis component of the light receiving unit 120 is determined to be a positive normal, and the surface corresponding to the dashed arrow W1 is determined to be a hidden surface.

[0150] exist Figure 11 In step SB8 shown, the extraction unit 290B extracts the shape features and brightness features of each point from the first three-dimensional data corresponding to the first placement posture and the second three-dimensional data corresponding to the second placement posture obtained by the data acquisition unit 291.

[0151] In step SB10, the feature values ​​obtained in step SB9 are input to extraction unit 290B to detect corresponding points. In the absence of an initial estimate, candidate corresponding points are obtained through feature differences between the first and second 3D data, correlations between the first and second 3D data, etc. At this time, points initially located at a distance but with similar features in shape and brightness can be selected as corresponding points. On the other hand, if an initial estimate exists, only nearby points and points with similar orientations of normal vectors can be candidate corresponding points, and the correlation values ​​of the features can be weighted based on the distance between points and the inner product of the normal vectors.

[0152] Assuming the i-th point of the point cloud P of the alignment source is pi, and the point obtained by converting pi to an initial estimate is p'i, and the j-th point of the point cloud Q of the alignment destination is qj, with eigenvector F(*) and normal vector N(*), the evaluation value w_d(dist(p'i, qj))*wn(1-dot(N(p'i), N(qj))*CORR(F(p'i), FU(qj)) can be obtained. wd is a distance-related weight, and wn is a weight related to the orientation of the normal. When the weight is a step function of [1, 0], only the nearest weight is selected, and when the weight is a monotonically decreasing function, continuous weights can be applied. Points with high evaluation values ​​are set as corresponding points, thus eliminating erroneous corresponding points with similar shapes and brightness distributions. The matrix C(pi, qj) representing the corresponding point candidates is obtained as the output of the corresponding point detection unit. It can be represented by a matrix, in which the corresponding point is determined to be 1, and other points are 0.

[0153] In step SB11, candidate poses are obtained from these corresponding candidate points. The method for obtaining the final placement pose of workpiece W in steps SB12 to SB17 will be described. For example, the Random Sample Consensus (RANSAC) method can be used as a method for obtaining the final placement pose of workpiece W. An example of using RANSAC will be described below. In the RANSAC pose candidate calculation, candidates are calculated from multiple randomly extracted corresponding points (step SB12). As a result, candidate poses can be obtained (step SB13). In the case of using only the coordinates of the points, the correspondence between three points is required, and in the case of using the normal and the coordinates of the points, the correspondence between two points is required. Taking the correspondence between three points as an example, the rotation matrix R and translation vector t that minimize the squared distance between the corresponding p' and q are obtained through linear and nonlinear optimization (step SB14).

[0154] Sum_C(p_i,q_j)(q_j-p'_i*R+t)^2

[0155] This implementation is not limited to this candidate calculation method, and candidates can be calculated using any method. For example, the method of using graphic segmentation is also applicable to this implementation.

[0156] To determine whether the rotation matrix and translation vector obtained in this way are close to the initial estimate, we need to check if the correct candidate has been obtained. When using the sum of squares of each element, the initial estimate is t, and the candidate is t', the translation vector can be defined as dist(t,t')=(t_x-t'_x)^2+(t_y-t'_y)^2+(t_z-t'_z)^2. Additionally, the rotation matrix can be obtained as Dist(R,R')=arccos[(trace(R^T*R')-1) / 2].

[0157] Placement postures with translational component errors within a predetermined size and angular errors within a predetermined angle can be extracted as correct candidates. In step SB16, the detected postures are obtained from these candidates through a final posture selection process, and the selected candidates are obtained (step SB17). This is obtained by a method for evaluation using a predetermined evaluation value. For example, by using a rotation matrix R representing the candidate posture and a translation component t, the transformation is performed as P” = R*P’ + t, and the number of points in the point cloud Q that satisfy dist(P” - Q) < a threshold can be set as the evaluation value. However, there are no particular limitations on the calculation of the evaluation value to be combined in this embodiment. Since the evaluation value calculation can also be performed only in the overlapping area, the phenomenon that the evaluation value is incorrectly increased due to the presence of corresponding points nearby, even if some points do not overlap, can be suppressed. The second placement posture is specified in this way, and the workpiece W placed in the specified second placement posture is scanned by the measurement unit 100. Therefore, second three-dimensional data with high-precision alignment with the first three-dimensional data and a large amount of additional data can be obtained.

[0158] When performing alignment between the first and second 3D data, the alignment unit 290A can perform alignment based on the shape features and image features of the 3D data. That is, the analysis module 290 includes a shape feature extraction unit (first extraction unit) 290C that extracts shape features from the first and second 3D data acquired by the data acquisition unit 291, and an image feature extraction unit (second extraction unit) 290D. The shape feature extraction unit 290C includes a neural network comprising an input layer, multiple intermediate layers, and an output layer. The input layer of the neural network is a component that receives input from the first and second 3D data acquired by the data acquisition unit 291. The multiple intermediate layers of the neural network are components that extract shape features based on the input received by the input layer. The output layer of the neural network is a component that outputs the shape features extracted in the intermediate layers. On the other hand, the image feature extraction unit 290D is a component that extracts image features from the texture information included in each of the first and second 3D data acquired by the data acquisition unit 291.

[0159] As described above, in this embodiment, when performing alignment between the first and second 3D data, deep learning is used to incorporate image features such as brightness and color into the feature vector. In calculating the feature vector for each point in an image or point cloud, a "convolution" arithmetic operation can be performed, multiplying the coordinates and colors—information of the point itself and information of surrounding points—by coefficients to obtain their sum or maximum value. For example, in a convolutional neural network (CNN) in an image, a large number of 3×3 filters with coefficients determined through pre-learning are provided, the output value of each filter is set, and the set output vector is set as the output vector of the point, and the result obtained by repeatedly using the output vector with different filters is set as the feature vector.

[0160] When point clouds are included in first or second 3D data, since the data cannot be guaranteed to exist at intervals with different rules than those in the image, other methods can be used. Examples include methods for applying a CNN in 3D by sampling points in voxels, multiplying points by coefficients when they exist, and outputting 0 when they do not; PointNet++ for applying the same coefficients to each point to obtain the maximum output; and methods such as KPConv for performing interpolation arithmetic relative to the deviation of the point cloud's position from coefficients determined on a grid to obtain coefficients and then convolving those coefficients.

[0161] In processing point clouds included in the first or second 3D data, the computational load becomes enormous and the processing time becomes very long because the arrangement of points is unknown. In the technical field of the 3D scanner 1 in this example, when the processing time becomes long, work is delayed and not performed, so it is necessary to set the processing time without problems in practical use. Therefore, the number of points that can be processed is naturally limited, and as an example, the upper limit for the number of keypoints can be set to several thousand points, and approximately tens of thousands of points in the entire point cloud. Since the measurement unit 100 can scan millions of points in the point cloud at a time, the points used in deep learning need to be sparsified to approximately 1%.

[0162] In the simplest case where image information is input into the input layer of a neural network, image information (brightness and color) can be input in addition to coordinates and normal vectors. However, in this case, the amount of data input to the input layer is too small, and useful information about the image is lost. Therefore, there are cases where performance is not improved.

[0163] In the following text, reference will be made to Figure 13The flowchart shown describes the process of extracting shape features and image features. In step SC1, the analysis module 290 acquires the point cloud (input point cloud) of the first 3D data or the second 3D data. In steps SC2 and SC3, coarse sparsification and dense sparsification are performed, respectively. That is, as shown... Figure 5 As shown, the analysis module 290 includes a resolution conversion unit 290E, which converts the resolution of the first three-dimensional data and the resolution of the second three-dimensional data acquired by the data acquisition unit 291. The resolution conversion unit 290E performs a first conversion process (coarse sparsification) to convert the first and second three-dimensional data into three-dimensional data with a first resolution lower than that acquired by the data acquisition unit 291. That is, the resolution conversion unit 290E converts the first three-dimensional data acquired by the data acquisition unit 291 into third three-dimensional data with a first resolution, and converts the second three-dimensional data acquired by the data acquisition unit 291 into fourth three-dimensional data with a first resolution. In the first conversion process performed by the resolution conversion unit 290E, since the resolution of the three-dimensional data acquired by the data acquisition unit 291 is reduced to low-resolution three-dimensional data, the number of points in the three-dimensional data is less than the number of points acquired by the data acquisition unit 291.

[0164] The first conversion process of the resolution conversion unit 290E corresponds to step SC2. Step SC2 generates third 3D data (third point cloud) and fourth 3D data (fourth point cloud) as shape feature extraction point clouds (Q), and the shape feature extraction unit 290C acquires the shape feature extraction point clouds (Q) in step SC4. The shape feature extraction unit 290C extracts shape features from the shape information included in each of the first 3D data with a first resolution and the second 3D data with a first resolution converted by the resolution conversion unit 290E.

[0165] The resolution conversion unit 290E performs a second conversion process (dense-sparse processing) that transforms the first and second 3D data acquired by the data acquisition unit 291 into 3D data with a second resolution that is lower than the resolution acquired by the data acquisition unit 291 but higher than the first resolution. Specifically, the resolution conversion unit 290E converts the first 3D data acquired by the data acquisition unit 291 into fifth 3D data with a second resolution, and converts the second 3D data acquired by the data acquisition unit 291 into sixth 3D data with a second resolution. In the second conversion process performed by the resolution conversion unit 290E, since the resolution of the 3D data acquired by the data acquisition unit 291 is reduced to low-resolution 3D data, the number of points in the 3D data is less than the number of points acquired by the data acquisition unit 291. However, since the resolution is higher than the resolution of the first conversion process, the number of points is greater than the number of points acquired in the first conversion process. The second conversion process of the resolution conversion unit 290E corresponds to step SC3.

[0166] In step SC3, fifth 3D data (fifth point cloud) and sixth 3D data (sixth point cloud) (Q') are generated as image feature extraction point clouds. In step SC5, image feature extraction unit 290D acquires image feature extraction point clouds (Q'). Image feature extraction unit 290D extracts image features from the texture information included in each of the first and second 3D data having a second resolution, which are converted by resolution conversion unit 290E.

[0167] Note that step SC3 above can be skipped, and the image feature extraction unit 290D can generate an image feature extraction point cloud by using the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit 291.

[0168] In the first and second transformation processes, for example, random sparsification for randomly selecting a predetermined number of data points, or voxel sparsification for calculating a voxel grid and reducing multiple point clouds entering the voxels to only one average value, can be used. Here, the point cloud before sparsification (the point cloud before the second transformation process) can be used as the point cloud for image feature extraction.

[0169] In step SC6, the shape feature extraction unit 290C extracts key points from the low-resolution first 3D data and second 3D data. For example, when extracting key points, there are methods for randomly selecting and extracting key points, methods for extracting feature points as key points through deep learning or rule-based point cloud analysis, etc., and any method can be used. The key points extracted by the shape feature extraction unit 290C are obtained by the shape feature extraction unit 290C (step SC7).

[0170] In step SC8, the analysis module 290 acquires the shape feature extraction point cloud (Q) obtained in step SC4 and the key points obtained in step SC7, and the analysis module 290 samples the points around the key points. At this time, the points used for shape feature extraction are sampled based on the shape feature extraction point cloud (Q) obtained in step SC4.

[0171] In step SC12, which continues after step SC5, the analysis module 290 acquires the image feature extraction point cloud (Q') obtained in step SC5 and the key points obtained in step SC7, and samples the points around the key points. Points for image feature calculation are sampled based on the image feature extraction point cloud (Q') obtained in step SC5. Furthermore, because the image feature extraction point cloud (Q') has a large number of points, the surrounding area to be sampled can be narrowed compared to the shape feature extraction point cloud (Q).

[0172] In step SC9, which continues after step SC8, the shape features are input to the shape feature extraction unit 290C. For example, the shape feature vector can be calculated only from the vicinity of the points sampled for the shape features. There are no particular limitations on the method used to calculate the shape feature vector; for example, methods can be used to project the image information of the point cloud based on the normals of the key points to obtain a patch image and calculate the feature vector through computation such as CNN or Scale Invariant Feature Transform (SIFT), or methods can be used to create a histogram based on the RGB or brightness information of the sampled points and use the histogram as the feature vector, etc. In step SC10, the shape feature extraction unit 290C calculates the shape feature vector of the key points.

[0173] On the other hand, in step SC13, which continues after step SC12, image features are input to the image feature extraction unit 290D. Here, the feature vector of the image is calculated. The feature vector of the keypoint is calculated by using all points around the keypoint sampled from the image feature extraction point cloud (Q'). In step SC14, the image feature extraction unit 290D calculates the image feature vector of the keypoint, and in step SC15, the image feature vector calculated by the image feature extraction unit 290D is obtained.

[0174] In other words, the image feature extraction unit 290D extracts key points for calculating shape features from low-resolution first and second 3D data. The image feature extraction unit 290D can specify a region corresponding to the key points extracted from the first and second 3D data before conversion by the resolution conversion unit 290E. The image feature extraction unit 290D extracts image features from the specified corresponding region.

[0175] It should be noted that when calculating the feature vector of an image, the image feature extraction unit 290D can specify the region corresponding to the key points extracted by the shape feature extraction unit 290C from the first three-dimensional data and the second three-dimensional data, and extract image features from the region corresponding to the key points specified by the image feature extraction unit 290D.

[0176] In the calculation of the shape feature vector in step SC11, a three-dimensional to six-dimensional vector with X, Y, and Z coordinates and, if necessary, normal information Nx, Ny, and Nz can be input. As described above, the shape feature extraction unit 290C extracts key points from the low-resolution first three-dimensional data and the second three-dimensional data as partial regions for calculating shape features, and extracts shape features for the extracted key points.

[0177] In step SC16, candidate pairs are obtained by comparing shape feature vectors, image feature vectors, or shape feature vectors including image features with key points of the 3D data of the alignment source and the 3D data of the alignment destination. There are no particular limitations on the methods used to compute candidate pairs, and examples include methods for obtaining the distance between feature vectors and forming pairs based on smaller distances, and methods for forming pairs based on higher output values ​​of deep learning modules that correspond to each other. Mean square (L2 distance) or cosine similarity is used as the distance. Furthermore, at this point, to compare shape feature vectors and image feature vectors, a simple method can be used to obtain a pair by combining the two vectors as input, the sum of distances calculated from the shape feature vectors and image feature vectors, the maximum value, the minimum value, etc. Since the points of the candidate pairs are used, the positional relationships (rotation and translation) between point clouds can be obtained through RANSAC or deep learning-based methods.

[0178] Figure 14 This illustrates the case where a vector with added image features is input to the shape feature extraction unit during the processing of shape feature extraction and image feature extraction. The processing involves... Figure 13 Step SC8 is performed until... Figure 14 Step SC9' in the process, and the processing has been completed. Figure 13 Step SC12 is performed until... Figure 14 Step SC13' in the middle. Steps SC13' to SC15' are related to... Figure 13 Steps SC13 to SC15 are the same. In step SC10', the image feature vector is input to the shape feature extraction unit 290C. In this case, the output shape feature vector is a shape feature vector that already includes the image features (step SC11'). Step SC16' is the same as... Figure 13 The steps are the same as in step SC16.

[0179] Alignment unit 290A performs a combined process of global alignment and local alignment, namely, aligning the point clouds of the first 3D data and the point clouds of the second 3D data with high precision. In this process, global alignment is performed using low-resolution point clouds, and local alignment, which involves transformation of their positions, is performed using high-resolution point clouds. For example, local alignment is performed by using point cloud data with a first resolution (low resolution) for shape feature extraction in global alignment, using point cloud data with a third resolution higher than the first resolution for image feature extraction, and using point cloud data with a second resolution higher than both the first and third resolutions as input. Note that the point cloud data used for local alignment can be point cloud data with a second resolution, or it can be point cloud data with the same resolution as the point cloud generated by receiving a light receiving signal in point cloud acquisition unit 263a.

[0180] Specifically, alignment unit 290A acquires shape features extracted by shape feature extraction unit 290C and image features extracted by image feature extraction unit 290D. In this case, alignment unit 290A can perform global alignment based on the shape features extracted by shape feature extraction unit 290C and the image features extracted by image feature extraction unit 290D. In this case, alignment unit 290A performs a first alignment process that calculates low-precision alignment parameters indicating the relative position and pose of the second three-dimensional data with respect to the first three-dimensional data.

[0181] When calculating low-precision alignment parameters, perform steps SD1 to SD7, as follows: Figure 15 As shown in the diagram. Steps SD1 to SD5 and Figure 13 Steps SC1 to SC5 shown are the same. Figure 15 In step SD6, the shape feature extraction point cloud obtained in step SD4 and the image feature extraction point cloud obtained in step SD5 are input into the alignment unit 290A. The alignment unit 290A performs global alignment of the input point clouds and calculates low-precision alignment parameters.

[0182] After the first alignment process, the alignment unit 290A acquires the low-precision alignment parameters calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data. The alignment unit 290A can perform local alignment based on the low-precision alignment parameters, the first three-dimensional data, and the second three-dimensional data. In this case, the alignment unit 290A performs a second alignment process that calculates high-precision alignment parameters indicating the relative position and orientation of the second three-dimensional data with respect to the first three-dimensional data.

[0183] When calculating high-precision alignment parameters, such as Figure 16As shown, in step SE1, the analysis module 290 acquires the input point cloud of the first 3D data. In step SE2, the analysis module 290 acquires the input point cloud of the second 3D data. In step SE3, the analysis module 290 acquires the input point cloud of the second 3D data. Figure 15 The flowchart shown illustrates the low-precision alignment parameters calculated during the processing.

[0184] Analysis module 290 includes coordinate transformation unit 290F, which is based on... Figure 15 The flowchart shown illustrates the process of calculating low-precision alignment parameters to perform coordinate transformation of the first 3D data. In step SE4, the coordinate transformation unit 290F performs coordinate transformation of the input point cloud of the first 3D data based on the low-precision alignment parameters. In step SE5, the point cloud after coordinate transformation is acquired.

[0185] In step SE6, the input point cloud of the second 3D data obtained in step SE2 and the point cloud after coordinate transformation obtained in step SE4 are input to the alignment unit 290A. The alignment unit 290A performs local alignment of the input point cloud and calculates high-precision alignment parameters. In step SE7, the high-precision alignment parameters are obtained. As described above, in the second alignment process, the alignment unit 290A obtains the first 3D data and the second 3D data that have undergone coordinate transformation by the coordinate transformation unit 290F, and performs local alignment based on the first 3D data and the second 3D data.

[0186] Executed as described above Figure 6 After the alignment process in step SA14 shown, the combination process is performed. That is, as... Figure 5 As shown, the analysis module 290 includes a combination unit 290G. The combination unit 290G is a component that combines first three-dimensional data of a workpiece W placed in a first placement posture and second three-dimensional data of a workpiece W placed in a second placement posture, aligned by the alignment unit 290A, to generate combined three-dimensional data. The combination unit 290G can also acquire first mesh data as first three-dimensional data and acquire second mesh data as second three-dimensional data. In this case, the combination unit 290G combines the first mesh data and the second mesh data to generate combined mesh data as combined three-dimensional data.

[0187] in addition, Figure 12The removal of the hidden surfaces shown is performed before generating the combined 3D data and is used for alignment by the alignment unit 290A. That is, the alignment unit 290A can perform alignment using the 3D data from which the hidden surfaces have been removed. Then, the combination unit 290G generates the combined 3D data based on the alignment performed using the 3D data from which the hidden surfaces have been removed. Here, the generation of the combined 3D data is performed before the removal of the hidden surfaces, i.e., using the 3D data including the hidden surfaces.

[0188] The combining unit 290G can acquire shape information of the workpiece W based on a first light receiving signal output from the light receiving unit 120, and can acquire texture information of the workpiece W based on a second light receiving signal output from the light receiving unit 120. Having acquired both the shape and texture information of the workpiece W, the combining unit 290G generates three-dimensional data including both the shape and texture information of the workpiece W.

[0189] Furthermore, when the data acquisition unit 291 acquires combined three-dimensional data obtained by combining the first three-dimensional data and the second three-dimensional data, the attitude calculation unit 293 can calculate a placement attitude different from the first placement attitude based on the combined three-dimensional data acquired by the data acquisition unit 291. As described above, in addition to the already generated combined three-dimensional data, the placement attitude during scanning can also be presented to the user. In this case, the additional data quantity estimation unit 296 estimates the amount of additional data to be added to the combined three-dimensional data. In addition, the overlap region estimation unit 295 estimates the overlap region of the combined three-dimensional data. Therefore, an evaluation index based on the additional data quantity and the overlap region quantity can be presented to the user when scanning is performed with the additional placement attitude.

[0190] When the combination process is complete, the processing progresses to... Figure 6 The step SA15 is shown. In step SA15, the user determines whether there is a part on the workpiece W that needs to be scanned. If there is a part on the workpiece W that needs to be scanned, the process proceeds to step SA2, and the workpiece W is placed on the mounting surface 142 in a third placement posture and scanned by the measuring unit 100. By repeating this process, all necessary parts can be scanned. If it is determined in step SA15 that no part is scanned, a full-surface scan model is obtained in step SA16. Note that a full-surface scan model is not required, and any model can be used as long as the user can scan the necessary parts.

[0191] Next, we will refer to Figure 17The flowchart describes an example of scanning processing in the presence of a reference model with CAD data or measured 3D data of a workpiece W. In step SG1, the reading unit 292 of the analysis module 290 reads the CAD model (reference model) from the storage device 240, and in step SG2, the CAD model is acquired. In step SG3, the scanning process is performed in a manner similar to... Figure 6 The evaluation value calculation and attitude candidate suggestion are performed in step SA4. In step SG3, since CAD data exists, the attitude calculation unit 293 analyzes the CAD data read by the reading unit 292. The attitude calculation unit 293 estimates the amount of data acquired (additional data amount) by analyzing the CAD data, and calculates the recommended placement attitude based on the estimated data amount. In addition, the attitude calculation unit 293 analyzes the CAD data read by the reading unit 292 and estimates the overlap area between the 3D data and the CAD data acquired when the device is placed in the recommended placement attitude. The attitude calculation unit 293 calculates the recommended placement attitude based on the estimated overlap area.

[0192] In step SG4, for example, the user selects a candidate pose on the user interface screen 700, such as... Figure 8 As shown. In step SG5, the user determines whether the candidate pose displayed on the display unit 400 is the desired pose. If the pose is not the desired pose, the process proceeds to step SG6, and the placement pose of the workpiece W is adjusted according to the computer graphics. In step SG7, the overlap region estimation unit 295 estimates the amount of overlap region in the adjusted placement pose, and the additional data quantity estimation unit 296 estimates the additional data quantity. The analysis module 290 recalculates the evaluation index based on the estimated overlap region and the estimated additional data quantity, and presents the evaluation index to the user.

[0193] If the orientation is determined to be the desired orientation in step SG5, the process proceeds to step SG8, ​​and the analysis module 290 generates a computer-generated graphic workpiece (model) with the determined placement orientation. The display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400.

[0194] In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W along with the mounting surface 142, and the data acquisition unit 291 acquires a real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until it overlaps with the workpiece in the computer graphics.

[0195] In step SG10, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG4, and another placement orientation is selected. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the first placement orientation, the process proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement orientation. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the first three-dimensional data and the CAD data used as a reference model. In step SG14, the analysis module 290 acquires the aligned CAD model.

[0196] In step SG15, evaluation value calculation and attitude candidate suggestions are performed. In step SG16, the user selects a candidate attitude, and then processing proceeds to step SG17. If the attitude is not the desired attitude, adjustments are made in step SG18, the evaluation value is updated in step SG19, and then processing proceeds to step SG17. If the attitude is the desired attitude, processing proceeds to step SG20, and the display control unit 255 overlays the computer graphics artifact generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. Here, the computer graphics artifact can be the first 3D data or reference model acquired in step SG12. Alternatively, the first 3D data and the reference model can be displayed simultaneously on the display unit 400, or they can be switched and displayed interchangeably.

[0197] In step SG21, while viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the workpiece in the computer graphics.

[0198] In step SG22, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG16, and another placement posture is selected. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement posture (second placement posture). In step SG24, the alignment unit 290A performs alignment with the acquired second 3D data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the 3D model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a part that needs to be scanned. If there is a part that needs to be scanned, the process proceeds to step SG12. If there is no part to be scanned, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model for which alignment processing was performed by the alignment unit 290A in step SG26. In addition, the full-surface scan model and the CAD model can be displayed on the display unit 400 in an aligned state.

[0199] In this implementation plan, such as Figure 19 As shown, alignment unit 290A performs alignment between first 3D data A and second 3D data B based on a first alignment parameter (first positional relationship), where the first alignment parameter is the positional relationship between the first 3D data A and the second 3D data B. Combination unit 290G combines the first 3D data (first mesh data) A and the second 3D data (second mesh data) B acquired by data acquisition unit 291 to generate combined 3D data AB. Subsequently, when data acquisition unit 291 acquires third 3D data C, alignment unit 290A performs alignment between the combined 3D data AB obtained by combining the first 3D data A and the second 3D data B, and the third 3D data C, based on a second alignment parameter (second positional relationship), where the second alignment parameter is the positional relationship between the combined 3D data AB and the third 3D data C. Combination unit 290G combines the combined 3D data AB with the third 3D data (third mesh data) C acquired by data acquisition unit 291 to generate combined 3D data ABC. At this time, for example, when the editing unit 290H receives input for editing the position or shape of at least one of the first, second, and third grid data, the storage device 240 stores the first, second, and third grid data. Editing the shape of the grid data includes, for example, removing a portion of the point cloud.

[0200] Furthermore, when the data acquisition unit 291 acquires the fourth three-dimensional data D, the alignment unit 290A performs alignment between the combined three-dimensional data ABC and the fourth three-dimensional data D based on a third alignment parameter (third positional relationship). The third alignment parameter is the positional relationship between the combined three-dimensional data ABC and the fourth three-dimensional data D. The combination unit 290G combines the combined three-dimensional data ABC with the fourth three-dimensional data (fourth mesh data) D acquired by the data acquisition unit 291 to generate combined three-dimensional data ABCD. When the editing unit 290H receives input for editing the position or shape of the fourth mesh data, the storage device 240 stores the fourth mesh data.

[0201] As described above, users can obtain the 3D data of the entire workpiece W by sequentially combining two 3D data sets. Performing sequential combination ensures that the processing load for transforming the mesh data from the point cloud remains constant, as does the processing load for combining the mesh data to which sparsification is applied. For example, the processing load can be reduced compared to processing all the original point clouds that are jointly transformed to form the combined 3D data ABCD. Note that the combined 3D data ABCD is not limited to the method of sequentially adding a fourth 3D data set to the combined 3D data ABCD; the combined 3D data ABCD can be generated by combining the first, second, third, and fourth 3D data sets based on the positional relationships between multiple 3D data sets.

[0202] The display control unit 255 displays the combined three-dimensional data generated by the combination unit 290G on the display unit 400, and displays the first three-dimensional data and the second three-dimensional data in a recognizable manner on the display unit 400. That is, the display control unit 255 generates data such as... Figure 20 The user interface screen 800 is shown and displayed on the display unit 400. The user interface screen 800 provides a first display area 801 for displaying first 3D data (first mesh data), a second display area 802 for displaying second 3D data (second mesh data), and a third display area 803 for displaying combined 3D data (combined mesh data). Since the first display area 801 and the second display area 802 are distinct from each other, the first mesh data and the second mesh data before being reassembled by the combining unit 290G can be displayed in a recognizable manner on the display unit 400. The X, Y, and Z coordinate systems are displayed in the first display area 801, the second display area 802, and the third display area 803, respectively.

[0203] In this embodiment, the combined processing of the first and second three-dimensional data performed by the combining unit 290G can be edited. That is, the analysis module 290 includes an editing unit 290H, which edits the combined processing of the first and second three-dimensional data performed by the combining unit 290G. The editing unit 290H accepts input for editing the position or shape of at least one of the first and second three-dimensional data. The editing unit 290H can edit the position of the first three-dimensional data in any of the X, Y, and Z directions. The second three-dimensional shape data can be edited similarly.

[0204] Editing unit 290H accepts input for editing first or second 3D data, and edits the first or second 3D data based on the accepted input. Then, combining unit 290G recombines the first and second 3D data based on the input received by editing unit 290H. At this time, alignment unit 290A can perform alignment between the first and second 3D data based on a first alignment parameter that represents the positional relationship between the first 3D data A and the second 3D data B. Therefore, it eliminates the need for the user to accept new alignment specifications, improving convenience. Similarly, when input for editing third and fourth 3D data is received, editing unit 290H edits the third and fourth 3D data based on the accepted input. Then, combining unit 290G updates and regenerates the combined 3D data based on the input received by editing unit 290H.

[0205] For example, when the first three-dimensional data and the second three-dimensional data are combined, the first three-dimensional data and the second three-dimensional data with different initial positions are aligned on the background of the display processing of the user interface screen 810, as an example, such as Figure 21 As shown, the shapes are then combined by the combination unit 290G. A combination result display button 811, an additional shape display button 812, and an original shape display button 813 are provided on the user interface screen 810.

[0206] When the user operates the combination result display button 811, the display control unit 255 displays the combined three-dimensional data at the time point when the combination of the first and second three-dimensional data is completed, and displays the combined three-dimensional data in the display area 814 of the user interface screen 810. This combined three-dimensional data is the combination result. The user can confirm the quality of the combination result by viewing the combined three-dimensional data. When there is a defective part, the occurrence part and its causes can also be specified. When specifying, when the user operates as follows... Figure 22 When the additional shape display button 812 is displayed on the user interface screen 810, the display control unit 255 displays the second three-dimensional data in the display area 814 of the user interface screen 810, and when the user operates as shown... Figure 21When the original shape display button 813 is pressed on the user interface screen 810, the display control unit 255 displays the first three-dimensional data in the display area 814 of the user interface screen 810. As described above, since the first and second three-dimensional data can be displayed separately through the user's switching operation, if a problem is found in the combined result of the combined three-dimensional data, the user can select either the first or the second three-dimensional data, and partially cut off or repair the three-dimensional data while confirming the selected three-dimensional data. Note that the combined three-dimensional data, the first three-dimensional data, and the second three-dimensional data can be displayed on one user interface screen.

[0207] exist Figure 23 The user interface screen 810 shown provides an edit button 815. When the user operates the edit button 815, the display control unit 255 generates and displays, as shown in the image. Figure 20 The user interface screen 820 shown is for data editing and accepts instructions from the editing unit 290H to edit 3D data. The user interface screen 820 displays the following areas: a display area 821, in which instructions such as partial cutting or repair are received, and the 3D data to be edited by the editing unit 290H is displayed based on these instructions; a process display area 822, in which the editing process is displayed; and a setting area 823, in which the specified method, selection method, and editing method of the editing area can be set. When editing such as partial cutting or repair of 3D data is performed on the user interface screen 820, the edited 3D data is stored in the storage device 240.

[0208] If the edited 3D data is, for example, the second 3D data, the alignment unit 290A performs alignment between the edited second 3D data and the first 3D data again. At this time, the alignment unit 290A can perform alignment between the edited second 3D data and the first 3D data based on a first alignment parameter. Then, the combination unit 290G generates combined 3D data by combining the edited second 3D data and the first 3D data. The combined 3D data generated in this way is displayed in the display area 814.

[0209] Furthermore, if the quality of the combined 3D data deteriorates due to editing of the second 3D data, the user issues a re-acquisition command to reacquire the second 3D data acquired by the data acquisition unit 291. When the user operates... Figure 20When the recapture button 816 is pressed on the user interface screen 810, the receiving unit 298 receives a recapture instruction to recapture the second 3D data. In this case, the data acquisition unit 291 acquires the fifth grid data (grid data for updating) based on the recapture instruction received by the receiving unit 298, and acquires the first grid data from the storage device 240, while the alignment unit 290A performs alignment between the first grid data and the fifth grid data. The fifth grid data is acquired to update the second grid data. Therefore, the alignment unit 290A can perform alignment between the grid data for updating (i.e., the fifth grid data) acquired as a replacement for the second grid data and the first grid data based on a first alignment parameter. Figure 19 When the alignment button 817 is pressed on the user interface screen 810 shown, alignment begins.

[0210] When the receiving unit 298 receives a reacquisition command, the display control unit 255 can also display a recommended placement posture based on the first three-dimensional data on the display unit 400. The recommended placement posture is calculated by the posture calculation unit 293 as described above. Since the recommended placement posture is displayed on the display unit 400, a large amount of additional data can be used to capture the placement posture.

[0211] Then, the combination unit 290G combines the first grid data and the fifth grid data aligned by the alignment unit 290A and updates the combined 3D data. The updated combined 3D data is displayed in the display area 814. As described above, the first alignment parameter, which serves as the alignment parameter for the first grid data and the second grid data, can be used to combine the first grid data and the fifth grid data aligned by the alignment unit 290A. That is, the first alignment parameter, which serves as the alignment parameter for the first 3D data A and the second 3D data B, is associated with the combined 3D data AB, and when a reacquisition instruction to reacquire the second 3D data is received, the alignment between the first 3D data A and the newly acquired 3D data B', rather than the second 3D data B, can be performed using the first alignment parameter associated with the combined 3D data AB to generate new combined 3D data AB'. In the case of updated combined 3D data, the combination unit 290G can discard the combined 3D data before updating. Furthermore, when the first grid data and the third grid data aligned by the alignment unit 290A are combined, the combination unit 290G can discard the second grid data, which is the grid data before the update. In other words, since grid data that does not constitute the combined 3D data is unnecessary, its wasteful use in storage space can be mitigated by discarding it. User confirmation is required when discarding unnecessary data.

[0212] Furthermore, alignment unit 290A performs alignment between the combined 3D data AB' obtained by combining the first and fifth mesh data and the third mesh data. At this time, alignment unit 290A can perform alignment between the combined mesh data AB' obtained by combining the first and fifth mesh data and the third mesh data based on a second alignment parameter. Then, combination unit 290G combines the combined mesh data AB' aligned by alignment unit 290A and the third mesh data to generate combined mesh data AB'C. The generated combined mesh data AB'C is displayed in display area 814.

[0213] On the other hand, since the first, fifth, and third 3D data constituting the combined 3D data are automatically stored as necessary data in the storage device 240, at least one of the first, fifth, and third 3D data can be read later. For noise or misalignment that is ignored during repeated combination processing, the individual scan results can be automatically stored, and the combination processing can be performed again. For example, as... Figure 24 As shown, in the case of generating combined 3D data AB, then generating combined 3D data ABC, and then generating combined 3D data ABCD, and in the case where the third 3D data C is combined with the fourth 3D data D without noticing noise contamination, the step can be recovered from the middle by recombining the third 3D data C with the combined 3D data AB.

[0214] As described above, the combined 3D data may include information indicating the combination order of the 3D data, and this information can be stored in the storage device 240 in a state associated with the combined 3D data. The combination unit 290G combines the first grid data and the third grid data aligned by the alignment unit 290A based on the information indicating the combination order corresponding to the second 3D data.

[0215] (Alignment of CAD data)

[0216] When the actual workpiece W is arranged to match the CAD data displayed on the display unit 400, the user needs to move the workpiece W on the rotating stage 143. However, the present invention is not limited to this, and the CAD data can be moved to match the actual workpiece W. That is, it may be difficult for the user to arrange the actual workpiece W to match the CAD data while viewing the display unit 400, because the arrangement needs to be performed simultaneously while viewing both the display unit 400 and the rotating stage 143. Furthermore, since the camera capturing the actual workpiece W and the user's line of sight are facing each other, the user performs the operation while viewing a mirror image, which leads to difficulties. Therefore, to improve user convenience, a function (alignment function) can be installed that enables the actual workpiece and CAD data to be aligned by the user moving the CAD data while viewing the display unit 400 without moving the actual workpiece W on the 3D scanner 1.

[0217] In the following text, reference will be made to Figure 25 The flowchart shown illustrates the details of the alignment function. In the following description, alignment is referred to as overlay alignment, and CAD data is referred to as "virtual objects." In step S100, the controller 200 reads the center position of the rotating platform 143 and the CAD data. In step S101, the controller 200 calculates a virtual ground based on the center position of the rotating platform 143 read in step S100, displays the virtual ground on the display unit 400, and displays virtual objects on the display unit 400 based on the CAD data.

[0218] In step S102, the controller 200 determines whether the mouse button of the operation unit 250 is pressed near the display position of the virtual object. If the mouse button is not pressed near the display position of the virtual object, the overlay alignment is terminated. On the other hand, if the mouse button is pressed near the display position of the virtual object, the process proceeds to step S103, and the controller 200 performs virtual object rotation processing by dragging the mouse.

[0219] Reference Figure 24The following flowchart describes the virtual object rotation process. In step S200, the controller 200 determines whether the rotation button of the operation unit 250 is pressed. If the controller 200 determines that the rotation button of the operation unit 250 is pressed, in step S201, the controller 200 determines whether the movement direction of the mouse of the operation unit 250 is close to horizontal. If the movement direction of the mouse is not close to horizontal, the process proceeds to step S202. On the other hand, if the movement direction of the mouse is close to horizontal, the process proceeds to step S203, and the controller 200 fixes the rotation axis of the virtual object to an axis perpendicular to the virtual ground. In step S204, the controller 200 extracts the horizontal component of the mouse movement. In step S205, the controller 200 calculates the posture of rotation based on the horizontal component extracted around the fixed axis. In step S206, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S207, the display of the virtual object is updated. In step S208, the controller 200 determines whether the mouse button is released. If the mouse button is not released, the process returns to step S201.

[0220] In step S202, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to vertical. If the mouse movement direction is close to vertical, the process proceeds to step S209, and the controller 200 fixes the rotation axis of the virtual object to the right and left axes in the line of sight. In step S210, the controller 200 extracts the vertical component of the mouse movement. In step S211, the controller 200 calculates the posture of rotation based on the vertical component extracted around the fixed axis, and the process proceeds to step S206.

[0221] If it is determined in step S202 that the mouse movement direction is not close to vertical, the process proceeds to step S212, and the amount of mouse movement is extracted. In step S213, the orientation for rotation in any direction based on the amount of mouse movement is calculated, and the process proceeds to step S206.

[0222] If no result is found in step S200, the process proceeds to step S214, and the mouse movement amount is extracted. In step S215, the orientation for rotation in any direction based on the mouse movement amount is calculated, and the process proceeds to step S206.

[0223] The process then proceeded to... Figure 26In step S104, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S105, the controller 200 extracts surfaces of the virtual object that are close to the virtual ground. In step S106, the controller 200 calculates the grounding degree between the surface of the virtual object and the virtual ground. In step S107, the controller 200 determines whether there are extracted surfaces for which the grounding degree has not been calculated. If there are extracted surfaces for which the grounding degree has not been calculated, the next surface is selected in step S108, and the process proceeds to step S106.

[0224] If no extraction surface has been found whose grounding degree has not been calculated, the process proceeds to step S109, and the controller 200 selects the surface with the highest grounding degree. In step S110, the controller 200 calculates the orientation of the selected surface and the virtual ground ground. In step S111, the display of the virtual object is updated to the calculated orientation.

[0225] Figure 27 The flowchart illustrates the process of adjusting the positional relationship between a virtual object and the virtual ground. In step S300, the controller 200 calculates the distance between the virtual object and the virtual ground. In step S301, the controller 200 determines whether the virtual object is buried in the virtual ground. If it is determined to be yes in step S301, the process proceeds to step S302, pushing the virtual object upwards above the virtual ground, and then proceeds to step S303. If it is determined not to be yes in step S301, the process proceeds to step S303. In step S303, the controller 200 determines whether the virtual object floats off the virtual ground. If it is determined to be yes in step S303, the process proceeds to step S304, and the virtual object is placed on the virtual ground.

[0226] Figure 28 The flowchart illustrates the stage surface detection process. In step S400, the controller 200 measures and acquires the three-dimensional shape of the rotating stage 143. In step S401, the controller 200 obtains the position of the stage surface (mounting surface 142) from the three-dimensional shape of the rotating stage 143. In step S402, the controller 200 calculates the height information of the stage surface relative to the camera position based on the acquired planar position. In step S403, the calculated stage surface height information is stored in a storage device 240 or the like.

[0227] In step S404, the controller 200 determines whether to save the dedicated chart. The dedicated chart may be, for example, a calibration plate. If it is determined not to save in step S404, the process proceeds to step S405, and the center position of the stage is calculated based on the irregular shape on the stage surface having known design values. If it is determined to save in step S404, the process proceeds to step S406, and the dedicated chart is rotated and measured from multiple directions to calculate the center position of the stage. In step S407, the center position of the stage is stored in a storage device 240 or the like.

[0228] Figure 24 This is a flowchart illustrating an example of the alignment function processing when performing rotation and movement of virtual objects. Steps S500, S501, and S502 are respectively related to... Figure 24 Steps S100, S101, and S102 are the same. Furthermore, steps S505 to S512 are respectively... Figure 28 Steps S104 to S111 are the same.

[0229] In step S503, the virtual object is rotated and moved by dragging with the mouse. Figure 28 This is a flowchart illustrating an example of the processing when rotating and moving a virtual object. In step S600, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to horizontal. If the mouse movement direction is not close to horizontal, the process proceeds to step S601. On the other hand, if the mouse movement direction is close to horizontal, the process proceeds to step S602, and the controller 200 fixes the rotation axis of the virtual object to an axis perpendicular to the virtual ground. In step S603, the controller 200 extracts the horizontal component of the mouse movement. In step S604, the controller 200 calculates the posture of rotation based on the horizontal component extracted around the fixed axis. In step S605, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S606, the display of the virtual object is updated. In step S607, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S603.

[0230] In step S601, the controller 200 determines whether the movement direction of the mouse in the operation unit 250 is close to vertical. If the mouse movement direction is close to vertical, the process proceeds to step S608, and the controller 200 fixes the rotation axis of the virtual object to the right and left axes in the line of sight. In step S609, the controller 200 extracts the vertical component of the mouse movement. In step S610, the controller 200 calculates the rotation posture based on the vertical component extracted around the fixed axis, and the process proceeds to step S611. In step S611, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S612, the display of the virtual object is updated. In step S613, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S609.

[0231] If the mouse movement direction is not close to vertical, the process proceeds to step S614, and the controller 200 extracts the amount of mouse movement. In step S615, the controller 200 calculates the rotational posture in any direction based on the amount of mouse movement, and the process proceeds to step S616. In step S616, the controller 200 adjusts the positional relationship between the virtual object and the virtual ground. In step S617, the display of the virtual object is updated. In step S618, the controller 200 determines whether the mouse button has been released. If the mouse button has not been released, the process returns to step S614.

[0232] exist Figure 30 In step S513, the controller 200 calculates the distance between the virtual object and the virtual tilting platform. In step S514, surfaces close to the virtual tilting platform are extracted from the surfaces of the virtual object. In step S515, the grounding degree between the surface of the virtual object and the virtual tilting platform is calculated. In step S516, it is determined whether there are extracted surfaces whose grounding degree has not been calculated. If there are extracted surfaces that have not been calculated, the process proceeds to step S517, the next surface is selected, and the process proceeds to step S515. When there are no extracted surfaces that have not been calculated, the process proceeds to step S518, and the surface with the highest grounding degree is selected. In step S519, the orientation of the selected surface grounding with the virtual tilting platform is calculated. In step S520, the positional relationship between the virtual object, the virtual ground, and the virtual tilting platform is adjusted.

[0233] Figure 25 This is a flowchart illustrating another example of the processing in the case of performing rotation and movement of a virtual object. Steps S700 to S705 are respectively related to... Figure 25 Steps S200 to S205 are the same. Furthermore, steps S707 to S715 are respectively... Figure 31Steps S207 to S215 are the same. In step S706, the controller 200 adjusts the positional relationship between the virtual object, the virtual ground, and the virtual tilting platform. The virtual tilting platform virtually refers to the tilting platform set on the rotating stage 143. The tilting platform set on the rotating stage 143 is configured such that, for example, the tilt angle relative to the horizontal plane can be changed in various ways, and the workpiece W can be tilted by mounting the workpiece W on the tilting platform.

[0234] Figure 6 This is a flowchart illustrating an example of the process when adjusting the positional relationship between a virtual object, a virtual ground, and a virtual tilting platform. In step S800, the controller 200 calculates the distance between the virtual object, the virtual ground, and the virtual tilting platform. In step S801, the controller 200 determines whether the virtual object is buried in the virtual ground or the virtual tilting platform. If it is determined to be yes in step S801, the process proceeds to step S802, pushing the virtual object upwards above the virtual ground or the virtual tilting platform, and then proceeds to step S803. If it is determined not to be yes in step S801, the process proceeds to step S803. In step S803, the controller 200 determines whether the virtual object is floating from the virtual ground or the virtual tilting platform. If it is determined to be yes in step S803, the process proceeds to step S804, and the virtual object is grounded to the virtual ground or the virtual tilting platform.

[0235] As described above, the same determinations as those for the rotating stage 143 are performed in the space where the virtual object exists, and physical constraints are introduced to ground the virtual object to the rotating stage 143. Therefore, since rotation and translation are limited to the same degrees of freedom as the actual workpiece W, alignment by the user becomes easy.

[0236] Furthermore, although unstable postures may occur during mouse dragging on the operation unit 250, a stable posture is calculated at the moment the mouse is released, and this posture is automatically corrected to one where the grounding degree between the rotating stage 143 and the virtual object increases. Since the actual possible postures of the workpiece W also exist with a finite number of degrees of freedom, alignment becomes easier.

[0237] Furthermore, rotation operations performed by dragging the mouse through the operation unit 250 are restricted, allowing rotation to occur only relative to one of the workpiece's roll, pitch, and yaw axes at a time. Therefore, virtual objects on the rotary stage 143 can rotate while maintaining the rotary stage 143 in a grounded state, and alignment becomes easier.

[0238] Furthermore, when the rotation of the rotating stage 143 and the display of the virtual object are interlocked, it may be difficult to capture the positional relationship in the depth direction when capturing images from a fixed camera, as one of the difficulties in alignment. On the other hand, since the rotation of the rotating stage 143 and the display state of the virtual object are interlocked, when capturing objects on the rotating stage 143 from different angles, the positional relationship with the virtual object can be captured, and alignment can be performed using the captured images.

[0239] Furthermore, in the case of a structure that allows the rotating stage 143 to tilt as described above, tilt information, etc., can be interlocked with the alignment function. For example, the application provides a section for inputting tilt information (tilt angle information) of the rotating stage 143, and the controller 200 acquires this information. Therefore, the virtual object can be interlocked with the virtual space, and the virtual object can take into account the tilt angle of the rotating stage 143 to adopt a posture.

[0240] In addition, without Figure 17 In the case of the reference model shown, the processing can proceed to step SA5 without calculating the evaluation value and the recommended placement posture calculated in SA4. In this case, in step SA5, for example, the posture calculation unit 293 calculates a posture as the recommended placement posture, which is obtained by rotating a portion of the scanned model of the workpiece W placed in the first placement posture, acquired by the data acquisition unit 291, around a predetermined axis (such as a rotation axis horizontal with respect to the mounting surface 140) by a constant rotation angle. Then, the display control unit 255 can display the recommended placement posture calculated by the posture calculation unit 293 on the display unit 400. Here, for example, the posture calculation unit 293 can calculate the recommended placement posture by estimating the rotation axis horizontal with respect to the mounting surface 140 and rotating the workpiece W around the rotation axis by a specific rotation angle, such as 60 degrees or 90 degrees. In step SA6, the user determines whether the posture displayed on the display unit 400 is the desired posture. If the posture is not the desired posture, the processing proceeds to step SA7, and the placement posture of the workpiece W is adjusted on the computer graphics. In step SA8, the attitude calculation unit 293 can calculate and update the evaluation value of the adjusted attitude, but can skip to step SA6.

[0241] If the orientation is determined to be the desired orientation in step SA6, the process proceeds to step SA9, and the analysis module 290 generates a computer-generated workpiece (model) with the determined placement orientation. In step SA10, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires a real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the computer-generated workpiece.

[0242] In step SA11, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SA5, and the placement posture is adjusted. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the second placement posture, the process proceeds to step SA12, and the measurement unit 100 scans the workpiece W placed in the second placement posture. In step SA12, the light projection unit 110 of the measurement unit 100 illuminates the workpiece W placed in the second placement posture with measurement light. The light receiving unit 120 of the measurement unit 100 receives the measurement light reflected by the workpiece W. The light receiving signal output from the light receiving unit 120 is received by the point cloud acquisition unit 263a, and second point cloud data of the workpiece W is generated. The mesh data generation unit 263b acquires the second point cloud data acquired by the point cloud acquisition unit 263a, processes the acquired second point cloud data, and converts the data into second mesh data. In step SA12, the second mesh data obtained through the processing in step SA12 is acquired as the second three-dimensional data (step SA13). The second three-dimensional data is stored in the storage device 240.

[0243] In step SA14, the alignment unit 290A, included in the analysis module 290, aligns the first three-dimensional data (three-dimensional data of the workpiece placed in a first placement posture) and the second three-dimensional data acquired by the data acquisition unit 291 based on the relative positional relationship calculated by the arithmetic unit 294. During this alignment, the overlapping area extracted by the extraction unit 290B, included in the analysis module 290, is used.

[0244] In addition, for the existence of such Figure 32In the example of scanning processing using a reference model of the CAD data or measured 3D data of the workpiece W, the processing can proceed to step SG4 without the posture calculation unit 293 calculating the evaluation value and the recommended placement posture in step SG3. In this case, in step SG4, the display control unit 255 displays the CAD data or measured 3D data on the display unit 400. The posture of the CAD data or measured 3D data at this time can be, for example, the posture of the CAD data or measured 3D data relative to the scanner, determined by matching the coordinate system of the CAD data or measured 3D data with the scanner coordinate system. In step SG4, the user can adjust the posture of the CAD data or measured 3D data based on this posture. In step SG5, the user determines whether the candidate posture displayed on the display unit 400 is the desired posture. If the posture is not the desired posture, the processing proceeds to step SG6, and the placement posture of the workpiece W is adjusted according to the computer graphics. In step SG7, the posture calculation unit 293 calculates and updates the evaluation value for the adjusted posture, but this process can be skipped, and the processing can proceed to step SG5.

[0245] If the orientation is determined to be the desired orientation in step SG5, the process proceeds to step SG8, ​​and the analysis module 290 generates a computer-generated graphic workpiece (model) with the determined placement orientation. The display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. In step SG9, the user mounts the workpiece W at a temporary position on the mounting surface 142. Then, the light receiving unit 120 captures the workpiece W together with the mounting surface 142, and the data acquisition unit 291 acquires the real-time image and displays it on the display unit 400. While viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the computer-generated graphic workpiece.

[0246] In step SG10, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG4, and the placement posture is adjusted. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, since the workpiece W is in the first placement posture, the process proceeds to step SG11, and the measurement unit 100 scans the workpiece W placed in the first placement posture. In step SG12, first three-dimensional data is acquired. In step SG13, the alignment unit 290A performs alignment between the measured first three-dimensional data and the CAD data used as a reference model. In step SG14, the analysis module 290 acquires the aligned CAD model or the measured three-dimensional data.

[0247] In step SG15, a candidate pose is suggested, and processing proceeds to step SG16. For example, for a candidate pose, the pose of the CAD data or measured 3D data can be changed according to a predetermined rule regarding the desired pose determined in step SG5. As an example of the prescribed rule, there is a case where the CAD data or measured 3D data is rotated by 60 degrees relative to the desired pose determined in step SG5, with one axis of the scanner coordinate system serving as the axis of rotation. Alternatively, for example, the user can select a rotation angle range within 90 degrees. In step SG16, the user selects a candidate pose, and processing proceeds to step SG17. If the pose is not the desired pose, after adjustment in step SG18, the evaluation value is updated in step SG19, and processing proceeds to step SG17. Step SG19 can be skipped.

[0248] If the pose is the desired pose, the processing proceeds to step SG20, and the display control unit 255 overlays the computer-generated graphic workpiece generated by the analysis module 290 onto the real-time image acquired by the data acquisition unit 291, and displays the overlaid image on the display unit 400. Here, the computer-generated graphic workpiece can be the first 3D data or reference model acquired in step SG12. Furthermore, the first 3D data and the reference model can be displayed simultaneously on the display unit 400, or they can be switched and displayed separately.

[0249] In step SG21, while viewing the real-time image on the display unit 400, the user moves or rotates the actual workpiece W until the actual workpiece W overlaps with the workpiece in the computer graphics.

[0250] In step SG22, the user determines whether the actual workpiece W is mounted to overlap with the workpiece in the computer graphics. If the actual workpiece W cannot be mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG16, and another placement orientation is selected. If the actual workpiece W is mounted to overlap with the workpiece in the computer graphics, the process proceeds to step SG23, and the measurement unit 100 scans the workpiece W in the placement orientation (second placement orientation). In step SG24, the alignment unit 290A performs alignment using the acquired second 3D data, and in step SG25, a partial scan model is acquired. In step SG26, the alignment unit 290A performs alignment processing of the 3D model, and the combination unit 290G performs combination processing. In step SG27, the user determines whether there is a part that needs to be scanned. If there is a part that needs to be scanned, the process proceeds to step SG12. If there is no part to be scanned, a full-surface scan model is acquired in step SG28. The full-surface scan model acquired in step SG28 can be output together with the CAD model or the measured 3D data, wherein the alignment processing is performed by the alignment unit 290A in step SG26. In addition, full-surface scan models and CAD models or measured 3D data can be displayed on display unit 400 in an aligned state.

[0251] In this embodiment, CAD data or measured 3D data is used as a reference model to obtain the combined 3D data of workpiece W. However, even after obtaining the combined 3D data of workpiece W without a reference model, alignment with the CAD data or measured 3D data can still be performed. The reading unit 292 reads the 3D data and CAD data of workpiece W from the storage unit 240. Subsequently, the resolution conversion unit 290E, read by the reading unit 292, converts at least the 3D data of workpiece W into 3D data with a first resolution lower than the resolution obtained by the data acquisition unit 291. Then, the shape feature extraction unit 290C extracts the partial regions (key points) to be used for calculating shape features from the 3D data and CAD data at the first resolution. Subsequently, the analysis module 290 samples the periphery of the partial regions, and the shape feature extraction unit 290C calculates shape feature vectors from the vicinity of the points sampled by the analysis module 290. Subsequently, the analysis module 290 compares the shape feature vectors of the 3D data obtained by the shape feature extraction unit 290C with the shape feature vectors of the CAD data, and the alignment unit 290A aligns the 3D data with the CAD data based on the comparison of the shape feature vectors. The shape feature vectors of candidate pairs can be extracted by comparing their shape feature vectors, and the positional relationships (rotation and translation) between point clouds can be obtained by using the points of the candidate pairs through RANSAC or deep learning-based methods.

[0252] Furthermore, after the alignment between the combined 3D data and CAD data is completed, the analysis module 290 can automatically perform a comparison between the CAD data and the 3D data. Figure 33A An example of the analysis is shown, in which the dimensional differences between the scanned data and CAD data of workpiece W are calculated by comparing their three-dimensional shapes, and a color map is displayed by assigning colors corresponding to the degree of difference. The analysis module 290 calculates the shape difference between the three-dimensional data of workpiece W obtained by the data acquisition unit 291 and the CAD data for each grid based on the alignment result of the alignment unit 290A and the CAD data read by the reading unit 292, and assigns color information corresponding to the degree of difference to each grid. Then, the display control unit 255 displays the color map on the display unit 400, where a color is assigned to each grid based on the color information of at least one of the three-dimensional data and the CAD data. Additionally, after alignment by the alignment unit 290A, the analysis module 290 can compare the three-dimensional data with the CAD data, and the display control unit 255 can display the color map on the display unit 400 based on an instruction to assign analysis settings or start comparison analysis received by the receiving unit 298. In this case, the user can assign detailed settings for the comparison.

[0253] in addition, Figure 33B An example of performing cross-sectional measurements on three-dimensional data of workpiece W is shown. The user specifies the surface on which the cross-sectional measurement will be performed on the three-dimensional data of workpiece W, and indicates the type and location of the analysis tool to be performed on the cross-section. The analysis tool may be, for example, a measurement of the distance between two points and the angle formed by surfaces. Analysis module 290 accepts the instruction and performs analysis on the specified surface of the three-dimensional data of workpiece W based on the instruction.

[0254] ​ An example of performing cross-sectional measurements on CAD data is shown. The user specifies the surface on which the cross-sectional measurement is performed on the CAD data and indicates the type and assignment location of the analysis tool to be performed on the cross-section. The analysis module 290 accepts the instructions and performs analysis on the indicated surface of the CAD data based on the instructions. The analysis module 290 can compare the results obtained by performing cross-sectional measurements on the three-dimensional data with the results obtained by performing cross-sectional measurements on the CAD data, and the display control unit 255 displays the comparison results on the display unit 400.

[0255] Furthermore, cross-sectional measurements can be performed on the data where the 3D data and CAD data of workpiece W are aligned. Alignment unit 290A aligns the 3D data and CAD data of workpiece W to obtain the data where the 3D data and CAD data of workpiece W are aligned. The user specifies the surface on which the cross-sectional measurement will be performed on the data where the workpiece W and CAD data are aligned, and indicates the type and location of the analysis tool to be performed on the cross-section. Analysis tools may include, for example, measurements such as the distance between two points and the angle formed by surfaces. Analysis module 290 receives instructions and performs analysis on the indicated surface of the data where the 3D data and CAD data of workpiece W are aligned, based on the instructions. Analysis module 290 can perform cross-sectional measurements on the data where the 3D data and CAD data are aligned. For example, a comparison can be performed based on the dimensional differences between the CAD data and the scanned data. Display control unit 255 can display the comparison results on display unit 400.

[0256] The above embodiments are merely illustrative in all respects and should not be construed as limiting. Furthermore, all modifications and alterations falling within the equivalent scope of the claims are within the scope of this invention. As described above, this invention can be used to generate multiple 3D data sets for various workpieces.

Claims

1. A 3D scanner, which generates combined 3D data of a workpiece by generating multiple 3D data of a workpiece placed in different orientations and combining the multiple 3D data, the 3D scanner comprising: The data acquisition unit acquires first three-dimensional data and second three-dimensional data. The first three-dimensional data includes shape information and texture information of the workpiece placed in a first placement posture, and the second three-dimensional data includes shape information and texture information of the workpiece placed in a second placement posture. The first extraction unit includes an input layer of a neural network, multiple intermediate layers, and an output layer, wherein the input layer receives input from the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit, the multiple intermediate layers extract shape features based on the input received by the input layer, and the output layer outputs the shape features extracted by the intermediate layers. The second extraction unit extracts image features from the texture information included in each of the three-dimensional data in the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit. An alignment unit performs alignment between the first 3D data and the second 3D data based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit. as well as A combining unit that combines the first three-dimensional data and the second three-dimensional data aligned by the alignment unit to generate the combined three-dimensional data.

2. The 3D scanner according to claim 1 further includes: A resolution conversion unit converts the first three-dimensional data and the second three-dimensional data acquired by the data acquisition unit into three-dimensional data with a first resolution lower than that acquired by the data acquisition unit. The first extraction unit extracts the shape features from the shape information included in each of the first three-dimensional data and the second three-dimensional data having the first resolution, which are converted by the resolution conversion unit.

3. The three-dimensional scanner according to claim 2, wherein The resolution conversion unit converts the first 3D data and the second 3D data acquired by the data acquisition unit into 3D data with a second resolution. This second resolution is lower than the resolution acquired by the data acquisition unit but higher than the first resolution. The second extraction unit extracts the image features from the texture information included in each of the first three-dimensional data and the second three-dimensional data having the second resolution, which are converted by the resolution conversion unit.

4. The three-dimensional scanner of claim 2, wherein the first extraction unit extracts a partial region for calculating the shape feature from the first three-dimensional data and the second three-dimensional data having the first resolution, and extracts the shape feature for each extracted partial region.

5. The 3D scanner of claim 4, wherein the second extraction unit designates a region corresponding to each partial region extracted by the first extraction unit from the first 3D data and the second 3D data before conversion by the resolution conversion unit, and extracts the image features from the designated corresponding region.

6. The three-dimensional scanner according to claim 1, wherein, The alignment unit performs global alignment based on the shape features extracted by the first extraction unit and the image features extracted by the second extraction unit, and performs a first alignment process to calculate low-precision alignment parameters, which indicate the relative position and pose of the second three-dimensional data with respect to the first three-dimensional data.

7. The three-dimensional scanner according to claim 6, wherein, The alignment unit performs local alignment based on the low-precision alignment parameters calculated by the first alignment process, the first three-dimensional data, and the second three-dimensional data, and performs a second alignment process to calculate high-precision alignment parameters, which indicate the relative position and orientation of the second three-dimensional data with respect to the first three-dimensional data.

8. The 3D scanner of claim 7 further includes a coordinate transformation unit, the coordinate transformation unit performing coordinate transformation of the first 3D data based on the low-precision alignment parameters calculated by the first alignment process.

9. The 3D scanner of claim 8, wherein the alignment unit performs local alignment in the second alignment process based on the first 3D data and the second 3D data after coordinate transformation performed by the coordinate transformation unit.

10. The three-dimensional scanner according to claim 1, wherein, The alignment unit specifies the corresponding points of the points included in the first three-dimensional data from the second three-dimensional data, and adjusts the position and orientation of the first three-dimensional data and the second three-dimensional data based on the specified corresponding points.

11. The three-dimensional scanner according to claim 1, further comprising: A light projection unit that illuminates the workpiece with measuring light and uniform light at different time intervals; as well as A light receiving unit receives the measurement light emitted by the light projection unit and reflected by the workpiece, outputs a first light receiving signal for measurement, receives the uniform light emitted by the light projection unit and reflected by the workpiece, and outputs a second light receiving signal for texture acquisition. The data acquisition unit acquires three-dimensional data, including shape and texture information of the workpiece generated based on the first light receiving signal and the second light receiving signal output by the light receiving unit, as the first three-dimensional data and the second three-dimensional data.

12. The three-dimensional scanner according to claim 1, further comprising: The first light projection unit illuminates the workpiece with patterned light used for measurement; The second light projection unit illuminates the workpiece with illumination light; as well as A light receiving unit receives the patterned light emitted by the first light projection unit and reflected by the workpiece, and outputs a first light receiving signal; it also receives the illumination light emitted by the second light projection unit and reflected by the workpiece, and outputs a second light receiving signal. The data acquisition unit acquires three-dimensional data, including shape and texture information of the workpiece generated based on the first light receiving signal and the second light receiving signal output by the light receiving unit, as the first three-dimensional data and the second three-dimensional data.

13. A three-dimensional measurement method for generating combined three-dimensional data of a workpiece by generating multiple three-dimensional data of a workpiece placed in different orientations and combining the multiple three-dimensional data, the three-dimensional measurement method comprising: Acquire first three-dimensional data and second three-dimensional data, wherein the first three-dimensional data is three-dimensional data including shape information and texture information of the workpiece placed in a first placement posture, and the second three-dimensional data is three-dimensional data including shape information and texture information of the workpiece placed in a second placement posture; The acquired first and second 3D data are input into the input layer of the neural network; Based on the first three-dimensional data and the second three-dimensional data input to the input layer of the neural network, shape features are extracted in multiple intermediate layers of the neural network; The shape features extracted from the intermediate layer of the neural network are output from the output layer of the neural network; Image features are extracted from the texture information included in each of the first and second three-dimensional data. Alignment is performed between the first 3D data and the second 3D data based on the extracted shape and image features; as well as The aligned first 3D data and the second 3D data are combined to generate the combined 3D data.

14. A storage medium for storing a three-dimensional measurement program, the three-dimensional measurement program being used to cause a computer to execute a three-dimensional measurement method, the three-dimensional measurement method being used to generate combined three-dimensional data of the workpiece by generating multiple three-dimensional data of a workpiece placed in different placement postures and combining the multiple three-dimensional data, wherein the three-dimensional measurement method includes: Acquire first three-dimensional data and second three-dimensional data. The first three-dimensional data includes shape and texture information of the workpiece placed in a first placement posture, and the second three-dimensional data includes shape and texture information of the workpiece placed in a second placement posture. The acquired first and second 3D data are input into the input layer of the neural network. Based on the first three-dimensional data and the second three-dimensional data input to the input layer of the neural network, shape features are extracted in multiple intermediate layers of the neural network. The shape features extracted from the intermediate layer of the neural network are output from the output layer of the neural network. Image features are extracted from the texture information included in each of the first and second three-dimensional data sets. Alignment is performed between the first 3D data and the second 3D data based on the extracted shape and image features, and The aligned first 3D data and the second 3D data are combined to generate the combined 3D data.

Images