3D shape measuring device

Abstract

To provide a three-dimensional shape measuring device that can reduce the burden in calculation processing, and also provide a three-dimensional shape measuring device that can perform more accurate measurements.SOLUTION: A three-dimensional shape measuring device 1 includes: a projection unit 20 configured to project, onto a measurement object, a measurement pattern 21 composed of a graph having nodes and edges that have characteristics distinguishable by the connection state of line segments; an imaging unit 30 configured to capture an image of the measurement pattern 21 projected onto the measurement object; and a calculation unit 120 configured to calculate a three-dimensional shape of the measurement object based on the image captured by the imaging unit 30. The measurement pattern 21 projected by the projection unit 20 has a lattice pattern that is distorted into a pincushion shape when projected onto a plane from the normal direction of the plane.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional shape measurement device.

Background Art

[0002] In recent years, three-dimensional shape measurement has been used in various fields such as smartphone face authentication, autonomous driving technology, medical, and industrial fields, and high-precision and stable restoration are required. Three-dimensional shape measurement mainly uses three methods: passive stereo method, active stereo method, and TOF camera. These methods are well known and are used according to conditions. Among them, the one-shot active stereo method can be photographed only with a projector and a camera, and measurement can be performed from a single input image. Therefore, high-speed measurement is possible with a small device.

[0003] A technique for applying a three-dimensional shape measurement device using the one-shot active stereo method to an endoscope is disclosed in Patent Document 1.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the imaging lens provided in the imaging unit of the endoscope uses an ultra-wide-angle lens or a fish-eye lens with a very wide angle of view to secure a wide field of view, the distortion aberration generated in the captured image is very large. Therefore, in order to perform accurate measurement, it was necessary to correct the distortion aberration by image processing or correct the measurement result. Such correction processing has increased the burden of arithmetic processing. In particular, in the process of calculating the three-dimensional shape of the measurement object, when increasing the number of measurement points to measure the three-dimensional shape in more detail or estimating the position of the measurement points using deep learning, it is required to reduce the burden of arithmetic processing of the correction processing.

[0006] Furthermore, the method disclosed in Patent Document 1 only allows for the representation of three types of features by the nodes of the projected pattern, resulting in a small amount of information being obtained, which is insufficient for measuring three-dimensional shapes with higher accuracy.

[0007] The objective of this disclosure is to provide a three-dimensional shape measuring device that can reduce the burden of computational processing. Furthermore, the second objective of this disclosure is to provide a three-dimensional shape measuring device that can perform measurements with higher accuracy. [Means for solving the problem]

[0008] This disclosure solves the aforementioned problems by the following solutions. For ease of understanding, the embodiments of this disclosure will be described using corresponding reference numerals, but are not limited thereto.

[0009] The first disclosure is a three-dimensional shape measuring device (1) comprising: a projection unit (20) that projects a two-dimensional pattern (21) composed of a graph having nodes and edges having features distinguishable by the connection state of line segments onto a measurement target; an imaging unit (30) that photographs the two-dimensional pattern (21) projected onto the measurement target; and a calculation unit (120) that calculates the three-dimensional shape of the measurement target based on the image captured by the imaging unit (30), wherein the two-dimensional pattern (21) projected by the projection unit (20) has a spool-shaped distorted grid pattern when projected onto a plane from the normal direction of the plane.

[0010] The second disclosure is a three-dimensional shape measuring device (1) as described in the first disclosure, wherein the two-dimensional pattern (21) is a graph structure pattern consisting of vertical and horizontal line segments, and each node is characterized by the step difference of the left and right horizontal line segments connected to the vertical line segments, and the step difference of the upper and lower vertical line segments connected to the horizontal line segments.

[0011] The third disclosure is a three-dimensional shape measuring device (1) as described in the first or second disclosure, wherein the calculation unit (120) comprises: a graph creation unit (121) that creates a graph by connecting nodes based on an image captured by the imaging unit (30); a feature estimation unit (122) that estimates the feature quantities of each node in the graph created by the graph creation unit (121); a correspondence point estimation unit (123) that estimates a plurality of candidate correspondence points in the captured image corresponding to the two-dimensional pattern (21) for each node based on the graph created by the graph creation unit (121) and the feature quantities of each node estimated by the feature estimation unit (122); and a correspondence point selection unit (124) that selects a correspondence point from the candidate correspondence points estimated by the correspondence point estimation unit (123) using the relationship between adjacent correspondence points.

[0012] The fourth disclosure is a three-dimensional shape measuring device (1) described in the third disclosure, further comprising a self-calibration unit (125) that calibrates information regarding the relative position between the projection unit (20) and the imaging unit (30) using information about the corresponding points selected by the corresponding point selection unit (124).

[0013] The fifth disclosure is a three-dimensional shape measuring device (1) described in the third or fourth disclosure, wherein at least one of the graph creation unit (121), the feature estimation unit (122), and the corresponding point estimation unit (123) uses deep learning with a modified training image obtained by modifying the training image in addition to the training image.

[0014] The sixth disclosure is a three-dimensional shape measuring device (1) comprising: a projection unit (20) that projects a two-dimensional pattern (21) composed of a graph having nodes and edges that have features distinguishable by the connection state of line segments onto a measurement target; an imaging unit (30) that photographs the two-dimensional pattern (21) projected onto the measurement target; and a calculation unit (120) that calculates the three-dimensional shape of the measurement target based on the image captured by the imaging unit (30), wherein the two-dimensional pattern (21) projected by the projection unit (20) is a graph structure pattern consisting of vertical and horizontal line segments, and each node is characterized by the step difference of the left and right horizontal line segments connected to the vertical line segments and the step difference of the upper and lower vertical line segments connected to the horizontal line segments.

[0015] The seventh disclosure is a three-dimensional shape measuring device (1) as described in the sixth disclosure, wherein the two-dimensional pattern (21) has a lattice pattern that is distorted in a spool-like shape when projected onto a plane from the direction normal to the plane.

[0016] The eighth disclosure is a three-dimensional shape measuring device (1) as described in the sixth or seventh disclosure, wherein the calculation unit (120) comprises: a graph creation unit (121) that creates a graph by connecting nodes based on an image captured by the imaging unit (30); a feature estimation unit (122) that estimates the feature quantities of each node in the graph created by the graph creation unit (121); a correspondence point estimation unit (123) that estimates a plurality of candidate correspondence points in the captured image corresponding to the two-dimensional pattern (21) for each node based on the graph created by the graph creation unit (121) and the feature quantities of each node estimated by the feature estimation unit (122); and a correspondence point selection unit (124) that selects a correspondence point from the candidate correspondence points estimated by the correspondence point estimation unit (123) using the relationship between adjacent correspondence points.

[0017] The ninth disclosure is a three-dimensional shape measurement device (1) described in the eighth disclosure, further comprising a self-calibration unit (125) that calibrates information regarding the relative position between the projection unit (20) and the imaging unit (30) using the information of the corresponding points selected by the corresponding point selection unit (124).

[0018] The tenth disclosure is a three-dimensional shape measurement device (1) described in the eighth disclosure or the ninth disclosure, wherein at least one of the graph creation unit (121), the feature amount estimation unit (122), and the corresponding point estimation unit (123) uses deep learning with a modified teacher image obtained by modifying the teacher image in addition to the teacher image.

Advantages of the Invention

[0019] According to the present disclosure, a three-dimensional shape measurement device capable of reducing the burden of arithmetic processing can be provided. Also, according to the present disclosure, a three-dimensional shape measurement device capable of performing more accurate measurement can be provided.

Brief Description of the Drawings

[0020] [Figure 1] It is a diagram showing an embodiment of the three-dimensional shape measurement device 1 according to the present disclosure. [Figure 2] It is an enlarged cross-sectional view around the forceps hole at the insertion end of the endoscope 50. [Figure 3] It is a diagram showing a state where an image of the measurement pattern 21 is projected onto the measurement object. [Figure 4] It is a diagram showing the measurement pattern 21 when the projection unit 20 projects from the normal direction of the plane onto the plane. [Figure 5] It is a diagram showing the originally designed pattern. [Figure 6] It is a diagram showing the measurement pattern 21 photographed by the imaging unit 30, which is the measurement pattern 21 projected from the projection unit 20 from the normal direction of the plane onto the plane. [Figure 7] It is a diagram showing the originally designed pattern before distortion with the line interval adjusted. [Figure 8] The projection unit 20 is a view showing the measurement pattern 22 when FIG. 7 is projected onto a plane from the normal direction of the plane. [Figure 9] It is a view showing the measurement pattern 22 obtained by photographing the measurement pattern 22 projected from the projection unit 20 onto a plane from the normal direction of the plane by the photographing unit 30. [Figure 10] It is a view showing the code assigned to the step of the measurement pattern 21. [Figure 11] It is a block diagram showing the overall configuration of the three-dimensional shape measurement device. [Figure 12] It is a view for explaining the state in which the feature quantity estimation unit 122 estimates the feature quantity. [Figure 13] It is a view for explaining the method of selecting corresponding points performed by the corresponding point selection unit 124. [Figure 14] It is a view for explaining the teacher image and the modified teacher image. [Figure 15] It is a view showing the sample used for measurement and the measurement range. [Figure 16] It is a view showing the result of measuring the range R1 shown in FIG. 15(a). [Figure 17] It is a view showing the result of measuring the range R2 shown in FIG. 15(b). [Figure 18] It is a view showing the result of measuring the range R3 shown in FIG. 15(b). [Figure 19] It is a view showing an example in which the endoscope 50 is inserted into the stomach of a pig for measurement.

Mode for Carrying Out the Invention

[0021] Hereinafter, the best mode for implementing the present disclosure will be described with reference to the drawings and the like.

[0022] (Embodiment) FIG. 1 is a view showing an embodiment of the three-dimensional shape measurement device 1 according to the present disclosure. Please note that the following figures, including Figure 1, are schematic representations, and the size and shape of each part have been exaggerated or omitted as appropriate to facilitate understanding. Furthermore, the following explanation will include specific numerical values, shapes, etc., but these may be changed as appropriate.

[0023] As shown in Figure 1, the three-dimensional shape measuring device 1 of this embodiment comprises a laser light source 10 and a projection unit (pattern projector) 20 that projects a measurement pattern 21 (see Figures 4 and 5) onto the object to be measured. The three-dimensional shape measuring device 1 is used together with an endoscope 50. The endoscope 50 has a forceps hole 51. Normally, forceps for taking a sample of the observation site are inserted through the forceps hole 51. However, in this embodiment, the projection unit 20 is inserted through the forceps hole 51 instead of forceps. In this embodiment, the projection unit 20 is inserted through the forceps hole 51, but for example, the projection unit 20 may be fixed to the endoscope 50 as part of the endoscope 50, such as the illumination unit 40 described later.

[0024] The projection unit 20 is connected to the laser light source 10. The laser light source 10 comprises a laser module 10a, a diffuser plate 10b, an illumination aperture 10c, and a collimator lens 10d.

[0025] The laser module 10a outputs a green laser beam LB (for example, a laser beam with a wavelength of 517 mm). The diffuser plate 10b diffuses the laser beam LB emitted from the laser module 10a, making the intensity of the laser beam LB uniform across its cross-section. The collimator lens 10d converts the incident laser beam LB into parallel light. The laser beam LB, converted into parallel light by the collimator lens 10d, is incident on the optical fiber 20a that constitutes the projection unit 20.

[0026] Figure 2 is a magnified cross-sectional view of the area around the forceps channel at the insertion end of the endoscope 50. As shown in Figure 2, the projection unit 20 comprises an optical fiber 20a, a green lens 20b, a diffractive optical element 20c, and a cylindrical portion 20d. The size of the projection unit 20 is, for example, 1.8 mm in diameter and 19 mm in length.

[0027] The optical fiber 20a is a single-mode optical fiber made of plastic or glass. The green laser beam LB emitted from the collimator lens 10d enters the optical fiber 20a from one end and exits from the other end.

[0028] The green lens 20b focuses the laser light emitted from the optical fiber 20a and irradiates the region of the diffractive optical element 20c where the diffraction grating is located. The diffractive optical element 20c is an optical element called a DOE (diffractive optional element). The DOE forms a pattern by diffracting the laser light passing through it. Therefore, it can reduce optical loss compared to a mask pattern made of film; for example, the optical energy loss rate is less than 5%. Because the DOE can project a clear pattern regardless of distance, it can extend the measurement range compared to an imaging optical system. Furthermore, since the optical efficiency of the DOE is over 90%, the reduction in brightness of the pattern projected onto the measurement target is suppressed. The size of the projected pattern is, for example, approximately 30mm at a distance of 30mm. 2 That is the case.

[0029] The diffractive optical element 20c is placed on the optical path of the laser beam LB emitted from the green lens 20b. Therefore, the laser beam LB projected onto the biological tissue becomes projected light containing an image of the measurement pattern 21 (2D pattern) shown in Figure 4, which will be described later. This measurement pattern 21 is composed of a graph consisting of nodes and edges that have distinguishable features based on the connection state of line segments, as shown below. Details of the measurement pattern 21 will be described later.

[0030] The cylindrical portion 20d is a cylindrical member. The cylindrical portion 20d protrudes from the insertion end 52 of the endoscope 50. The outer diameter of the cylindrical portion 20d is sized to fit into the forceps hole 51. The inner diameter of the cylindrical portion 20d is large enough to accommodate the optical fiber 20a, the green lens 20b, and the diffractive optical element 20c.

[0031] Figure 3 shows how the image of the measurement pattern 21 is projected onto the object to be measured. The laser beam LB emitted from the green lens 20b is emitted to the outside via the diffractive optical element 20c provided at the end of the cylindrical portion 20d. The laser beam LB is diffracted by the diffractive optical element 20c to form a projection image of the measurement pattern 21. As shown in Figure 3, the measurement pattern 21 is projected onto the measurement target (target T) irradiated by the laser beam LB.

[0032] The insertion end 52 of the endoscope 50 is equipped with a forceps channel 51, as well as an imaging unit 30 and an illumination unit 40. The imaging unit 30 includes an imaging optical system and image sensor (not shown) for imaging the target T. The illumination unit 40 irradiates the target T with illumination light. The endoscope 50 has the function of irradiating the target T with illumination light using the illumination unit 40 and imaging the irradiated target T using the imaging unit 30. In this embodiment, the imaging unit 30 provided in the endoscope 50 is used to photograph the target T and the projected image of the measurement pattern 21 projected onto the target T, thereby performing three-dimensional shape measurement.

[0033] Here, we will describe the measurement pattern 21 projected by the projection unit 20. Figure 4 shows the measurement pattern 21 when the projection unit 20 projects onto a plane from the direction normal to the plane. The measurement pattern 21 (two-dimensional pattern) projected by the projection unit 20 has a pincushion-shaped distorted grid pattern when projected onto a plane from the normal direction of the plane, as shown in Figure 4. The reason why the measurement pattern 21 projected by the projection unit 20 is a pattern in which orthogonal grid patterns are distorted into a pincushion shape is to take into consideration that the imaging optical system of the imaging unit 30 of the endoscope 50 is a fisheye lens. Even if an orthogonal grid pattern is arranged (projected) on a plane, barrel distortion occurs in the image captured by the fisheye lens. Correcting this, as explained earlier, places a heavy burden on the computational processing. Therefore, in this embodiment, the diffraction grating of the diffractive optical element 20c is configured such that the projection unit 20 projects a measurement pattern 21 with a shape corresponding to pincushion distortion, which is the opposite of the barrel distortion that occurs when the image is captured by the imaging unit 30. As shown in Figure 4, when the projection unit 20 projects onto a plane from the direction normal to the plane, the measurement pattern 21 has a grid pattern that is intentionally distorted in a pincushion shape. However, Figure 5 shows the target pattern that was originally designed to be obtained as the photographed result when the measurement pattern 21 in Figure 4 is photographed by the photography unit 30, and the pincushion distortion is canceled out by the optical distortion (barrel distortion) of the fisheye lens.

[0034] Figure 6 shows the measurement pattern 21, which is projected from the projection unit 20 onto a plane from the direction of the plane's normal vector, and then photographed by the imaging unit 30. When the orthogonal grid pattern shown in Figure 4 is distorted into a pincushion shape and photographed by the imaging unit 30, the measurement pattern 21 is photographed in a form having an orthogonal grid pattern, as shown in Figure 6. In this way, the barrel distortion aberration that occurs when the pincushion grid pattern is photographed using the fisheye lens of the imaging unit 30 results in a photographic result having an orthogonal grid pattern, as if the distortion aberration had been corrected. This is an excellent effect obtained by the projection unit 20 projecting a grid pattern that includes a pincushion correction value in advance so that it can cancel out the "barrel distortion" of the fisheye lens of the imaging unit 30. Therefore, there is no need to perform image processing calculations to correct the distortion aberration, and the burden of computation can be reduced.

[0035] Furthermore, as shown in Figure 4, the grid spacing is set so that the grid spacing during projection is approximately equal. This allows nodes to be arranged at approximately equal intervals on the target T. To achieve this, the initially designed target pattern shown in Figure 5 has a grid spacing at the periphery that is narrower than the grid spacing in the center. When this pattern is captured with an endoscope camera, as shown in Figure 6, due to fisheye lens distortion, pincushion distortion is eliminated, but the spacing at the periphery becomes narrow again. Therefore, Figure 7 shows a target pattern in which the line spacing has been further adjusted to account for fisheye distortion. In the pattern shown in Figure 7, the grid size at the periphery is larger than the grid size at the periphery of the pattern shown in Figure 5. In other words, the pattern shown in Figure 7 has a smaller difference in size between the grid near the center and the grid at the periphery than the pattern shown in Figure 5. Figure 8 shows the measurement pattern 22 when the projection unit 20 projects Figure 7 onto a plane from the direction of the plane's normal vector. Figure 9 is an image of Figure 8 captured by the imaging unit 30. In Figure 9, it can be seen that the grid spacing (grid size) at the periphery is almost the same as near the center.

[0036] Furthermore, the measurement pattern 21 in this embodiment is a pattern that includes widely spaced lines and is relatively robust against blurring due to subsurface scattering, similar to the pattern described in Japanese Patent Application Publication No. 2017-217215 (hereinafter referred to as Patent Document 1). As shown in Figures 4 and 5, the measurement pattern 21 is based on a grid-like pattern consisting of vertical and horizontal line segments. The measurement pattern 21 of this embodiment is based on a grid pattern and has vertical and horizontal line segments (edges), with nodes at the intersections of each line segment. Each node is characterized by a graph structure pattern formed by the step difference between the horizontal line segments to the left and right that connect to the vertical line segments, and the step difference between the vertical line segments above and below that connect to the horizontal line segments. In measurement pattern 21, horizontal line segments have small steps added between adjacent horizontal line segments (horizontal line segments connected to vertical line segments) at each grid point (intersection of line segments). Similarly, vertical line segments also have small steps added between adjacent vertical line segments (vertical line segments connected to horizontal line segments) at each grid point. The size of these steps includes zero. Since the positions of line segments longer than the blur size are less likely to be lost due to subsurface scattering, information about the steps can be reliably acquired.

[0037] Figure 10 shows the codes assigned to the steps of the measurement pattern 21. As shown in Figure 10, due to the stepped configuration of the measurement pattern 21 described above, each grid point is assigned one of five types of codes (features): A, B, C, D, and E. CODE_A is assigned to grid points where the endpoints of the vertical and horizontal line segments are all continuous, that is, where there are no steps or gaps in any direction (up, down, left, or right). CODE_B is assigned to a grid point where the endpoint of the rightmost horizontal segment is higher than the endpoint of the leftmost horizontal segment. CODE_C is assigned to a grid point where the endpoint of the left horizontal line segment is higher than the endpoint of the right horizontal line segment. CODE_D is assigned to a grid point where the endpoint of the upper vertical line segment is to the left of the endpoint of the lower vertical line segment. CODE_E is assigned to a grid point where the endpoint of the lower vertical line segment is to the left of the endpoint of the upper vertical line segment.

[0038] In this embodiment, each node is coded not only by the step difference between the horizontal lines on the left and right that connect to the vertical lines, but also by the step difference between the vertical lines above and below that connect to the horizontal lines. The measurement pattern 21 is a graph structure pattern in which nodes are characterized by the step differences between the horizontal lines on the left and right, and the step differences between the vertical lines above and below. According to this embodiment, while Patent Document 1 used three types of codes (feature quantities), five types of codes (feature quantities) can be assigned to each node, increasing the amount of positional information obtained from the measurement pattern 21 and improving measurement accuracy.

[0039] Figure 11 is a block diagram showing the overall configuration of the 3D shape measuring device. Figure 11 shows the overall configuration of the 3D shape measuring device 1. As shown in Figure 11, in addition to the laser light source 10 and projection unit 20 described above, the 3D shape measuring device 1 further includes a control unit 100. The control unit 100 is a computer equipped with a CPU (central processing unit), storage device, input / output device, pointing device or display, and other human-machine interface components. The functions of the control unit 100 are realized when the CPU executes a program stored in the storage device according to the operator's operation information input from the human-machine interface. That is, according to the program being executed, the control unit 100 controls the laser light source 10 via the input / output device, controls the illumination unit 40 of the endoscope 50, and inputs imaging data from the imaging unit 30 of the endoscope 50 via the input / output device. Furthermore, this explanation assumes that the control unit 100 contains all the components of the calculation unit 120. However, this is not the only option; for example, some components of the calculation unit 120 (such as the graph creation unit 121, feature estimation unit 122, and corresponding point estimation unit 123, which will be described later) may be provided on a server in the cloud.

[0040] The endoscope 50 also includes an imaging unit 30 for photographing the object to be measured and an illumination unit 40 for illuminating the object to be measured. The imaging unit 30 images the object to be measured along with the measurement pattern 21 projected onto the object to be measured.

[0041] The control unit 100 comprises an operation unit 110, a calculation unit 120, and a display unit 130. The operation unit 110 outputs an operation signal to the calculation unit 120 in accordance with the operator's operation.

[0042] The calculation unit 120 receives imaging data of the object to be measured obtained from the imaging unit 30 of the endoscope 50 in accordance with the operation signal from the operation unit 110, and performs various calculations on the input imaging data to calculate the three-dimensional shape of the object to be measured.

[0043] The display unit 130 displays various images, such as imaging data of the measurement target obtained from the imaging unit 30 of the endoscope 50.

[0044] Next, we will explain the calculation unit 120 in more detail. The calculation unit 120 includes a graph creation unit 121, a feature estimation unit 122, a corresponding point estimation unit 123, a corresponding point selection unit 124, a self-calibration unit 125, and a measurement unit 126.

[0045] The graph creation unit 121 creates a graph (grid graph) by connecting nodes based on the captured images obtained by the imaging unit 30. Specifically, the graph creation unit 121 takes an image of an object onto which the measurement pattern 21 is projected as input, estimates the vertical and horizontal pixel-level phases using U-Net (hereinafter referred to as U-Net-1), and further obtains a grid-divided image by dividing it into one-wavelength segments. At this time, the centroid of each divided grid is used as a node, and a graph is created by connecting adjacent nodes. Here, U-Net is a type of FCN (fully convolutional network) and is known as a network for estimating image segmentation. The phases here are obtained as separate phases corresponding to the vertical and horizontal line segments of the measurement pattern 21. That is, a vertically striped vertical phase map and a horizontally striped horizontal phase map are generated from a single captured image. The graph creation unit 121 combines these vertical and horizontal phase maps to create a graph (grid graph) by connecting nodes as described above. Before the graph creation unit 121 performs the graph creation process, it may also perform known image processing, such as reducing noise in the image using a Gaussian filter or a median filter, as appropriate.

[0046] The feature estimation unit 122 estimates the feature quantities of each node in the graph created by the graph creation unit 121. Specifically, the feature estimation unit 122 uses a U-Net (hereinafter referred to as U-Net-2) different from the U-Net-1 used by the graph creation unit 121 described above to estimate the feature quantities of each node from the measurement pattern 21 in the captured image, i.e., from CODE_A to CODE_E shown in Figure 10. Figure 12 illustrates the state in which the feature estimation unit 122 estimates features. Figure 12(a) shows the measurement pattern 21 obtained from the captured image, and Figure 12(b) shows the estimated features (codes) superimposed on the measurement pattern 21. As shown in Figure 12, the feature estimation unit 122 estimates which of CODE_A to CODE_E shown in Figure 10 corresponds to each node, depending on the state of the vertical and horizontal steps at each node.

[0047] The correspondence point estimation unit 123 estimates multiple candidate correspondence points in the captured image corresponding to the measurement pattern 21 (2D pattern) for each node, based on the graph created by the graph creation unit 121 and the feature quantities of each node estimated by the feature quantity estimation unit 122. Each node in the measurement pattern 21 is pre-assigned a unique number (hereinafter referred to as ID). This ID makes it possible to identify which node in the measurement pattern 21 is being used. The correspondence point estimation unit 123 uses a graph convolutional network (GCN) as input, taking the graph created by the graph creation unit 121 and the feature quantities of each node estimated by the feature quantity estimation unit 122 as input, to estimate which ID each node of the graph obtained from the measurement pattern 21 projected onto the object corresponds to. In this embodiment, the correspondence point estimation unit 123 outputs multiple candidate IDs that have been estimated, in descending order of the probability that they are correct estimations (hereinafter also simply referred to as probability).

[0048] The corresponding point selection unit 124 selects one corresponding point (ID) from the candidate corresponding points (IDs) estimated by the corresponding point estimation unit 123, using the relationships between adjacent corresponding points (IDs). Figure 13 illustrates the method by which the corresponding point selection unit 124 selects corresponding points. Figure 13(a) shows a graph of the ID output by the corresponding point estimation unit 123 as having the highest probability for a given node, including the estimated IDs of surrounding nodes. Figure 13(b) shows a graph of the sequence of IDs pre-assigned to the measurement pattern 21, showing the top three IDs estimated by the corresponding point estimation unit 123 to have the highest probability. Figure 13(c) shows the results of scoring the top three IDs estimated by the corresponding point estimation unit 123 to have the highest probability in relation to the surrounding arrangement. Note that only the top three with the highest probability are shown here, but the number of candidates output by the corresponding point estimation unit 123 is not limited to three and can be set as appropriate.

[0049] For example, suppose that, as shown in Figure 13(a), the correspondence point estimation unit 123 estimates ID=183 as the candidate with the highest probability for a certain node. At this time, suppose that the IDs surrounding this node are 260 above, 262 below, 256 to the left, and 188 to the right. The correspondence point estimation unit 123 estimates 261 as the ID with the next highest probability after ID=183, and then estimates 224 as the ID with the next highest probability. Figure 13(b) shows the arrangement of IDs that have been pre-assigned to the measurement pattern 21 for the surrounding nodes of ID=183, 261, and 224, i.e., the correct arrangement of surrounding IDs. In Figure 13(a) above, there are many places where the surrounding IDs differ from the correct arrangement of ID=183, so it is highly likely that the correspondence point estimation unit 123's estimation of ID=183 as having the highest probability was incorrect. To properly evaluate this, the corresponding point selection unit 124 counts how many of the arrangements of the four surrounding nodes estimated by the corresponding point estimation unit 123 in Figure 13(a) match the correct arrangement in Figure 13(b), and uses this as a score (evaluation value). As shown in Figure 13(c), ID=261, which had the second highest probability in the estimation results of the corresponding point estimation unit 123, has a higher score than ID=183, which had the highest probability. Therefore, in the example in Figure 13, it is highly likely that the ID=183 estimated by the corresponding point estimation unit 123 is incorrect. Accordingly, the corresponding point selection unit 124 selects ID=261 and replaces the estimation result of the corresponding point estimation unit 123.

[0050] The self-calibration unit 125 uses the information of the corresponding points selected by the corresponding point selection unit 124 to calibrate the information regarding the relative position between the projection unit 20 and the imaging unit 30. The projection unit 20 is simply inserted into the forceps channel 51 of the endoscope 50 and is not fixed to the endoscope 50. Therefore, the relative position between the projection unit 20 and the imaging unit 30 may change even during use. If the relative position between the projection unit 20 and the imaging unit 30 is not correctly obtained, the measurement unit 126 cannot correctly determine the three-dimensional shape. Therefore, in this embodiment, a self-calibration unit 125 is provided, and the information regarding the relative position between the projection unit 20 and the imaging unit 30 is calibrated in a timely manner using the information of the corresponding points selected by the corresponding point selection unit 124. The timing of this self-calibration can be freely set, but for example, it may be performed each time a measurement is taken.

[0051] The self-calibration unit 125 performs self-calibration by using the captured image taken by projecting the measurement pattern 21 and the information of the corresponding points selected by the corresponding point selection unit 124 to perform outlier removal using the RANSAC method, thereby outputting the position parameters (6-degree-of-freedom 3D rigid body transformation) of the projection unit 20 relative to the imaging unit 30. Specifically, it randomly selects a node from all nodes and performs fitting by minimizing the epipolar error for the correspondence relationships at the selected node. The estimated variable in the fitting is the 6-degree-of-freedom 3D rigid body transformation mentioned above. Next, it estimates the error in the correspondence relationships at the remaining nodes using the estimated 3D rigid body transformation, and nodes whose errors are above a certain value are considered outliers. If the "proportion of outliers" is below a certain value, the solution is stored. The above process up to storing the solution is repeated a certain number of times, and the solution with the smallest fitting error among the stored solutions is output as the calibration value.

[0052] The measurement unit 126 performs image processing using the active stereo method based on the image of the measurement pattern 21 projected onto the object to be measured, in order to determine the three-dimensional shape of the object to be measured. In this embodiment, the measurement unit 126 measures the three-dimensional shape of the object to be measured based on the light section method. The light section method is one of the active stereo methods and is a method that applies the principle of triangulation. In the light section method, the three-dimensional shape of the object to be measured is measured based on the parallax (baseline length) between the projection center of the object to be measured by the projection unit 20 and the imaging center of the camera (imaging unit 30) that images the object to be measured. In this embodiment, the three-dimensional shape of the object to be measured is determined based on the information of corresponding points for each node in the captured image corresponding to the measurement pattern 21 obtained by the calculation unit 120. The three-dimensional shape measurement method performed by the measurement unit 126 is the same as that described in Patent Document 1 and the following References 2, 3, and 4, so a detailed explanation will be omitted.

[0053] Reference 2 describes how to detect and estimate the vertical and horizontal lines of a grid pattern and the codes embedded in the grid pattern from captured images using a deep learning network called U-Net. Reference 2 describes the code estimation method using U-Net that is used in the feature estimation unit 122. [Reference 2] Furukawa, R., Miyazaki, D., Baba, M., Hiura, S., & Kawasaki, H. (2018). Robust structured light system against subsurface scattering effects achieved by CNN-based pattern detection and decoding algorithm. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops.

[0054] Reference 3 describes how to estimate the pattern ID corresponding to each node in a grid pattern using a deep learning network called GCN. Reference 3 describes the GCN-based correspondence estimation method used in the correspondence point estimation unit 123. [Reference 3] Furukawa, R., Oka, S., Kotachi, T., Okamoto, Y., Tanaka, S., Sagawa, R., & Kawasaki, H. (2020, July). Fully auto-calibrated active-stereo-based 3d endoscopic system using correspondence estimation with graph convolutional network. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (pp. 4357-4360). IEEE.

[0055] Reference 4 describes a method using U-Net to estimate the relative positions of pixels not only along the lines of a grid pattern but also between them, and to perform high-density reconstruction. Reference 4 describes a method in which the graph generation unit 121 estimates the vertical and horizontal pixel-level phases and obtains a correspondence for each pixel. [Reference 4] Furukawa, R., Mikamo, M., Sagawa, R., & Kawasaki, H. (2022). Single-shot dense active stereo with pixel-wise phase estimation based on grid-structure using CNN and correspondence estimation using GCN. In Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision (pp. 4001-4011).

[0056] As mentioned above, U-Net-1, U-Net-2, and GCN all utilize deep learning, and the training data used for training is crucial. In this embodiment, in addition to using images obtained by projecting the measurement pattern 21 onto an object as training images for these networks, training is also performed using modified training images obtained by altering the training images. Figure 14 illustrates the training image and the modified training image. Figure 14(a) shows the training image from which the modification was made, and Figures 14(b) to 14(d) show the modified training images. Note that objects are not shown in Figure 14. Modified training images are created by processing the training image shown in Figure 14(a) using computer graphics (CG). For example, a modified training image can be created by shrinking, enlarging, or rotating the training image, as shown in Figure 14(b). Furthermore, a modified training image can be created by adding various types of noise to the training image, as shown in Figure 14(c). Additionally, a modified training image can be created by adding deformation processing such as shear deformation to the training image, as shown in Figure 14(d). Besides the examples above, the brightness of the training image may be increased or decreased, color changes may be added, or a modified training image may be created by combining these as appropriate. By performing this kind of training data augmentation, training accuracy can be improved, misrecognition can be reduced, and accurate shape measurement becomes possible under a wider range of conditions.

[0057] As described above, according to this embodiment, the measurement pattern 21 (two-dimensional pattern) projected by the projection unit 20 has a spool-shaped distorted grid pattern when projected onto a plane from the normal direction of the plane. This makes it possible to provide a three-dimensional shape measuring device that can reduce the burden of computational processing. Furthermore, the measurement pattern 21 is a graph structure pattern in which each node is characterized by the step difference between the left and right horizontal line segments connected to the vertical line segments, and the step difference between the upper and lower vertical line segments connected to the horizontal line segments. This allows for a larger number of features to be assigned to each node, providing a 3D shape measurement device that can perform measurements with higher accuracy. Furthermore, by configuring the calculation unit 120 to include a graph creation unit (U-Net-1), a feature estimation unit (U-Net-2), a corresponding point estimation unit (GCN), and a corresponding point selection unit, it is possible to provide a 3D shape measurement device that can perform measurements with even higher accuracy. Furthermore, since the 3D shape measuring device 1 is equipped with a self-calibration unit 125, it can perform highly accurate measurements even if the position of the projection unit 20 is not fixed relative to the imaging unit 30. Furthermore, by using modified training images in the deep learning of the graph generation unit (U-Net-1), the feature estimation unit (U-Net-2), and the corresponding point estimation unit (GCN), estimation in these networks can be performed more accurately, and as a result, a 3D shape measurement device that can perform measurements with even higher accuracy can be provided.

[0058] (Example 1) This document shows an example of actual measurements performed using the 3D shape measuring device 1 described herein. Figure 15 shows the sample and measurement range used for the measurement. Measurements were actually performed for range R1 in Figure 15(a), and ranges R2 and R3 in Figure 15(b). Figure 16 shows the results of measuring the range R1 shown in Figure 15(a). Figure 16(a) shows the measurement pattern captured by the imaging unit 30, and Figure 16(b) shows the result of representing the 3D shape of the measurement target obtained by calculation using computer graphics (hereinafter referred to as the reconstruction result). Figure 17 shows the results of measuring the range R2 shown in Figure 15(b). Figure 17(a) shows the measurement pattern captured by the imaging unit 30, and Figure 17(b) shows the reconstruction result. Figure 18 shows the results of measuring the range R3 shown in Figure 15(b). Figure 18(a) shows the measurement pattern captured by the imaging unit 30, and Figure 18(b) shows the reconstruction result. Referring to Figures 15 through 18, it can be seen that despite the wide field of view due to the fisheye lens, the pattern covers a large portion of the area, and that all of it is appropriately reconstructed as a three-dimensional shape.

[0059] (Example 2) Furthermore, we will show an example in which an endoscope 50 was actually inserted into a pig's stomach and the surface of the pig's stomach was measured. Figure 19 shows an example of measurement performed by inserting an endoscope 50 into the stomach of a pig. Figure 19(a) shows the state of inserting the endoscope 50 into the stomach of a pig, Figure 19(b) shows the measurement pattern inside the pig's stomach as captured by the imaging unit 30, and Figure 19(c) shows the reconstructed result. As shown in Figure 19, it can be seen that shape reconstruction was successful despite the fact that the pattern features are not clearly defined due to scattering and absorption on the biological surface.

[0060] (Transformed form) The embodiments described above are not limited to those described above, and various modifications and changes are possible, which are also within the scope of this disclosure.

[0061] (1) In the embodiment, a three-dimensional shape measuring device used in conjunction with an endoscope 50 was described as an example. However, the embodiment is not limited to this, and may also be a three-dimensional shape measuring device used in conjunction with a robot or a drone, for example, and the specific application of the three-dimensional shape measuring device can be changed as appropriate.

[0062] (2) In this embodiment, U-Net and GCN were given as examples of networks used by the arithmetic unit 120. However, the system is not limited to these, and other networks such as GCNII (Graph Convolutional Network via Initial residual and Identity mapping) may be used instead of Graph Convolutional Network (GCN). Alternatively, a learning model different from neural networks, such as Random Forest, may be used. Furthermore, other networks such as ResU-Net, R2U-Net, and SegNet may be used instead of U-Net. These can be changed as appropriate depending on the resources available for training and execution.

[0063] (3) In the embodiment, the measurement pattern 21 has a grid pattern that is distorted in a spool shape when projected onto a plane from the direction normal to the plane, and each node is characterized by the step difference between the left and right horizontal line segments connected to the vertical line segments, and the step difference between the upper and lower vertical line segments connected to the horizontal line segments. However, the embodiment is not limited to this, and for example, the measurement pattern may not be distorted in a spool shape when projected onto a plane from the direction normal to the plane, and each node may be characterized by the step difference between the left and right horizontal line segments connected to the vertical line segments, and the step difference between the upper and lower vertical line segments connected to the horizontal line segments. Furthermore, for example, the measurement pattern may have a grid pattern that is distorted in a pincushion shape when projected onto a plane from the direction normal to the plane, and each node may be characterized by only one of either the step difference between the left and right horizontal line segments connected to the vertical line segments, or the step difference between the upper and lower vertical line segments connected to the horizontal line segments.

[0064] The embodiments and variations may be used in combination as appropriate, but a detailed explanation is omitted. Furthermore, this disclosure is not limited to the embodiments described above. [Explanation of Symbols]

[0065] 1. Three-dimensional shape measuring device 10 Laser light source 10a Laser Module 10b Diffuser 10c illumination aperture 10D Collimator Lens 20 Projection section 20a optical fiber 20b Green Lens 20c diffractive optical element 20d Cylindrical section 21 Measurement Patterns 30 Photography Department 40 Lighting Section 50 Endoscopes 51 Forceps holes 52 Insertion end 100 Control Unit 110 Operation section 120 Arithmetic section 121 Graph Creation Department 122 Feature Estimation Unit 123 Corresponding Point Estimation Unit 124 Corresponding Point Selection Section 125 Self-calibration section 126 Measurement Unit 130 Display section

Claims

1. A projection unit that projects a two-dimensional pattern, consisting of a graph having nodes and edges with distinguishable features based on the connection state of line segments, onto a measurement target, A camera unit that captures the two-dimensional pattern projected onto the object to be measured, A calculation unit that calculates the three-dimensional shape of the object to be measured based on the image captured by the aforementioned imaging unit, Equipped with, A three-dimensional shape measuring device wherein the two-dimensional pattern projected by the projection unit has a spool-shaped grid pattern when projected onto a plane from the normal direction of the plane.

2. In the three-dimensional shape measuring device according to claim 1, The three-dimensional shape measuring device is a graph structure pattern in which the two-dimensional pattern consists of vertical and horizontal line segments, and each node is characterized by the step difference of the horizontal line segments to the left and right that connect to the vertical line segments, and the step difference of the vertical line segments above and below that connect to the horizontal line segments.

3. In the three-dimensional shape measuring device according to claim 1 or claim 2, The aforementioned arithmetic unit, A graph creation unit creates a graph by connecting nodes based on the image captured by the aforementioned imaging unit, A feature estimation unit that estimates the feature quantities of each node in the graph created by the graph creation unit, A correspondence point estimation unit estimates multiple candidate corresponding points in the captured image corresponding to the two-dimensional pattern for each node, based on the graph created by the graph creation unit and the feature quantities of each node estimated by the feature quantity estimation unit. A corresponding point selection unit selects a corresponding point from the candidate corresponding points estimated by the corresponding point estimation unit using the relationship between adjacent corresponding points, A three-dimensional shape measuring device equipped with the following features.

4. In the three-dimensional shape measuring device according to claim 3, A three-dimensional shape measuring device comprising a self-calibration unit that calibrates information regarding the relative position between the projection unit and the imaging unit using information about the corresponding points selected by the corresponding point selection unit.

5. In the three-dimensional shape measuring device according to claim 3, A three-dimensional shape measuring device in which at least one of the graph generation unit, the feature estimation unit, and the corresponding point estimation unit uses deep learning that uses a modified training image obtained by modifying the training image in addition to the training image.

6. A projection unit that projects a two-dimensional pattern, consisting of a graph having nodes and edges with distinguishable features based on the connection state of line segments, onto a measurement target, A camera unit that captures the two-dimensional pattern projected onto the object to be measured, A calculation unit that calculates the three-dimensional shape of the object to be measured based on the image captured by the aforementioned imaging unit, Equipped with, The two-dimensional pattern projected by the projection unit is a graph structure pattern consisting of vertical and horizontal line segments, where each node is characterized by the step difference between the horizontal line segments connected to the vertical line segments and the step difference between the vertical line segments connected to the horizontal line segments. The three-dimensional shape measuring device has a 2D pattern that, when projected onto a plane from the normal direction of the plane, has a grid pattern that is distorted in a spool-like shape.

7. In the three-dimensional shape measuring device according to claim 6, The aforementioned arithmetic unit, A graph creation unit creates a graph by connecting nodes based on the image captured by the aforementioned imaging unit, A feature estimation unit that estimates the feature quantities of each node in the graph created by the graph creation unit, A correspondence point estimation unit estimates multiple candidate corresponding points in the captured image corresponding to the two-dimensional pattern for each node, based on the graph created by the graph creation unit and the feature quantities of each node estimated by the feature quantity estimation unit. A corresponding point selection unit selects a corresponding point from the candidate corresponding points estimated by the corresponding point estimation unit using the relationship between adjacent corresponding points, A three-dimensional shape measuring device equipped with the following features.

8. In the three-dimensional shape measuring device according to claim 7, A three-dimensional shape measuring device comprising a self-calibration unit that calibrates information regarding the relative position between the projection unit and the imaging unit using information about the corresponding points selected by the corresponding point selection unit.

9. In the three-dimensional shape measuring device according to claim 7, A three-dimensional shape measuring device in which at least one of the graph generation unit, the feature estimation unit, and the corresponding point estimation unit uses deep learning that uses a modified training image obtained by modifying the training image in addition to the training image.

Images