3D measuring device

Abstract

This invention provides a 3D shape measuring device capable of accurately measuring the 3D shape of an object, including parts that are transparent to visible light. [Solution] A 3D measuring device (100) that creates 3D shape data of an object to be measured (140) from captured images of the object to be measured comprises an image capture unit (110) sensitive to ultraviolet light, and a processing unit (120) that creates 3D shape data of the object to be measured based on a plurality of images of the object to be measured captured by the image capture unit. The processing unit comprises a correspondence point estimation unit (124) that estimates correspondence points between the plurality of images, a shape calculation unit (125) that calculates 3D spatial information for creating 3D shape data of the object based on the estimation results by the correspondence point estimation unit, and a shape output unit (126) that creates 3D shape data of the object based on the 3D spatial information calculated by the shape calculation unit. The correspondence point estimation unit estimates the correspondence points using a deep learning model.

Description

Technical Field

[0001] The present invention relates to a three-dimensional measuring device.

Background Art

[0002] In recent years, with the development of machine vision technology, it has been increasingly required to measure the three-dimensional shape of an object. For example, in the logistics and industrial fields, three-dimensional measurement is used to measure the distance to an object, automatically pick the object with a robot, or measure the size of the object, so as to perform packaging according to the shape of the object. This is strongly required.

[0003] However, when the measurement object has a transparent part with respect to visible light, it is generally difficult to perform three-dimensional measurement of the transparent part using a visible light camera. For example, consider a case where a product is packaged with transparent plastic like a blister pack. At this time, the three-dimensional shape to be measured is the shape of the entire product including the transparent plastic. However, in a three-dimensional measuring device using visible light or near-infrared light, usually, only the three-dimensional shape of the product inside is acquired, and the three-dimensional shape of the transparent packaging material cannot be acquired. At this time, the shape measured by the three-dimensional measuring device is estimated to be smaller than the size of the shape that should be truly measured. Therefore, there is a risk of destroying the packaging during picking by a picking robot, and when performing packaging according to the shape of the object, there is a risk of packaging errors due to an underestimated measurement size.

[0004] As a technology that enables measurement of the three-dimensional shape of an object including such a transparent part, for example, in Patent Document 1, information obtained from a far-infrared camera and a heating unit is complemented to the distance information obtained by edge detection using a stereo camera, thereby enabling measurement of the three-dimensional shape of a transparent object. In addition, Patent Document 2 discloses a technology that enables measurement of the three-dimensional shape of an object including a transparent object by using ultraviolet light diffusely reflected on the object surface.

Prior Art Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2024-19989 [Patent Document 2] Special Publication No. 2013-540999 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, in Patent Document 1, the far-infrared camera needs to capture the changes before and after heating by the heating element, which takes time for measurement and makes real-time processing impossible. Furthermore, since the stereo camera captures only edge information and the far-infrared camera supplements it, it is considered difficult to measure complex shapes.

[0007] Furthermore, Patent Document 2 requires that the surface of the object to be measured diffusely reflect an illumination light beam with a purple or ultraviolet wavelength. However, transparent glass and transparent plastics that are not translucent specularly reflect light cannot be used for objects to be measured that include transparent parts that do not diffusely reflect light.

[0008] One aspect of the present invention aims to provide a three-dimensional shape measuring device capable of accurately measuring the three-dimensional shape of an object to be measured, including a portion that is transparent to visible light. [Means for solving the problem]

[0009] To solve the above problems, a three-dimensional measuring device according to one aspect of the present invention is a three-dimensional measuring device that creates three-dimensional shape data of a measuring object from captured images of the measuring object, comprising: an image capture unit sensitive to ultraviolet light; and a processing unit that creates three-dimensional shape data of the measuring object based on a plurality of images of the measuring object captured by the image capture unit, wherein the processing unit comprises: a correspondence point estimation unit that estimates correspondence points between the plurality of images; a shape calculation unit that calculates three-dimensional spatial information for creating three-dimensional shape data of the measuring object based on the estimation results by the correspondence point estimation unit; and a shape output unit that creates three-dimensional shape data of the measuring object based on the three-dimensional spatial information calculated by the shape calculation unit, wherein the correspondence point estimation unit estimates the correspondence points using a deep learning model. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to accurately measure the three-dimensional shape of an object to be measured, including a portion that is transparent to visible light. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to the first embodiment of the present invention. [Figure 2] This is a schematic diagram showing examples of images captured using visible light and ultraviolet light, and the corresponding three-dimensional shapes. [Figure 3] This figure shows the transmittance of various transparent plastics when using light sources of different wavelengths. [Figure 4] This is a block diagram showing the configuration of the corresponding point estimation unit 124. [Figure 5] This is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to a second embodiment of the present invention. [Figure 6] This is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to a third embodiment of the present invention. [Figure 7] This is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to a third embodiment of the present invention.

Embodiments for Carrying out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] 〔First Embodiment〕 FIG. 1 is a schematic diagram for explaining the configuration of a three-dimensional measuring device according to the first embodiment of the present invention.

[0014] [Configuration of Three-Dimensional Measuring Device 100] As shown in FIG. 1, the three-dimensional measuring device 100 according to the present embodiment measures the three-dimensional shape of the measurement object 140, and includes an image capturing unit 110, a processing unit 120, and a light source unit 130.

[0015] [Image Capturing Unit 110] The image capturing unit 110 captures an image of the measurement object 140. The image capturing unit 110 is sensitive to ultraviolet rays, and in the present embodiment, it is composed of two cameras 111 and 112 that are sensitive to ultraviolet rays. Note that three or more cameras may be provided in the image capturing unit 110, and any two of the three or more cameras may be selected and used as the two cameras 111 and 112.

[0016] The two cameras 111 and 112 form a stereo camera, and are arranged side by side in the horizontal direction so that the imaging surfaces are parallel to each other. <00OO090> Also, in the present embodiment, each of the cameras 111 and 112 is composed of an ultraviolet camera provided with a monochrome imaging element having sensitivity to ultraviolet rays of the wavelength to be imaged.

[0018] Each camera 111, 112 has sensitivity to ultraviolet light, and it may be a camera that has sensitivity only in the ultraviolet region or a camera that has sensitivity in both the ultraviolet region and the visible to near-infrared regions. However, when using a camera that has sensitivity in both the ultraviolet region and the visible to near-infrared regions, it is desirable to ensure that visible to near-infrared light does not have an adverse effect on the measurement. As a method for excluding the adverse effect of visible to near-infrared light, it is conceivable to use a filter (for example, a band-pass filter) that can selectively extract only the ultraviolet light of the wavelength used for the measurement.

[0019] Also, the image capturing unit 110 (each camera 111, 112) is connected to the processing unit 120 via a predetermined interface (for example, a USB interface), and for example, periodically transmits the captured image to the processing unit 120.

[0020] [Processing unit 120] The processing unit 120 creates three-dimensional shape data of the measurement object 140 based on a plurality of images of the measurement object 140 captured by the image capturing unit 110. Details of the processing unit 120 will be described later.

[0021] [Light source unit 130] The light source unit 130 irradiates the measurement object 140 with ultraviolet light 131 (an ultraviolet light source). As the ultraviolet light 131, for example, those having a wavelength of 200 to 405 nm can be used. Note that the light source unit 130 only needs to include ultraviolet light in a part of the emitted light, and the emitted light may include visible light or near-infrared light. The light source unit 130 is constituted by, for example, an LED, an excimer lamp, or a mercury lamp. The shorter the wavelength, the smaller the transmittance for transparent objects, and the more the measurable objects increase.

[0022] Next, three-dimensional measurement in the three-dimensional measurement apparatus 100 having the above configuration will be described.

[0023] <Here, as shown in Figure 1, we assume that the object to be measured 140 is a container 141 wrapped in transparent packaging material 142. Furthermore, during the photography process, the object to be measured 140 is assumed to be placed on a mounting surface 143. In this case, the mounting surface 143 serves as the background. We then consider measuring the three-dimensional shape of the transparent packaging material 142 that forms the outermost shape of the object to be measured 140.

[0024] In this situation, if the object to be measured 140 is photographed with a visible light camera, the captured image will show the contents 141 of the object to be measured 140, as shown in Figure 2(a). On the other hand, the transparent packaging material 142 may or may not be partially visible depending on conditions such as the position of the light source. Regardless of whether the transparent packaging material 142 is partially visible or not, if the object to be measured 140 is photographed with a visible light camera, an image showing the contents 141 will be obtained. If the 3D shape is calculated using this captured image, generally, as shown in Figure 2(b), only the 3D shape derived from the contents 141 is obtained from the portion of the captured image where the contents 141 are visible, and the 3D shape derived from the transparent packaging material 142, which should be measured, is not obtained.

[0025] To measure the three-dimensional shape of the transparent packaging material 142, it is necessary that the contents 141 behind the transparent packaging material 142 are not visible in the captured image. In other words, it is important that the light used for measurement does not pass through the transparent packaging material 142.

[0026] In this embodiment, ultraviolet light is used as the light that does not pass through the transparent material.

[0027] When the object to be measured 140 is photographed using a camera sensitive to ultraviolet light and ultraviolet light of a wavelength with low transmittance to the transparent packaging material 142, an image is obtained in which the transparent packaging material 142 appears black, as shown in Figure 2(c). When the transmittance of ultraviolet light to the transparent packaging material 142 is sufficiently low, the ultraviolet light does not reach the contents 141. In addition, ultraviolet light scattered from the surface of the object to be measured 140 does not reach the camera. Therefore, by using ultraviolet light with sufficiently low transmittance to the transparent packaging material, it is possible to measure only the three-dimensional shape originating from the transparent packaging material 142 without obtaining information about the contents 141, as shown in Figure 2(d).

[0028] Thus, by using a camera sensitive to ultraviolet light and ultraviolet light of a wavelength that has low transmittance through the transparent packaging material 142, it is not possible to obtain information about the contents 141. However, this does not necessarily mean that the three-dimensional shape of the transparent packaging material 142 can be accurately measured.

[0029] In particular, in disparity calculation using stereo matching, which is used in 3D measurement, corresponding points are found between multiple images from different viewpoints, and the disparity, which is the difference between the corresponding points in each image, is calculated to obtain the 3D shape. However, generally speaking, it is considered extremely difficult, and practically impossible, to find corresponding points in areas without texture or in areas with repeating patterns. Therefore, it becomes practically impossible to measure the 3D shape of parts where texture does not exist.

[0030] Images of the transparent packaging material 142 taken with a camera sensitive to ultraviolet light and using ultraviolet light of a wavelength that has low transmittance through the transparent packaging material 142 are uniformly black and lack texture. Therefore, it is not possible to perform three-dimensional measurement of the transparent packaging material 142 using general calculation methods.

[0031] Therefore, in this embodiment, inference based on deep learning performed in advance is used as a method for determining parallax from multiple ultraviolet images. In this embodiment, by using a deep learning model that can estimate areas where texture does not exist, it is possible to measure the three-dimensional shape of the transparent packaging material 142.

[0032] In this embodiment, the stereo model IGEV, a deep learning model for visible light images, is extended to ultraviolet images to enable the measurement of the three-dimensional shape of the transparent packaging material 142.

[0033] Furthermore, it is also possible to extend and apply other deep learning stereo models for visible light images (e.g., MonSter, FoundationStereo, etc.).

[0034] Furthermore, while transparent packaging materials were used as an example here, the above-mentioned phenomena generally occur with objects that contain transparent parts, either entirely or partially, and are not limited to transparent packaging materials.

[0035] Furthermore, even if the object being measured does not contain transparent parts, its three-dimensional shape can be measured in the same way as when it does contain transparent parts.

[0036] Figure 3 shows the transmittance of various transparent plastics when using light sources of different wavelengths.

[0037] Here, as shown in Figure 3(a), a marker 10 was prepared by attaching black tape in a cross shape to a polytetrafluoroethylene (PTFE) plate. Various transparent plastic plates 11-14 were then placed on top of the marker 10, and images were taken using light sources of different wavelengths. If the light of the wavelength used for imaging passed through the transparent plastic plate, the marker was captured in the image. If all of the light of the wavelength used for imaging was absorbed by the transparent plastic plate, the marker was not captured in the image. The fact that the marker behind the transparent plastic plate was not captured in the image means that three-dimensional measurement of the transparent plastic portion is possible.

[0038] Here, as transparent plastic sheets, we used polypropylene (11), polyethylene terephthalate (PET) (12), rigid polyvinyl chloride (13), and acrylic (14). As light sources, we used ultraviolet light with wavelengths of 222 nm, 254 nm, 280 nm, 365 nm, and 395 nm, and blue light with a wavelength of 470 nm. Figure 3(b) shows the results when using ultraviolet light with a wavelength of 222 nm, Figure 3(c) shows the results when using ultraviolet light with a wavelength of 254 nm, Figure 3(d) shows the results when using ultraviolet light with a wavelength of 280 nm, Figure 3(e) shows the results when using ultraviolet light with a wavelength of 365 nm, Figure 3(f) shows the results when using ultraviolet light with a wavelength of 395 nm, and Figure 3(g) shows the results when using blue light with a wavelength of 470 nm.

[0039] As shown in Figure 3(g), when light with a wavelength of 470 nm is used, all plastic plates appear transparent, indicating that light is transmitted. As shown in Figure 3(f), when light with a wavelength of 395 nm is used, rigid PVC and acrylic plates appear dark due to absorption, but the markers are still visible. As shown in Figure 3(e), when light with a wavelength of 365 nm is used, the markers do not appear on the rigid PVC and acrylic plates, indicating that 3D measurement is possible for these two types of plastic plates. As shown in Figures 3(b) to (d), at wavelengths of 280 nm, 254 nm, and 222 nm, the markers do not appear on all four types of plastic plates, indicating that 3D measurement is possible. As described above, using short-wavelength ultraviolet light is effective for 3D measurement of objects, including transparent ones.

[0040] Furthermore, while ultraviolet (UV) radiation generally has greater adverse effects on the human body as its wavelength decreases, UV radiation with wavelengths of 200-235 nm has been reported to be relatively safe for humans. Therefore, using UV radiation with wavelengths of 200-235 nm makes it possible to perform safe measurements even in environments with people present. Examples of UV light sources that emit UV radiation of this wavelength include, for example, an excimer lamp that emits UV radiation with a wavelength of 222 nm, a fluorescent excimer lamp (a combination of an excimer lamp and a phosphor) that emits UV radiation with a wavelength of 228 nm, an LED that emits UV radiation with a wavelength of 230 nm, and an LED that emits UV radiation with a wavelength of 235 nm.

[0041] Next, we will describe the details of the processing unit 120.

[0042] [Configuration of Processing Unit 120] As shown in Figure 1, the processing unit 120 includes an image acquisition unit 121, an aberration correction unit 122, a parallelization processing unit 123, a corresponding point estimation unit 124, a shape calculation unit 125, and a shape output unit 126.

[0043] [Image acquisition unit 121] The image acquisition unit 121 acquires images from each of the cameras 111 and 112 that make up the image shooting unit 110.

[0044] [Aberration correction unit 122 and parallelization processing unit 123] The aberration correction unit 122 corrects image distortion caused by optical aberrations occurring in the lenses and optical system, based on distortion information of the lenses of cameras 111 and 112 obtained in advance. The parallelization processing unit 123 performs processing so that the image planes of the two images obtained by cameras 111 and 112, which constitute the stereo camera, lie on the same plane and the rows of images are aligned. Note that the aberration correction unit 122 and the parallelization processing unit 123 can be omitted.

[0045] [Correspondence point estimation unit 124] The correspondence point estimation unit 124 estimates corresponding points in each image from a plurality of images (two in this embodiment) captured by the image capture unit 110 and acquired by the image acquisition unit 121. In this embodiment, the correspondence point estimation unit 124 utilizes a deep learning model and uses inference based on deep learning performed in advance to estimate (calculate) corresponding points in each image from two ultraviolet images taken by cameras 111 and 112 from different viewpoints.

[0046] [Shape calculation section 125] The shape calculation unit 125 calculates three-dimensional spatial information (in this embodiment, the parallax of each pixel in the reference image) for creating three-dimensional shape data of an object, based on the estimation results (correspondence information obtained by the correspondence point estimation unit 124) from the correspondence point estimation unit 124.

[0047] [Shape output unit 126] The shape output unit 126 creates and outputs three-dimensional shape data of the object to be measured based on the three-dimensional spatial information (parallax in this embodiment) calculated by the shape calculation unit 125 and the camera parameters obtained in advance.

[0048] Next, we will explain the details of the corresponding point estimation unit 124.

[0049] Figure 4 is a block diagram showing the configuration of the corresponding point estimation unit 124.

[0050] [Configuration of the Corresponding Point Estimation Unit 124] As shown in Figure 4, the correspondence point estimation unit 124 comprises a feature extraction unit 1241 and a cost volume calculation unit 1242.

[0051] [Feature extraction unit 1241] The feature extraction unit 1241 uses a convolutional neural network (CNN) to extract features from two ultraviolet images taken from different viewpoints by cameras 111 and 112.

[0052] Depending on the deep learning model used, other configurations (for example, Vision Transformer (ViT), etc.) may also be considered.

[0053] Furthermore, in this embodiment, existing CG image datasets (for example, the Sceneflow dataset or Tartan Air) are used as training data for the convolutional neural network. Alternatively, a dataset of visible light images actually captured by a camera may be used.

[0054] It is also possible to use a dataset of ultraviolet images as training data.

[0055] [Cost volume calculation unit 1242] The cost volume calculation unit 1242 calculates the similarity between two ultraviolet images taken from different viewpoints, based on the features extracted by the feature extraction unit 1241.

[0056] The cost volume calculated by the cost volume calculation unit 1242 is passed to the shape calculation unit 125.

[0057] The shape calculation unit 125 calculates three-dimensional spatial information (in this embodiment, parallax) based on the cost volume calculated by the cost volume calculation unit 1242.

[0058] The processing unit 120 is configured, for example, by a regular personal computer or other computer. The processing unit 120 includes, as hardware, a CPU (Central Processing Unit), main memory such as RAM, auxiliary storage devices such as HDDs and SSDs, an external interface unit such as a USB interface unit, a display device, an input device, etc. The display device is configured, for example, by an LCD display device, and the input device is configured, for example, by a keyboard, mouse or other pointing device. The processing unit 120 may further include one or more GPUs (Graphics Processing Units).

[0059] The above parts 121 to 126 are basically implemented by the CPU executing a program loaded into main memory.

[0060] In this embodiment, which has the above configuration, ultraviolet light is used as the light for shooting, and a deep learning model is used to estimate corresponding points in multiple images, so that even if the object to be measured includes transparent parts, it is possible to create 3D shape data.

[0061] [Second Embodiment] Next, a second embodiment of the present invention will be described.

[0062] Figure 5 is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to a second embodiment of the present invention.

[0063] [Configuration of the 3D measuring device 200] As shown in Figure 5, the 3D measuring device 200 according to this embodiment measures the 3D shape of the object to be measured 140, and comprises an image capture unit 210, a processing unit 220, and a light source unit 130.

[0064] [Light source unit 130 and object to be measured 140] The light source unit 130 and the object to be measured 140 are the same as those in the first embodiment described above, so further explanation is omitted.

[0065] [Image Capture Unit 210] The image capture unit 210, like the image capture unit 110, captures images of the object to be measured 140 and is sensitive to ultraviolet light. On the other hand, in this embodiment, the image capture unit 210 captures images from three or more different viewpoints and is composed of three cameras 211 to 213 that are sensitive to ultraviolet light. Since each of the cameras 211 to 213 is the same as the cameras 111 and 112 in the first embodiment described above, a detailed explanation of each camera is omitted.

[0066] In this embodiment, the image capture unit 210 is configured with three cameras 211 to 213, but it is also conceivable to configure it with four or more cameras.

[0067] Furthermore, the image capture unit 210 (each camera 211-213) is connected to the processing unit 220 via a predetermined interface (for example, a USB interface), and periodically transmits captured images to the processing unit 220.

[0068] [Configuration of Processing Unit 220] As shown in Figure 5, the processing unit 220 includes an image acquisition unit 221, an aberration correction unit 222, a camera position estimation unit 223, a corresponding point estimation unit 224, a shape calculation unit 225, and a shape output unit 226.

[0069] [Image acquisition unit 221 and aberration correction unit 222] The image acquisition unit 221 and the aberration correction unit 222 are the same as the image acquisition unit 121 and the aberration correction unit 122 in the first embodiment described above, so a detailed explanation will be omitted.

[0070] [Camera position estimation unit 223] The camera position estimation unit 223 estimates the position and orientation of each camera 211 to 213 from multiple images acquired by the image acquisition unit 221. The camera position estimation unit 223 can be omitted, for example, if the camera positions can be determined separately.

[0071] [Correspondence point estimation unit 224] The correspondence point estimation unit 224 estimates corresponding points in each of the three images captured by the image capture unit 210 and acquired by the image acquisition unit 221. In this embodiment, the correspondence point estimation unit 224 utilizes a deep learning model and uses inference based on pre-performed deep learning to estimate (calculate) corresponding points in each of the three ultraviolet images taken by cameras 211 to 213 from different viewpoints. The configuration of the correspondence point estimation unit 224 is the same as that of the correspondence point estimation unit 124 in the first embodiment described above, so a detailed explanation is omitted.

[0072] [Shape calculation section 225] The shape calculation unit 225 calculates three-dimensional spatial information (depth in this embodiment) for creating three-dimensional shape data of the object to be measured, based on the estimation results (correspondence information obtained by the correspondence point estimation unit 224) from the correspondence point estimation unit 224.

[0073] [Shape output unit 226] The shape output unit 226 creates and outputs three-dimensional shape data of the object to be measured based on the three-dimensional spatial information (depth in this embodiment) calculated by the shape calculation unit 225 and the camera parameters obtained in advance.

[0074] In this embodiment, which has the above configuration, ultraviolet light is used as the light for shooting, and a deep learning model is used to estimate corresponding points in multiple images, so that even if the object to be measured includes transparent parts, it is possible to create 3D shape data.

[0075] Furthermore, this embodiment uses images captured from three or more different viewpoints. Therefore, compared to the first embodiment which uses two viewpoints, the ambiguity of corresponding point estimation can be reduced. Also, in the case of two viewpoints, the problem of occlusion may occur where a part of the object being measured is not captured in the image, but this problem can be solved by using multiple viewpoints.

[0076] [Third Embodiment] Next, a third embodiment of the present invention will be described.

[0077] Figure 6 is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to a third embodiment of the present invention.

[0078] [Configuration of the 3D measuring device 300] As shown in Figure 6, the 3D measuring device 300 according to this embodiment measures the 3D shape of the object to be measured 140, and comprises an image capture unit 310, a processing unit 320, and a light source unit 130.

[0079] [Light source unit 130 and object to be measured 140] The light source unit 130 and the object to be measured 140 are the same as those in the first embodiment described above, so further explanation is omitted.

[0080] [Image Capture Unit 310] The image capture unit 310, like the image capture unit 110, captures images of the object to be measured 140, is sensitive to ultraviolet light, and, like the image capture unit 210, captures images from three or more different viewpoints. On the other hand, in this embodiment, the image capture unit 310 is composed of a single camera 311 that is sensitive to ultraviolet light and is configured to be movable. In this embodiment, multiple images are captured while changing the relative position between the single camera 311 and the object to be measured 140, that is, by moving the single camera 311 to three or more different positions and taking images, it is possible to capture images from three or more different viewpoints. The camera 311 itself is the same as the cameras 111 and 112 in the first embodiment described above, so a detailed explanation is omitted. Furthermore, any known mechanism (for example, a robot arm) can be used as the mechanism for moving the camera 311.

[0081] Furthermore, the image capture unit 310 (camera 311) is connected to the processing unit 320 via a predetermined interface (for example, a USB interface), and periodically transmits captured images to the processing unit 320.

[0082] [Configuration of Processing Unit 320] As shown in Figure 5, the processing unit 320 includes an image acquisition unit 321, an aberration correction unit 322, a camera position estimation unit 323, a corresponding point estimation unit 324, a shape calculation unit 325, and a shape output unit 326.

[0083] [Image acquisition unit 321 and aberration correction unit 322] The image acquisition unit 321 and the aberration correction unit 322 are the same as the image acquisition unit 121 and the aberration correction unit 122 in the first embodiment described above, so a detailed explanation will be omitted.

[0084] [Camera position estimation unit 323] The camera position estimation unit 323 estimates the position and orientation of the camera 311 at the time of shooting from multiple images captured by the image shooting unit 310 and acquired by the image acquisition unit 321. The camera position estimation unit 323 can be omitted, for example, if the camera's position can be determined separately.

[0085] [Correspondence point estimation unit 324] The correspondence point estimation unit 324 estimates corresponding points in each image from multiple images captured by the image capture unit 310 and acquired by the image acquisition unit 321. In this embodiment, the correspondence point estimation unit 324 utilizes a deep learning model and uses inference based on deep learning performed in advance to estimate (calculate) corresponding points in each image from multiple ultraviolet images with different viewpoints. The configuration of the correspondence point estimation unit 324 is the same as that of the correspondence point estimation unit 124 in the first embodiment described above, so a detailed explanation is omitted.

[0086] [Shape calculation section 325] The shape calculation unit 325 calculates three-dimensional spatial information (depth in this embodiment) for creating three-dimensional shape data of the object to be measured, based on the estimation results (correspondence information obtained by the correspondence point estimation unit 324) from the correspondence point estimation unit 324.

[0087] [Shape output unit 326] The shape output unit 326 creates and outputs three-dimensional shape data of the object to be measured based on the three-dimensional spatial information (depth in this embodiment) calculated by the shape calculation unit 325 and the camera parameters obtained in advance.

[0088] In this embodiment, which has the above configuration, ultraviolet light is used as the light for shooting, and a deep learning model is used to estimate corresponding points in multiple images, so that even if the object to be measured includes transparent parts, it is possible to create 3D shape data.

[0089] Furthermore, this embodiment uses images captured from three or more different viewpoints. Therefore, compared to the first embodiment which uses two viewpoints, the ambiguity of corresponding point estimation can be reduced. Also, in the case of two viewpoints, the problem of occlusion may occur where a part of the object being measured is not captured in the image, but this problem can be solved by using multiple viewpoints.

[0090] [Fourth Embodiment] Next, a fourth embodiment of the present invention will be described.

[0091] Figure 7 is a schematic diagram illustrating the configuration of a three-dimensional measuring device according to the fourth embodiment of the present invention.

[0092] [Configuration of the 3D measuring device 400] As shown in Figure 7, the 3D measuring device 400 according to this embodiment measures the 3D shape of the object to be measured 140, and comprises an image capture unit 410, a processing unit 420, and a light source unit 130.

[0093] [Light source unit 130 and object to be measured 140] The light source unit 130 and the object to be measured 140 are the same as those in the first embodiment described above, so further explanation is omitted. However, in this embodiment, a moving mechanism (object to be measured movement unit) is provided (not shown), and the position of the object to be measured 140 can be changed by this moving mechanism. Any known mechanism (for example, an XY table or a conveyor) can be used as the mechanism to move the object to be measured 140.

[0094] [Image Capture Unit 410] The image capture unit 410, like the image capture unit 110, captures images of the object to be measured 140, is sensitive to ultraviolet light, and, like the image capture unit 210, captures images from three or more different viewpoints, and, like the image capture unit 310, is composed of a single camera 411 that is sensitive to ultraviolet light. On the other hand, in this embodiment, in order to move the camera 411 relative to the object to be measured 140, instead of moving the camera 411, the object to be measured 140 is moved to three or more different positions and then captured, thereby achieving capture from three or more different viewpoints. The camera 411 itself is the same as the cameras 111 and 112 in the first embodiment described above, so a detailed explanation is omitted.

[0095] Furthermore, the image acquisition unit 410 (camera 411) is connected to the processing unit 420 via a predetermined interface (for example, a USB interface), and periodically transmits the captured images to the processing unit 420.

[0096] [Configuration of Meter 420] As shown in Figure 7, the processing unit 420 includes an image acquisition unit 421, an aberration correction unit 422, a camera position estimation unit 423, a corresponding point estimation unit 424, a shape calculation unit 425, and a shape output unit 426.

[0097] [Image acquisition unit 421 and aberration correction unit 422] The image acquisition unit 421 and the aberration correction unit 422 are the same as those in the first embodiment described above, so a detailed explanation will be omitted.

[0098] [Correspondence point estimation unit 424] The correspondence point estimation unit 424 estimates corresponding points in each image from multiple images captured by the image capture unit 410 and acquired by the image acquisition unit 421. In this embodiment, the correspondence point estimation unit 424 utilizes a deep learning model and uses inference based on deep learning performed in advance to estimate (calculate) corresponding points in each image from multiple ultraviolet images with different viewpoints. The configuration of the correspondence point estimation unit 424 is the same as that of the correspondence point estimation unit 124 in the first embodiment described above, so a detailed explanation is omitted.

[0099] [Shape calculation section 425] The shape calculation unit 425 calculates three-dimensional spatial information (in this embodiment, parallax or depth) for creating three-dimensional shape data of the object to be measured, based on the estimation results (correspondence information obtained by the correspondence point estimation unit 424) from the correspondence point estimation unit 424.

[0100] [Shape output unit 426] The shape output unit 426 creates and outputs three-dimensional shape data of the object to be measured based on the three-dimensional spatial information (depth in this embodiment) calculated by the shape calculation unit 425 and the camera parameters obtained in advance.

[0101] In this embodiment, which has the above configuration, ultraviolet light is used as the light for shooting, and a deep learning model is used to estimate corresponding points in multiple images, so that even if the object to be measured includes transparent parts, it is possible to create 3D shape data.

[0102] Furthermore, this embodiment uses images captured from three or more different viewpoints. Therefore, compared to the first embodiment which uses two viewpoints, the ambiguity of corresponding point estimation can be reduced. Also, in the case of two viewpoints, the problem of occlusion may occur where a part of the object being measured is not captured in the image, but this problem can be solved by using multiple viewpoints.

[0103] [Examples of implementation using software] The functions of the three-dimensional measuring devices (hereinafter also referred to as "devices") according to each of the above embodiments can be realized by programs that cause a computer to function as the device, and by programs that cause a computer to function as each functional block of the device.

[0104] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0105] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0106] Furthermore, some or all of the functions of each of the above-mentioned functional blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits that function as each of the above-mentioned functional blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above-mentioned functional blocks by, for example, a quantum computer.

[0107] 〔summary〕 This specification describes at least the following configurations:

[0108] The 3D measuring device according to Embodiment 1 is a 3D measuring device that creates 3D shape data of an object to be measured from captured images of the object to be measured, comprising: an image capture unit sensitive to ultraviolet light; and a processing unit that creates 3D shape data of the object to be measured based on a plurality of images of the object to be measured captured by the image capture unit, wherein the processing unit comprises: a correspondence point estimation unit that estimates correspondence points between the plurality of images; a shape calculation unit that calculates 3D spatial information for creating 3D shape data of the object to be measured based on the estimation results by the correspondence point estimation unit; and a shape output unit that creates 3D shape data of the object to be measured based on the 3D spatial information calculated by the shape calculation unit, wherein the correspondence point estimation unit estimates the correspondence points using a deep learning model.

[0109] With the above configuration, ultraviolet light is used as the light source for shooting, and a deep learning model is used to estimate corresponding points in multiple images, making it possible to create 3D shape data even for objects that include transparent parts.

[0110] The three-dimensional measuring device according to Embodiment 2 is characterized in that, in the three-dimensional measuring device according to Embodiment 1, it further comprises an ultraviolet light source for irradiating the object to be measured with ultraviolet light, and the wavelength of the ultraviolet light is 200 to 405 nm.

[0111] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0112] The three-dimensional measuring device according to Embodiment 3 is characterized in that, in the three-dimensional measuring device according to Embodiment 2, the wavelength of the ultraviolet light is 200 to 235 nm.

[0113] According to the above configuration, since ultraviolet light of a specific wavelength is used as the light for shooting, adverse effects on the human body can be relatively reduced, making safe measurements possible even in environments with people present.

[0114] The three-dimensional measuring device according to Embodiment 4 is characterized in that, in the three-dimensional measuring device according to any of Embodiments 1 to 3, the image acquisition unit is equipped with two cameras, and the two cameras constitute a stereo camera.

[0115] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0116] The three-dimensional measuring device according to embodiment 5 is characterized in that, in the three-dimensional measuring device according to embodiment 4, the shape calculation unit calculates the parallax based on the estimation result by the corresponding point estimation unit.

[0117] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0118] The 3D measuring device according to embodiment 6 is characterized in that, in the 3D measuring device according to any of embodiments 1 to 3, the image acquisition unit is equipped with three or more cameras.

[0119] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0120] The three-dimensional measuring device according to embodiment 7 is characterized in that, in the three-dimensional measuring device according to embodiment 6, the shape calculation unit calculates the depth based on the estimation result by the corresponding point estimation unit.

[0121] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0122] The 3D measuring device according to embodiment 8 is a 3D measuring device according to any of embodiments 1 to 3, characterized in that the image acquisition unit includes a movable camera and acquires multiple images while changing the relative position between the camera and the object to be measured.

[0123] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0124] The 3D measuring device according to embodiment 9 is a 3D measuring device according to any of embodiments 1 to 3, further comprising a measuring object moving unit for moving the position of the measuring object, and the image capturing unit is characterized in that it captures multiple images at multiple different positions of the measuring object.

[0125] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0126] The three-dimensional measuring device according to embodiment 10 is characterized in that, in the three-dimensional measuring device according to any of embodiments 1 to 9, the image acquisition unit is composed of a camera that is sensitive only to ultraviolet light.

[0127] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts.

[0128] The three-dimensional measuring device according to embodiment 11 is characterized in that, in the three-dimensional measuring device according to any of embodiments 1 to 9, the image acquisition unit is composed of a filter that can selectively extract ultraviolet light and a camera that is sensitive to at least visible light and infrared light.

[0129] With the above configuration, similar to embodiment 1, it is possible to create 3D shape data even for objects that include transparent parts. [Explanation of Symbols]

[0130] 100, 200, 300, 400 3D measuring devices 110, 210, 310, 410, Image Capture Department 111, 112, 211, 311, 411 cameras 120, 220, 320, 420 processing units 121, 221, 321, 421 Image acquisition unit 122, 222, 322, 422 Aberration Correction Section 123 Parallelization Processing Unit 124, 224, 324, 424 Corresponding Point Estimation Unit 125, 225, 325, 425 Shape calculation section 126, 226, 326, 426 Shape output section 130 Light source section 131 Ultraviolet rays 140 Objects to be measured 141 Contents 142 Transparent packaging materials 143 Mounting surface 223, 323, 323], 423 Camera position estimation unit 1241 Feature extraction unit 1242 Cost Volume Calculation Unit

Claims

1. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit which creates three-dimensional shape data of the object to be measured based on a plurality of images of the object to be measured taken by the image acquisition unit. Equipped with, The aforementioned image capture unit is equipped with two cameras, The two cameras in question constitute a stereo camera. The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

2. The shape calculation unit calculates the parallax based on the estimation results from the corresponding point estimation unit. The three-dimensional measuring device according to feature 1.

3. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit which creates three-dimensional shape data of the object to be measured based on a plurality of images of the object to be measured taken by the image acquisition unit. Equipped with, The aforementioned image capture unit is equipped with three or more cameras, The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

4. The shape calculation unit calculates the depth based on the estimation result by the corresponding point estimation unit. The three-dimensional measuring device according to feature 3.

5. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit which creates three-dimensional shape data of the object to be measured based on a plurality of images of the object to be measured taken by the image acquisition unit. Equipped with, The aforementioned image capture unit is Equipped with a single portable camera, Multiple images are taken while changing the relative position between the camera and the object to be measured. The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

6. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit that creates three-dimensional shape data of the object to be measured based on multiple images of the object to be measured taken by the image acquisition unit, A measuring object moving unit that moves the position of the object to be measured, Equipped with, The aforementioned image capture unit is Multiple images are taken at multiple different locations on the object to be measured. The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

7. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit which creates three-dimensional shape data of the object to be measured based on a plurality of images of the object to be measured taken by the image acquisition unit. Equipped with, The image capture unit is comprised of a camera that is sensitive only to ultraviolet light. The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

8. A three-dimensional measuring device that creates three-dimensional shape data of an object to be measured from an image of the object to be measured, An image capture unit that is sensitive to ultraviolet light, A processing unit which creates three-dimensional shape data of the object to be measured based on a plurality of images of the object to be measured taken by the image acquisition unit. Equipped with, The aforementioned image capture unit is A filter that can selectively extract ultraviolet light, A camera sensitive to ultraviolet light and at least visible light and infrared light. It is composed of, The aforementioned processing unit, A correspondence point estimation unit that estimates corresponding points between the plurality of images, A shape calculation unit calculates three-dimensional spatial information for creating three-dimensional shape data of the object to be measured, based on the estimation results by the corresponding point estimation unit. A shape output unit creates three-dimensional shape data of the object to be measured based on the three-dimensional spatial information calculated by the shape calculation unit. Equipped with, The corresponding point estimation unit estimates the corresponding points using a deep learning model. A three-dimensional measuring device characterized by the following features.

9. The measurement object is further provided with an ultraviolet light source that irradiates ultraviolet light onto the object to be measured. The wavelength of the ultraviolet light is 200 to 405 nm. A three-dimensional measuring device according to any one of claims 1 to 8.

10. The wavelength of the ultraviolet light is 200 to 235 nm. The three-dimensional measuring device according to feature 9.

Images