Sensing device and sensing method

Abstract

To provide a sensing device and a sensing method which can correctly measure a three-dimensional shape of an object including a transparent raw material.SOLUTION: A computing machine 1 generates an image 35 on the basis of visual information of an object 30, extracts a region occupied by the object 30 in the image 35 as an object region 36, generates edge distance information 37 by extracting distance information of an edge portion of the object 30 in the distance information of the object 30, generates a far-infrared image 41 corresponding to the object region 36 on the basis of far-infrared ray information of the object 30, controls a heating device 4 so as to heat the object 30, estimates partial surface shape information 44, 45 of the object 30 from the far-infrared images 41, 42 before and after heating of the object 30, generates interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 by using the partial surface shape information 44, 45, and converts the interpolated edge distance information 57 into the three-dimensional shape information to output the converted information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a sensing device and a sensing method. [Background technology]

[0002] In recent years, with the declining labor force due to a declining birthrate and aging population, expectations are growing for automated and autonomous system control to alleviate labor shortages and improve productivity. For example, in the industrial and logistics fields, there is a high demand for robots that can automatically pick workpieces. Automating picking tasks requires the placement of sensors such as cameras to measure the 3D shape of the workpiece and teach the robot information such as the position where the workpiece should be grasped. A common method for measuring 3D shapes is to install multiple cameras or a camera and a projector and measure using the principle of triangulation. In this case, the camera often uses a visible light sensor, which can accurately recognize the 3D shape of workpieces packaged in materials such as paper. However, when workpieces are packaged in transparent materials such as blister packs, the shape of the packaging cannot be captured and only the shape of the workpiece itself is captured, which can lead to the risk of crushing and destroying the transparent packaging during picking. Therefore, a method using both a visible light sensor and a far-infrared camera is expected to capture the 3D shape of transparent parts. For example, in Patent Document 1, a three-dimensional shape is acquired by measuring a workpiece from multiple locations using an imaging device equipped with a fixed visible light camera and far-infrared camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-032600 Summary of the Invention [Problem to be solved by the invention]

[0004] By utilizing the technology of Patent Document 1, it is possible to correctly recognize the shape of the transparent part of a workpiece whose body is wrapped in a transparent material, and pick the workpiece without destroying the wrapping. However, moving the imaging device to capture multiple images not only depends on the accuracy of calibration, but also takes time to pick one workpiece, which may reduce the efficiency of the entire system.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a sensing device and a sensing method that are capable of accurately measuring the three-dimensional shape of an object that includes a transparent material. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a sensing device for measuring the three-dimensional shape of an object, comprising a computer and a heating device, wherein the computer has an image generation unit that generates an image based on visual information of the object, an object area extraction unit that extracts an area of ​​the image occupied by the object as an object area, an edge distance information generation unit that extracts distance information of an edge portion of the object from distance information of the object to generate edge distance information, a far-infrared image generation unit that generates a far-infrared image corresponding to the object area based on far-infrared information of the object, a heating device control unit that controls the heating device to heat the object, a partial surface shape estimation unit that estimates partial surface shape information of the object from far-infrared images of the object before and after heating, a shape interpolation unit that generates interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information, and an object shape output unit that converts the interpolated edge distance information into three-dimensional shape information and outputs it.

[0007] Furthermore, a sensing method for measuring the three-dimensional shape of an object includes the steps of generating an image of the object based on visual information of the object, extracting an area of ​​the image occupied by the object as an object area, extracting distance information of an edge portion of the object from the distance information of the object to generate edge distance information, generating a far-infrared image corresponding to the object area based on far-infrared information of the object, heating the object, generating a far-infrared image of the object after heating, estimating partial surface shape information of the object from the far-infrared images of the object before and after heating, generating interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information, and converting the interpolated edge distance information into three-dimensional shape information.

[0008] According to the present invention configured as described above, partial surface shape information of an object is generated from far-infrared images of the object before and after heating, and edge distance information of the object is interpolated using the partial surface shape information, thereby making it possible to accurately measure the three-dimensional shape of an object, including transparent materials. [Effects of the Invention]

[0009] According to the present invention, it is possible to accurately measure the three-dimensional shape of an object that includes a transparent material. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a sensing device and a sensing system according to a first embodiment of the present invention. [Figure 2] Functional block diagram of the object region extraction unit [Figure 3] FIG. 10 is a diagram illustrating the processing of an edge distance information generating unit. [Figure 4] FIG. 10 is a diagram showing an example of edge distance information generated by an edge distance information generating unit. [Figure 5] A diagram explaining the processing of the far-infrared image generation unit [Figure 6] FIG. 10 is a diagram for explaining the processing of the partial surface shape estimation unit. [Figure 7]Functional block diagram of shape interpolation unit [Figure 8] FIG. 10 is a diagram illustrating the processing of the shape interpolation unit. [Figure 9] 1 is a configuration diagram of a picking robot system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same elements are designated by the same reference numerals, and duplicated explanations will be omitted as appropriate. [Example]

[0012] Fig. 1 is a configuration diagram of a sensing device and a sensing system according to a first embodiment of the present invention. The sensing device 100 shown in Fig. 1 is a device that measures the three-dimensional shape of an object, and includes a computer 1 and a heating device 4. In Fig. 1, the functions of functional units 5 to 12 of the computer 1 are realized in the computer 1, which has an arithmetic unit, a main memory device, and an external memory device.

[0013] The sensing device 100 measures the three-dimensional shape of an object based on visual information, distance information, and far-infrared information of the object. A visible light sensor is preferably used as a means for acquiring the visual information and distance information of the object. In this embodiment, a stereo camera 2 is used as the visible light sensor, but the visible light sensor is not limited to this. For example, a sensor that measures distance information by installing a projector next to the visible light camera, or a sensor that estimates distance information from images captured by the visible light camera using machine learning or the like, can also be used instead. The sensing system 200 is composed of the sensing device 100, a visible light sensor 2, and a far-infrared sensor 3.

[0014] The following describes an overview of the functional units 5 to 12 shown in FIG. 1. The image generation unit 5 generates two images based on information acquired by the left and right visible light cameras of the stereo camera 2. The object region extraction unit 6 analyzes the two images generated by the image generation unit 5 to extract an object region in the image. The edge distance information generation unit 7 analyzes distance information corresponding to the object region and extracts distance information for the edge portion of the object. The far-infrared image generation unit 8 generates a far-infrared image corresponding to the object region using the far-infrared camera 3. The heating device control unit 9 controls the heating device 4 to heat the object region. The partial surface shape estimation unit 10 analyzes the far-infrared images of the object region before and after heating generated by the far-infrared image generation unit 8 and estimates a portion of the surface shape of the object region. The shape interpolation unit 11 interpolates the edge distance information generated by the edge distance information generation unit 7 using the estimated surface shape. The object shape output unit 12 converts the interpolated information into three-dimensional shape information and outputs it. The functional units 6 to 12 are described in detail below.

[0015] FIG. 2 is a functional block diagram of the object region extraction unit 6. The object region extraction unit 6 includes a 3D information acquisition unit 20 that acquires 3D information from the images captured by the stereo camera 2 generated by the image generation unit 5, and an object region extraction unit 21 that analyzes the acquired 3D information and extracts an object region in the image. The 3D information acquisition unit 20 acquires 3D information by calculating parallax from the images captured by the stereo camera 2. Parallax is calculated using a common method such as block matching. The calculated parallax information is converted into 3D information such as point cloud information in a 3D space based on parameters such as the camera installation position and orientation information. There are no particular limitations on this method, as long as it is possible to acquire 3D information from two camera images.

[0016] The object region extraction unit 21 extracts an object region from the camera image and the three-dimensional information. The extraction method is not particularly limited, and may include, for example, a method of extracting, as the object region, a difference region between a camera image in which an object is present and a camera image of a background in which the object is not present that has been previously captured, a method of similarly extracting, as the object region, a difference region between three-dimensional information calculated when an object is present and three-dimensional information calculated when an object is not present, or a method of extracting, as the final object region, a common portion between the object region found from the image and the object region found from the three-dimensional information.

[0017] Fig. 3 is a diagram illustrating the processing of the edge distance information generation unit 7. In Fig. 3, 30 denotes an example of an object (hereinafter referred to as a workpiece) to be picked up by the robot, 31 denotes a portion of the packaging material for the workpiece 30 made of paper, 32 denotes a portion of the packaging material for the workpiece 30 made of a transparent material (plastic, glass, etc.), 33 denotes the workpiece itself packed in the packaging materials 31 and 32, 34 denotes a base on which the workpiece 30 is placed when measurement is performed by the sensing device 100, 35 denotes an example of an image captured by one of the visible light cameras 2 when the stereo camera 2 captures an image of the workpiece 30 on the base 34, 36 denotes an object region extracted from the captured image 35 by the object region extraction unit 6, and 37 denotes edge distance information generated by the edge distance information generation unit 7.

[0018] The edge distance information generation unit 7 generates edge distance information for the workpiece 30 by analyzing the three-dimensional information within the object region 36. An example of a method for generating edge distance information will be described below. First, the distance value from the stereo camera 2 to a position corresponding to each pixel of the captured image 35 is calculated from the three-dimensional information, and information linking the distance value to each pixel is generated as a distance image. Next, the distance value of each pixel in the distance image is compared with the distance values ​​of adjacent pixels above, below, left, and right, and if there is a difference between the two that is greater than or equal to a threshold, the pixel is extracted as an edge part, and the smaller of the compared distance values ​​is linked to the pixel in the edge part to generate information as edge distance information.

[0019] FIG. 4 is a diagram showing an example of edge distance information 37 generated by the edge distance information generation unit 7. Each pixel in the edge portion stores a distance value (unit: cm) from the camera, and no value is stored in each pixel outside the edge portion. The edge distance information 37 makes it possible to represent distance information for the outer periphery of the workpiece 30, and distance information for the outer frame of the transparent packaging materials 31 and 32 can also be obtained. Note that there are no particular limitations on the method other than this example as long as it is possible to obtain distance information corresponding to the edge portion of the workpiece 30. Furthermore, the resolution of the distance image may be the same as that of the captured image, or a distance image with reduced resolution by downsampling processing or the like may be used.

[0020] FIG. 5 is a diagram illustrating the processing of the far-infrared image generating unit 8. In FIG. 5, 40 denotes a far-infrared image of the workpiece 30 and base 34 generated based on information from the far-infrared camera 3, and 41 denotes a far-infrared image corresponding to the object region 36 extracted from the far-infrared image 40. The far-infrared image contains temperature information, which varies depending on the material, etc. The far-infrared image is information in which a temperature value is associated with each pixel, and is treated as image information, just like the captured image and distance image. The resolution of the far-infrared image is not particularly limited and may be the same as that of the captured image or may be lower.

[0021] Returning to FIG. 1 , the heating device control unit 9 heats the workpiece 30 using the heating device 4. Ideally, the heating device 4 is installed in a position close to the stereo camera 2 and the far-infrared camera 3, and the direction in which the workpiece 30 is photographed is the same (directly above the workpiece 30 in this example). However, there are no particular limitations on the heating device 4 as long as it is capable of heating the workpiece 30 from the same direction as the camera photographs the workpiece 30. In addition, in this embodiment, a hot air device is assumed as the heating device 4, but there are no particular limitations on the heating device 4 as long as it is capable of promoting a temperature increase in the workpiece 30. In the sensing device 100, when heating of the workpiece 30 by the heating device 4 is completed, the heating device control unit 9 sends a command to the far-infrared image generation unit 8 to measure the workpiece 30 again using the far-infrared camera 3, thereby generating far-infrared images of the workpiece 30 before and after heating. The heating method using the heating device 4 is not particularly limited and may include a method of heating for a predetermined time each time, a method of estimating the approximate distance from the heating device 4 to the object 30 using edge distance information and far-infrared images generated in advance and changing the heating time according to that distance, a method of determining that the workpiece 30 is made of a material that heats easily if the temperature values ​​in the far-infrared images are generally high and shortening the heating time, or a method of adjusting the heating time or heating direction according to prior information about the workpiece 30 (shape, dimensions, material, etc.).

[0022] FIG. 6 is a diagram illustrating the processing performed by the partial surface shape estimation unit 10. In FIG. 6, 41 is an example of a far-infrared image corresponding to the work area (object area) before heating shown in FIG. 5, 42 and 43 are examples of far-infrared images corresponding to the work area after heating by the heating device control unit 9, and 44 and 45 are examples of partial surface shape information output by the partial surface shape estimation unit 10. The heating device 4 is installed in close proximity to the far-infrared camera 3 and heats the workpiece 30 from directly above. Therefore, the higher the portion of the workpiece 30 (the portion closer to the far-infrared camera 3 and the stereo camera 2), the faster it heats and the greater the temperature change. For example, in the case of the workpiece 30 in this example, the transparent packaging portion is closest to the heating device 4, so the temperature of the transparent packaging portion is the highest, as indicated by 42. Furthermore, the paper portion of the workpiece 30 itself is heated, causing a temperature change and allowing the paper material region to be measured. On the other hand, if the workpiece body 33 is made of a material that heats slowly, its temperature will be lower than that of the transparent packaging portion, as shown in 42, and the area of ​​the workpiece body 33 cannot be measured. If the material heats easily, its temperature will be higher than that of the transparent packaging portion, as shown in far-infrared image 43, and the area of ​​the workpiece body 33 can be measured. In the present invention, since the area of ​​the transparent packaging portion is to be acquired, the area of ​​the workpiece body 33 in far-infrared image 43 becomes noise. Therefore, a far-infrared image 41 generated before heating may be used, and noise removal processing may be performed, such as removing areas with temperature values ​​above a threshold before heating from the far-infrared image 43 generated after heating. The partial surface shape estimation unit 10 outputs areas with similar temperature values ​​from the final post-heating far-infrared image 42 after noise removal, as partial surface shape information 44, 45. Note that when extracting partial surface shape information, not only temperature values ​​but also position information of the areas may be used. If there are multiple areas with similar temperature values ​​and these areas are separated by more than a predetermined threshold, they may be output as separate partial surface shape information; this is not particularly limited.

[0023] 7 is a functional block diagram of the shape interpolation unit 11. The shape interpolation unit 11 includes a measurement information calibration unit 50 that calibrates two pieces of measurement information, namely, edge distance information generated based on information from the stereo camera 2 and partial surface shape information estimated based on information from the far-infrared camera 3, to information in the same sensor space, and a shape interpolation execution unit 51 that compares the calibrated measurement information to interpolate the three-dimensional shape of the workpiece. The measurement information calibration unit 50 and the shape interpolation execution unit 51 will be described below.

[0024] The measurement information calibration unit 50 calibrates the edge distance information and the partial surface shape information to information in the same sensor space based on the installation positions, orientation information, and resolution information of the stereo camera 2 and the far-infrared camera 3. In this example, since each measurement information is treated as image information, the calibration method involves resizing the image so that the resolution matches the size of one of the images, and then converting the coordinate information of each measurement information into the same space using a rotation matrix and translation vector between the sensors obtained by a calibration method using a general check marker or the like. Regarding the resolution resizing, there is no particular limitation on which resolution is matched to the edge distance information or the partial surface shape information. The calibration method is not particularly limited as long as it is a method that can perform coordinate conversion that allows comparison between the edge distance information shown in FIG. 4 and the partial surface shape information 44, 45 shown in FIG. 6. When each measurement information is treated as image information as in this example, a coordinate conversion method in two-dimensional space, such as image registration, may be used.

[0025] FIG. 8 is a diagram illustrating the processing of the shape interpolation execution unit 51. In FIG. 8, reference numeral 55 denotes an example of initial edge distance information in which distance values ​​of edge portions are described in the edge distance information 37, and reference numerals 56 and 57 denote examples of interpolated edge distance information interpolated by the shape interpolation execution unit 51. The shape interpolation execution unit 51 interpolates the initial edge distance information 55 using the partial surface shape information 44 and 45 calibrated by the measurement information calibration unit 50. The interpolation process involves comparing the partial surface shape information 44 and 45 with the edge distance information 37 and determining, by fitting or the like, which edge portion of the edge distance information 37 the outer periphery of each piece of partial surface shape information corresponds to. Then, by comparing the values ​​of the initial edge distance information 55 corresponding to the outer periphery of each piece of partial surface shape information, the interpolation process of the initial edge distance information 55 is performed while sequentially using partial surface shape information that is farther from the stereo camera 2. Note that when determining the order of the partial surface shape information to be used for interpolation, the size of the partial surface shape information may be used instead of the distance from the stereo camera 2. The interpolation process is not particularly limited as long as it is a method of filling in blanks using the values ​​of the initial edge distance information 55, such as a method of using the distance value of the edge distance information corresponding to the outer periphery of the partial surface shape information as the reference distance value, and filling in blanks around each pixel (at the top, bottom, left, and right positions) using the reference distance value. This process is repeated within the object area until a pixel containing a distance value smaller than the reference distance value is found. In this example, when comparing the edge distance information of the outer periphery of the partial surface shape information 44 and 45, the distance value of the partial surface shape information 44 is found to be 15 cm larger, so the initial edge distance information 55 is interpolated using the partial surface shape information 44 to generate interpolated edge distance information 56. Then, edge distance information is interpolated for the interpolated edge distance information 56 using the partial surface shape information 45 to generate interpolated edge distance information 57. In this example, when filling in the blanks in the interpolated edge distance information 56, a pixel containing a distance value (85) larger than the reference distance value (80) is found, so the distance value of that pixel is overwritten with the reference distance value (80) to generate the final interpolated edge distance information 57.

[0026] The object shape output unit 12 generates a three-dimensional point group from the interpolated edge distance information generated by the shape interpolation unit 11, and outputs it as final three-dimensional shape information of the workpiece.

[0027] (summary) In this embodiment, a sensing device 100 for measuring the three-dimensional shape of an object 30 includes a computer 1 and a heating device 4. The computer 1 includes an image generation unit 5 for generating an image 35 of the object 30 based on visual information of the object 30, an object region extraction unit 6 for extracting an area of ​​the image 35 occupied by the object 30 as an object region 36, an edge distance information generation unit 7 for extracting distance information of an edge portion of the object 30 from the distance information of the object 30 to generate edge distance information 37, and a far-infrared image generation unit 8 for generating a far-infrared image corresponding to the object region 36 based on far-infrared information of the object 30. the heating device control unit 9 controlling the heating device 4 to heat the object 30; a partial surface shape estimation unit 10 estimating partial surface shape information 44, 45 of the object 30 from the far-infrared images 41, 42, 43 of the object 30 before and after heating; a shape interpolation unit 11 generating interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 using the partial surface shape information 44, 45; and an object shape output unit 12 converting the interpolated edge distance information 57 into three-dimensional shape information and outputting it.

[0028] The sensing system 200 in this embodiment also includes the sensing device 100, a visible light sensor 2 that acquires visual information and distance information of the object 30, and a far-infrared sensor 3 that acquires far-infrared information of the object 30.

[0029] In addition, in this embodiment, the sensing method for measuring the three-dimensional shape of an object includes the steps of generating an image 35 of the object 30 based on visual information of the object 30, extracting the area of ​​the image 35 occupied by the object 30 as an object area 36, ​​extracting distance information of the edge portion of the object 30 from the distance information of the object 30 to generate edge distance information 37, generating a far-infrared image 41 corresponding to the object area 36 based on the far-infrared information of the object 30, heating the object 30, generating far-infrared images 42 and 43 after heating the object 30, estimating partial surface shape information 44 and 45 of the object 30 from the far-infrared images 41 to 43 before and after heating the object 30, generating interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 using the partial surface shape information 44 and 45, and converting the interpolated edge distance information 57 into three-dimensional shape information.

[0030] According to the present embodiment configured as described above, partial surface shape information 44, 45 of object 30 is generated from far-infrared images 41 to 43 of object 30 before and after heating, and edge distance information 37 of object 30 is interpolated using partial surface shape information 44, 45, thereby making it possible to accurately measure the three-dimensional shape of object 30 including transparent materials.

[0031] Furthermore, the edge distance information generator 7 in this embodiment calculates the distance value from the visible light sensor 2 to a position corresponding to each pixel of the image 35, generates information linking the distance value to each pixel of the image 35 as a distance image, calculates the difference in distance values ​​between adjacent pixels in the distance image, extracts pixels where the difference is equal to or greater than a predetermined threshold as pixels that make up the edge portion, and generates information linking each pixel that makes up the edge portion with the smaller of the distance value of each pixel and the distance value of its adjacent pixel as edge distance information 37. This makes it possible to generate the edge distance information 37 from the information acquired by the visible light sensor 2.

[0032] Furthermore, the far-infrared image generating unit 8 in this embodiment generates information in which far-infrared information of the object 30 is linked to each pixel of the object region 36 as a far-infrared image 41. This makes it possible to acquire the far-infrared image 41 corresponding to the object region 36.

[0033] Furthermore, the heating device control unit 9 in this embodiment adjusts the heating time or heating direction of the object 30 by the heating device 4 based on at least one of the object region 36, the edge distance information 37, the far-infrared image 41 before heating the object 30, and the prior information of the object 30. This makes it possible to heat the object 30 to a temperature state suitable for estimating the partial surface shape information 44, 45.

[0034] Furthermore, when estimating the partial surface shape information 44, 45 of the object 30, the partial surface shape estimation unit 10 in this embodiment uses the far-infrared image 41 before heating the object 30 to remove noise contained in the far-infrared image 43 after heating the object 30. This makes it possible to improve the estimation accuracy of the partial surface shape information 44, 45.

[0035] Furthermore, the shape interpolation unit 11 in this embodiment has a measurement information calibration unit 50 that calibrates the edge distance information 37 and the partial surface shape information 44, 45 by converting the coordinates of the edge distance information 37 and the partial surface shape information 44, 45 into coordinates in the same space, and a shape interpolation execution unit 51 that interpolates the calibrated edge distance information 37 using the calibrated partial surface shape information 44, 45. This makes it possible to improve the interpolation accuracy of the edge distance information 37.

[0036] Furthermore, the partial surface shape information 44, 45 in this embodiment includes a plurality of pieces of partial surface shape information 44, 45 corresponding to a plurality of partial surface shapes, and the shape interpolation unit 11 determines the order in which the plurality of pieces of partial surface shape information 44, 45 are used to interpolate the edge distance information 37 based on the distances from the visible light camera 2 to the plurality of partial surface shapes or the sizes of the plurality of partial surface shapes. This makes it possible to improve the accuracy of interpolation of the edge distance information 37.

[0037] In addition, the visible light sensor 2 in this embodiment is configured as either a stereo camera, a visible light camera equipped with a projector, or a visible light camera with a function to estimate distance from an image, which makes it possible to simultaneously obtain visual information and distance information of the object 30.

[0038] In this embodiment, a workpiece having a rectangular partial surface shape has been described as an example, but the shape of the workpiece is not particularly limited. For example, if the transparent packaging part is hemispherical, the far-infrared information after heating will indicate that the temperature is highest at the vertex closest to the camera and decreases as the distance from the camera increases. In this case, edge distance information for the part with the lowest temperature can be generated. The partial surface shape information can be estimated by using this reference distance value and interpolating with 3D information estimated from the far-infrared image. Alternatively, a method can be used in which a model of a far-infrared image corresponding to each object shape category is stored, a 3D model is generated from the far-infrared image, and the partial surface shape information is estimated using the edge distance information. [Example]

[0039] Fig. 9 is a configuration diagram of a picking robot system according to a second embodiment of the present invention. The picking robot system 300 shown in Fig. 9 includes a picking robot 60, a belt conveyor 61, and a sensing system 200. The picking robot system 300 is a system in which the sensing system 200 measures the three-dimensional shape of the workpiece 30 flowing on the belt conveyor 61, and the picking robot 60 grasps and transports the workpiece 30.

[0040] In FIG. 9, the far-infrared camera 3a, the heating device 4, and the picking robot 60 are installed in the vicinity of the belt conveyor 61 in the order from the upstream to the downstream of the belt conveyor 61. The stereo camera 2 and the far-infrared camera 3b are installed near the picking robot 60. 0 It is installed near the

[0041] 9, the image generation unit 5, object region extraction unit 6, edge distance information generation unit 7, far-infrared image generation unit 8, heating device control unit 9, shape interpolation unit 11, and object shape output unit 12 have the same or similar functions as those in the first embodiment. The robot control unit 13 controls the picking robot 6 using the three-dimensional shape information of the workpiece 30 output by the object shape output unit 12. 0 The robot has the function of controlling the far-infrared cameras 3a and 3b to grasp and transport the workpiece 30 to a predetermined position. The preliminary far-infrared image 14 is a far-infrared image of the workpiece 30 generated before the workpiece 30 is heated. The calibration information 15 is parameter information for calibrating the measurement information of the far-infrared cameras 3a and 3b. The partial surface shape estimation unit 10 and the robot control unit 13 will be described below.

[0042] The partial surface shape estimation unit 10 substantially follows the functions of the partial surface shape estimation unit 10 (shown in FIG. 1) in the first embodiment, and generates partial surface shape information from a preliminary far-infrared image 14 and a far-infrared image measured by the far-infrared camera 3b near the picking robot 60. The far-infrared cameras 3a and 3b are calibrated in advance using the method described in the measurement information calibration unit 50 (shown in FIG. 7) to generate calibration information 15, so that the measurement information of each camera can be handled in the same coordinate space. By using the calibration information 15, the measurement information of the far-infrared camera 3a and the stereo camera 2 can also be expressed in the same coordinate space. Therefore, by comparing the preliminary far-infrared image 14 with the object region extracted by the object region extraction unit 6, a far-infrared image before heating corresponding to the object region can be generated, and partial surface shape information can be estimated using the same flow as in the first embodiment.

[0043] The robot control unit 13 controls the picking robot 6 using the three-dimensional shape information of the workpiece 30 output by the object shape output unit 12. 0 In this example, the three-dimensional shape information is obtained by measuring the workpiece 30 with the stereo camera 2, so the picking robot 6 0 The stereo camera 2 is calibrated in advance. The calibration method is0 The method is not particularly limited, but may be a method of installing a calibration board on the arm of the picking robot 6, measuring with the stereo camera 2, and estimating the calibration information. The robot control unit 13 handles the three-dimensional shape information as three-dimensional point cloud coordinates, and 0 The workpiece 30 is transported by instructing the coordinate information of the gripping position and the transporting position to the robot. The method of determining the gripping position and the transporting position is not particularly limited. The arm portion used for gripping is also not limited to a shape like a human hand, and may be a suction arm using a vacuum pump, and is not particularly limited.

[0044] (summary) In this embodiment, in a picking robot system 300 including a sensing system 200, a picking robot 60, and a belt conveyor 61 for transporting an object 30 to the picking robot 60, the far-infrared sensors 3a and 3b include a first far-infrared sensor 3a for acquiring far-infrared information of the object 30 before it is heated by a heating device 4, and a second far-infrared sensor 3b for acquiring far-infrared information of the object 30 after it has been heated by the heating device 4, and the computer 1 has a robot control unit 13 for controlling the picking robot 60 using three-dimensional shape information of the object 30.

[0045] According to the present embodiment configured as described above, a partial surface shape of the object 30 is generated from the far-infrared information of the object 30 before heating measured by the first far-infrared sensor 3a and the far-infrared information of the object 30 after heating measured by the second far-infrared sensor 3b, and the partial surface shape is used to interpolate the edge distance information of the object 30 measured by the visible light sensor 2, thereby making it possible to accurately measure the three-dimensional shape of the object 30 including a transparent material while maintaining the efficiency of the entire system.

[0046] Furthermore, the calculator 1 in this embodiment may store the edge distance information and the partial surface shape information in association with each other each time the object 30 is measured, and if the newly generated edge distance information matches the previously generated edge distance information, the heating of the object 30 by the heating device 4 and the acquisition of far-infrared information by the far-infrared sensors 3a and 3b may be omitted, and the newly generated edge distance information may be interpolated using the partial surface shape information corresponding to the previously generated edge distance information to generate the three-dimensional shape information of the object 30. This makes it possible to improve the efficiency of the entire system.

[0047] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to add part of the configuration of one embodiment to the configuration of another embodiment, or to delete part of the configuration of one embodiment or replace it with part of another embodiment. [Explanation of symbols]

[0048] 1...computer, 2...stereo camera (visible light sensor), 3...far-infrared camera (far-infrared sensor), 3a...far-infrared camera (first far-infrared sensor), 3b...far-infrared camera (second far-infrared sensor), 4...heating device, 5...image generation unit, 6...object region extraction unit, 7...edge distance information generation unit, 8...far-infrared image generation unit, 9...heating device control unit, 10...partial surface shape estimation unit, 11...shape interpolation unit, 12...object shape output unit, 13...robot control unit, 14...preliminary far-infrared image, 15...calibration information, 21...object Area extraction unit, 30... work (object), 31, 32... packaging material, 33... work body, 34... base, 35... captured image, 36... object area, 37... edge distance information, 40-43... far-infrared image, 44, 45... partial surface shape information, 50... measurement information calibration unit, 51... shape interpolation execution unit, 55... initial edge distance information, 56, 57... interpolated edge distance information, 60... picking robot, 61... belt conveyor, 100... sensing device, 200... sensing system, 300... picking robot system.

Claims

1. In a sensing device for measuring the three-dimensional shape of an object, A calculator and a heating device, The computer an image generation unit that generates an image based on visual information of the object; an object region extraction unit that extracts an area occupied by the object from the image as an object region; an edge distance information generating unit that extracts distance information of an edge portion of the object from the distance information of the object and generates edge distance information; a far-infrared image generating unit that generates a far-infrared image corresponding to the object region based on the far-infrared information of the object; a heating device control unit that controls the heating device to heat the object; a partial surface shape estimation unit that estimates partial surface shape information of the object from far-infrared images before and after heating of the object; a shape interpolation unit that generates interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information; an object shape output unit that converts the interpolated edge distance information into three-dimensional shape information and outputs the three-dimensional shape information; A sensing device characterized by:

2. The sensing device according to claim 1 , The far-infrared image generating unit generates information in which far-infrared information of the object is associated with each pixel of the object region as the far-infrared image. A sensing device characterized by:

3. The sensing device according to claim 1 , The heating device control unit adjusts a heating time or a heating direction of the object by the heating device based on at least one of the object region, the edge distance information, the far-infrared image before heating the object, and prior information of the object. A sensing device characterized by:

4. The sensing device according to claim 1 , The partial surface shape estimation unit, when estimating the partial surface shape of the object, removes noise included in the far-infrared image after heating the object by using the far-infrared image before heating the object. A sensing device characterized by:

5. The sensing device according to claim 1 , The shape interpolation unit a measurement information calibration unit that calibrates the edge distance information and the partial surface shape information by converting each coordinate of the edge distance information and the partial surface shape information into coordinates in the same space; a shape interpolation execution unit that interpolates the calibrated edge distance information using the calibrated partial surface shape information; A sensing device characterized by:

6. The sensing device according to claim 1 ; a visible light sensor for acquiring visual information and distance information of the object; a far-infrared sensor for acquiring far-infrared information of the object; A sensing system characterized by:

7. 7. The sensing system according to claim 6, The edge distance information generation unit calculating a distance value from the visible light sensor to a position corresponding to each pixel of the image; generating information in which each pixel of the image is associated with the distance value as a distance image; calculating a difference in distance values ​​between adjacent pixels of the distance image; extracting pixels whose difference is equal to or greater than a predetermined threshold as pixels that constitute the edge portion; The edge distance information is generated by associating each pixel constituting the edge portion with the smaller value of the distance value of each pixel and the distance value of an adjacent pixel. A sensing system characterized by:

8. 7. The sensing system according to claim 6, the partial surface shape information includes a plurality of pieces of partial surface shape information corresponding to a plurality of partial surface shapes, The shape interpolation unit determines an order in which the plurality of pieces of partial surface shape information are used to interpolate the edge distance information based on distances from the visible light sensor to the plurality of partial surface shapes or sizes of the plurality of partial surface shapes. A sensing system characterized by:

9. 7. The sensing system according to claim 6, The visible light sensor is composed of either a stereo camera, a visible light camera equipped with a projector, or a visible light camera with a function to estimate distance from an image. A sensing system characterized by:

10. The sensing system according to claim 6 ; Picking robots and a belt conveyor that transports the object to the picking robot, the far-infrared sensor includes a first far-infrared sensor that acquires far-infrared information of the object before heating by the heating device, and a second far-infrared sensor that acquires far-infrared information of the object after heating by the heating device; The computer has a robot control unit that controls the picking robot using three-dimensional shape information of the object. A picking robot system characterized by:

11. The picking robot system according to claim 10, The computer storing the edge distance information and the partial surface shape information in association with each other each time the object is measured; When the newly generated edge distance information matches the previously generated edge distance information, heating of the object by the heating device and acquisition of the far-infrared information by the far-infrared sensor are omitted, and three-dimensional shape information of the object is generated by interpolating the newly generated edge distance information using the partial surface shape information corresponding to the previously generated edge distance information. A picking robot system characterized by:

12. A sensing method for measuring a three-dimensional shape of an object, generating an image of the object based on visual information of the object; extracting an area of ​​the image occupied by the object as an object area; a step of extracting distance information of an edge portion of the object from the distance information of the object to generate edge distance information; generating a far-infrared image corresponding to the object region based on far-infrared information of the object; heating the object; generating a far-infrared image of the object after heating the object; a step of estimating partial surface shape information of the object from far-infrared images of the object before and after heating; generating interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information; and converting the interpolated edge distance information into three-dimensional shape information. A sensing method comprising:

Images