Sensing device and sensing method

Abstract

To provide a sensing device and a sensing method which can correctly measure a three-dimensional shape of an object including a transparent raw material.SOLUTION: A computing machine 1 generates an image 35 on the basis of visual information of an object 30, extracts a region occupied by the object 30 in the image 35 as an object region 36, generates edge distance information 37 by extracting distance information of an edge portion of the object 30 in the distance information of the object 30, generates a far-infrared image 41 corresponding to the object region 36 on the basis of far-infrared ray information of the object 30, controls a heating device 4 so as to heat the object 30, estimates partial surface shape information 44, 45 of the object 30 from the far-infrared images 41, 42 before and after heating of the object 30, generates interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 by using the partial surface shape information 44, 45, and converts the interpolated edge distance information 57 into the three-dimensional shape information to output the converted information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sensing device and a sensing method. [Background technology]

[0002] In recent years, with the decline in the working population due to the low birthrate and aging population, there are growing expectations for automated and autonomous system control to resolve labor shortages and improve productivity. For example, in the industrial and logistics fields, there is a high demand for robots that automatically pick up workpieces, which are the objects of work. To automate picking work, it is necessary to place sensors such as cameras to measure the three-dimensional shape of the workpiece and teach the robot the position information for grasping the workpiece. A common method of measuring three-dimensional shapes is to install multiple cameras or a camera and a projector and measure based on the principle of triangulation. In this case, the camera often uses a visible light sensor, and for workpieces packed in materials such as paper, the three-dimensional shape of the packaging material can be accurately recognized. However, if the workpiece is packed in a transparent material such as a blister pack, the shape of the packaging part cannot be obtained, and only the shape of the workpiece itself inside is obtained, which may cause the transparent packaging part to be crushed and destroyed when picking. Therefore, a method using not only a visible light sensor but also a far-infrared camera is expected to be used to obtain the three-dimensional shape of the transparent part. For example, in Patent Document 1, a three-dimensional shape is acquired by measuring a workpiece from multiple points using an imaging device equipped with a fixed visible light camera and far-infrared camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-032600 A Summary of the Invention [Problem to be solved by the invention]

[0004] By utilizing the technology of Patent Document 1, the shape of the transparent part of a workpiece that is wrapped in a transparent material can be correctly recognized, and the workpiece can be picked without destroying the wrapping. However, moving the imaging device to capture multiple images not only depends on the accuracy of the calibration, but also requires time to pick one workpiece, which may reduce the efficiency of the entire system.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a sensing device and a sensing method capable of accurately measuring the three-dimensional shape of an object including a transparent material. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a sensing device for measuring the three-dimensional shape of an object, comprising a computer and a heating device, wherein the computer has an image generation unit for generating an image based on visual information of the object, an object area extraction unit for extracting an area occupied by the object from the image as an object area, an edge distance information generation unit for extracting distance information of an edge portion of the object from the distance information of the object and generating edge distance information, a far-infrared image generation unit for generating a far-infrared image corresponding to the object area based on far-infrared information of the object, a heating device control unit for controlling the heating device to heat the object, a partial surface shape estimation unit for estimating partial surface shape information of the object from far-infrared images of the object before and after heating, a shape interpolation unit for generating interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information, and an object shape output unit for converting the interpolated edge distance information into three-dimensional shape information and outputting it.

[0007] Further, a sensing method for measuring the three-dimensional shape of an object includes the steps of generating an image of the object based on visual information of the object, extracting an area of ​​the image occupied by the object as an object area, extracting distance information of an edge portion of the object from the distance information of the object to generate edge distance information, generating a far-infrared image corresponding to the object area based on far-infrared information of the object, heating the object, generating a far-infrared image of the object after the object is heated, estimating partial surface shape information of the object from the far-infrared images of the object before and after heating, generating interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information, and converting the interpolated edge distance information into three-dimensional shape information.

[0008] According to the present invention configured as described above, partial surface shape information of an object is generated from far-infrared images before and after heating of the object, and edge distance information of the object is interpolated using the partial surface shape information, making it possible to accurately measure the three-dimensional shape of an object, including transparent materials. Effect of the Invention

[0009] According to the present invention, it is possible to accurately measure the three-dimensional shape of an object including a transparent material. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of a sensing device and a sensing system according to a first embodiment of the present invention. [Diagram 2] Functional block diagram of the object region extraction unit [Diagram 3] FIG. 13 is a diagram for explaining the processing of an edge distance information generating unit. [Figure 4] FIG. 13 is a diagram showing an example of edge distance information generated by an edge distance information generating unit; [Diagram 5] FIG. 1 is a diagram for explaining the processing of the far-infrared image generating unit. [Figure 6] FIG. 13 is a diagram for explaining the processing of a partial surface shape estimation unit. [Figure 7]Functional block diagram of shape interpolation unit [Figure 8] FIG. 2 is a diagram for explaining the processing of the shape interpolation unit; [Figure 9] FIG. 1 is a configuration diagram of a picking robot system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same reference numerals are used to denote the same elements, and duplicated explanations will be omitted as appropriate. EXAMPLES

[0012] Fig. 1 is a configuration diagram of a sensing device and a sensing system in a first embodiment of the present invention. The sensing device 100 shown in Fig. 1 is a device that measures the three-dimensional shape of an object, and includes a computer 1 and a heating device 4. In Fig. 1, the functions of functional units 5 to 12 of the computer 1 are realized in the computer 1 that has an arithmetic unit, a main memory device, and an external memory device.

[0013] The sensing device 100 measures the three-dimensional shape of an object based on the visual information, distance information, and far-infrared information of the object. A visible light sensor is preferably used as a means for acquiring the visual information and distance information of the object. In this embodiment, a stereo camera 2 is used as the visible light sensor, but the visible light sensor is not limited to this. For example, a sensor that measures distance information by installing a projector next to the visible light camera, or a sensor that estimates distance information from an image of the visible light camera by machine learning or the like can be used instead. The sensing system 200 is composed of the sensing device 100, a visible light sensor 2, and a far-infrared sensor 3.

[0014] The functional units 5 to 12 shown in FIG. 1 will be outlined below. The image generating unit 5 has a function of generating two images based on information acquired by the left and right visible light cameras of the stereo camera 2. The object area extracting unit 6 has a function of extracting an object area in an image by analyzing the two images generated by the image generating unit 5. The edge distance information generating unit 7 has a function of analyzing distance information corresponding to the object area and extracting distance information of an edge part of the object. The far-infrared image generating unit 8 has a function of generating a far-infrared image corresponding to the object area by the far-infrared camera 3. The heating device control unit 9 has a function of controlling the heating device 4 to heat the object area. The partial surface shape estimating unit 10 has a function of estimating a part of the surface shape of the object area by analyzing the far-infrared images of the object area before and after heating generated by the far-infrared image generating unit 8. The shape interpolating unit 11 has a function of interpolating the edge distance information generated by the edge distance information generating unit 7 using the estimated surface shape. The object shape output unit 12 has a function of converting the interpolated three-dimensional shape information into three-dimensional shape information and outputting it. The functional units 6 to 12 will be described in detail below.

[0015] FIG. 2 is a functional block diagram of the object region extraction unit 6. The object region extraction unit 6 includes a three-dimensional information acquisition unit 20 that acquires three-dimensional information from the captured images of the stereo camera 2 generated by the image generation unit 5, and an object region extraction unit 21 that analyzes the acquired three-dimensional information and extracts an object region in the image. The three-dimensional information acquisition unit 20 acquires three-dimensional information by calculating parallax from the captured images of the stereo camera 2. The parallax is calculated by using a general method such as block matching. The calculated parallax information is converted into three-dimensional information such as point cloud information in a three-dimensional space from parameters of the installation position and orientation information of the camera. In addition to this method, any method that can acquire three-dimensional information from two camera images is not particularly limited.

[0016] The object region extraction unit 21 extracts an object region from the camera image and the three-dimensional information. The extraction method is not particularly limited, and may be, for example, a method of extracting a difference region between a camera image in which an object exists and a camera image of a background in which no object exists, captured in advance, as the object region, a method of similarly extracting a difference region between three-dimensional information calculated when an object exists and three-dimensional information calculated when an object does not exist, as the object region, or a method of extracting a common portion between an object region obtained from an image and an object region obtained from three-dimensional information as the final object region.

[0017] Fig. 3 is a diagram for explaining the processing of the edge distance information generating unit 7. In Fig. 3, 30 denotes an example of an object (hereinafter, work) to be picked up by the robot, 31 denotes a part of the packaging material of the work 30 formed of a paper material, 32 denotes a part of the packaging material of the work 30 formed of a transparent material (plastic, glass, etc.), 33 denotes a work body packed with the packaging materials 31 and 32, 34 denotes a base on which the work 30 is placed when performing measurement with the sensing device 100, 35 denotes an example of an image captured by one of the visible light cameras 2 when the work 30 on the base 34 is captured by the stereo camera 2, 36 denotes an object region extracted from the captured image 35 by the object region extracting unit 6, and 37 denotes edge distance information generated by the edge distance information generating unit 7.

[0018] The edge distance information generating unit 7 generates edge distance information of the workpiece 30 by analyzing the three-dimensional information in the object region 36. An example of a method for generating edge distance information will be described. First, a distance value from the stereo camera 2 at a position corresponding to each pixel of the captured image 35 is obtained from the three-dimensional information, and information linking the distance value to each pixel is generated as a distance image. Next, the distance value of each pixel in the distance image is compared with the distance values ​​of adjacent pixels above, below, left, and right, and if there is a difference between the two that is equal to or greater than a threshold, the pixel is extracted as an edge part, and information linking the smaller of the compared distance values ​​to the pixel of the edge part is generated as edge distance information.

[0019] FIG. 4 is a diagram showing an example of edge distance information 37 generated by the edge distance information generating unit 7. A distance value (unit: cm) from the camera is stored in each pixel of the edge portion, and no value is stored in each pixel other than the edge portion. The edge distance information 37 makes it possible to express distance information of the outer shape of the workpiece 30, and distance information of the outer frame can be obtained even for the transparent packaging materials 31 and 32. Note that there is no particular limitation on the method other than this example as long as it is a method capable of acquiring distance information corresponding to the edge portion of the workpiece 30. In addition, the resolution of the distance image may be the same as that of the captured image, or a distance image with a lowered resolution by downsampling processing or the like may be used.

[0020] FIG. 5 is a diagram for explaining the processing of the far-infrared image generating unit 8. In FIG. 5, 40 indicates a far-infrared image of the workpiece 30 and the base 34 generated based on information from the far-infrared camera 3, and 41 indicates a far-infrared image corresponding to the object region 36 extracted from the far-infrared image 40. The far-infrared image includes temperature information, and the temperature information differs depending on the material. The far-infrared image is information in which a temperature value is associated with each pixel, and is treated as image information, like the captured image and the distance image. The resolution of the far-infrared image is not particularly limited and may be the same as that of the captured image or may be lower.

[0021] Returning to FIG. 1, the heating device control unit 9 heats the workpiece 30 by the heating device 4. The heating device 4 is installed in a position close to the stereo camera 2 and the far-infrared camera 3, and the direction in which the workpiece 30 is photographed is ideally the same as the direction in which the workpiece 30 is photographed by the camera (directly above the workpiece 30 in this example), but there is no particular limitation as long as the means can heat the workpiece 30 from the same direction as the direction in which the workpiece 30 is photographed by the camera. In addition, in this embodiment, the heating device 4 is assumed to be a hot air device, but there is no particular limitation as long as the device can promote the temperature rise of the workpiece 30. In the sensing device 100, when the heating of the workpiece 30 by the heating device 4 is completed, the heating device control unit 9 transmits a command to the far-infrared image generation unit 8 to measure the workpiece 30 again by the far-infrared camera 3, thereby generating far-infrared images of the workpiece 30 before and after heating. As a heating method using the heating device 4, there may be used a method of heating for a preset time each time, a method of estimating the approximate distance from the heating device 4 to the object 30 using edge distance information and far-infrared images generated in advance and changing the heating time according to that distance, a method of determining that the workpiece 30 is made of a material that heats easily if the temperature values ​​in the far-infrared image are generally high and shortening the heating time, or a method of adjusting the heating time or heating direction according to prior information about the workpiece 30 (shape, dimensions, material, etc.), and there are no particular limitations to this.

[0022] FIG. 6 is a diagram for explaining the processing of the partial surface shape estimation unit 10. In FIG. 6, 41 is an example of a far-infrared image corresponding to the work area (object area) before heating shown in FIG. 5, 42 and 43 are examples of far-infrared images corresponding to the work area after heating by the heating device control unit 9, and 44 and 45 are examples of partial surface shape information output by the partial surface shape estimation unit 10. The heating device 4 is installed in a position close to the far-infrared camera 3 and heats the work 30 from directly above, so the higher the part of the work 30 (the part closer to the far-infrared camera 3 or the stereo camera 2), the faster it is heated, and the greater the temperature change. For example, in the case of the work 30 in this example, the transparent packaging part is located closest to the heating device 4, so the temperature of the transparent packaging part is the highest as shown by 42. In addition, the paper part of the work 30 itself is heated, so a temperature change occurs and the area of ​​the paper material can be measured. On the other hand, if the workpiece body 33 is made of a material that is difficult to heat up, the temperature will be lower than the transparent packaging part as shown in 42, and the area of ​​the workpiece body 33 cannot be measured, and if the material is easy to heat up, the temperature will be higher than the transparent packaging part as shown in far-infrared image 43, and the area of ​​the workpiece body 33 can be measured. In the present invention, since it is desired to obtain the area of ​​the transparent packaging part, the area of ​​the workpiece body 33 in the far-infrared image 43 becomes noise. Therefore, a process such as noise removal may be added, in which a far-infrared image 41 generated before heating is used and an area having a temperature value equal to or higher than a threshold before heating is deleted from the far-infrared image 43 generated after heating. The partial surface shape estimation unit 10 outputs areas with similar temperature values ​​as partial surface shape information 44, 45 from the final far-infrared image 42 after heating after noise removal, etc. Note that when extracting the partial surface shape information, not only the temperature value but also the position information of the area may be used, and if there are multiple areas with similar temperature values ​​and these areas are separated by a predetermined threshold, they may be output as separate partial surface shape information, and there is no particular limitation.

[0023] 7 is a functional block diagram of the shape interpolation unit 11. The shape interpolation unit 11 includes a measurement information calibration unit 50 that calibrates two pieces of measurement information, namely edge distance information generated based on information from the stereo camera 2 and partial surface shape information estimated based on information from the far-infrared camera 3, to information in the same sensor space, and a shape interpolation execution unit 51 that compares the calibrated measurement information to interpolate the three-dimensional shape of the workpiece. The measurement information calibration unit 50 and the shape interpolation execution unit 51 will be described below.

[0024] The measurement information calibration unit 50 calibrates the edge distance information and the partial surface shape information to information in the same sensor space based on the installation positions, attitude information, and resolution information of the stereo camera 2 and the far-infrared camera 3. In this example, since each measurement information is treated as image information, the calibration method involves resizing the image so that the resolution matches the size of one side, and then converting the coordinate information of each measurement information into the same space using a rotation matrix and translation vector between sensors obtained by a calibration method using a general check marker or the like. There is no particular limit to which resolution is to be matched to the edge distance information or the partial surface shape information. There is no particular limit to the calibration method as long as it is a method capable of performing coordinate conversion that allows the edge distance information shown in FIG. 4 to be compared with the partial surface shape information 44, 45 shown in FIG. 6. When each measurement information is treated as image information as in this example, a coordinate conversion method in a two-dimensional space such as image registration may be used.

[0025] FIG. 8 is a diagram for explaining the processing of the shape interpolation execution unit 51. In FIG. 8, 55 is an example of initial edge distance information in which the distance value of the edge part is described in the edge distance information 37, and 56 and 57 are examples of interpolated edge distance information interpolated by the shape interpolation execution unit 51. The shape interpolation execution unit 51 interpolates the initial edge distance information 55 using the partial surface shape information 44 and 45 calibrated by the measurement information calibration unit 50. The flow of the interpolation is to compare the partial surface shape information 44 and 45 with the edge distance information 37, and determine which edge part of the edge distance information 37 corresponds to the outer periphery part of each partial surface shape information by fitting processing or the like. Then, by comparing the values ​​of the initial edge distance information 55 corresponding to the outer periphery part of the partial surface shape information, the partial surface shape information far from the stereo camera 2 is used in order to perform the interpolation processing of the initial edge distance information 55. Note that when determining the order of the partial surface shape information to be used for the interpolation, the size of the partial surface shape information may be used instead of the distance from the stereo camera 2. The interpolation process is not particularly limited as long as it is a method of filling blanks using the value of the initial edge distance information 55, such as a method of using the distance value of the edge distance information corresponding to the outer periphery of the partial surface shape information as the reference distance value, and filling in blanks around each pixel (at the top, bottom, left, and right positions) with the reference distance value, and repeating this process within the object area until a pixel containing a distance value smaller than the reference distance value is found. In this example, when comparing the edge distance information of the outer periphery of the partial surface shape information 44 and 45, the distance value of the partial surface shape information 44 is 15 cm larger, so the initial edge distance information 55 is interpolated using the partial surface shape information 44 to generate the interpolated edge distance information 56. Then, the edge distance information of the interpolated edge distance information 56 is interpolated using the partial surface shape information 45 to generate the interpolated edge distance information 57. In this example, when filling in blanks in the interpolated edge distance information 56, a pixel containing a distance value (85) larger than the reference distance value (80) is found, so the distance value of the pixel is overwritten with the reference distance value (80) to generate the final interpolated edge distance information 57.

[0026] The object shape output unit 12 generates a three-dimensional point group from the interpolated edge distance information generated by the shape interpolation unit 11, and outputs it as final three-dimensional shape information of the workpiece.

[0027] (summary) In this embodiment, a sensing device 100 for measuring the three-dimensional shape of an object 30 includes a computer 1 and a heating device 4. The computer 1 includes an image generating unit 5 for generating an image 35 of the object 30 based on visual information of the object 30, an object region extracting unit 6 for extracting an area of ​​the image 35 occupied by the object 30 as an object region 36, an edge distance information generating unit 7 for extracting distance information of an edge portion of the object 30 from the distance information of the object 30 to generate edge distance information 37, and a far-infrared image generating unit 8 for generating a far-infrared image corresponding to the object region 36 based on far-infrared information of the object 30. the heating device control unit 9 controlling the heating device 4 to heat the object 30; a partial surface shape estimation unit 10 estimating partial surface shape information 44, 45 of the object 30 from the far-infrared images 41, 43 of the object 30 before and after heating; a shape interpolation unit 11 generating interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 using the partial surface shape information 44, 45; and an object shape output unit 12 converting the interpolated edge distance information 57 into three-dimensional shape information and outputting it.

[0028] The sensing system 200 in this embodiment also includes the sensing device 100, a visible light sensor 2 that obtains visual information and distance information of the object 30, and a far-infrared sensor 3 that obtains far-infrared information of the object 30.

[0029] In addition, in this embodiment, the sensing method for measuring the three-dimensional shape of an object includes the steps of generating an image 35 of the object 30 based on visual information of the object 30, extracting the area of ​​the image 35 occupied by the object 30 as an object area 36, ​​extracting distance information of the edge portion of the object 30 from the distance information of the object 30 to generate edge distance information 37, generating a far-infrared image 41 corresponding to the object area 36 based on the far-infrared information of the object 30, heating the object 30, generating far-infrared images 42, 43 after the object 30 is heated, estimating partial surface shape information 44, 45 of the object 30 from the far-infrared images 41-43 before and after heating the object 30, generating interpolated edge distance information 57 of the object 30 by interpolating the edge distance information 37 using the partial surface shape information 44, 45, and converting the interpolated edge distance information 57 into three-dimensional shape information.

[0030] According to the present embodiment configured as described above, partial surface shape information 44, 45 of object 30 is generated from far-infrared images 41-43 before and after heating of object 30, and edge distance information 37 of object 30 is interpolated using partial surface shape information 44, 45, making it possible to accurately measure the three-dimensional shape of object 30 including transparent materials.

[0031] Moreover, the edge distance information generator 7 in this embodiment calculates a distance value from the visible light sensor 2 to a position corresponding to each pixel of the image 35, generates information linking the distance value to each pixel of the image 35 as a distance image, calculates a difference in distance values ​​between adjacent pixels in the distance image, extracts pixels where the difference is equal to or greater than a predetermined threshold as pixels that constitute the edge portion, and generates information linking each pixel that constitutes the edge portion with the smaller of the distance value of each pixel and the distance value of its adjacent pixel as edge distance information 37. This makes it possible to generate edge distance information 37 from information acquired by the visible light sensor 2.

[0032] Further, the far-infrared image generating unit 8 in this embodiment generates information in which the far-infrared information of the object 30 is linked to each pixel of the object region 36 as a far-infrared image 41. This makes it possible to obtain the far-infrared image 41 corresponding to the object region 36.

[0033] Furthermore, the heating device control unit 9 in this embodiment adjusts the heating time or heating direction of the object 30 by the heating device 4 based on at least one of the object region 36, the edge distance information 37, the far-infrared image 41 before the object 30 is heated, and the prior information of the object 30. This makes it possible to heat the object 30 to a temperature state suitable for estimating the partial surface shape information 44, 45.

[0034] Furthermore, when estimating the partial surface shape information 44, 45 of the object 30, the partial surface shape estimation unit 10 in this embodiment uses the far-infrared image 41 before heating the object 30 to remove noise contained in the far-infrared image 43 after heating the object 30. This makes it possible to improve the estimation accuracy of the partial surface shape information 44, 45.

[0035] Moreover, the shape interpolation unit 11 in this embodiment has a measurement information calibration unit 50 that calibrates the edge distance information 37 and the partial surface shape information 44, 45 by converting each coordinate of the edge distance information 37 and the partial surface shape information 44, 45 into coordinates in the same space, and a shape interpolation execution unit 51 that interpolates the calibrated edge distance information 37 using the calibrated partial surface shape information 44, 45. This makes it possible to improve the interpolation accuracy of the edge distance information 37.

[0036] Furthermore, the partial surface shape information 44, 45 in this embodiment includes multiple pieces of partial surface shape information 44, 45 corresponding to multiple partial surface shapes respectively, and the shape interpolation unit 11 determines the order in which the multiple pieces of partial surface shape information 44, 45 are used to interpolate the edge distance information 37 based on each distance from the visible light camera 2 to the multiple partial surface shapes or each size of the multiple partial surface shapes. This makes it possible to improve the interpolation accuracy of the edge distance information 37.

[0037] In addition, the visible light sensor 2 in this embodiment is composed of either a stereo camera, a visible light camera equipped with a projector, or a visible light camera equipped with a function of estimating distance from an image, which makes it possible to simultaneously obtain visual information and distance information of the object 30.

[0038] In this embodiment, the workpiece has a rectangular partial surface shape, but the shape of the workpiece is not particularly limited. For example, if the transparent packaging part is hemispherical, the far-infrared information after heating indicates that the temperature of the apex closest to the camera is the highest, and the temperature decreases as the distance from the camera increases. In this case, the edge distance information of the part with the lowest temperature can be generated, so that the partial surface shape information may be estimated by a method such as interpolating the reference distance value with the three-dimensional information estimated from the far-infrared image, or a method of storing a model of the far-infrared image corresponding to each category of object shape, generating a three-dimensional model from the far-infrared image, and estimating the partial surface shape information using the edge distance information may be used. EXAMPLES

[0039] Fig. 9 is a configuration diagram of a picking robot system in a second embodiment of the present invention. The picking robot system 300 shown in Fig. 9 includes a picking robot 60, a belt conveyor 61, and a sensing system 200. The picking robot system 300 is a system in which the sensing system 200 measures the three-dimensional shape of the workpiece 30 flowing on the belt conveyor 61, and the picking robot 60 grasps and transports the workpiece 30.

[0040] 9, the far-infrared camera 3a, the heating device 4, and the picking robot 60 are installed in the vicinity of the belt conveyor 61, in that order from the upstream to the downstream of the belt conveyor 61. The stereo camera 2 and the far-infrared camera 3b are installed in the vicinity of the picking robot 64.

[0041] In FIG. 9, the image generating unit 5, the object region extracting unit 6, the edge distance information generating unit 7, the far-infrared image generating unit 8, the heating device control unit 9, the shape interpolating unit 11, and the object shape output unit 12 have the same or similar functions as those in the first embodiment. The robot control unit 13 has a function of controlling the picking robot 64 using the three-dimensional shape information of the workpiece 30 output by the object shape output unit 12, and gripping and transporting the workpiece 30 to a predetermined position. The preliminary far-infrared image 14 is a far-infrared image of the workpiece 30 generated before the workpiece 30 is heated. The calibration information 15 is parameter information for calibrating the measurement information of the far-infrared cameras 3a and 3b. The partial surface shape estimating unit 10 and the robot control unit 13 will be described below.

[0042] The partial surface shape estimation unit 10 has almost the same functions as the partial surface shape estimation unit 10 (shown in FIG. 1) in the first embodiment, and generates partial surface shape information from the preliminary far-infrared image 14 and a far-infrared image measured by the far-infrared camera 3b near the picking robot 60. The far-infrared cameras 3a and 3b are calibrated in advance by the method described in the measurement information calibration unit 50 (shown in FIG. 7) to generate calibration information 15, so that the measurement information of each camera can be handled in the same coordinate space. By using the calibration information 15, the measurement information of the far-infrared camera 3a and the stereo camera 2 can also be expressed in the same coordinate space, so that by comparing the preliminary far-infrared image 14 with the object region extracted by the object region extraction unit 6, a far-infrared image before heating corresponding to the object region can be generated, and partial surface shape information can be estimated in the same flow as in the first embodiment.

[0043] The robot control unit 13 uses the three-dimensional shape information of the workpiece 30 output by the object shape output unit 12 to carry out transportation of the workpiece 30 by the picking robot 64. In this example, the obtained three-dimensional shape information is measured by the stereo camera 2, so the picking robot 64 and the stereo camera 2 are calibrated in advance. The calibration method is not particularly limited, and may be a method in which a calibration board is installed on the arm part of the picking robot 64, measurement is performed by the stereo camera 2, and calibration information is estimated. The robot control unit 13 handles the three-dimensional shape information as three-dimensional point cloud coordinates, and transports the workpiece 30 by teaching the picking robot 64 coordinate information of the gripping position and the transporting position. The method of determining the gripping position and the transporting position is not particularly limited. In addition, the arm part used for gripping is not limited to a shape like a human hand, and may be a suction arm using a vacuum pump, and is not particularly limited.

[0044] (summary) In this embodiment, in a picking robot system 300 including a sensing system 200, a picking robot 60, and a belt conveyor 61 for transporting an object 30 to the picking robot 60, the far-infrared sensors 3a and 3b include a first far-infrared sensor 3a for acquiring far-infrared information of the object 30 before it is heated by a heating device 4, and a second far-infrared sensor 3b for acquiring far-infrared information of the object 30 after it is heated by the heating device 4, and the computer 1 has a robot control unit 13 for controlling the picking robot 60 using three-dimensional shape information of the object 30.

[0045] According to the present embodiment configured as described above, a partial surface shape of the object 30 is generated from the far-infrared information of the object 30 before heating measured by the first far-infrared sensor 3a and the far-infrared information of the object 30 after heating measured by the second far-infrared sensor 3b, and the partial surface shape is used to interpolate the edge distance information of the object 30 measured by the visible light sensor 2, making it possible to accurately measure the three-dimensional shape of the object 30 including a transparent material while maintaining the efficiency of the entire system.

[0046] Furthermore, the calculator 1 in this embodiment may store the edge distance information and the partial surface shape information in association with each other every time the object 30 is measured, and when the newly generated edge distance information matches the previously generated edge distance information, the heating of the object 30 by the heating device 4 and the acquisition of the far-infrared information by the far-infrared sensors 3a and 3b may be omitted, and the newly generated edge distance information may be interpolated using the partial surface shape information corresponding to the previously generated edge distance information to generate the three-dimensional shape information of the object 30. This makes it possible to improve the efficiency of the entire system.

[0047] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-mentioned embodiments, and various modifications are included. For example, the above-mentioned embodiments have been described in detail to easily explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to add a part of the configuration of one embodiment to the configuration of another embodiment, and it is also possible to delete a part of the configuration of one embodiment, or to replace it with a part of another embodiment. [Explanation of symbols]

[0048] Reference Signs List 1...computer, 2...stereo camera (visible light sensor), 3...far-infrared camera (far-infrared sensor), 3a...far-infrared camera (first far-infrared sensor), 3b...far-infrared camera (second far-infrared sensor), 4...heating device, 5...image generation unit, 6...object area extraction unit, 7...edge distance information generation unit, 8...far-infrared image generation unit, 9...heating device control unit, 10...partial surface shape estimation unit, 11...shape interpolation unit, 12...object shape output unit, 13...robot control unit, 14...preliminary far-infrared image, 15...calibration information, 21...object Area extraction unit, 30... work (object), 31, 32... packaging material, 33... work body, 34... base, 35... captured image, 36... object area, 37... edge distance information, 40-43... far-infrared images, 44, 45... partial surface shape information, 50... measurement information calibration unit, 51... shape interpolation execution unit, 55... initial edge distance information, 56, 57... interpolated edge distance information, 60... picking robot, 61... conveyor belt, 100... sensing device, 200... sensing system, 300... picking robot system.

Claims

1. In a sensing device for measuring a three-dimensional shape of an object, A computer, A heating device, The computer includes: an image generating unit that generates an image based on visual information of the object; an object region extraction unit that extracts an area occupied by the object from the image as an object region; an edge distance information generating unit that extracts distance information of an edge portion of the object from the distance information of the object to generate edge distance information; a far-infrared image generating unit that generates a far-infrared image corresponding to the object region based on far-infrared information of the object; a heating device control unit that controls the heating device to heat the object; a partial surface shape estimation unit that estimates partial surface shape information of the object from far-infrared images before and after heating of the object; a shape interpolation unit that generates interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information; an object shape output unit that converts the interpolated edge distance information into three-dimensional shape information and outputs the three-dimensional shape information; A sensing device comprising:

2. The sensing device according to claim 1 , The far-infrared image generating unit generates information in which far-infrared information of the object is associated with each pixel of the object region as the far-infrared image. A sensing device comprising:

3. The sensing device according to claim 1 , The heating device control unit adjusts a heating time or a heating direction of the object by the heating device based on at least one of the object region, the edge distance information, the far-infrared image before heating the object, and prior information of the object. A sensing device comprising:

4. The sensing device according to claim 1 , The partial surface shape estimating unit, when estimating the partial surface shape of the object, removes noise included in the far-infrared image after the object is heated by using the far-infrared image before the object is heated. A sensing device comprising:

5. The sensing device according to claim 1 , The shape interpolation unit a measurement information calibration unit that calibrates the edge distance information and the partial surface shape information by converting each coordinate of the edge distance information and the partial surface shape information into coordinates in the same space; a shape interpolation execution unit that interpolates the calibrated edge distance information using the calibrated partial surface shape information; A sensing device comprising:

6. The sensing device according to claim 1 ; a visible light sensor for acquiring visual and distance information of the object; a far-infrared sensor for acquiring far-infrared information of the object; A sensing system comprising:

7. 7. The sensing system according to claim 6, The edge distance information generating unit Calculating a distance value from the visible light sensor to a position corresponding to each pixel of the image; generating information in which each pixel of the image is associated with the distance value as a distance image; Calculating a difference in distance values ​​between adjacent pixels of the distance image; extracting pixels whose difference is equal to or greater than a predetermined threshold as pixels constituting the edge portion; The edge distance information is generated by associating each pixel constituting the edge portion with the smaller one of the distance value of each pixel and the distance value of an adjacent pixel. A sensing system comprising:

8. 7. The sensing system according to claim 6, the partial surface shape information includes a plurality of pieces of partial surface shape information respectively corresponding to a plurality of partial surface shapes; The shape interpolation unit determines an order in which the plurality of pieces of partial surface shape information are used for interpolating the edge distance information, based on each distance from the visible light sensor to the plurality of partial surface shapes or each size of the plurality of partial surface shapes. A sensing system comprising:

9. 7. The sensing system according to claim 6, The visible light sensor is composed of a stereo camera, a visible light camera equipped with a projector, or a visible light camera having a function of estimating distance from an image. A sensing system comprising:

10. A sensing system according to claim 6; Picking robots and a belt conveyor that transports the object to the picking robot, The far-infrared sensor includes a first far-infrared sensor that acquires far-infrared information of the object before it is heated by the heating device, and a second far-infrared sensor that acquires far-infrared information of the object after it is heated by the heating device, The computer has a robot control unit that controls the picking robot using three-dimensional shape information of the object. A picking robot system comprising:

11. The picking robot system according to claim 10, The computer includes: storing the edge distance information and the partial surface shape information in association with each other each time the object is measured; When the newly generated edge distance information coincides with the previously generated edge distance information, heating of the object by the heating device and acquisition of the far-infrared information by the far-infrared sensor are omitted, and the newly generated edge distance information is interpolated using the partial surface shape information corresponding to the previously generated edge distance information, thereby generating three-dimensional shape information of the object. A picking robot system comprising:

12. 1. A sensing method for measuring a three-dimensional shape of an object, comprising: generating an image of the object based on visual information of the object; extracting an area of ​​the image occupied by the object as an object area; A step of extracting distance information of an edge portion of the object from the distance information of the object to generate edge distance information; generating a far-infrared image corresponding to the object region based on far-infrared information of the object; heating the object; generating a far infrared image of the object after heating the object; A step of estimating partial surface shape information of the object from far-infrared images before and after heating of the object; generating interpolated edge distance information of the object by interpolating the edge distance information using the partial surface shape information; and converting the interpolated edge distance information into three-dimensional shape information. A sensing method comprising:

Images