Thickness estimation apparatus and thickness estimation method

Abstract

The present invention provides a thickness estimation device and estimation method that can improve the accuracy of estimating the thickness of iron pieces that may be fed into a shredder. [Solution] The thickness estimation device is a thickness estimation device for estimating the thickness of an object, comprising: a data acquisition unit that acquires an image generated by imaging the object and 3D shape data of the object; a surface identification unit that inputs the image of the object into a trained model and acquires region data of the surface of the object used for estimating the thickness of the object from the trained model, thereby identifying the surface used for thickness estimation in the image; and a thickness estimation unit that estimates the thickness of the object based on the coordinates of the 3D shape data of the surface used for thickness estimation identified by the surface identification unit in the image.

Description

Technical Field

[0001] The present invention relates to a thickness estimation device and a thickness estimation method.

Background Art

[0002] In recent years' steel industry, there has been an increasing demand for recycling iron in order to achieve effects such as reducing carbon dioxide emissions. In order to produce high-quality steel using iron pieces to be recycled, it is necessary to separate (remove) in advance objects that are not suitable for crushing (hereinafter referred to as "non-crushable objects") from the pile of iron scrap containing the iron pieces. Patent Document 1 discloses an information processing device for managing iron scrap in quantitative categories.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] If non-crushable objects are not separated from the iron scrap in advance, they will be put into the shredder together with the iron scrap. For example, when an iron piece with a thickness greater than or equal to a threshold value is put into the shredder, such an iron piece cannot be crushed by the shredder hammers and stays in the crushing chamber for a long time, resulting in problems such as wear of the shredder hammers and shortening of the shredder's lifespan, or problems such as a decrease in work efficiency due to the occurrence of work for removing the iron piece from the crushing chamber. Therefore, an iron piece with a thickness greater than or equal to a threshold value is not suitable for crushing and can be said to be an example of a non-crushable object. In order to prevent the shredder from being damaged, it is desirable to estimate the thickness of the iron piece based on the captured image of the iron piece and separate in advance the iron pieces with a thickness greater than or equal to the threshold value from the iron scrap.

[0005] However, for example, on a conveyor belt that transports iron pieces, the iron pieces are placed in random orientations, so the apparent thickness of the iron piece in the captured image changes depending on its orientation. Therefore, there is a problem in that it is difficult to improve the accuracy of estimating the thickness of the iron piece before it is fed into the shredder.

[0006] The present invention has been made in view of the above circumstances, and provides a thickness estimation device and a thickness estimation method that can improve the accuracy of estimating the thickness of an iron piece before it is fed into a shredder. [Means for solving the problem]

[0007] (1) One aspect of the present invention is a thickness estimation device for estimating the thickness of an object, comprising: an image capture unit that generates an image of the object; a data acquisition unit that acquires three-dimensional shape data of the object; a surface identification unit that inputs the image capture unit of the object into a trained model and acquires region data of a surface used for estimating the thickness of the object from the trained model, thereby identifying the surface used for estimating the thickness in the image capture unit; and a thickness estimation unit that estimates the thickness of the object based on the coordinates of the three-dimensional shape data of the surface used for estimating the thickness identified by the surface identification unit in the image capture unit.

[0008] (2) One aspect of the present invention is a thickness estimation device for estimating the thickness of an object, comprising: a data acquisition unit that acquires an image generated by imaging the object; a surface identification unit that identifies a surface used for estimating the thickness in the image by inputting the image into a trained model and acquiring region data of a surface used for estimating the thickness of the object from the trained model; an orientation correction unit that corrects the orientation of the surface used for estimating the thickness identified by the surface identification unit in the image; and a thickness estimation unit that estimates the thickness of the object based on the length in the image of the surface used for estimating the thickness, whose orientation has been corrected by the orientation correction unit.

[0009] (3) In one aspect of the present invention, in the thickness estimation apparatus described in (2) above, the orientation correction unit inputs the captured image to a depth estimation model, obtains three-dimensional shape data of the object from the depth estimation model in the captured image, determines the normal of the surface used for thickness estimation based on the three-dimensional shape data, and corrects the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide.

[0010] (4) In one aspect of the present invention, in the thickness estimation device described in (2) above, the orientation correction unit acquires three-dimensional shape data of the object from a sensor capable of measuring the three-dimensional shape data, determines the normal of the surface used for thickness estimation based on the three-dimensional shape data, and corrects the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide.

[0011] (5) One aspect of the present invention is a thickness estimation method for estimating the thickness of an object, comprising: an image capture generated by imaging the object; a data acquisition step of acquiring three-dimensional shape data of the object; a surface identification step of inputting the image capture of the object into a trained model and acquiring region data of the surface of the object used for estimating the thickness of the object from the trained model, thereby identifying the surface used for estimating the thickness in the image capture; and a thickness estimation step of estimating the thickness of the object in the image capture based on the coordinates of the three-dimensional shape data of the surface used for estimating the thickness identified by the surface identification step.

[0012] (6) One aspect of the present invention is a thickness estimation method for estimating the thickness of an object, comprising: a data acquisition step of acquiring an image generated by imaging the object; a surface identification step of identifying a surface used for estimating the thickness in the image by inputting the image of the object into a trained model and acquiring region data of a surface used for estimating the thickness of the object from the trained model; an orientation correction step of correcting the orientation of the surface used for estimating the thickness identified in the surface identification step in the image; and a thickness estimation step of estimating the thickness of the object based on the length in the image of the surface used for estimating the thickness whose orientation has been corrected in the orientation correction step.

[0013] (7) One aspect of the present invention is the thickness estimation method described in (6) above, wherein the orientation correction step involves inputting the captured image into a depth estimation model, obtaining three-dimensional shape data of the object from the depth estimation model in the captured image, determining the normal of the surface used for thickness estimation based on the three-dimensional shape data, and correcting the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide.

[0014] (8) One aspect of the present invention is the thickness estimation method described in (6) above, wherein the orientation correction step involves acquiring three-dimensional shape data of the object from a sensor capable of measuring the three-dimensional shape data, determining the normal of the surface used for thickness estimation based on the three-dimensional shape data, and correcting the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide. [Effects of the Invention]

[0015] This invention makes it possible to improve the accuracy of estimating the thickness of iron pieces before they are fed into a shredder. [Brief explanation of the drawing]

[0016] [Figure 1]It is a diagram showing a configuration example of a separation system in the first embodiment. [Figure 2] It is a diagram showing an example of a learning image in the first embodiment. [Figure 3] It is a diagram showing an example of a process using the separation system in the first embodiment. [Figure 4] It is a diagram showing an example of a captured image of an iron piece on a conveying device in the first embodiment. [Figure 5] It is a diagram showing an example of a captured image of an iron piece having a surface with corrected orientation in the first embodiment. [Figure 6] It is a diagram showing a configuration example of a separation device in the first embodiment. [Figure 7] It is a diagram showing a specific example of a separation process in the first embodiment. [Figure 8] It is a flowchart showing an operation example of a thickness estimation device in the first embodiment. [Figure 9] It is a diagram showing an example of a learning image in the second embodiment. [Figure 10] It is a diagram showing a specific example of a heavy machine in the second embodiment. [Figure 11] It is a diagram showing an example of a captured image of an iron piece in the second embodiment. [Figure 12] It is a diagram showing an example of an iron piece having a surface with corrected orientation in a captured image in the second embodiment. [Figure 13] It is a diagram for explaining a thickness estimation process in the third embodiment. [Figure 14] It is a diagram for explaining a thickness estimation process in the third embodiment. [Figure 15] It is a diagram for explaining a thickness estimation process in the third embodiment. [Figure 16] It is a flowchart showing an operation example of a thickness estimation device in the third embodiment.

Embodiments for Carrying Out the Invention

[0017] Embodiments of the present invention will be described in detail with reference to the drawings. (First Embodiment) Figure 1 shows an example of the configuration of the separation system 1 in the first embodiment. The separation system 1 is a system for separating unsuitable materials from a collection of materials that include desired materials and unsuitable materials. Desired materials are relatively thin pieces of iron that can be shredded and fed into a shredder. Unsuitable materials are pieces of iron with a thickness above a threshold (for example, 6 mm or more) that are difficult to shred and cannot be fed into a shredder. In the following, we will explain using the example of a case where multiple desired materials and unsuitable materials are mixed together to form iron scrap (a collection of materials).

[0018] The separation system 1 comprises a learning device 10, one or more sensors 20, a thickness estimation device 30, a separation device 40, and a communication line 50. As shown in Figure 1, the learning device 10 is configured separately from the thickness estimation device 30, but is not limited to this and may be provided within the thickness estimation device 30. The learning device 10 comprises a learning communication unit 11 and a learning unit 12. The thickness estimation device 30 comprises an estimation communication unit 31, a storage device 32, and a control unit 33. The separation device 40 comprises a separation processing unit 41 and a separation execution unit 42.

[0019] The learning device 10 and the thickness estimation device 30 are configured using one or more hardware processors such as a CPU (Central Processing Unit) and one or more memories (main memory). The memories are configured using storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory). The learning device 10 and the thickness estimation device 30 perform various calculations by having one or more hardware processors execute one or more programs stored in the memories.

[0020] Furthermore, some or all of the functions of the learning device 10 and the thickness estimation device 30 may be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (such as portable media like solid-state drives (SSDs), hard disks and semiconductor storage devices built into computer systems). The above program may be transmitted via a communication line 50.

[0021] First, let me explain the overview of separation system 1. The learning device 10 generates a trained model by training a deep learning model consisting of a neural network using annotated training images (teacher images) and labels corresponding to the annotated content. The trained model comprises multiple layers of the neural network. The trained model is, for example, a segmentation model that detects objects in an image on a pixel-by-pixel basis. The explanatory variables of the trained model are the captured images of a set of objects. The objective variable of the trained model is the surface (region) among the faces of the iron piece in the captured image that is used to estimate the thickness of the iron piece. The learning device 10 transmits the generated trained model to the thickness estimation device 30.

[0022] Sensor 20 is a device that captures images of an object, generates captured images, and acquires three-dimensional shape data of that object. Sensor 20 is installed in a position where it can capture images from above (a position overlooking the conveying device) of multiple objects (iron pieces) extracted from a group of iron scrap from which materials other than iron pieces have been removed in advance, and which are being transported by a conveying device (belt conveyor). The multiple objects captured by Sensor 20 may include iron pieces of various thicknesses. There may be one or more sensors 20. Any of the multiple sensors 20 may be installed in a position where they can capture images of the group of objects being transported by the conveying device from the side or diagonally.

[0023] Sensor 20 may include, for example, a camera (2D image camera). This camera continuously images the object in a time-series order and generates captured images (frames). The camera included in sensor 20 may also be, for example, a stereo camera (3D image camera). This stereo camera generates a set of captured images (stereo pair) taken from the same location from different angles. The camera included in sensor 20 may also be, for example, a light section camera. This light section camera acquires point cloud data of a set of objects based on the displacement of linear light irradiated onto the surface of objects being transported by a belt conveyor, and generates 3D shape data corresponding to the shape of the surface of the objects based on this point cloud data. The point cloud data may be generated using a light detection and ranging (LiDAR: Light Detection And Ranging) method. The 3D shape data generated by sensor 20 includes spatial coordinate values ​​corresponding to each position.

[0024] The thickness estimation device 30 functions as a data acquisition unit that acquires captured images of a collection of objects (one or more objects) from the sensor 20 (2D image camera). By inputting the captured images of one or more objects into a trained model, the thickness estimation device 30 identifies the surface of the iron piece contained in the object that will be used to estimate the thickness of the iron piece.

[0025] The thickness estimation device 30 may also acquire depth data of the iron pieces in the captured image by inputting the captured image, which is an image of a collection of objects, into a depth estimation model when it acquires an image from the sensor 20. In this case, a known depth estimation model is used, and depth data is acquired using a trained depth estimation model obtained by training the model to output depth data, which is the distance in the depth direction of the iron pieces, when an image is input. Based on the depth data of the iron pieces, the thickness estimation device 30 can obtain three-dimensional shape data (the orientation of the specified surface) of the iron pieces.

[0026] The thickness estimation device 30 may also function as a data acquisition unit that acquires, in addition to the image generated by imaging an object, a set of image data (stereo pair) of an object image taken from the sensor 20 (3D image camera). If a set of image data (stereo pair) is acquired, the device may also acquire 3D shape data of the iron piece based on the acquired stereo pair.

[0027] Furthermore, the thickness estimation device 30 may also function as a data acquisition unit that acquires point cloud data (3D shape data) of a collection of objects from the sensor 20 (light section camera), in addition to the captured image generated by imaging the object. If point cloud data (3D shape data) of a collection of objects is acquired, the device may also acquire 3D shape data of the iron piece based on the acquired point cloud data.

[0028] The thickness estimation device 30 uses a surface identification unit 331 (described later) to identify a surface used for estimating the thickness of the iron piece (the surface having the shortest side when the iron piece is modeled as a three-dimensional rectangular parallelepiped).

[0029] Next, the thickness estimation device 30 uses the orientation correction unit 332 (described later) to correct the orientation of the surface used for thickness estimation in the captured image acquired from the sensor 20. Here, the thickness estimation device 30 corrects the orientation of the surface used for thickness estimation in the captured image so that the normal of the surface used for thickness estimation matches the imaging direction of the captured image. In other words, the thickness estimation device 30 corrects the orientation of the surface used for thickness estimation in the captured image so that the normal of the surface used for thickness estimation points in the direction of the sensor 20.

[0030] The thickness estimation device 30 uses the thickness estimation unit 333 (described later) to estimate the thickness of an iron piece having a surface, based on the length of that surface in the captured image (the number of pixels in a predetermined direction perpendicular to the normal facing the sensor 20) of the surface used for thickness estimation, whose orientation has been corrected in the captured image.

[0031] Thus, the thickness estimation device 30 estimates the thickness of an iron piece based on the length of the surface used for thickness estimation, which has been corrected for orientation. Therefore, it is possible to improve the estimation accuracy of the thickness of the iron piece regardless of the orientation of the iron piece.

[0032] Furthermore, the thickness estimation unit 333 may also estimate the thickness of a surface used for thickness estimation by using the 3D shape data of the surface used for thickness estimation from the acquired 3D shape data. In this case as well, since the distance can be calculated using the coordinate information contained in the 3D shape data, it is possible to improve the estimation accuracy of the thickness of the iron piece regardless of the orientation of the iron piece.

[0033] The thickness estimation device 30 uses the thickness estimation unit 333 to determine whether the thickness of the object (especially the surface used for thickness estimation) is above a threshold (for example, 6 mm or more). If it is determined that the estimated thickness is above the threshold, the thickness estimation device 30 controls the operation of the separation device 40 (for example, a suspended magnetic separator) and uses the separation device 40 to remove iron pieces with a thickness above the threshold from the multiple pieces of scrap iron on the conveying surface of the conveying device (belt conveyor). This makes it possible to separate iron pieces with a thickness above the threshold from the multiple pieces of scrap iron that constitute the object before they are fed into the shredder.

[0034] Next, we will describe the details of separation system 1. The learning communication unit 11 is connected to the thickness estimation device 30 via a communication line 50 and transmits the trained model generated by the learning unit 12 to the thickness estimation device 30. The learning unit 12 generates a trained model by training a deep learning model consisting of a neural network using annotated training images (teacher images) and labels corresponding to the annotated content. The training images, for example, capture multiple iron pieces taken from a pile of objects containing iron pieces (a group of iron scraps) and placed on a conveyor belt. In these training images, the surfaces (regions) used to estimate the thickness of the iron pieces are specified as correct labels by annotation.

[0035] Figure 2 shows an example of a training image 13 in the first embodiment. The training image 13 includes, as an example, an iron piece 201 having a surface 202 (region) used for estimating the thickness of the iron piece, and iron scrap 203 other than the iron piece 201. The surface 202 (ground truth label) used for estimating the thickness of the iron piece 201 is specified by annotation.

[0036] The learning unit 12 inputs the annotated training image 13 as explanatory variables into a learning model comprising multiple layers of a neural network. The learning unit 12 retrieves the segmented image as the target variable from the learning model. The learning unit 12 adjusts the weighting of each node in the neural network by backpropagation of the error between the annotated faces (ground truth labels) in the training image 13 and the images segmented by the learning model (identified faces). In this way, the learning unit 12 generates a trained model (segmentation model) from the learning model.

[0037] Figure 3 shows an example of processing using the separation system 1 in the first embodiment. The separation system 1 further comprises a conveying device 60, a heavy machine 70, and a supplying device 80. The heavy machine 70 is equipped with a lift magnet 71. In the separation system 1, a first region 91 is predetermined downstream of the conveying device 60. The first region 91 is the region where, after unsuitable materials for crushing have been removed from the iron scrap and only iron pieces that meet the conditions (such as thickness) remain, these iron pieces are conveyed and ultimately arrive. In addition, in the separation system 1, a second region 92 is predetermined at a different location from the first region 91 (for example, next to the conveying device 60). The second region 92 is the region where iron pieces 401 that are unsuitable for crushing and have a thickness greater than a threshold ultimately arrive.

[0038] The lift magnet 71 is an electromagnet installed on the heavy machinery 70, and moves in conjunction with the movement of the heavy machinery 70 or the movement of the arm of the heavy machinery 70. The state of the electromagnet of the lift magnet 71 (on and off) is switched according to the operation of the operator of the heavy machinery 70. The lift magnet 71 places iron scrap on the upper surface of the dispensing device 80 according to the operation of the operator of the heavy machinery 70. The dispensing device 80 vibrates itself, causing the object to move on the conveying surface of the conveying device 60, and gradually moves the object in the conveying direction for conveyance.

[0039] Sensor 20 captures an image of the collection of objects (iron pieces 301 and iron scrap) being transported by the transport device 60, from a position overlooking the transport device 60, within the sensor's detection range. Sensor 20 transmits the captured image of the collection of objects to the estimation communication unit 31. Sensor 20 may also acquire a set of captured images (stereo pair) and, based on the set of captured images (stereo pair), acquire 3D shape data, and transmit the acquired 3D shape data to the estimation communication unit 31. Sensor 20 may also acquire point cloud data (3D shape data) of the collection of objects and transmit it to the estimation communication unit 31.

[0040] In the thickness estimation device 30, the estimation communication unit 31 functions as a data acquisition unit that acquires captured images of the collection of objects from the sensor 20. The estimation communication unit 31 may also acquire 3D shape data in addition to the captured images 21. The estimation communication unit 31 records the captured images of the collection of objects (time-series captured images) and the 3D shape data of the collection of objects in the storage device 32.

[0041] In the thickness estimation device 30, the memory device 32 stores the trained model generated by the learning unit 12. The memory device 32 may also store a model for estimating the depth data of an object in an captured image (depth estimation model).

[0042] The storage device 32 stores captured images of a set of objects (images in which a set of objects has been captured) and programs executed by the control unit 33. The storage device 32 may also store sets of captured images (stereo pairs). The storage device 32 may also store point cloud data (3D shape data) of a set of objects.

[0043] Figure 4 is a diagram (overhead view) showing an example of an image 21 of an iron piece 301 on a transport device 60 in the first embodiment. The surface identification unit 331 acquires a trained model (segmentation model) from the storage device 32. The surface identification unit 331 acquires the image 21 from the storage device 32. In the image 21, as an example, an iron piece 301 placed on the transport device 60 is imaged from above. The surface identification unit 331 inputs the image 21 into the trained model and acquires region data of the surface 302 used to estimate the thickness of the iron piece 301 from the trained model, thereby identifying the surface 302 used to estimate the thickness of the iron piece 301 in the image 21.

[0044] The orientation correction unit 332 inputs the captured image 21 (2D image) into a depth estimation model to obtain depth data for each surface of the iron piece 301 in the captured image 21 from the depth estimation model. The orientation correction unit 332 also determines the normal vector 303 of the identified surface 302 based on the depth data of the iron piece 301.

[0045] If the sensor 20 is equipped with a stereo camera or a light section camera, the orientation correction unit 332 may acquire three-dimensional shape data of the iron piece 301 from the storage device 32. The orientation correction unit 332 may determine the normal 303 of the identified surface 302 based on the acquired three-dimensional shape data.

[0046] The orientation correction unit 332 corrects the orientation of the identified surface 302 in the captured image 21, for example by projection transformation, so that the determined normal 303 matches the imaging direction of the captured image 21. Figure 5 shows an example of a captured image 21 of an iron piece 301 having an orientation-corrected surface 302 in the first embodiment. As a result of this correction, the normal 303 of the surface 302 faces the sensor 20 in the captured image 21, so the thickness estimation unit 333 can accurately estimate the thickness of the iron piece 301 based on the length "L" of the surface 302 in the captured image 21 (the number of pixels in a predetermined direction perpendicular to the normal 303 facing the sensor 20).

[0047] Furthermore, if the estimation communication unit 31 has acquired 3D shape data in addition to the captured image, the thickness estimation unit 333 can also estimate the thickness of the iron piece using the coordinate information included in the 3D shape data without going through the processing of the surface identification unit 331 and the orientation correction unit 332.

[0048] Figure 6 shows an example of the configuration of the separation device 40 in the first embodiment. The separation device 40 (suspended magnetic separator) includes a separation processing unit 41. The separation device 40 also includes a coil 421, a motor 422, and a conveying belt 423 as a separation execution unit 42. Partitions at predetermined intervals are arranged on the conveying belt 423.

[0049] Figure 7 shows a specific example of the separation process in the first embodiment. In the separation system 1, the conveying device 60 conveys the collection of objects (for example, a group of iron scrap containing iron pieces with a thickness greater than a threshold (unsuitable for crushing)) placed on the conveying surface 101 in the conveying direction.

[0050] The transport device 60 of the separation system 1 carries a collection of objects. The transport direction of the transport device 60 is indicated by arrow 111. A sensor 20 is installed on the upstream side of the transport device 60. The sensor 20 is installed so that the transport surface 101 on the upstream side of the transport device 60 is included in the detection range 22. The sensor 20 may also be installed so that the side of the transport device 60 is included in the detection range 22.

[0051] The sensor 20 generates predetermined information about an object (e.g., a piece of iron) located inside the detection range 22. This predetermined information about the object may be, for example, an image of the object, or an image plus a set of images of the object (stereo pair), or three-dimensional shape data of the object (point cloud data). The sensor 20 transmits this predetermined information about the object to the thickness estimation device 30.

[0052] The separation device 40 is installed downstream of the detection range 22 in the transport direction of the transport device 60. The separation device 40 determines whether an object placed on the separation region 61 on the transport surface 101 of the transport device 60 is unsuitable for crushing by estimating the thickness of the object using the thickness estimation device 30, based on the acquired image of the object, a set of imaged images of the object (stereo pair), or the object's three-dimensional shape data (point cloud data). If it is determined to be unsuitable for crushing (i.e., an object to be separated), it is separated from the group of objects on the transport surface 101. The separation region 61 is provided downstream of the detection range 22.

[0053] Based on the control by the separation control unit 334, the separation processing unit 41 transitions the state of the coil 421 (electromagnet) from off to on when the object to be separated reaches the separation region 61. The separation processing unit 41 also drives the transport belt 423 by controlling the drive of the motor 422 based on the control by the separation control unit 334. Each partition placed on the transport belt 423 separates the object to be separated (iron piece with a thickness greater than or equal to the threshold) that has been attracted to the coil 421 by magnetic force, in the direction of the second region 92 as the transport belt 423 is driven.

[0054] In this way, the object to be separated, which was on the separation region 61 of the transport surface 101, is attracted by magnetic force to the separation device 40 located above the transport surface 101. The separation control unit 334 may control the magnetic force of the separation device 40 so that the magnetic force becomes stronger the thicker the object to be separated (iron piece with a thickness above a threshold). The transport belt 423 of the separation device 40 rotates to push the attracted object to be separated in the direction of arrow 402. As a result, the object to be separated, attracted to the separation device 40, moves to the second region 92 along arrow 131.

[0055] Furthermore, the second region 92 may be divided into multiple sections depending on the thickness of the iron pieces. For example, an iron piece with a thickness of 6 mm or more but less than 12 mm may be placed in the first section predetermined in the second region 92. For example, an iron piece with a thickness of 12 mm or more may be placed in the second section of the second region 92.

[0056] When a predetermined time has elapsed since the object to be separated reached the separation region 61, the separation processing unit 41 transitions the state of the coil 421 (electromagnet) from on to off based on the control of the separation control unit 334. The transport device 60 transports the object (iron piece with a thickness less than the threshold) to the first region 91.

[0057] In this way, the materials that were not separated by the separation device 40 (materials that are not unsuitable for crushing) are transported by the conveying device 60, moving along arrow 121 to the first region 91 located downstream of the conveying device 60. The materials separated by the separation device 40 (materials unsuitable for crushing) move along arrow 131 and reach the second region 92, which is different from the first region 91.

[0058] Next, an example of the operation of the thickness estimation device 30 will be described. Figure 8 is a flowchart showing an example of the operation (estimation process) of the thickness estimation device 30 in the first embodiment. The surface identification unit 331 acquires an image 21 of the iron piece 301 (object) from the sensor 20 (step S101). The surface identification unit 331 inputs the acquired image 21 into a trained model (segmentation model) (step S102). Based on the output of the trained model, the surface identification unit 331 identifies the surface 302 used to estimate the thickness of the iron piece 301 in the image 21 (step S103).

[0059] The orientation correction unit 332 corrects the orientation of the identified surface 302 in the captured image 21 based on, for example, depth data of the iron piece 301 (output of a depth estimation model) so that the normal 303 of the identified surface 302 matches the imaging direction of the captured image 21 (step S104). The thickness estimation unit 333 estimates the thickness of the iron piece 301 having the orientation-corrected surface 302 based on the length "L" of the orientation-corrected surface 302 in the captured image 21 (for example, the number of pixels on a predetermined side of the identified surface). For example, the thickness estimation unit 333 may use the result of multiplying the length "L" of the surface 302 by a predetermined coefficient as the thickness of the iron piece 301 (step S105).

[0060] The thickness estimation unit 333 determines whether the thickness of the iron piece 301 is greater than or equal to a threshold (step S106). If it is determined that the thickness of the iron piece 301 is less than the threshold (step S106: NO), the control unit 33 terminates the thickness estimation process. If it is determined that the thickness of the iron piece 301 is greater than or equal to the threshold (step S106: YES), the separation control unit 334 uses the separation device 40 to separate the iron piece 301, which has a thickness greater than or equal to the threshold, from the collection of iron pieces (group of iron scrap) as a target for separation (unsuitable for crushing) (step S107).

[0061] As described above, the thickness estimation device 30 estimates the thickness of the object. Here, the sensor 20 captures an image of the object and generates an image. The surface identification unit 331 inputs the image 21 of the iron piece 301 (object) into a trained model (segmentation model). The surface identification unit 331 identifies the surface 302 used for estimating the thickness of the iron piece 301 in the image 21 by obtaining region data of the surface 302 used for estimating the thickness of the iron piece 301 from the trained model. The orientation correction unit 332 corrects the orientation of the surface 302 (the surface used for thickness estimation) identified by the surface identification unit 331 in the image 21. The thickness estimation unit 333 estimates the thickness of the iron piece 301 having the orientation-corrected surface 302 (the surface used for thickness estimation) based on the length "L" of the surface 302 in the captured image 21, which has been orientation-corrected by the orientation correction unit 332.

[0062] In this way, the surface identification unit 331 identifies the surface 302 used for estimating the thickness of the iron piece 301 (object) in the captured image. The orientation correction unit 332 corrects the orientation of the identified surface 302 in the captured image 21. This makes it possible to improve the accuracy of estimating the thickness of the iron piece before it is fed into the shredder. It is possible to obtain the base material to be fed into the shredder with reduced working time and cost. In addition, it is possible to obtain the desired amount of iron scrap with reduced impurities.

[0063] (Second Embodiment) In the second embodiment, the difference from the first embodiment is that the thickness of the iron pieces in the iron scrap attracted by the magnetic force of the lift magnet 71 of the heavy machinery 70 is estimated. The second embodiment will be explained focusing on the differences from the first embodiment.

[0064] Figure 9 shows an example of a training image 14 in the second embodiment. The training image 14, as an example, captures an iron piece 201 having a surface 202 and a lift magnet 71. The iron piece 201 is magnetically attracted to the lift magnet 71. The surface 202 (ground truth label) is also specified on the iron piece 201 by annotation. Surface 202 is a surface (region) used to estimate the thickness of the iron piece 201.

[0065] The learning unit 12 inputs the annotated training image 14 as explanatory variables into a learning model equipped with a neural network, and obtains the segmented image as the target variable. The learning unit 12 adjusts the weighting of each node of the neural network by backpropagation for the error between the annotated surface in the training image 13 and the image segmented by the learning model. In this way, the learning unit 12 generates a trained model from the learning model.

[0066] Figure 10 shows a specific example of the heavy machinery 70 in the second embodiment. The heavy machinery 70 is equipped with a sensor 20 and a lift magnet 71. The sensor 20 is installed, for example, in front of the driver's seat of the heavy machinery 70. In the iron scrap accumulation area, during sorting work by the heavy machinery 70, the sensor 20 images the lift magnet 71 located inside the detection range 22 and the object that is magnetically attracted to the lift magnet 71. In Figure 10, as an example, iron pieces 301 and iron scrap 304 are magnetically attracted to the lift magnet 71.

[0067] Figure 11 shows an example of an image 23 of an iron piece 301 in the second embodiment. The sensor 20 transmits the image 23 of the collection of objects to the estimation communication unit 31. The surface identification unit 331 acquires a trained model (segmentation model) from the storage device 32. The surface identification unit 331 acquires the image 23 from the storage device 32. The image 23, as an example, captures the lift magnet 71 and the iron piece 301 which is magnetically attracted to the lift magnet 71. The surface identification unit 331 inputs the image 23 to the trained model. The surface identification unit 331 identifies the surface 302 used for estimating the thickness of the iron piece 301 in the image 21 by acquiring region data of the surface 302 used for estimating the thickness of the iron piece 301 from the trained model.

[0068] The orientation correction unit 332 may input the captured image 23 (2D image) into a depth estimation model to obtain depth data for each surface of the iron piece 301 in the captured image 23 from the depth estimation model. The orientation correction unit 332 may determine the normal 303 of the identified surface 302 based on the depth data of the iron piece 301.

[0069] If the sensor 20 is equipped with a stereo camera or a light section camera, the orientation correction unit 332 may acquire three-dimensional shape data of the iron piece 301 from the storage device 32. The orientation correction unit 332 may determine the normal 303 of the identified surface 302 based on the acquired three-dimensional shape data.

[0070] Figure 12 shows an example of an iron piece 301 having a surface 302 whose orientation has been corrected in the captured image 23, according to the second embodiment. The orientation correction unit 332 corrects the orientation of the identified surface 302 in the captured image 23, for example by projection transformation, so that the determined normal 303 and the imaging direction of the captured image 21 coincide. As a result, the normal 303 of the surface 302 faces the sensor 20 in the captured image 23, so that the thickness of the iron piece 301 can be accurately estimated based on the length "L" of the surface 302 in the captured image 23 (the number of pixels in a predetermined direction perpendicular to the normal 303 facing the sensor 20).

[0071] The thickness estimation unit 333 determines whether the thickness of the iron piece 301 is greater than or equal to a threshold. If it is determined that the thickness of the iron piece 301 is greater than or equal to a threshold, the separation control unit 334 notifies the operator (person) of the heavy machinery 70 that the thickness is greater than or equal to a threshold. The operator of the heavy machinery 70 may manually separate the iron piece 301 with a thickness greater than or equal to the threshold from the lift magnet 71 and the iron scrap 304 as an object to be separated (unsuitable for crushing). The operator may also separate the iron piece 301 with a thickness greater than or equal to the threshold from the iron scrap 304 by manipulating the magnetic force of the lift magnet 71.

[0072] As described above, the surface identification unit 331 inputs the captured image 23 of one or more iron pieces 301 into a trained model (segmentation model). The surface identification unit 331 obtains region data of the surface 302 used to estimate the thickness of the iron piece 301 from the trained model, thereby identifying the surface 302 used to estimate the thickness of the iron piece 301 that is magnetically attracted to the lift magnet 71 in the captured image 23. The orientation correction unit 332 corrects the orientation of the identified surface 302 in the captured image 23. The thickness estimation unit 333 estimates the thickness of the iron piece 301 having the orientation-corrected surface 302 based on the length "L" of the orientation-corrected surface 302 in the captured image 23.

[0073] This makes it possible to improve the accuracy of estimating the thickness of iron pieces magnetically attracted to the lift magnet. It is also possible to obtain the base material to be fed into the shredder with reduced working time and cost. Furthermore, it is possible to obtain the desired amount of iron scrap with reduced impurities.

[0074] (Third embodiment) In the third embodiment, the difference from the first and second embodiments is that the length of the surface of the object is estimated as thickness, even without relying on the length in the captured image. The third embodiment will be explained primarily in terms of the differences from the first and second embodiments.

[0075] In the above description, the orientation of the surface used for thickness estimation is corrected by the orientation correction unit 332, and then the length of the orientation-corrected surface in the captured image is estimated as the thickness of the object by the thickness estimation unit 333. However, the present invention is not limited to this. For example, the thickness estimation device 30 may omit the processing in the orientation correction unit 332 and calculate the distance between each point (in 3D space) from the spatial coordinate values ​​of each coordinate in the 3D shape data (point cloud data) acquired by the sensor 20, and estimate that distance as the thickness of the object.

[0076] In other words, the thickness estimation unit 333 determines two points (a pair of start and end points) for calculating the thickness width on the thickness region boundary, and by directly calculating the distance between these two points based on their coordinates in three-dimensional space, the thickness of the object can be estimated. More specifically, for example, the following calculation procedures (1) to (5) can be used.

[0077] (1) The surface identification unit 331 extracts a surface 306 (Figure 13(a)) used for estimating the thickness of the iron piece 305 from the captured image (Figure 13(b)), and performs a thinning process on the surface 306 used for thickness estimation (by repeating an expansion process and a contraction process a predetermined number of times, etc.) (Figure 13(c)). (2) The thickness estimation unit 333 determines two normals (two normals perpendicular to the contour) that originate from the thinned surface 306 (contour) for each search point on the captured image (which is essentially a linear contour line "skeleton") (Figure 14(a)), and determines a pair of intersection points (referred to as a "pair of thickness measurement candidate points") between these normals and a line "boundary" that corresponds to the outer perimeter of the surface 306 used for estimating the thickness before thinning (Figure 14(b)). (3) The thickness estimation unit 333 generates three-dimensional coordinates for the set of thickness measurement candidate points by assigning spatial coordinate information along the depth direction of the captured image to the two-dimensional coordinates of each point included in the set of thickness measurement candidate points, based on the coordinates in three-dimensional space (Figure 15(a)). (4) The thickness estimation unit 333 calculates the distance between each point (from the starting point "start_point" to the ending point "end_point") in the set of thickness measurement candidate points with three-dimensional coordinates generated in (3) above, and generates a histogram (Figure 15(b)). (5) The thickness estimation unit 333 uses statistical methods (e.g., mean or median of distance) on the generated histogram to identify a set of thickness measurement candidate points that is suitable for calculating the final thickness from among the set of thickness measurement candidate points, and calculates the distance in three-dimensional space of the identified set of thickness measurement candidate points as the representative length of the surface 306 used for thickness estimation. This calculated length is taken as the thickness of the iron piece 305.

[0078] Figure 16 is a flowchart illustrating an example of the operation of the thickness estimation device. The surface identification unit 331 acquires an image 21 of the iron piece 305 (object) from the sensor 20 (step S201). The surface identification unit 331 acquires three-dimensional shape data of the iron piece 305 from the sensor 20 (step S202). The surface identification unit 331 inputs the acquired image 24 into a trained model (segmentation model) (step S203). Based on the output of the trained model, the surface identification unit 331 identifies the surface 306 used to estimate the thickness of the iron piece 305 in the image 24 (step S204).

[0079] The thickness estimation unit 333 estimates the thickness of the iron piece 305 based on the coordinates of the 3D shape data (step S205). The thickness estimation unit 333 determines whether the thickness of the iron piece 305 is greater than or equal to a threshold (step S206). If it is determined that the thickness of the iron piece 305 is less than the threshold (step S206: NO), the control unit 33 terminates the thickness estimation process. If it is determined that the thickness of the iron piece 305 is greater than or equal to the threshold (step S206: YES), the separation control unit 334 uses the separation device 40 to separate the iron piece 305, which has a thickness greater than or equal to the threshold, from the collection of iron pieces (iron scrap group) as an object to be separated (unsuitable for crushing) (step S207).

[0080] As described above, the thickness estimation device 30 estimates the thickness of the object. Here, the sensor 20 is capable of measuring the three-dimensional shape data of the iron piece. The sensor 20 captures an image of the iron piece 305 (object) to generate an image 24 and acquires the three-dimensional shape data of the iron piece 305. The surface identification unit 331 inputs the image 24 of the iron piece 305 into a trained model and acquires region data of the surface 306 used for estimating the thickness of the iron piece 305 from the trained model, thereby identifying the surface 306 used for thickness estimation in the image 24. The thickness estimation unit 333 estimates the thickness of the iron piece 305 based on the coordinates of the three-dimensional shape data of the surface 306 used for thickness estimation identified by the surface identification unit 331 in the image 24. This makes it possible to improve the accuracy of thickness estimation of the iron piece before it is fed into the shredder, even if the shape of the iron piece is not approximated as a rectangular parallelepiped.

[0081] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of symbols]

[0082] 1...Separation system, 10...Learning device, 11...Learning communication unit, 12...Learning unit, 13...Learning image, 14...Learning image, 20...Sensor, 21...Captured image, 22...Detection range, 23...Captured image, 24...Captured image, 30...Thickness estimation device, 31...Estimation communication unit, 32...Storage device, 33...Control unit, 40...Separation device, 41...Separation processing unit, 42...Separation execution unit, 50...Communication line, 60...Transportation device, 61...Separation area, 70...Heavy machinery, 71...Lift magnet, 80 ...Providing device, 91...First area, 92...Second area, 101...Conveying surface, 111...Arrow, 121...Arrow, 131...Arrow, 201...Iron piece, 202...Surface, 203...Iron scrap, 301...Iron piece, 302...Surface, 303...Normal, 304...Iron scrap, 305...Iron piece, 306...Surface, 331...Surface identification unit, 332...Orientation correction unit, 333...Thickness estimation unit, 334...Separation control unit, 401...Iron piece, 402...Arrow, 421...Coil, 422...Motor, 423...Conveying belt

Claims

1. A thickness estimation device for estimating the thickness of an object, A data acquisition unit that acquires an image generated by imaging the aforementioned object, and three-dimensional shape data of the aforementioned object, A surface identification unit inputs the captured image of the object into a trained model and obtains region data of the surface of the object used for estimating the thickness of the object from the trained model, thereby identifying the surface used for estimating the thickness in the captured image. A thickness estimation unit estimates the thickness of the object based on the coordinates of the three-dimensional shape data of the surface used for estimating the thickness, which has been identified by the surface identification unit in the captured image. A thickness estimation device equipped with the following features.

2. A thickness estimation device for estimating the thickness of an object, A data acquisition unit that acquires an image generated by imaging the aforementioned object, A surface identification unit identifies the surface used for estimating the thickness of the object in the captured image by inputting the captured image into a trained model and obtaining region data of the surface used for estimating the thickness of the object from the trained model, The captured image includes an orientation correction unit that corrects the orientation of the surface used to estimate the thickness identified by the surface identification unit, A thickness estimation unit estimates the thickness of the object based on the length in the captured image of the surface used for thickness estimation, whose orientation has been corrected by the orientation correction unit, A thickness estimation device equipped with the following features.

3. The thickness estimation apparatus according to claim 2, wherein the orientation correction unit inputs the captured image to a depth estimation model, obtains three-dimensional shape data of the object from the depth estimation model in the captured image, determines the normal of the surface used for thickness estimation based on the three-dimensional shape data, and corrects the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide.

4. The thickness estimation apparatus according to claim 2, wherein the orientation correction unit acquires three-dimensional shape data of the object from a sensor capable of measuring the three-dimensional shape data, determines the normal of the surface used for estimating the thickness based on the three-dimensional shape data, and corrects the orientation of the surface used for estimating the thickness so that the normal and the imaging direction of the captured image coincide.

5. A thickness estimation method for estimating the thickness of an object, A data acquisition step involves capturing an image of the object and generating an image of it, and acquiring three-dimensional shape data of the object. A surface identification step involves inputting an image of the object into a trained model and obtaining region data from the trained model for the surface of the object used to estimate its thickness, thereby identifying the surface in the image used to estimate the thickness. A thickness estimation step is performed in which the thickness of the object is estimated based on the coordinates of the three-dimensional shape data of the surface used for estimating the thickness, which was identified in the surface identification step, in the captured image. A thickness estimation method including the following.

6. A thickness estimation method for estimating the thickness of an object, A data acquisition step involves capturing an image of the object and obtaining the resulting image. A surface identification step involves inputting an image of the object into a trained model and obtaining region data from the trained model for the surface of the object used to estimate its thickness, thereby identifying the surface in the image used to estimate the thickness. The captured image includes an orientation correction step for correcting the orientation of the surface used to estimate the thickness identified in the surface identification step, A thickness estimation step in which the thickness of the object is estimated based on the length in the captured image of the surface used for thickness estimation, whose orientation has been corrected in the orientation correction step, A thickness estimation method including the following.

7. The thickness estimation method according to claim 6, wherein the orientation correction step involves inputting the captured image into a depth estimation model, obtaining three-dimensional shape data of the object from the depth estimation model in the captured image, determining the normal of the surface used for thickness estimation based on the three-dimensional shape data, and correcting the orientation of the surface used for thickness estimation so that the normal and the imaging direction of the captured image coincide.

8. The thickness estimation method according to claim 6, wherein the orientation correction step involves acquiring three-dimensional shape data of the object from a sensor capable of measuring the three-dimensional shape data, determining the normal of the surface used for estimating the thickness based on the three-dimensional shape data, and correcting the orientation of the surface used for estimating the thickness so that the normal and the imaging direction of the captured image coincide.

Images