Image processing method, image processing system, and program

Abstract

The objective of the present invention is to enable accurate identification of piled matter piled in each of a plurality of piling regions within a piled matter storage warehouse in which the piling regions are located. The present invention provides a method for processing images obtained by imaging, by means of an imaging device, the inside of a piled matter storage warehouse having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which piled matter is piled, wherein, for each area 4, a predetermined best shot region B that can be imaged by the imaging device in a state in which the area 4 is not blocked by an obstacle is set, and the method includes a piled matter extraction step for extracting the piled matter in a best shot image captured within the best shot region B.

Description

Image processing method, image processing system, and program

[0001] The present invention relates to an image processing method, an image processing system, and a program.

[0002] Scrap is piled up and stored in a storage facility (scrap yard) for scrap metals, etc. As a device for estimating the weight of the scrap piled up in such a storage facility, for example, Patent Document 1 discloses a method for measuring the height of the piles of scrap in each area stored in the scrap yard using parallax caused by stereo vision using a pair of cameras, and estimating the remaining amount of scrap based on the pile heights.

[0003] JP 2007-197170 A

[0004] Here, scrap storage facilities may be divided into multiple areas by partition walls. In scrap yards where obstacles such as partition walls exist, scrap may be in the blind spot of the obstacle depending on the camera's imaging position. Furthermore, in scrap yards where multiple sections are adjacent to each other and separated by partition walls, adjacent scrap may be identified as one piece depending on the camera's imaging position and the height of the scrap stack relative to the height of the partition wall. If the remaining amount of scrap is estimated based on such images, the scrap cannot be accurately identified, resulting in a problem of an estimated amount being less than the actual remaining amount.

[0005] The present invention has been developed in consideration of the above-mentioned problems, and its purpose is to enable accurate identification of sediments deposited in each deposition area in a sediment storage facility having multiple deposition areas.

[0006] According to one aspect of the present invention, there is provided a method for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple sediment-accumulating areas, the image processing method including: setting a predetermined best-shot area for each sediment storage area so that the image of the sediment storage area can be captured by the imaging device without being blocked by obstacles; and extracting sediments from a best-shot image captured within the best-shot area. According to the above aspect, by setting the best-shot area, the sediments are prevented from being in blind spots of obstacles, so that the entire sediment can be captured and the sediments can be accurately identified.

[0007] According to one aspect of the present invention, the best shot area is set based on the size of the imaging area, which is determined based on the imaging height and field of view of the imaging device, and the size of the accumulation area. According to the above aspect, since the best shot area is set based on the size of the imaging area and the size of the accumulation area, it is possible to capture an image so that the accumulation does not enter the blind spot of an obstacle from the imaging position, and the accumulation can be accurately identified.

[0008] According to one aspect of the present invention, the method further includes an imaging step of capturing image data within the sediment storage facility while moving or changing the angle of the imaging device, a position information generation step of generating position information regarding the position or angle at which the image was captured by the imaging device, and a best-shot image extraction step of extracting best-shot images captured within best-shot areas based on the position information. When capturing images while moving the imaging device, the amount of captured image data becomes enormous, and processing all of the image data places a heavy load on the computer. In contrast, according to the above aspect, image data captured within best-shot areas is extracted from the captured image data, thereby reducing the load on the computer. Furthermore, each sediment storage area within the sediment storage facility can be captured even with a small number of cameras.

[0009] According to one aspect of the present invention, the method further includes a target deposit identification step of identifying only deposits in a target deposit area from the deposits extracted in the deposit extraction step, and the target deposit identification step extracts only deposits in the target deposit area within the best shot image based on position information. According to the above aspect, only deposits in the target deposit area can be identified based on the position information, and the deposits can be extracted separately from deposits in other deposit areas.

[0010] According to one aspect of the present invention, the method further includes a deposit classification step of classifying the deposits extracted in the deposit extraction step by deposit area. According to this aspect, by classifying the deposits by deposit area, even when an image is captured at an imaging position where multiple deposit areas are included in the image, it is possible to prevent deposits deposited in different deposit areas from being identified as deposits from the same deposit area, and it is possible to accurately identify the deposits.

[0011] According to one aspect of the present invention, the deposit extraction step extracts partition walls from the captured best shot image. According to this aspect, by extracting the partition walls and deposits, even if adjacent deposit areas exist in the deposit storage facility, it is possible to recognize the partition walls separating the adjacent deposit areas, and it is possible to prevent deposits deposited in adjacent deposit areas from being identified as deposits from the same deposit area, thereby enabling accurate identification of the deposits.

[0012] According to one aspect of the present invention, the deposit classification step determines whether at least a portion of a partition wall is included inside the deposit, and divides the deposit if it is determined that at least a portion of the partition wall is included inside the deposit. Even if deposits of multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, it is possible to recognize the partition walls separating the deposit areas by determining whether a portion of the partition wall is included inside the deposit, and to separate the deposit by deposit area.

[0013] According to one aspect of the present invention, the deposit classification step determines whether at least a portion of the deposit is contained on the upper ends of the partition walls, and if it is determined that at least a portion of the deposit is contained on the upper ends of the partition walls, divides the deposit. Even if the deposits of multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, by determining whether at least a portion of the deposit is contained on the upper ends of the partition walls, it is possible to recognize the partition walls separating the deposit areas and separate the deposits for each deposit area.

[0014] According to one aspect of the present invention, the method further comprises a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit areas based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as the basis, differences in height of the deposit among the multiple deposit areas can be utilized, and the deposit can be classified into each deposit area.

[0015] According to one aspect of the present invention, the method further includes a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and a weight estimation step of estimating a weight of the deposit based on the three-dimensional shape. According to the above aspect, the three-dimensional shape of the deposit can be constructed, and the weight of the deposit can be estimated.

[0016] According to one aspect of the present invention, the weight estimation step calculates the weight of the pile based on a variable corresponding to the height of the pile. Although the density of the pile varies depending on the height of the pile because the pile is compressed by its own weight, according to the above aspect, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.

[0017] According to one aspect of the present invention, the variable corresponding to height is larger as the height increases. As the height of a pile increases, the density of the pile increases due to compression caused by its own weight. However, according to the above aspect, since the variable that increases as the height increases is used, the weight of the pile can be estimated more accurately.

[0018] According to one aspect of the present invention, the height-dependent variable has a smaller increase per unit height as the height increases. As the height of the pile increases, the pile is compressed by its own weight, increasing its density, but the increase in density becomes smaller as the height increases. According to the above aspect, the variable with a smaller increase per unit height is used as the height increases, allowing for more accurate estimation of the weight of the pile.

[0019] According to one aspect of the present invention, in the weight estimation step, the three-dimensional shape of the deposit is divided into a plurality of parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby estimating the weight of the deposit. According to the above aspect, since the three-dimensional shape of the deposit is divided into a plurality of parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, the weight of the deposit can be estimated more accurately.

[0020] According to one aspect of the present invention, there is provided an image processing system for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, wherein a predetermined best shot area is set for each deposition area so that the imaging device can capture an image of the deposition area without obstructing it, and the image processing system is equipped with a sediment extraction unit that identifies the sediments in the best shot image captured within the best shot area and extracts the sediments.

[0021] According to one aspect of the present invention, there is provided a program for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, wherein a predetermined best shot area is set for each deposition area so that the imaging device can capture an image of the deposition area without blocking the deposition area by any obstacles, and the program causes a computer to execute a sediment extraction step of identifying the sediment in the best shot image captured within the best shot area and extracting the sediment.

[0022] According to one aspect of the present invention, a method for processing an image captured by an imaging device in a sediment storage facility having a plurality of sediment deposit areas, at least a portion of which is partitioned by partition walls, in which at least a portion of the plurality of sediment deposit areas is included in an image captured by the imaging device, the method comprising: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting deposits from the image captured in the image acquisition step; and a deposit classification step for classifying the deposits extracted in the deposit extraction step into each of the deposit areas. According to the above aspect, by extracting deposits from an image captured at an imaging position where the image includes a plurality of deposit areas and classifying the extracted deposits into each of the deposit areas, it is possible to prevent deposits deposited in different deposit areas from being identified as deposits from the same deposit area, and to accurately identify the deposits.

[0023] According to one aspect of the present invention, the deposit extraction step extracts partition walls from the image acquired in the image acquisition step. According to this aspect, by extracting the partition walls and deposits, even if adjacent deposit areas exist in the deposit storage facility, it is possible to recognize the partition walls separating the adjacent deposit areas, and it is possible to prevent deposits deposited in adjacent deposit areas from being identified as deposits from the same deposit area, thereby enabling accurate identification of the deposits.

[0024] According to one aspect of the present invention, the deposit classification step determines whether at least a portion of a partition wall is included inside the deposit, and divides the deposit if it is determined that at least a portion of the partition wall is included inside the deposit. Even if deposits of multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, it is possible to recognize the partition walls separating the deposit areas by determining whether a portion of the partition wall is included inside the deposit, and to separate the deposit by deposit area.

[0025] According to one aspect of the present invention, the deposit classification step determines whether at least a portion of the deposit is contained on the upper ends of the partition walls, and if it is determined that at least a portion of the deposit is contained on the upper ends of the partition walls, divides the deposit. Even if the deposits of multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, by determining whether at least a portion of the deposit is contained on the upper ends of the partition walls, it is possible to recognize the partition walls separating the deposit areas and separate the deposits for each deposit area.

[0026] According to one aspect of the present invention, the method further comprises a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit areas based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as the basis, differences in height of the deposit among the multiple deposit areas can be utilized, and the deposit can be classified into each deposit area.

[0027] According to one aspect of the present invention, the method further includes a weight estimation step of estimating weights of the sediments classified in the sediment classification step based on the three-dimensional shapes of the sediments. According to the above aspect, by estimating the weights of the sediments based on the three-dimensional shapes of the sediments classified into each sediment area, it is possible to estimate weights according to the type of sediment, and to accurately estimate the weights of the sediments.

[0028] According to one aspect of the present invention, the weight estimation step calculates the weight of the pile based on a variable corresponding to the height of the pile. Although the density of the pile varies depending on the height of the pile because the pile is compressed by its own weight, according to the above aspect, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.

[0029] According to one aspect of the present invention, a method for processing an image captured by an imaging device inside a deposit storage facility having a deposit area where deposits are deposited includes: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting deposits from the image acquired in the image acquisition step; a deposit height information acquisition step for acquiring the height of the deposit extracted in the deposit extraction step; and a weight estimation step for calculating the weight of the deposit based on the height of the deposit acquired in the deposit height information acquisition step and a variable corresponding to the height of the deposit. Different heights of deposits result in different densities because the deposits are compressed by their own weight. However, according to the above aspect, by extracting the deposits from the captured image and acquiring the height of the deposit extracted from the captured image, the weight of the deposit can be calculated based on the variable corresponding to the height, thereby enabling accurate estimation of the weight of the deposit.

[0030] According to one aspect of the present invention, the pile height information acquisition step constructs a three-dimensional shape of the pile, and the weight estimation step divides the three-dimensional shape of the pile into a plurality of vertical sections, calculates the weight of each section based on the height of each section and a variable corresponding to the height of the pile, and estimates the weight of the pile. According to the above aspect, since the three-dimensional shape of the pile is divided into a plurality of vertical sections and the weight of each section is calculated based on the height of each section, the weight of the pile can be estimated more accurately.

[0031] According to one aspect of the present invention, a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being obstructed by obstacles, and the deposit extraction step extracts the deposit from the best shot image captured within the best shot area. According to the above aspect, by setting the best shot area, the deposit is prevented from being in a blind spot of the obstacle, so that the entire deposit can be imaged and the deposit can be accurately identified.

[0032] According to the present invention, it is possible to accurately identify the sediments deposited in each deposition area in a sediment storage facility having a plurality of deposition areas separated by partition walls.

[0033] 1 is a plan view showing the configuration of a scrap yard that is the subject of an image processing method according to one embodiment of the present invention. FIG. 2 is a diagram showing the hardware configuration of an image processing system according to one embodiment of the present invention. FIG. 3 is a diagram showing the software configuration of an image processing system according to one embodiment of the present invention. FIG. 4 is a diagram showing map information within a scrap yard and best shot areas in each area. FIG. 5 is a diagram for explaining the basic principle of a method for estimating the weight of scrap according to one embodiment of the present invention. FIG. 6 is a diagram for explaining the basic principle of a method for estimating the weight of scrap according to one embodiment of the present invention. FIG. 7 is a flowchart showing a method for estimating the weight of scrap according to one embodiment of the present invention. FIG. 8 is a flowchart showing a detailed flow of an approximate formula creation step. FIG. 9 is a graph showing an approximate formula for a density variable. FIG. 10 is a flowchart showing a detailed flow of a three-dimensional shape acquisition step. FIG. 11 is a diagram showing a state in which the constructed three-dimensional shape data includes three-dimensional shapes of multiple scrap piles S. FIG. 12 is a flowchart showing a detailed flow of a weight estimation step. FIG. 13 is a diagram showing how the three-dimensional shape of a scrap pile is divided.

[0034] An image processing method according to one embodiment of the present invention will be described in detail below with reference to the drawings. While the following description will be given using an example in which the weight of scrap piled up in each area of ​​a scrap yard partitioned by partition walls is measured, the present invention is not limited to this and can be applied to any case in which the weight of piled up material is measured in a pile storage facility partitioned by partition walls. The piled up material is not limited to scrap, but can be various other materials such as product waste, waste material from product processing, waste material from building demolition, and powder, and the material is not limited to metal.

[0035] FIG. 1 is a plan view showing the configuration of a scrap yard that is the subject of an image processing method according to one embodiment of the present invention. As shown in FIG. 1, the scrap yard 1 is surrounded by an exterior wall 3, and the interior of the exterior wall 3 is divided into multiple areas 4 (4A-4F) by partition walls 2. The partition walls 2 have a predetermined height and are open at the top. Each partition wall 2 extends linearly in a plan view. Each area 4 is rectangular in a plan view. It is not limited to areas in which all sides of the rectangle are divided by partition walls 2 or exterior walls 3, but may also include areas in which at least one side of the rectangle does not have a partition wall 2 or exterior wall 3. Various types of scrap metal are piled in each area 4, forming a scrap pile S. Alternatively, the area 4 may be a container, and the side of the container corresponds to the partition wall 2. The area 4 may have partition walls 2 that separate areas (e.g., corridors) where scrap metal is not accumulated inside or outside the area 4. The exterior wall 3 may be a wall separating areas other than the scrap yard 1 within the building, and may be open at the top. In this case, the outer wall 3 may be treated in the same manner as the partition wall 2 in the process described below.

[0036] An overhead crane is installed above the scrap yard 1. The overhead crane is equipped with a girder 6, which can move horizontally across the scrap yard 1 in the lateral direction of FIG. 1. The girder 6 is equipped with multiple cameras 8 (two cameras 8A and 8B in this embodiment) as imaging devices and a position information acquisition device 9. In this embodiment, the camera 8 used as the imaging device is a stereo camera, and two imaging units (two imaging units 8A1 and 8A2 of camera 8A and two imaging units 8B1 and 8B2 of camera 8B) composed of imaging elements such as CCD or CMOS capture two-dimensional images. The camera 8 only needs to move along with the movement of the girder 6, and an imaging position adjustment jig may be provided between the girder 6 and the camera 8. The position information acquisition device 9 is a device that acquires the position of the image captured by the camera 8 when it captures an image. In this embodiment, the position of the camera 8 in the vertical direction in FIG. 1 is known, so the horizontal position in FIG. 1 is acquired. However, depending on the installation location and type of the position information acquisition device 9, both the vertical and horizontal positions in FIG. 1 may be acquired. The position information acquisition device 9 may be, for example, a rangefinder that measures the distance to the side wall of the scrap yard 1, a device that detects the position of the girder 6 of the overhead crane, a GPS, or a sensor or camera that detects a self-position detection marker installed in the scrap yard 1. When multiple cameras 8 are used, multiple position information acquisition devices 9 corresponding to each of the multiple cameras 8 may be provided. When using SLAM technology, which creates a map from images captured by an imaging device and estimates its own position, the imaging device serves as the position information acquisition device. Furthermore, the movement speed of the camera 8 can be calculated based on the position and time detected by the position information acquisition device 9.

[0037] In this embodiment, the camera 8 is movable, but the camera 8 may be installed so that its angle can be changed so that its imaging direction can be changed, and the angle of the imaging direction may be detected by the position information acquisition device 9. Furthermore, if the lighting and sunlight within the scrap yard 1 cause areas within the field of view of the image acquired by the camera 8 to have large differences in brightness, a shadow-forming member or lighting member may be provided above the camera 8.

[0038] FIG. 2 is a diagram illustrating the hardware configuration of an image processing system according to an embodiment of the present invention. As shown in FIG. 2, the image processing system 10 includes a computer 12 as an information processing device. The computer 12 includes a CPU 13, a GPU 18, a RAM 14, a ROM 15, and an interface 16. The RAM 14, the ROM 15, and the interface 16 are connected to the CPU 13 and the GPU 18 via a system bus 17. The interface 16 is connected to a camera 8, a location information acquisition device 9, and a display 7. In this embodiment, a stereo camera is used as the camera 8 to construct a three-dimensional shape of the scrap pile, as described below. However, this is not limited to this. Alternatively, multiple general cameras 8 capturing two-dimensional images with a single imaging unit may be used to construct the three-dimensional shape of the scrap pile by utilizing the parallax caused by the stereoscopic vision of the cameras. Alternatively, the three-dimensional shape of the scrap pile may be constructed using a camera 8 capturing two-dimensional images with a single imaging unit and an imaging distance information acquisition device such as a rangefinder or a time-of-flight (TOF) imaging device. Note that, to accurately identify the scrap, the images captured by the camera 8 are preferably color images. In this embodiment, the image processing system 10 is configured by a computer, but is not limited to this, and each function may be distributed using a plurality of terminals.

[0039] 3 is a diagram showing the software configuration of an image processing system according to an embodiment of the present invention. As shown in FIG. 3, the image processing system 10 includes an approximate expression creation unit 120, an image processing unit 140, a weight estimation unit 160, and an area information storage unit 130. These units are implemented by the CPU 13 and GPU 18 executing programs stored in the ROM 15.

[0040] The approximate expression creating unit 120 creates an approximate expression used to estimate the weight of the scrap pile based on the three-dimensional shape of the scrap pile.

[0041] The area information storage unit 130 stores map information for the scrap yard 1, information on each area 4, each partition wall 2, and each camera 8. The map information for the scrap yard 1 includes the position and size (e.g., vertical and horizontal width) of each area 4 and the position and size of each partition wall 2. Information for each area 4 includes the type of scrap to be piled, the type of area 4 (e.g., when floor areas and container areas are mixed), the camera 8 that captures the images, and information on the presence or absence of partition walls 2 that separate the area 4 into areas where scrap is not to be piled. Information on the partition walls 2 includes height information. Information on the camera 8 includes the position of the camera 8 within the scrap yard 1, the camera height (e.g., the vertical distance from the camera 8 to the bottom of the area 4), the movement speed or upper limit of the movement speed of the camera 8, and the like. When multiple cameras 8 are provided, the area 4 to be imaged may also be stored. The bottom of the area 4 refers to the floor of the area 4. If the area 4 is a container, this corresponds to the bottom of the container. Furthermore, if the area 4 is a container, it is preferable to maintain either the vertical distance between the floor of the scrap yard 1 and the bottom of the container, or the vertical distance between the camera 8 and the bottom of the container.

[0042] In this embodiment, multiple cameras 8 are provided, and areas corresponding to the cameras 8 are set. For example, camera 8A is set to capture areas 4A, 4B, 4C, and 4D in the scrap yard 1 shown in FIG. 1 , and camera 8B is set to capture areas 4E and 4F. If the entire area of ​​area 4F falls within the images of both cameras 8A and 8B, it is set to be the capture target of either camera 8A or 8B. If the entire area of ​​area 4F does not fall within the images of either camera 8A or 8B, it is set to be the capture target of both cameras 8A and 8B, and area 4F may be divided to correspond to each of cameras 8A and 8B, and image processing, as described below, may be performed. Alternatively, after the image is acquired, the image may be combined with 3D construction data of the scrap pile, as described below.

[0043] The image processing unit 140 processes the images captured by the camera 8 to obtain the three-dimensional shape of the scrap pile in each area.

[0044] The image processing unit 140 has a captured image acquisition unit 142, an image capture position information acquisition unit 144, a best shot image extraction unit 146, and a deposit identification unit 147. The deposit identification unit 147 has a partition wall / scrap extraction unit 148, a three-dimensional shape construction unit 150, a scrap classification unit 151, and a target scrap identification unit 152.

[0045] The captured image acquisition unit 142 can communicate with the camera 8 and acquires image data captured by the camera 8. The image capture position information acquisition unit 144 can communicate with the position information acquisition device 9 and acquires position information of the camera 8 at the time the camera 8 captured the image. The image data acquired by the captured image acquisition unit 142 is recorded in association with the position information acquired by the image capture position information acquisition unit 144. In this embodiment, the horizontal position in FIG. 1 acquired by the position information acquisition device 9 is recorded in association with the image data. However, the vertical position in FIG. 1, which is the position of the camera 8 on the girder 6, may also be recorded in association with the image data. In addition, corresponding imaging target areas may be set for each of the multiple cameras 8. This makes it possible to discard image data if the position information of the image data acquired by the image capture position information acquisition unit 144 is outside the best shot area of ​​all imaging target areas, thereby reducing the computer load of the best shot image extraction unit 146, which will be described later.

[0046] The best shot image extraction unit 146 stores in advance map information for the scrap yard 1 and the best shot areas for each area 4. FIG. 4 shows the map information for the scrap yard 1 and the best shot areas for each area. As shown in FIG. 4, a best shot area B is set for each area 4. The best shot area B is an area in which the image data captured by the camera 8 does not include any blind spots of the scrap piles in the area 4, and the entire area is included in the image. In other words, the best shot area B is an area in which the scrap piles S in the area 4 are not obstructed by obstacles such as the partition walls 2 when captured by the camera 8. Obstacles include not only the partition walls 2 but also the loading / unloading jigs, scrap transport pipes or conveyors connected to the loading / unloading jigs, loading / unloading jigs attached to the crane and extending downward from the crane (such as lifting magnets or arms or buckets capable of grabbing piles), and piles of piles in other areas that are piled high.

[0047] In addition, the best shot area B is set based on the size of the imaging area determined based on the installation height and field of view of the camera 8 and the size of the area 4, and is set so that the scrap pile S in the area 4 is not obstructed by the partition wall 2 when imaged by the camera 8. Furthermore, in this embodiment, the best shot area B for each area is set based on the position of the camera 8 on the girder 6. For example, in the case of areas 4A and 4E, where the difference between the width of the image field of view in the movement direction of the camera 8 (the horizontal direction in FIG. 1 in this embodiment) and the width of the area in the image is small, the best shot area B is set near the center of the areas 4A and 4E. In the case of areas 4B and 4C, where the difference between the width of the image field of view in the movement direction of the camera 8 and the width of the area in the image is sufficiently large, the best shot area B is set to a predetermined range larger than the areas 4B and 4C. Note that, for an area where the entire area does not fit within the image field of view, for example, in the case of an area larger than the image field of view such as area 4D, multiple best shot areas B may be set for the area, or a best shot area B corresponding to each camera 8 may be set to combine images captured by multiple cameras 8.

[0048] The best shot area B is set so that, when captured by the camera 8, the scrap pile S is not obstructed by obstacles, such as a loading / unloading jig or a scrap transport pipe or conveyor connected to the loading / unloading jig, that exist between the camera 8 and the area 4. Furthermore, if the installation angle of the camera 8 is such that the imaging unit is not parallel to the floor, the best shot area B is set taking into consideration the possibility of blind spots due to piles of material in the surrounding area. For example, the best shot area B is set assuming that the scrap pile S accumulated in the surrounding area of ​​the target area 4 is piled to a predetermined height higher than the height of the partition wall. The best shot area B may be set taking into consideration the lighting and sunlight in the scrap yard 1, or the width of the best shot area B may be set to change over time taking into consideration times when shadows are likely to appear in the scrap yard 1. Furthermore, the best shot area B for an area where the camera 8 does not move directly above at least a portion of the area due to the movable range of the girder 6 may be set using a method different from the setting method for the other best shot areas B.

[0049] 1, the best shot area can be set as a rectangular area with a predetermined length in the vertical direction and a predetermined width in the horizontal direction. Also, if the camera 8 is installed so that the angle can be changed to change the imaging direction, the best shot area can be set as an angle.

[0050] The best shot image extraction unit 146 extracts images captured within the best shot region of each area from the image data acquired by the captured image acquisition unit 142. In this embodiment, since the camera 8 is a stereo camera, both of the two-dimensional images captured by the two imaging units are extracted.

[0051] The partition wall / scrap extraction unit 148 extracts the partition walls 2 and scrap from the image data extracted by the best shot image extraction unit 146. The partition wall / scrap extraction unit 148 is previously trained with training data regarding the metals constituting the scrap and the partition walls 2. This allows the partition walls 2 and scrap to be extracted from the best shot image. By recognizing the scrap and the partition walls, the partition wall / scrap extraction unit 148 can learn about the metals constituting the scrap and the partition walls, thereby reducing the possibility of erroneously recognizing the partition walls as scrap. Note that in this embodiment, because the camera 8 is a stereo camera, the best shot image extraction unit 146 extracts two sets of image data. However, the partition walls 2 and scrap may be extracted from one or both sets of image data, or scrap may be extracted from both sets of image data and the partition walls 2 from one of the sets of image data.

[0052] The three-dimensional shape construction unit 150 constructs three-dimensional shape data of the scrap pile from the image data extracted by the best shot image extraction unit 146. In this embodiment, the camera 8 is a stereo camera, and thus the three-dimensional shape data is constructed from two two-dimensional images. However, if, for example, a camera capturing two-dimensional images using a single imaging unit and an imaging distance information acquisition device are used, the three-dimensional shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. Note that the three-dimensional shape only needs to be the shape of the upper part of the scrap pile captured in the two-dimensional image. If three-dimensional shape data is also to be constructed for the portion below the top of the scrap pile that cannot be captured by the camera 8, the data may be constructed based on vertical distance information from the camera 8 to the bottom of the area 4, which is pre-stored in the area information storage unit 130. The three-dimensional shape data also includes height information of the scrap pile. In this embodiment, the height of the scrap pile is calculated from the vertical distance from camera 8 to the bottom of area 4, which is stored in advance in the area information storage unit 130, and the vertical distance from camera 8 to the scrap pile, and is added to the three-dimensional shape data, but this is not limiting, and scrap pile height information may be obtained by any appropriate method without constructing a three-dimensional shape. Note that, when using a best shot image in which area 4 is a container, the height of the scrap pile should be calculated taking into account the height of the bottom of the container.

[0053] The scrap sorting unit 151 divides the scrap pile into areas 4. In this embodiment, the scrap pile is divided into areas 4 based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148 and the three-dimensional data of the scrap pile S constructed by the three-dimensional shape construction unit 150. When multiple scrap piles are extracted by the partition wall / scrap extraction unit 148, the scrap sorting unit 151 determines whether the multiple scrap piles are scrap piles in the same area or scrap piles in different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. Furthermore, in order to detect whether scrap from adjacent areas extracted by the partition wall / scrap extraction unit 148 is connected, the scrap sorting unit 151 detects locations within the scrap pile S where the heights differ significantly (locations larger than a predetermined size). Furthermore, in order for the partition wall / scrap extraction unit 148 to detect that scrap from different adjacent areas has been extracted and connected across the partition wall 2, the scrap classification unit 151 may determine whether at least a portion of the partition wall is included inside the scrap pile, or may determine whether at least a portion of the scrap is included on the upper end of the partition wall. Furthermore, when the partition wall / scrap extraction unit 148 detects that scrap from different adjacent areas 4 has been extracted and connected across the partition wall 2, the scrap classification unit 151 divides the three-dimensional shape data of the scrap pile S into areas.

[0054] The target scrap identification unit 152 identifies the three-dimensional shape data of the scrap pile S in the target area 4 in the three-dimensional shape data.

[0055] The weight estimation unit 160 estimates the weight of the scrap pile S for each area 4 based on the three-dimensional shape of the scrap pile S for each area 4 acquired by the image processing unit 140 .

[0056] The following describes a method for estimating the weight of scrap by identifying the scrap piled in each area using the image processing method of this embodiment, acquiring its three-dimensional shape, and then estimating the weight of the scrap. Conventional methods for estimating the weight of scrap, etc., based on the three-dimensional shape of the scrap, etc., calculate the volume based on the three-dimensional shape of the scrap, etc., and then multiply this volume by a standard bulk density (assuming a uniform density). However, calculating the weight based on a uniform density for the volume in this manner has the problem of large discrepancies with the actual weight. Figures 5A and 5B are diagrams illustrating the basic principle of a method for estimating the weight of scrap according to one embodiment of the present invention. Note that in Figure 5A, areas with high density are indicated by darker colors. Also, in Figure 5B, areas with high average density when the bottom surface is divided by a predetermined area are indicated by darker colors. As shown in Figure 5A, in areas with a high stack height, the scrap is compressed and deformed by its own weight, resulting in a high density of the scrap pile. Therefore, the higher the stack height, the smaller the gaps between individual scrap pieces, and the larger the discrepancy between the weight calculated by multiplying the volume by the standard bulk gravity and the actual weight. Therefore, in this embodiment, based on the idea that the higher the stack height of the scrap, the higher the average density, the weight is estimated using a variable that increases as the stack height of the scrap increases, as shown in Fig. 5B. Specifically, an approximation formula for the variable that takes into account the density relative to the stack height of the scrap is created, and the weight is estimated based on a reference weight set for each type of scrap and the variable calculated using this approximation formula.

[0057] 6 is a flowchart illustrating a method for estimating the weight of scrap according to one embodiment of the present invention. As shown in FIG. 6 , the method of this embodiment includes an approximate expression creation step (S100) for creating an approximate expression of variables that takes into account the density of the scrap relative to the stack height, a three-dimensional shape acquisition step (S200) for acquiring the three-dimensional shape of the scrap pile S piled in each area 4 of the scrap yard 1, and a weight estimation step (S300) for estimating the weight of the scrap based on the acquired three-dimensional shape of the scrap pile S. Each step will be described in detail below.

[0058] 7 is a flowchart showing a detailed flow of the approximate equation creation step. As shown in FIG. 7, in the approximate equation creation step S100, first, reference weight data is acquired for each type of scrap stored in the scrap yard (reference weight acquisition step: S110). The reference weight data is obtained by loading the target metal to a height of 1 meter in a reference weight measurement container with bottom dimensions of 1 m x 1 m, and measuring the weight. This weight is used as the reference weight per unit volume. This reference weight is input to the approximate equation creation unit 120.

[0059] Next, the scrap is scanned with a 3D scanner to obtain shape information (S120). Specifically, for example, each type of scrap to be piled up in the scrap yard is scanned with a 3D scanner to obtain shape information for each individual scrap. If there are multiple different shapes for each scrap, or if the shapes are irregular enough to be classified as the same type under the condition that they are smaller than a specified size, shape information for any number of scraps is obtained. This shape information is input into the approximation formula creation unit 120.

[0060] Next, strength information of the scrap unit is acquired (S130). For example, the Young's modulus or the like is calculated by compressing the scrap unit. If the metal constituting the scrap unit is known, the physical property value of the metal can be used as the strength information of the scrap unit. This strength information is input to the approximation formula creation unit 120.

[0061] Next, the weight of each piece of scrap is measured to obtain weight information (S140). This weight information is input to the approximate formula creation unit 120. Note that the weight information of each piece of scrap may be calculated from the weight and shape information of the metal per unit volume.

[0062] Next, the approximation formula creation unit 120 simulates the gaps and deformations that will form between the scrap units when multiple scrap units are piled up based on the shape information, strength information, and weight information (S150). Specifically, a virtual container (1 m x 1 m bottom) is simulated, scrap units are randomly placed inside, and the stacking of the scrap units within the container is simulated based on the strength information and weight information of the scrap units, thereby determining the gaps and deformations in the scrap pile. This simulation determines the weight and height of the scrap pile within the container. Note that a virtual container (1 m x 1 m bottom) may be simulated, and the weight of the scrap pile when the height of the scrap pile within the container reaches 1 m may be used as the reference weight.

[0063] The approximate equation creating unit 120 repeats this simulation under various conditions to determine the relationship between the stack height of the scrap pile and the weight per unit area (S160).

[0064] Next, the approximation formula creation unit 120 calculates the relationship between the weight per unit area for each stacking height of the scrap pile obtained through the simulation and the estimated weight per unit area for each stacking height of the scrap pile without considering density changes from the reference weight. The weight obtained through the simulation is then divided by the estimated weight to calculate a density variable. An approximation formula approximating the relationship between stacking height and density variable is then created from the value of the density variable for each stacking height (S170). FIG. 8 is a graph showing the approximation formula for the density variable. As shown in the figure, the density variable increases with increasing height, and the increase in the density variable per unit height decreases with increasing height. Alternatively, the weight per unit area for each stacking height of the scrap pile obtained through the simulation may be used as a variable corresponding to each stacking height of the scrap pile, and an approximation formula approximating the relationship between stacking height and weight per unit area may be created. Even when weight per unit area is used as a variable, the weight increases with increasing height, and the increase in density per unit height decreases with increasing height.

[0065] In the above-described approximate formula creation step S100, the approximate formula is determined by simulation, but this is not limiting. Alternatively, an approximate formula may be created by repeatedly stacking actual scrap units without obtaining shape information or strength information of the scrap units, and determining the relationship between the stack height of the scrap pile and the weight per unit area. Furthermore, the approximate formula created by the above-described simulation may be corrected by comparing it with the results of actual experiments.

[0066] Next, the three-dimensional shape acquisition step S200 will be described in detail. Fig. 9 is a flowchart showing a detailed flow of the three-dimensional shape acquisition step. The area information holding unit 130 has pre-recorded therein map information of the scrap yard, area information, and camera information. In addition, a best shot area is set for each area of ​​the scrap yard, and information regarding the best shot area of ​​each area is pre-recorded in the image processing unit 140.

[0067] In the three-dimensional shape acquisition step, first, a jig for carrying in and out scrap, such as a lifting magnet extending downward from the girder 6 and the camera 8 movably mounted on the girder 6, is moved to the imaging start position (imaging preparation step S205). The imaging start position of the girder 6 may be a predetermined position, or, if at least one of the multiple areas 4 is the target area for three-dimensional shape acquisition, it may be a predetermined position around the target area. The imaging start position of the lifting magnet is outside the imaging area of ​​the camera 8. If the lifting magnet is present within the imaging area, the image will be captured with the lifting magnet present on top of the scrap pile S, and the deposits in each pile area cannot be accurately identified. Note that in the imaging preparation step, the girder 6 and the lifting magnet may be operated automatically or manually by the user. If the user operates them manually in the imaging preparation step, the imaging start position is displayed on the display 7.

[0068] Next, the camera 8 is moved by moving the girder 6, and images of the scrap yard 1 are captured by the camera 8 at predetermined intervals (image capturing step), generating image data. The image processing unit 140 then acquires the generated image data using the captured image acquisition unit 142 (image capturing step: S210). The image processing unit 140 may be configured to select a camera from among the multiple cameras 8 or a target area from among the multiple areas 4 when a user or the image processing unit 140 issues an image capture start command. The camera to capture the image may be determined based on the selected camera or area. In parallel with this, the position information acquisition device 9 generates position information regarding the position at which the camera 8 captured the image at the time the image was captured (position information generation step: S220). The image processing unit 140 acquires the position information generated by the image capturing position information acquisition unit 144. The image processing unit 140 associates the position information with the acquired image data and records it. In this embodiment, the movement speed of the camera 8 is the movement speed of the girder 6, and the upper limit of the movement speed is preferably a speed at which no blurring occurs in the image depending on the imaging interval and the performance of the camera 8. The movement speed of the camera 8 can be calculated from the acquisition time of the position information acquired by the imaging position information acquisition unit 144. The operation of moving the girder 6 in the imaging step may be performed automatically or manually by the user. When the user manually operates the girder 6 in the imaging step, the appropriate movement speed of the girder 6 and the current movement speed of the girder 6 may be displayed on the display 7, or when the movement speed of the girder 6 exceeds the upper limit, a deceleration operation instruction or an imaging error may be displayed on the display 7.

[0069] Next, the image processing unit 140, through the best shot image extraction unit 146, references the recorded image data for each area 4 and acquires image data (hereinafter referred to as best shot images) whose captured positions are included in the best shot areas based on the position information (best shot image extraction step: S230). The number of best shot images acquired for each area 4 is preferably one or more. Multiple best shot images may be acquired based on the position information so that the best shot areas are spaced evenly apart in width, or so that the image capture time intervals are evenly spaced, or the number of images acquired may be set according to the size and number of best shot areas. The best shot image extraction step is not limited to this. It may also be possible to thin out image data captured at predetermined time intervals so that the intervals are longer than the predetermined time interval, determine whether the position information of the thinned-out image data is included in the best shot area of ​​any of the areas 4, and acquire the image as the best shot image if it corresponds to the best shot area of ​​any of the areas 4.

[0070] Next, the image processing unit 140 extracts the partition walls 2 and scrap from the best shot image using the partition wall / scrap extraction unit 148 (scrap extraction step: S240). In the scrap extraction step 240, the partition walls 2 separating the areas 4 and the scrap from the best shot image are extracted using a neural network. The partition wall / scrap extraction unit 148 is previously trained with training data regarding the metals constituting the scrap and the partition walls 2. This allows the partition walls 2 and scrap to be extracted from the best shot image. Note that if the partition wall / scrap extraction unit 148 does not detect any metal components in any of the best shot images of the target area, it determines that the target area is empty of scrap. Furthermore, before, during, or simultaneously with the scrap extraction step, the image processing unit 140 preferably detects scrap transport jigs, such as lifting magnets, in the best shot image. For example, lifting magnets are detected using a neural network. If a lifting magnet is detected within the best shot image (particularly preferably within the target area), the three-dimensional shape acquisition step for the best shot image or the partition wall 2 and scrap extracted from the best shot image is terminated, and an imaging error is displayed on the display 7.

[0071] Next, the image processing unit 140 uses the three-dimensional shape construction unit 150 to construct a three-dimensional shape of the scrap pile in the best shot image (three-dimensional shape construction step: S250). In this embodiment, the camera 8 is a stereo camera, and thus three-dimensional shape data is constructed from two two-dimensional images. For example, if a camera capturing two-dimensional images using a single imaging unit and an imaging distance information acquisition device are used, the three-dimensional shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. In this case, the three-dimensional shape data includes height information of the scrap pile. The height of the scrap pile is calculated from the pre-stored vertical distance from the camera 8 to the bottom of the area 4 and the vertical distance from the camera 8 to the scrap pile. The three-dimensional shape construction step S250 corresponds to the scrap height information acquisition step.

[0072] Next, the image processing unit 140 causes the scrap classification unit 151 to classify the scrap piles into areas (scrap classification step: S260). Specifically, as shown in Fig. 10, when the constructed three-dimensional shape data includes three-dimensional shapes of multiple scrap piles S, it is determined whether the multiple scrap piles S are in the same area or different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. For example, when a partition wall 2 is extracted between multiple scrap piles S, it can be determined that the multiple scrap piles S are in different areas. The method of determining whether multiple scrap piles S are in the same area or different areas based on the partition walls 2 is not limited to this. For example, in the case of an area 4 whose periphery is entirely surrounded by partition walls 2, the upper ends of the partition walls 2 may be extracted as straight line components, a polygon surrounded by the straight line components may be detected, and the scrap within the polygon may be determined to be scrap piles S in the same area. Furthermore, in the case of adjacent areas 4 whose peripheries are entirely surrounded by partition walls 2, scrap in an area where a polygon cannot be formed by the straight line components of the upper ends of the partition walls (one side of the polygon cannot be detected) may be determined to be scrap outside the target area in the target scrap identification step S270 described below. In this embodiment, whether multiple scrap piles S are in the same area or different areas is determined based on the partition walls 2, but this is not limited thereto. The determination may also be based on the three-dimensional shape of the scrap piles S, in which case it may be based on any of the height, shape, or distance between the scrap piles S of the three-dimensional shape.

[0073] Furthermore, the scrap classification unit 151 detects areas in the scrap pile S where the height varies significantly (areas larger than a predetermined size) based on the height of the three-dimensional shape of the scrap pile S. When such areas where the height of the scrap pile varies significantly are detected, there is a high possibility that the scrap extracted by the partition wall / scrap extraction unit 148 is a combination of scrap from different adjacent areas. Therefore, when the scrap classification unit 151 detects areas where the height varies significantly, it divides the scrap pile S using the areas where the height varies significantly as boundaries.

[0074] The scrap classification unit 151 also determines whether at least some of the partition walls 2 are included inside the scrap pile S. If at least some of the partition walls are included inside such a scrap pile S, it is highly likely that the scrap extracted by the partition wall / scrap extraction unit 148 is scrap from different adjacent areas that are connected across the partition walls. Therefore, when the scrap classification unit 151 determines that at least some of the partition walls 2 are included inside the scrap pile S, it creates a boundary line that continues the partition walls 2 and divides the scrap pile S. Note that the upper ends of the partition walls 2 may be recognized, straight line components may be extracted from the upper ends of the partition walls 2, and the boundary line may be created by continuing the straight line components, or the junctions between the partition walls 2 at the upper ends of the partition walls 2 may be recognized as corners of the partition walls 2, and the boundary line may be created by connecting the corners with straight lines. Furthermore, if the area information storage unit 130 records that the target area of ​​the best shot image has a partition wall 2 that separates an area (e.g., an aisle) where scrap metal does not accumulate, the scrap pile S will not be divided even if a portion of the partition wall 2 is recognized inside the scrap pile S. The scrap sorting unit 151 is not limited to determining whether at least a portion of the partition wall 2 is included inside the scrap pile S, but may instead use a method of determining whether at least a portion of the scrap is included on the upper edge of the partition wall 2. Through the above process, a three-dimensional shape of the scrap piled in each area of ​​the scrap yard can be obtained. The scrap sorting step S260 is preferably performed after the three-dimensional shape construction step S250, but at least a portion of the scrap sorting step S260 may be performed before the three-dimensional shape construction step S250.

[0075] Next, the image processing unit 140 uses the target scrap identification unit 152 to identify only the scrap in the target area (target scrap identification step: S270). Specifically, as shown in FIG. 10 , if the constructed 3D shape data includes the 3D shapes of multiple scrap piles S, the positions of the target area and / or the scrap are estimated based on the location information recorded in association with the image data acquired by the captured image acquisition unit 142, and only the 3D shapes of the scrap in the target area are extracted from the constructed 3D shape data of the scrap pile S. Note that the target scrap identification step S270 is not limited to this. If the location of the target area in the captured image is recorded in advance, only the 3D shapes of the scrap in the target area may be extracted from the constructed 3D shape data of the scrap pile S without using the location information recorded in association with the image data. Note that the scrap classification step S260 and the target scrap identification step S270 may operate simultaneously, or the functions of the scrap classification unit 151 and the target scrap identification unit 152 may be integrated.

[0076] Next, a description will be given of the weight estimation step S300 in which the weight estimation unit 160 estimates the weight of the scrap based on the three-dimensional shape of the scrap acquired by the image processing unit 140. Fig. 11 is a flowchart showing the detailed flow of the weight estimation step.

[0077] First, the weight estimation unit 160 divides the three-dimensional shape of the scrap pile within the target area 4. Fig. 12 is a diagram showing how the three-dimensional shape of the scrap pile is divided. As shown in the figure, the scrap pile is divided vertically into, for example, a 15 cm x 15 cm grid (division step: S310).

[0078] Next, the weight estimation unit 160 estimates the height of each point (each grid) of the three-dimensional shape of the scrap from the bottom of the area 4. For example, the average value of the height coordinates within each point (each grid) may be used as the height of each point. Depending on the size of the grid, the height coordinate at the center of the grid may also be used. (Height estimation step: S320)

[0079] Next, the weight estimation unit 160 estimates the weight of each point using an approximation equation for a variable that takes into account the density relative to the stack height (weight calculation step: S330). Specifically, when a density variable is used as the variable, the weight of each point is calculated using the following equation: Weight = Base Area × Height × Reference Weight × Density Variable Corresponding to Height In the above equation, the base area is the area of ​​the grid divided in the division step. The height is the height estimated in the height estimation step. The reference weight obtained in the reference weight acquisition step S110 can be used as the reference weight. Furthermore, when weight per unit area is used as the variable, the weight of each point is calculated using the following equation: Weight = Base Area × Weight Per Unit Area Corresponding to Height In the above equation, the base area is the area of ​​the grid divided in the division step. In either case, the variable corresponding to the height can be calculated by substituting the height into the approximation equation created in the approximation equation creation step S100.

[0080] Next, the weight estimation unit 160 estimates the weight of the scrap in area 4 (total weight calculation step: S340). Specifically, the weight of each grid is added up to estimate the weight of the scrap pile. In this way, the weight estimation unit 160 estimates the weight of the scrap pile in each area 4 and outputs the result to the display 7 or the like.

[0081] Note that the weight estimation step S300 is not limited to the above method. For example, instead of performing the division step S310, the areas of points of the three-dimensional shape of the scrap pile that have the same height may be summed up, and the weight calculation step S330 may be performed for each height. In this case, the weight for each height is calculated using the following formula: Weight = Total base area × Height × Reference weight × Density variable corresponding to height Next, in the total weight calculation step, the weight for each height is summed up to estimate the weight of the scrap pile.

[0082] According to this embodiment, the following effects are achieved: The image processing method of this embodiment includes a scrap extraction step S240 for extracting scrap from a best-shot image captured within a best-shot area. This prevents the scrap pile S from being in the blind spot of an obstacle such as the partition wall 2, making it possible to capture an image of the entire scrap pile S and accurately identify the pile of scrap.

[0083] Furthermore, in the image processing method of this embodiment, the best shot area B is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the camera 8, and the size of area 4. In this way, the best shot area is set based on the size of the imaging area and the size of area 4. Therefore, the scrap pile S can be imaged from the imaging position of the camera 8 so that it does not fall into the blind spot of obstacles such as the partition wall 2, and the pile can be accurately identified.

[0084] The image processing method of this embodiment also includes an imaging step of capturing image data within the scrap yard 1 while moving the camera 8, a position information generation step S220 of generating the position where the image was captured by the camera 8, and a best shot image extraction step S230 of extracting best shot images captured within the best shot area based on the position information. When capturing images while moving the camera 8, the amount of captured image data becomes enormous, and processing all of the image data places a heavy load on the computer. In contrast, according to this embodiment, image data captured within the best shot area is extracted from the captured image data, thereby reducing the load on the computer. Furthermore, each area 4 within the scrap yard 1 can be captured even with a small number of cameras.

[0085] The image processing method of this embodiment also includes a target scrap identification step S270 for identifying only the scrap pile S in the target area 4 from the scrap piles S extracted in the scrap extraction step S240, and the target scrap identification step S270 extracts only the scrap pile S in the target area 4 within the best shot image based on the position information. This makes it possible to identify only the scrap pile S in the target area 4 based on the position information, and to extract the scrap pile S separately from the scrap piles S in other areas 4.

[0086] The image processing method of this embodiment also includes a scrap classification step S260 in which the scrap piles S extracted in the scrap extraction step S240 are classified into areas 4. As a result, even if an image is captured at an imaging position where multiple areas 4 are included in the image, by classifying the scrap piles S into areas 4, it is possible to prevent scrap piles S piled in different areas 4 from being identified as scrap piles S from the same area 4, and it is possible to accurately identify the piles.

[0087] Furthermore, in the image processing method of this embodiment, in the scrap extraction step S240, the partition walls 2 are extracted from the captured best shot image. As a result, even if adjacent areas 4 exist within the scrap yard 1, the partition walls 2 separating the adjacent areas 4 can be recognized by extracting the partition walls 2 and the scrap. This makes it possible to prevent scrap piles S piled up in adjacent areas 4 from being identified as scrap piles S from the same area 4, and enables accurate identification of piles.

[0088] Furthermore, in the image processing method of this embodiment, in the scrap classification step S260, it is determined whether at least some of the partition walls 2 are included inside the scrap pile S, and if it is determined that at least some of the partition walls 2 are included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted as a single unit in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether some of the partition walls 2 are included inside the scrap pile S, and the scrap pile S can be divided into individual areas 4.

[0089] Furthermore, in the image processing method of this embodiment, in the scrap classification step S260, it is determined whether at least some of the scrap is included above the upper ends of the partition walls 2, and if it is determined that at least some of the scrap is included above the upper ends of the partition walls 2, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted together in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether at least some of the scrap is included above the upper ends of the partition walls 2, and it is possible to separate the scrap piles S for each area 4.

[0090] The image processing method of this embodiment also includes a three-dimensional shape construction step S250 in which a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240 is constructed, and in the scrap classification step S260, the scrap pile S is classified into areas 4 based on the height of the three-dimensional shape of the scrap pile S. In this way, by using the height of the three-dimensional shape of the scrap pile S as the basis, the difference in height between the scrap piles S in the multiple areas 4 can be utilized, and the scrap pile S can be classified into areas 4.

[0091] The image processing method of this embodiment further includes a three-dimensional shape construction step S250 for constructing a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. In this way, the three-dimensional shape of the scrap pile S can be constructed, and the weight of the scrap pile S can be estimated.

[0092] Furthermore, in the weight estimation step S300 of the image processing method of this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S varies, the scrap pile S will be compressed by its own weight, resulting in different densities. However, according to this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so the weight of the scrap pile S can be estimated more accurately.

[0093] Furthermore, according to this embodiment, the variable corresponding to height is larger as the height increases. As the height of the scrap pile S increases, the density increases because the scrap pile S is compressed by its own weight. However, according to this embodiment, since a variable that increases as the height increases is used, the weight of the scrap pile S can be estimated more accurately.

[0094] Furthermore, according to this embodiment, the variable corresponding to height has a smaller increase in weight per unit height as the height increases. As the height of the scrap pile S increases, the pile is compressed by its own weight, resulting in an increase in density, but this increase becomes smaller as the height increases. According to this embodiment, the variable that has a smaller increase in weight per unit height is used as the height increases, so the weight of the scrap pile S can be estimated more accurately.

[0095] Furthermore, in the weight estimation step S300 of the image processing method of this embodiment, the three-dimensional shape of the scrap pile S is divided into multiple sections in the vertical direction, and the weight of each section is calculated based on the height of each divided section, thereby estimating the weight of the scrap pile S. In this way, because the three-dimensional shape of the scrap pile S is divided into multiple sections in the vertical direction, and the weight of each section is calculated based on the height of each divided section, the weight of the scrap pile S can be estimated more accurately.

[0096] Furthermore, in this embodiment, an image processing method for a scrap yard 1 having a plurality of areas 4, at least a portion of which is partitioned by partition walls 2 and in which scrap piles S are piled, in which an image captured by a camera 8 includes at least a portion of the plurality of areas 4, includes an image acquisition step S210 for acquiring an image captured by the camera 8, a scrap extraction step S240 for extracting scrap piles S from the image acquired in the image acquisition step S210, and a scrap classification step S260 for separating the scrap piles S extracted in the scrap extraction step S240 into each of the areas 4. In this way, by extracting scrap piles S from an image captured at an imaging position in which a plurality of areas 4 are included in the image and separating the extracted scrap piles S into each of the areas 4, it is possible to prevent scrap piles S piled in different areas 4 from being identified as scrap piles S from the same area 4, and to accurately identify the piles.

[0097] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes a scrap extraction step S240 in which the partition walls 2 are extracted from the captured best shot image. As a result, even if adjacent areas 4 exist within the scrap yard 1, the partition walls 2 separating the adjacent areas 4 can be recognized by extracting the partition walls 2 and the scrap. This makes it possible to prevent scrap piles S piled in adjacent areas 4 from being identified as scrap piles S from the same area 4, thereby enabling accurate identification of piles.

[0098] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes the following steps: in a scrap sorting step S260, it is determined whether at least a portion of the partition wall 2 is included inside the scrap pile S; and if it is determined that at least a portion of the partition wall 2 is included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of a plurality of areas 4 are extracted as a single unit in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether a portion of the partition wall 2 is included inside the scrap pile S, and the scrap pile S can be divided into each area 4.

[0099] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes the following steps: in a scrap division step S260, it is determined whether at least a portion of the scrap is included on the upper ends of the partition walls 2; and if it is determined that at least a portion of the scrap is included on the upper ends of the partition walls 2, the scrap piles S are divided. As a result, even if the scrap piles S of the plurality of areas 4 are extracted together in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether at least a portion of the scrap is included on the upper ends of the partition walls 2, and it is possible to separate the scrap piles S into individual areas 4.

[0100] Furthermore, in this embodiment, the image processing method for a scrap yard 1 having a plurality of areas 4 in which scrap piles S are piled, at least a portion of which is partitioned by partition walls 2, and in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8, further includes a three-dimensional shape construction step S250 for constructing a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240, and in a scrap sorting step S260, the scrap pile S is divided into areas 4 based on the height of the three-dimensional shape of the scrap pile S. In this way, by using the height of the three-dimensional shape of the scrap pile S as the basis, the difference in height between the scrap piles S in the plurality of areas 4 can be utilized, and the scrap pile S can be divided into areas 4.

[0101] In this embodiment, the image processing method for a scrap yard 1 having a plurality of areas 4, at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, in which at least some of the plurality of areas 4 are included in an image captured by a camera 8, further includes a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. By estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S divided into areas 4, it is possible to estimate the weight according to the type of scrap, and to accurately estimate the weight of the pile.

[0102] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least a portion of which is partitioned by partition walls 2 and in which scrap piles S are piled, an image captured by a camera 8 includes at least a portion of the plurality of areas 4. In the weight estimation step S300, the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S differs, the scrap pile S is compressed by its own weight, resulting in different densities. However, according to this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so the weight of the scrap pile S can be estimated more accurately.

[0103] In this embodiment, the image processing method for capturing an image of a scrap yard 1 having an area 4 where scrap piles S are piled up by a camera 8 includes an image capturing step S210 for capturing an image captured by the camera 8, a scrap extraction step S240 for extracting the scrap pile S from the image captured in the image capturing step S210, a scrap height information capturing step for capturing the height of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for calculating the weight of the scrap pile S based on the height of the scrap pile S captured in the scrap height information capturing step and a variable corresponding to the height of the scrap pile S. Different heights of the scrap pile S result in different densities because the scrap pile S is compressed by its own weight. However, according to this embodiment, by extracting the scrap pile S from the captured image and acquiring the height of the scrap pile S extracted from the captured image, the weight of the scrap pile S can be calculated based on the variable corresponding to the height of the scrap pile S, thereby accurately estimating the weight of the pile.

[0104] In this embodiment, the image processing method for capturing images of the inside of the scrap yard 1, which has an area 4 where the scrap pile S is piled up, using the camera 8, constructs a three-dimensional shape of the scrap pile S in the scrap height information acquisition step, and estimates the weight of the scrap pile S by dividing the three-dimensional shape of the scrap pile S into multiple vertical sections in the weight estimation step S300, and calculating the weight of each section based on the height of each divided section and a variable corresponding to the height of the scrap pile S. In this way, because the three-dimensional shape of the scrap pile S is divided into multiple vertical sections and the weight of each section is calculated based on the height of each divided section and a variable corresponding to the height of the scrap pile S, the weight of the scrap pile S can be estimated more accurately.

[0105] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least partially partitioned by partition walls 2 and where scrap piles S are piled, an image captured by a camera 8 includes at least a portion of the plurality of areas 4, or an image processing method of an image captured by a camera 8 of the scrap yard 1 having the area 4 where scrap piles S are piled, a predetermined best shot area is set for the area 4 so that the camera 8 can capture an image of the area 4 without obstructing the area 4, and in the scrap extraction step S240, the scrap pile S is extracted from the best shot image captured within the best shot area. By setting the best shot area in this way, the scrap pile S is not in the blind spot of obstacles such as the partition walls 2, so that the entire scrap pile S can be captured and the pile can be accurately identified.

[0106] 1: Scrap yard 2: Partition wall 3: Outer wall 4: Area 5: 3D shape of constructed scrap pile 6: Girder 7: Display 8: Camera 9: Position information acquisition device 10: Image processing system 12: Computer 13: CPU 14: RAM 15: ROM 16: Interface 17: System bus 18: GPU 100: Image processing system 120: Approximation formula creation unit 130: Area information storage unit 140: Image processing unit 142: Captured image acquisition unit 144: Captured position information acquisition unit 146: Best shot image extraction unit 147: Deposit identification unit 148: Partition wall / scrap extraction unit 150: 3D shape construction unit 151: Scrap classification unit 152: Target scrap identification unit 160 : Weight estimation area B: Best shot area S: Scrap pile

Claims

1. A method for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, the image processing method comprising: a predetermined best shot area set for each of the deposition areas, which can be imaged by the imaging device without the deposition area being obstructed by any obstacles; and a sediment extraction step for extracting the sediments from the best shot image captured within the best shot area.

2. The image processing method according to claim 1, wherein the best shot area is set based on the size of the imaging area determined based on the imaging height and viewing angle of the imaging device and the size of the accumulation area.

3. The image processing method according to claim 2, further comprising: an imaging step of capturing image data within the deposit storage facility while moving or changing the angle of the imaging device; a position information generation step of generating position information relating to the position or angle at which the image is captured by the imaging device; and a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information.

4. The image processing method of claim 3, further comprising a target deposit identification step of identifying only the deposits in a target deposit area from the deposits extracted in the deposit extraction step, wherein the target deposit identification step extracts only the deposits in a target deposit area within the best shot image based on the position information.

5. The image processing method according to claim 1, further comprising a deposit classification step of classifying the deposits extracted in the deposit extraction step into the deposit regions.

6. The image processing method according to claim 5, wherein the deposit extraction step extracts the partition wall in the captured best shot image.

7. The image processing method according to claim 6, wherein the deposit classification step determines whether at least a portion of the partition wall is included inside the deposit, and if it is determined that at least a portion of the partition wall is included inside the deposit, the deposit is divided.

8. The image processing method according to claim 6, wherein the deposit classification step determines whether at least a portion of the deposit is contained on the upper end of the partition wall, and if it is determined that at least a portion of the deposit is contained on the upper end of the partition wall, divides the deposit.

9. The image processing method according to claim 5, further comprising a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, wherein the deposit classification step divides the deposit into the deposit areas based on the height of the three-dimensional shape.

10. An image processing method according to any one of claims 1 to 8, further comprising: a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step; and a weight estimation step for estimating a weight of the deposit based on the three-dimensional shape.

11. The image processing method according to claim 10, wherein the weight estimation step calculates the weight of the pile based on a variable corresponding to the height of the pile.

12. The image processing method according to claim 11, wherein the variable according to the height is larger as the height is higher.

13. The image processing method according to claim 11, wherein the height-dependent variable has a smaller increase per unit height as the height increases.

14. The image processing method according to claim 11, wherein in the weight estimation step, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby estimating the weight of the deposit.

15. A system for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, wherein a predetermined best shot area is set for each of the deposition areas so that the imaging device can capture an image of the deposition area without obstructing the area, and the image processing system is equipped with a sediment extraction unit that identifies the sediments in the best shot image captured within the best shot area and extracts the sediments.

16. A program for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, the program causing a computer to execute the following steps: a predetermined best shot area is set for each of the deposition areas, which can be imaged by the imaging device without the deposition area being blocked by any obstacles; and a deposit extraction step of identifying the deposits in the best shot image captured within the best shot area and extracting the deposits.

17. A method for processing an image captured by an imaging device in a sediment storage facility having a plurality of deposition areas where sediments are deposited, at least a portion of which is partitioned by partition walls, and in which at least a portion of the plurality of deposition areas is included in the image, the image processing method comprising: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting the deposits in the image acquired in the image acquisition step; and a deposit classification step for dividing the deposits extracted in the deposit extraction step into each of the deposition areas.

18. The image processing method according to claim 17, wherein the deposit extraction step extracts the partition wall from the image acquired in the image acquisition step.

19. The image processing method of claim 18, wherein the deposit classification step determines whether at least a portion of the partition wall is included inside the deposit, and if it is determined that at least a portion of the partition wall is included inside the deposit, the deposit is divided.

20. The image processing method of claim 18, wherein the deposit classification step determines whether at least a portion of the deposit is contained on the upper end of the partition wall, and if it is determined that at least a portion of the deposit is contained on the upper end of the partition wall, divides the deposit.

21. An image processing method as described in claim 17, 19 or 20, further comprising a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, wherein the deposit classification step divides the deposit into the deposit areas based on the height of the three-dimensional shape.

22. The image processing method according to claim 21, further comprising a weight estimation step of estimating weights of the piles separated in the pile separation step based on the three-dimensional shapes of the piles.

23. The image processing method according to claim 22, wherein the weight estimation step calculates the weight of the pile based on a variable corresponding to the height of the pile.

24. A method for processing images captured by an imaging device inside a sediment storage facility having a deposition area where sediments are deposited, comprising: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting the sediment from the image acquired in the image acquisition step; a deposit height information acquisition step for acquiring the height of the sediment extracted in the deposit extraction step; and a weight estimation step for calculating the weight of the sediment based on the height of the sediment acquired in the deposit height information acquisition step and variables corresponding to the height of the sediment.

25. The image processing method of claim 24, wherein the deposit height information acquisition step constructs a three-dimensional shape of the deposit, and the weight estimation step divides the three-dimensional shape of the deposit into multiple parts vertically, calculates the weight of each part based on the height of each divided part and a variable corresponding to the height of the deposit, and estimates the weight of the deposit.

26. An image processing method as described in claim 17 or 24, wherein a predetermined best shot area is set for the accumulation area where the imaging device can capture an image of the accumulation area without obstructing the accumulation area, and the accumulation extraction step extracts the accumulation from a best shot image captured within the best shot area.

Images