Image processing method, image processing system and program

Abstract

To accurately identify sediments deposited in each deposition area in a sediment storage facility with multiple deposition areas.SOLUTION: A method for processing images, captured by an imaging device, inside a sediment storage facility, of which at least a portion is partitioned by partition walls 2 and which has multiple areas 4 in which sediments are deposited, includes a predetermined best shot area B set for each area 4 so that the area 4 can be captured by the imaging device without being blocked by any obstacles, and includes a sediment extraction step of extracting sediments from the best shot image captured within the best shot area B.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing system, and a program. [Background technology]

[0002] Scrap is piled up and stored in a storage facility (scrap yard) for scrap metals, etc. As a device for estimating the weight of the scrap piled up in such a storage facility, for example, Patent Document 1 discloses a method for measuring the height of the piles of scrap in each area stored in the scrap yard using the parallax caused by stereo vision from a pair of two cameras, and estimating the remaining amount of scrap based on the pile heights. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-197170 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, scrap storage facilities may be divided into multiple areas by partition walls. In scrap yards where obstacles such as partition walls exist, scrap may be in the blind spot of the obstacle depending on the camera's imaging position. Furthermore, in scrap yards where multiple sections are adjacent to each other and separated by partition walls, adjacent scrap may be identified as one piece depending on the camera's imaging position and the height of the scrap stack relative to the height of the partition wall. If the remaining amount of scrap is estimated based on such images, the scrap cannot be accurately identified, resulting in a problem of an estimated amount being less than the actual remaining amount.

[0005] The present invention has been developed in consideration of the above-mentioned problems, and its purpose is to enable accurate identification of sediments deposited in each deposition area in a sediment storage facility having multiple deposition areas. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a method for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, the image processing method including a predetermined best shot area set for each deposition area where the imaging device can capture an image of the deposition area without the deposition area being obstructed by any obstacles, and a sediment extraction step for extracting sediments from the best shot image captured within the best shot area. According to the above aspect, by setting the best shot area, the deposit will not be in the blind spot of the obstacle, so that the entire deposit can be captured and the deposit can be accurately identified.

[0007] According to one aspect of the present invention, the best shot area is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the imaging device, and the size of the deposition area. According to the above aspect, the best shot area is set based on the size of the imaging area and the size of the accumulation area, so that the image can be captured so that the accumulation does not enter the blind spot of an obstacle from the imaging position, and the accumulation can be accurately identified.

[0008] According to one aspect of the present invention, the method further includes an imaging step of capturing image data within the deposit storage facility while moving or changing the angle of the imaging device, a position information generation step of generating position information regarding the position or angle at which the image was captured by the imaging device, and a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information. When capturing images while moving the imaging device, the amount of captured image data becomes enormous, and processing all of the image data places a heavy load on the computer. In contrast, according to the above-described embodiment, image data captured within the best shot area is extracted from the captured image data, thereby reducing the load on the computer. Furthermore, each deposition area within the deposition repository can be captured even with a small number of cameras.

[0009] According to one aspect of the present invention, the method further includes a target deposit identification step for identifying only deposits in a target deposit area from the deposits extracted in the deposit extraction step, and the target deposit identification step extracts only deposits in the target deposit area within the best shot image based on position information. According to the above aspect, it is possible to identify only the deposits in the target deposit area based on the position information, and to extract the deposits separately from the deposits in other deposit areas.

[0010] According to one aspect of the present invention, the method further comprises a step of separating the sediments extracted in the step of extracting the sediments into sediment regions. According to the above aspect, by separating the deposits by deposition area, even when an image is captured at an imaging position where multiple deposition areas are included in the image, it is possible to prevent deposits deposited in different deposition areas from being identified as deposits from the same deposition area, and the deposits can be accurately identified.

[0011] According to one aspect of the present invention, the deposit extraction step extracts partition walls in the captured best shot image. According to the above aspect, by extracting the partition walls and sediments, even if there are adjacent deposition areas within the sediment storage facility, the partition walls separating the adjacent deposition areas can be recognized, and it is possible to prevent the sediments deposited in adjacent deposition areas from being identified as deposits from the same deposition area, thereby enabling the sediments to be accurately identified.

[0012] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the partition wall is contained inside the pile, and divides the pile if it is determined that at least a portion of the partition wall is contained inside the pile. Even if deposits from multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, the partition walls separating the deposit areas can be recognized by determining whether part of the partition wall is included inside the deposit, and the deposits can be separated by deposit area.

[0013] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the pile is contained on the upper end of the partition wall, and divides the pile if it is determined that at least a portion of the pile is contained on the upper end of the partition wall. Even if deposits from multiple deposit areas are extracted together in the deposit extraction step, according to the above aspect, the partition walls separating the deposit areas can be recognized by determining whether at least some of the deposits are contained on the upper ends of the partition walls, and the deposits can be separated by deposit area.

[0014] According to one aspect of the present invention, the method further comprises a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit areas based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as a basis, it is possible to utilize the difference in height of the deposit in a plurality of deposit areas, and to separate the deposit for each deposit area.

[0015] According to one aspect of the present invention, the method further includes a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and a weight estimation step for estimating the weight of the deposit based on the three-dimensional shape. According to the above aspect, it is possible to construct a three-dimensional shape of the pile and estimate the weight of the pile.

[0016] According to one aspect of the present invention, the weight estimation step calculates the weight of the pile based on a variable that depends on the height of the pile. When the height of a pile varies, the density of the pile varies because the pile is compressed by its own weight. However, according to the above-described embodiment, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.

[0017] According to one aspect of the present invention, the height-dependent variable is greater the higher the height. As the height of the pile increases, the density of the pile increases because it is compressed by its own weight. However, according to the above embodiment, the weight of the pile can be estimated more accurately because the variable used is larger as the height increases.

[0018] According to one aspect of the present invention, the height-dependent variable is such that the higher the height, the smaller the increase per unit height. As the height of the pile increases, the pile is compressed by its own weight, resulting in an increase in density, but the increase in density becomes smaller as the height increases. According to the above-described embodiment, the weight of the pile can be estimated more accurately because a variable with a smaller increase in weight per height is used as the height increases.

[0019] According to one aspect of the present invention, in the weight estimation step, the three-dimensional shape of the deposit is divided into multiple parts vertically, and the weight of each part is calculated based on the height of each divided part, thereby estimating the weight of the deposit. According to the above embodiment, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby making it possible to estimate the weight of the deposit more accurately.

[0020] According to one aspect of the present invention, there is provided an image processing system for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, wherein a predetermined best shot area is set for each deposition area so that the imaging device can capture an image of the deposition area without obstructing it, and the image processing system is equipped with a sediment extraction unit that identifies the sediments in the best shot image captured within the best shot area and extracts the sediments.

[0021] According to one aspect of the present invention, there is provided a program for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and which has multiple deposition areas where sediments are deposited, wherein a predetermined best shot area is set for each deposition area so that the imaging device can capture an image of the deposition area without blocking the deposition area by any obstacles, and the program causes a computer to execute a sediment extraction step of identifying the sediment in the best shot image captured within the best shot area and extracting the sediment.

[0022] According to one aspect of the present invention, there is provided a method for processing images captured by an imaging device in a sediment storage facility having multiple deposition areas, at least some of which are partitioned by partition walls and in which sediments are deposited, in which at least some of the multiple deposition areas are included in an image captured by the imaging device, the method comprising: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting deposits in the image acquired in the image acquisition step; and a deposit classification step for separating the deposits extracted in the deposit extraction step by deposition area. According to the above aspect, by extracting deposits from an image captured at an imaging position where multiple deposit areas are included in the image and separating the extracted deposits by deposit area, it is possible to prevent deposits deposited in different deposit areas from being identified as deposits from the same deposit area, and it is possible to accurately identify the deposits.

[0023] According to one aspect of the present invention, the deposit extraction step extracts partition walls from the images acquired in the image acquisition step. According to the above aspect, by extracting the partition walls and sediments, even if there are adjacent deposition areas within the sediment storage facility, the partition walls separating the adjacent deposition areas can be recognized, and it is possible to prevent the sediments deposited in adjacent deposition areas from being identified as deposits from the same deposition area, thereby enabling the sediments to be accurately identified.

[0024] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the partition wall is contained inside the pile, and divides the pile if it is determined that at least a portion of the partition wall is contained inside the pile. Even if deposits from multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above aspect, the partition walls separating the deposit areas can be recognized by determining whether part of the partition wall is included inside the deposit, and the deposits can be separated by deposit area.

[0025] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the pile is contained on the upper end of the partition wall, and divides the pile if it is determined that at least a portion of the pile is contained on the upper end of the partition wall. Even if deposits from multiple deposit areas are extracted together in the deposit extraction step, according to the above aspect, the partition walls separating the deposit areas can be recognized by determining whether at least some of the deposits are contained on the upper ends of the partition walls, and the deposits can be separated by deposit area.

[0026] According to one aspect of the present invention, the method further comprises a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit areas based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as a basis, it is possible to utilize the difference in height of the deposit in a plurality of deposit areas, and to separate the deposit for each deposit area.

[0027] According to one aspect of the present invention, the method further comprises a weight estimation step of estimating the weight of the sediment separated in the sediment separation step based on the three-dimensional shape of the sediment. According to the above aspect, by estimating the weight of the deposit based on the three-dimensional shape of the deposit divided into each deposition area, it is possible to estimate the weight according to the type of deposit, and to accurately estimate the weight of the deposit.

[0028] According to one aspect of the present invention, the weight estimation step calculates the weight of the pile based on a variable that depends on the height of the pile. When the height of a pile varies, the density of the pile varies because the pile is compressed by its own weight. However, according to the above-described embodiment, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.

[0029] According to one aspect of the present invention, a method for processing images captured by an imaging device inside a sediment storage facility having a deposition area where sediments are deposited comprises an image acquisition step for acquiring an image captured by the imaging device, a sediment extraction step for extracting sediments from the image acquired in the image acquisition step, a sediment height information acquisition step for acquiring the height of the sediment extracted in the sediment extraction step, and a weight estimation step for calculating the weight of the sediment based on the height of the sediment acquired in the sediment height information acquisition step and variables corresponding to the height of the sediment. When the height of a deposit differs, the deposit is compressed by its own weight, resulting in different densities. However, according to the above-described embodiment, by extracting the deposit from the captured image and obtaining the height of the deposit extracted from the captured image, the weight of the deposit can be calculated based on variables corresponding to the height, and the weight of the deposit can be accurately estimated.

[0030] According to one aspect of the present invention, the deposit height information acquisition step constructs a three-dimensional shape of the deposit, and the weight estimation step divides the three-dimensional shape of the deposit into multiple parts vertically, calculates the weight of each part based on the height of each divided part and variables corresponding to the height of the deposit, and estimates the weight of the deposit. According to the above embodiment, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby making it possible to estimate the weight of the deposit more accurately.

[0031] According to one aspect of the present invention, a predetermined best shot area is set for the accumulation area, where the imaging device can capture an image of the accumulation area without obstructing the accumulation area, and the accumulation extraction step extracts the accumulation in the best shot image captured within the best shot area. According to the above aspect, by setting the best shot area, the deposit will not be in the blind spot of the obstacle, so that the entire deposit can be captured and the deposit can be accurately identified. [Effects of the Invention]

[0032] According to the present invention, it is possible to accurately identify the sediments deposited in each deposition area in a sediment storage facility having a plurality of deposition areas separated by partition walls. [Brief explanation of the drawings]

[0033] [Figure 1] 1 is a plan view showing the configuration of a scrap yard that is the subject of an image processing method according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating a hardware configuration of an image processing system according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram showing a software configuration of an image processing system according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing map information within a scrap yard and best shot areas in each area. [Figure 5A] FIG. 2 is a diagram for explaining the principle underlying the method for estimating the weight of scrap according to an embodiment of the present invention. [Figure 5B] FIG. 2 is a diagram for explaining the principle underlying the method for estimating the weight of scrap according to an embodiment of the present invention. [Figure 6]1 is a flowchart illustrating a method for estimating the weight of scrap according to one embodiment of the present invention. [Figure 7] 10 is a flowchart showing a detailed flow of an approximate expression creating step. [Figure 8] 1 is a graph showing an approximation of a density variable. [Figure 9] 10 is a flowchart showing a detailed flow of a three-dimensional shape acquisition step. [Figure 10] 10 is a diagram showing a state in which the constructed three-dimensional shape data includes three-dimensional shapes of a plurality of scrap piles S. FIG. [Figure 11] 10 is a flowchart showing a detailed flow of a weight estimation step. [Figure 12] FIG. 10 is a diagram showing how the three-dimensional shape of a scrap pile is divided. DETAILED DESCRIPTION OF THE INVENTION

[0034] An image processing method according to one embodiment of the present invention will be described in detail below with reference to the drawings. While the following description will be given using an example in which the weight of scrap piled up in each area of ​​a scrap yard partitioned by partition walls is measured, the present invention is not limited to this and can be applied to any case in which the weight of piled up material is measured in a pile storage facility partitioned by partition walls. The piled up material is not limited to scrap, but can be various other materials such as product waste, waste material from product processing, waste material from building demolition, and powder, and the material is not limited to metal.

[0035] FIG. 1 is a plan view showing the configuration of a scrap yard that is the subject of an image processing method according to one embodiment of the present invention. As shown in FIG. 1, the scrap yard 1 is surrounded by an outer wall 3, and the interior of the outer wall 3 is divided into multiple areas 4 (4A-4F) by partition walls 2. The partition walls 2 have a predetermined height and are open at the top. Each partition wall 2 extends linearly in a plan view. Each area 4 is rectangular in a plan view, and is not limited to areas in which all sides of the rectangle are divided by partition walls 2 or outer walls 3. It may also include areas in which at least one side of the rectangle does not have a partition wall 2 or outer wall 3. Various types of scrap metal are piled in each area 4, forming a scrap pile S. The area 4 may also be a container, and the side of the container corresponds to the partition wall 2. The area 4 may have partition walls 2 inside or outside that partition areas (e.g., corridors) where scrap metal is not piled up. The exterior wall 3 may also be a wall separating areas other than the scrap yard 1 within the building, and may be open at the top. In this case, the outer wall 3 may be treated in the same manner as the partition wall 2 in the treatment described below.

[0036] An overhead crane is installed above the scrap yard 1. The overhead crane is equipped with a girder 6, which can move horizontally across the scrap yard 1 in the lateral direction of FIG. 1. The girder 6 is equipped with multiple cameras 8 (two cameras 8A and 8B in this embodiment) as imaging devices and a position information acquisition device 9. In this embodiment, the camera 8 used as the imaging device is a stereo camera, and two imaging units (two imaging units 8A1 and 8A2 of camera 8A and two imaging units 8B1 and 8B2 of camera 8B) composed of imaging elements such as CCD or CMOS capture two-dimensional images. The camera 8 only needs to move along with the movement of the girder 6, and an imaging position adjustment jig may be provided between the girder 6 and the camera 8. The position information acquisition device 9 is a device that acquires the position of the image captured by the camera 8 when it captures an image. In this embodiment, the position of the camera 8 in the vertical direction in FIG. 1 is known, so the horizontal position in FIG. 1 is acquired. However, depending on the installation location and type of the position information acquisition device 9, both the vertical and horizontal positions in FIG. 1 may be acquired. The position information acquisition device 9 may be, for example, a rangefinder that measures the distance to the side wall of the scrap yard 1, a device that detects the position of the girder 6 of the overhead crane, a GPS, or a sensor or camera that detects a self-position detection marker installed in the scrap yard 1. Note that when multiple cameras 8 are used, multiple position information acquisition devices 9 corresponding to each of the multiple cameras 8 may be provided. Note that when SLAM technology is used to create a map from captured images acquired by an imaging device and estimate the self-position, the imaging device serves as the position information acquisition device. Furthermore, the movement speed of the camera 8 can be calculated based on the position and time detected by the position information acquisition device 9.

[0037] In this embodiment, the camera 8 is moved, but the camera 8 may be installed so that its angle can be changed so that its imaging direction can be changed, and the angle of the imaging direction may be detected by the position information acquisition device 9. Furthermore, if the lighting and sunlight within the scrap yard 1 cause areas within the field of view of the image acquired by the camera 8 to have large differences in brightness, a shadow-forming member or lighting member may be provided above the camera 8.

[0038] FIG. 2 is a diagram illustrating the hardware configuration of an image processing system according to an embodiment of the present invention. As shown in FIG. 2, the image processing system 10 includes a computer 12 as an information processing device. The computer 12 includes a CPU 13, a GPU 18, a RAM 14, a ROM 15, and an interface 16. The RAM 14, the ROM 15, and the interface 16 are connected to the CPU 13 and the GPU 18 via a system bus 17. The interface 16 is connected to a camera 8, a location information acquisition device 9, and a display 7. In this embodiment, a stereo camera is used as the camera 8 to construct a 3D shape of the scrap pile, as described below. However, this is not limited to this. Alternatively, multiple general cameras 8 capturing 2D images with a single imaging unit may be used to construct the 3D shape of the scrap pile using the parallax caused by the stereo vision of the cameras. Alternatively, the 3D shape of the scrap pile may be constructed using a camera 8 capturing 2D images with a single imaging unit and an imaging distance information acquisition device such as a rangefinder or a TOF imaging device. Note that, for accurate scrap identification, the images captured by the camera 8 are preferably color images. In this embodiment, the image processing system 10 is configured by a computer, but is not limited to this, and each function may be distributed using a plurality of terminals.

[0039] Fig. 3 is a diagram showing the software configuration of an image processing system according to one embodiment of the present invention. As shown in Fig. 3, the image processing system 10 includes an approximate equation creation unit 120, an image processing unit 140, a weight estimation unit 160, and an area information storage unit 130. These units are implemented by the CPU 13 and GPU 18 executing a program stored in the ROM 15.

[0040] The approximate expression creating unit 120 creates an approximate expression used to estimate the weight of the scrap pile based on the three-dimensional shape of the scrap pile.

[0041] The area information storage unit 130 stores map information for the scrap yard 1, as well as information on each area 4, each partition wall 2, and each camera 8. The map information for the scrap yard 1 includes the position and size (e.g., vertical and horizontal width) of each area 4 and the position and size of each partition wall 2. Information for each area 4 includes the type of scrap to be piled up, the type of area 4 (e.g., when a floor area and a container area are mixed), the camera 8 that will capture the image, and information on the presence or absence of a partition wall 2 that separates the area 4 from areas where scrap is not to be piled up. Information on the partition wall 2 includes height information. Information on the camera 8 includes the position of the camera 8 within the scrap yard 1, the camera height (e.g., the vertical distance from the camera 8 to the bottom of the area 4), the movement speed or upper limit of the movement speed of the camera 8, and the like. If multiple cameras 8 are provided, the area 4 to be imaged may also be stored. The bottom of the area 4 refers to the floor of the area 4. If the area 4 is a container, this corresponds to the bottom of the container. Furthermore, if the area 4 is a container, it is preferable to maintain either the vertical distance between the floor of the scrap yard 1 and the bottom of the container, or the vertical distance between the camera 8 and the bottom of the container.

[0042] In this embodiment, multiple cameras 8 are provided, and areas corresponding to the cameras 8 are set. For example, camera 8A is set to capture areas 4A, 4B, 4C, and 4D in the scrap yard 1 shown in FIG. 1, and camera 8B is set to capture areas 4E and 4F. If the entire area of ​​area 4F falls within the images of both cameras 8A and 8B, it is set to be the capture target of either camera 8A or 8B. If the entire area of ​​area 4F does not fall within the images of either camera 8A or 8B, it is set to be the capture target of both cameras 8A and 8B, and area 4F may be divided to correspond to each of cameras 8A and 8B and subjected to image processing, as described below. Alternatively, after the images are acquired, the images may be combined with 3D construction data of the scrap pile, as described below.

[0043] The image processing unit 140 processes the images captured by the camera 8 to obtain the three-dimensional shape of the scrap pile in each area.

[0044] The image processing unit 140 has a captured image acquisition unit 142, an image capture position information acquisition unit 144, a best shot image extraction unit 146, and a deposit identification unit 147. The deposit identification unit 147 has a partition wall / scrap extraction unit 148, a 3D shape construction unit 150, a scrap classification unit 151, and a target scrap identification unit 152.

[0045] The captured image acquisition unit 142 can communicate with the camera 8 and acquires image data captured by the camera 8. The image capture position information acquisition unit 144 can communicate with the position information acquisition device 9 and acquires position information of the camera 8 at the time the camera 8 captured an image. The image data acquired by the captured image acquisition unit 142 is recorded in association with the position information acquired by the image capture position information acquisition unit 144. In this embodiment, the horizontal position in FIG. 1 acquired by the position information acquisition device 9 is recorded in association with the image data. However, the vertical position in FIG. 1, which is the position of the camera 8 on the girder 6, may also be recorded in association with the image data. In addition, corresponding imaging target areas may be set for each of the multiple cameras 8. This makes it possible to discard image data if the position information of the image data acquired by the image capture position information acquisition unit 144 is outside the best shot areas of all imaging target areas, thereby reducing the computer load of the best shot image extraction unit 146, which will be described later.

[0046] The best shot image extraction unit 146 stores in advance map information for the scrap yard 1 and the best shot areas for each area 4. FIG. 4 shows the map information for the scrap yard 1 and the best shot areas for each area. As shown in FIG. 4, a best shot area B is set for each area 4. The best shot area B is an area in which the image data captured by the camera 8 does not include any blind spots of the scrap piles in the area 4, and the entire area is included in the image. In other words, it is an area in which the scrap piles S in the area 4 are not obstructed by obstacles such as the partition walls 2 when captured by the camera 8. Obstacles include not only the partition walls 2 but also the loading / unloading jigs, scrap transport pipes or conveyors connected to the loading / unloading jigs, loading / unloading jigs attached to the crane and extending downward (such as lifting magnets or arms or buckets capable of grabbing piles), and piles of material piled high in another area.

[0047] The best shot area B is set based on the size of the image capture area, which is determined based on the installation height and field of view of the camera 8, and the size of the area 4. The best shot area B is set so that the scrap pile S in the area 4 is not blocked by the partition wall 2 when the camera 8 captures the image. Furthermore, in this embodiment, the best shot area B for each area is set based on the position of the camera 8 on the girder 6. For example, in the case of areas 4A and 4E, where the difference between the width of the image field of view in the movement direction of the camera 8 (the horizontal direction in FIG. 1 in this embodiment) and the width of the area in the image is small, the best shot area B is set near the center of the areas 4A and 4E. In the case of areas 4B and 4C, where the difference between the width of the image field of view in the movement direction of the camera 8 and the width of the area in the image is sufficiently large, the best shot area B is set to be larger by a predetermined range than the areas 4B and 4C. Note that, for an area where the entire area does not fit within the image field of view, such as area 4D, which is larger than the image field of view, multiple best shot areas B may be set for the area, or a best shot area B corresponding to each camera 8 may be set to combine images captured by multiple cameras 8.

[0048] The best shot area B is set so that the scrap pile S is not blocked by obstacles, such as a loading / unloading jig or a scrap transport pipe or conveyor connected to the loading / unloading jig, that exist between the camera 8 and the area 4 when the camera 8 captures the image. If the camera 8 is installed at an angle such that the imaging unit is not parallel to the floor, the best shot area B is set taking into consideration the possibility of blind spots due to piles of material in the surrounding area. For example, the best shot area B is set assuming that the scrap pile S accumulated in the surrounding area of ​​the target area 4 is piled to a predetermined height higher than the height of the partition wall. The best shot area B may be set taking into consideration the lighting and sunlight in the scrap yard 1, or the width of the best shot area B may be set to change over time to account for times when shadows are likely to appear in the scrap yard 1. Furthermore, the best shot area B for an area where the camera 8 does not move directly above due to the movable range of the girder 6 may be set using a method different from that for setting other best shot areas B.

[0049] 1, the best shot area can be set as a rectangular area with a predetermined length in the vertical direction and a predetermined width in the horizontal direction. Also, if the camera 8 is installed so that the angle can be changed to change the imaging direction, the best shot area can be set as an angle.

[0050] The best shot image extraction unit 146 extracts images captured within the best shot region of each area from the image data acquired by the captured image acquisition unit 142. In this embodiment, since the camera 8 is a stereo camera, both of the two-dimensional images captured by the two imaging units are extracted.

[0051] The partition wall / scrap extraction unit 148 extracts the partition walls 2 and scrap from the image data extracted by the best shot image extraction unit 146. The partition wall / scrap extraction unit 148 is previously trained with training data regarding the metals constituting the scrap and the partition walls 2. This allows the partition walls 2 and scrap to be extracted from the best shot image. By recognizing the scrap and the partition walls, the partition wall / scrap extraction unit 148 can learn about the metals constituting the scrap and the partition walls, thereby reducing the possibility of misidentifying the partition walls as scrap. Note that in this embodiment, because the camera 8 is a stereo camera, the best shot image extraction unit 146 extracts two sets of image data. However, the partition walls 2 and scrap may be extracted from one or both sets of image data, or scrap may be extracted from both sets of image data and the partition walls 2 from one of the sets of image data.

[0052] The 3D shape construction unit 150 constructs 3D shape data of the scrap pile from the image data extracted by the best shot image extraction unit 146. In this embodiment, the camera 8 is a stereo camera, so the 3D shape data is constructed from two 2D images. However, if a single camera capturing 2D images and an imaging distance information acquisition device are used, the 3D shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. Note that the 3D shape only needs to be the shape of the upper part of the scrap pile captured in the 2D image. If 3D shape data is also to be constructed for the portion below the top of the scrap pile that cannot be captured by the camera 8, the data may be constructed based on vertical distance information from the camera 8 to the bottom of the area 4, which is pre-stored in the area information storage unit 130. The 3D shape data also includes height information of the scrap pile. In this embodiment, the height of the scrap pile is calculated from the vertical distance from camera 8 to the bottom of area 4, which is stored in advance in the area information storage unit 130, and the vertical distance from camera 8 to the scrap pile, and is added to the 3D shape data, but this is not limiting, and scrap pile height information may be obtained by any appropriate method without constructing a 3D shape. Note that when using a best shot image of area 4 being a container, the height of the scrap pile should be calculated taking into account the height of the bottom of the container.

[0053] The scrap sorting unit 151 divides the scrap pile into areas 4. In this embodiment, the scrap pile is divided into areas 4 based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148 and the three-dimensional data of the scrap pile S constructed by the three-dimensional shape construction unit 150. When multiple scrap piles are extracted by the partition wall / scrap extraction unit 148, the scrap sorting unit 151 determines whether the multiple scrap piles are scrap piles in the same area or scrap piles in different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. Furthermore, in order to detect whether scrap from different adjacent areas has been extracted in a connected manner by the partition wall / scrap extraction unit 148, the scrap sorting unit 151 detects locations in the scrap pile S where the heights differ significantly (locations larger than a predetermined size). Furthermore, in order for the partition wall / scrap extraction unit 148 to detect that scrap from different adjacent areas has been extracted in a connected manner across the partition wall 2, the scrap classification unit 151 may determine whether at least a portion of the partition wall is included inside the scrap pile, or whether at least a portion of the scrap is included on the upper end of the partition wall. Furthermore, when the partition wall / scrap extraction unit 148 detects that scrap from different adjacent areas 4 has been extracted in a connected manner across the partition wall 2, the scrap classification unit 151 divides the three-dimensional shape data of the scrap pile S into areas.

[0054] The target scrap identifying unit 152 identifies the three-dimensional shape data of the scrap pile S in the target area 4 in the three-dimensional shape data.

[0055] The weight estimation unit 160 estimates the weight of the scrap pile S for each area 4 based on the three-dimensional shape of the scrap pile S for each area 4 acquired by the image processing unit 140.

[0056] Hereinafter, a method for identifying scrap piled up in each area, acquiring a three-dimensional shape, and estimating the weight of the scrap using the image processing method of this embodiment will be described. In a conventional method for estimating the weight of scrap or other materials based on their three-dimensional shape, the volume of the scrap or other materials is calculated based on the three-dimensional shape, and then the weight is calculated by multiplying this volume by a standard bulk density (assuming a uniform density). However, calculating the weight based on a uniform density for the volume in this manner has the problem of large discrepancies with the actual measured weight. Figures 5A and 5B are diagrams illustrating the basic principle of a method for estimating the weight of scrap according to one embodiment of the present invention. In Figure 5A, areas with high density are indicated by darker colors. In Figure 5B, areas with high average density when the bottom surface is divided by a predetermined area are indicated by darker colors. As shown in Figure 5A, in areas with a high stack height, the scrap is compressed and deformed by its own weight, resulting in a high density pile of scrap. Therefore, the higher the stack height, the smaller the gaps between individual scrap pieces, and the greater the discrepancy between the weight calculated by multiplying the volume by the standard bulk gravity and the actual weight. Therefore, in this embodiment, based on the idea that the higher the stack height of the scrap, the higher the average density, the weight is estimated using a variable that increases as the stack height of the scrap increases, as shown in Fig. 5B. Specifically, an approximation formula for the variable that takes into account the density relative to the stack height of the scrap is created, and the weight is estimated based on a reference weight set for each type of scrap and the variable calculated using this approximation formula.

[0057] Fig. 6 is a flowchart showing a method for estimating the weight of scrap according to one embodiment of the present invention. As shown in Fig. 6, the method of this embodiment includes an approximate expression creation step (S100) for creating an approximate expression of variables that takes into account the density of the scrap relative to the stack height, a three-dimensional shape acquisition step (S200) for acquiring the three-dimensional shape of the scrap pile S piled in each area 4 of the scrap yard 1, and a weight estimation step (S300) for estimating the weight of the scrap based on the acquired three-dimensional shape of the scrap pile S. Each step will be described in detail below.

[0058] 7 is a flowchart showing a detailed flow of the approximate equation creation step. As shown in FIG. 7, in the approximate equation creation step S100, first, reference weight data is acquired for each type of scrap stored in the scrap yard (reference weight acquisition step: S110). The reference weight data is obtained by loading the target metal to a height of 1 m in a reference weight measurement container with bottom dimensions of 1 m x 1 m, and measuring the weight. This weight is used as the reference weight per unit volume. This reference weight is input to the approximate equation creation unit 120.

[0059] Next, the scrap is scanned with a 3D scanner to obtain shape information (S120). Specifically, for example, each type of scrap to be piled up in the scrap yard is scanned with a 3D scanner to obtain shape information for each individual scrap. If there are multiple different shapes for each scrap, or if the shapes are irregular enough to be classified as the same type under the condition that they are smaller than a specified size, shape information for any number of scraps is obtained. This shape information is input into the approximation formula creation unit 120.

[0060] Next, strength information of the scrap unit is acquired (S130). For example, the Young's modulus or the like is calculated by compressing the scrap unit. If the metal constituting the scrap unit is known, the physical property value of the metal can be used as the strength information of the scrap unit. This strength information is input to the approximation formula creation unit 120.

[0061] Next, the weight of the scrap itself is measured to obtain weight information (S140). This weight information is input to the approximate formula creation unit 120. Note that the weight information of the scrap itself may be calculated from the weight and shape information of the metal per unit volume.

[0062] Next, the approximation formula creation unit 120 simulates the gaps and deformation that will form between scrap units when multiple scrap units are piled up based on the shape information, strength information, and weight information (S150). Specifically, a virtual container (1 m x 1 m bottom) is simulated, scrap units are randomly placed inside, and how the scrap units are piled up inside the container is simulated based on the strength information and weight information of the scrap units, thereby determining the gaps and deformation in the scrap pile. This simulation determines the weight and height of the scrap pile inside the container. Note that a virtual container (1 m x 1 m bottom) may be simulated, and the weight of the scrap pile when the height of the scrap pile inside the container is 1 m may be used as the reference weight.

[0063] The approximate equation creating unit 120 repeats this simulation under various conditions to determine the relationship between the stack height of the scrap pile and the weight per unit area (S160).

[0064] Next, the approximation formula creation unit 120 calculates the relationship between the weight per unit area for each stacking height of the scrap pile obtained through the simulation and the estimated weight per unit area for each stacking height of the scrap pile without considering density changes from the reference weight. The weight obtained through the simulation is then divided by the estimated weight to calculate a density variable. An approximation formula that approximates the relationship between stacking height and density variable is then created from the value of the density variable for each stacking height (S170). FIG. 8 is a graph showing the approximation formula for the density variable. As shown in the figure, the density variable increases as the height increases, and the increase in the density variable per unit height decreases as the height increases. Alternatively, the weight per unit area for each stacking height of the scrap pile obtained through the simulation may be used as a variable corresponding to each stacking height of the scrap pile, and an approximation formula that approximates the relationship between stacking height and weight per unit area may be created. Even when weight per unit area is used as a variable, the weight increases as the height increases, and the increase in density per unit height decreases as the height increases.

[0065] In the above-described approximate formula creation step S100, the approximate formula is determined by simulation, but this is not limiting. Alternatively, an approximate formula may be created by repeatedly stacking actual scrap units without obtaining shape information or strength information of the scrap units, and determining the relationship between the stack height of the scrap pile and the weight per unit area. Furthermore, the approximate formula created by the above-described simulation may be corrected by comparing it with the results of actual experiments.

[0066] Next, the three-dimensional shape acquisition step S200 will be described in detail. Fig. 9 is a flowchart showing a detailed flow of the three-dimensional shape acquisition step. The area information holding unit 130 has pre-recorded therein map information of the scrap yard, area information, and camera information. In addition, a best shot area is set for each area of ​​the scrap yard, and information regarding the best shot area of ​​each area is pre-recorded in the image processing unit 140.

[0067] In the 3D shape acquisition step, first, the girder 6 and a scrap transport jig, such as a lifting magnet extending downward from the camera 8 movably mounted on the girder 6, are moved to the image capture start position (image capture preparation step S205). The image capture start position of the girder 6 may be a predetermined position, or, if at least one of the multiple areas 4 is the 3D shape acquisition target area, it may be a predetermined position around the target area. The image capture start position of the lifting magnet is outside the image capture area of ​​the camera 8. If the lifting magnet is present within the image capture area, an image will be captured with the lifting magnet present on top of the scrap pile S, making it impossible to accurately identify the deposits in each pile area. Note that in the image capture preparation step, the girder 6 and lifting magnet may be operated automatically or manually by the user. If the user operates them manually in the image capture preparation step, the image capture start position is displayed on the display 7.

[0068] Next, the camera 8 is moved by moving the girder 6, and images of the scrap yard 1 are captured by the camera 8 at predetermined time intervals (image capturing step), and image data is generated. The image processing unit 140 acquires the generated image data by the captured image acquiring unit 142 (captured image acquiring step: S210). Note that the image processing unit 140 may be configured to select a camera from the multiple cameras 8 to capture or an area to be captured from the multiple areas 4 when an image capture start command is issued from the user or the image processing unit 140, and may determine the camera to capture based on the selected camera or area. In parallel with this, the position information acquiring device 9 generates position information regarding the position where the camera 8 captured the image at the time of capturing the image (position information generating step: S220). The image processing unit 140 acquires the position information generated by the image capturing position information acquiring unit 144. The image processing unit 140 associates the position information with the acquired image data and records it. In this embodiment, the movement speed of the camera 8 is the movement speed of the girder 6, and the upper limit of the movement speed is preferably a speed at which no blurring occurs in the image depending on the imaging interval and the performance of the camera 8. The movement speed of the camera 8 can be calculated from the acquisition time of the position information acquired by the imaging position information acquisition unit 144. Note that the operation of moving the girder 6 in the imaging step may be performed automatically or manually by the user. When the user manually operates the girder 6 in the imaging step, the appropriate movement speed of the girder 6 and the current movement speed of the girder 6 may be displayed on the display 7, and when the movement speed of the girder 6 exceeds the upper limit, a deceleration operation instruction or an imaging error may be displayed on the display 7.

[0069] Next, the image processing unit 140, through the best shot image extraction unit 146, references the recorded image data for each area 4 and acquires, based on the position information, image data (hereinafter referred to as best shot image) whose image capture position is included in a best shot area (best shot image extraction step: S230). The number of best shot images acquired for each area 4 is preferably one or more, and multiple best shot images may be acquired so that the width intervals between best shot areas are equal based on the position information, or multiple images may be acquired so that the image capture time intervals are equal, or the number of images acquired may be set according to the size and number of best shot areas. The best shot image extraction step is not limited to this, and may also thin out image data from image data captured at a predetermined time interval so that the intervals are longer than the predetermined time interval, determine whether the position information of the thinned-out image data is included in a best shot area of ​​any of the areas 4, and acquire the image as a best shot image if it corresponds to a best shot area of ​​any of the areas 4.

[0070] Next, the image processing unit 140 extracts the partition walls 2 and scrap in the best shot image using the partition wall / scrap extraction unit 148 (scrap extraction step: S240). In the scrap extraction step 240, the partition walls 2 separating the areas 4 and the scrap in the best shot image are extracted using a neural network. The partition wall / scrap extraction unit 148 is previously trained with training data regarding the metals constituting the scrap and the partition walls 2. This enables the partition walls 2 and scrap to be extracted in the best shot image. Note that if the partition wall / scrap extraction unit 148 does not detect any metal components in any of the best shot images of the target area, it determines that the target area is empty of scrap. Furthermore, before, during, or simultaneously with the scrap extraction step, the image processing unit 140 preferably detects scrap transport jigs, such as lifting magnets, in the best shot image. For example, lifting magnets are detected using a neural network. If a lifting magnet is detected in the best shot image (particularly preferably in the target area), the 3D shape acquisition step for the best shot image or the partition wall 2 and scrap extracted from the best shot image is terminated, and an imaging error is displayed on the display 7.

[0071] Next, the image processing unit 140 uses the 3D shape construction unit 150 to construct a 3D shape of the scrap pile in the best shot image (3D shape construction step: S250). In this embodiment, a stereo camera is used as the camera 8, and therefore 3D shape data is constructed from two 2D images. Note that, for example, when a camera that captures 2D images using a single imaging unit and an imaging distance information acquisition device are used, the 3D shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. In this case, the 3D shape data includes height information of the scrap pile. The height of the scrap pile is calculated from the vertical distance from the camera 8 to the bottom of the area 4, which is stored in advance, and the vertical distance from the camera 8 to the scrap pile. Note that the 3D shape construction step S250 corresponds to the scrap height information acquisition step.

[0072] Next, the image processing unit 140 causes the scrap classification unit 151 to classify the scrap piles into areas (scrap classification step: S260). Specifically, as shown in Fig. 10, when the constructed three-dimensional shape data includes three-dimensional shapes of multiple scrap piles S, it is determined whether the multiple scrap piles S are in the same area or different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. For example, when a partition wall 2 is extracted between multiple scrap piles S, it can be determined that the multiple scrap piles S are in different areas. The method of determining whether multiple scrap piles S are in the same area or different areas based on the partition walls 2 is not limited to this. For example, in the case of an area 4 whose periphery is entirely surrounded by partition walls 2, the upper ends of the partition walls 2 may be extracted as straight line components, a polygon surrounded by the straight line components may be detected, and scrap within the polygon may be determined to be scrap piles S in the same area. Furthermore, in the case of adjacent areas 4 whose peripheries are entirely surrounded by partition walls 2, scrap in an area where a polygon cannot be formed by the straight line components of the upper ends of the partition walls (one side of the polygon cannot be detected) may be determined to be scrap outside the target area in the target scrap identification step S270 described below. In this embodiment, whether multiple scrap piles S are in the same area or different areas is determined based on the partition walls 2, but this is not limiting. The determination may also be based on the three-dimensional shape of the scrap piles S, in which case it may be based on any of the height, shape, or distance between the scrap piles S of the three-dimensional shape of the scrap piles S.

[0073] Furthermore, the scrap classification unit 151 detects areas in the scrap pile S where the height varies significantly (areas larger than a predetermined size) based on the height of the three-dimensional shape of the scrap pile S. When such areas where the height of the scrap pile varies significantly are detected, there is a high possibility that the scrap extracted by the partition wall / scrap extraction unit 148 is a combination of scrap from different adjacent areas. Therefore, when the scrap classification unit 151 detects areas where the height varies significantly, it divides the scrap pile S using the areas where the height varies significantly as boundaries.

[0074] The scrap classification unit 151 also determines whether at least some of the partition walls 2 are included inside the scrap pile S. If at least some of the partition walls are included inside such a scrap pile S, it is highly likely that the scrap extracted by the partition wall / scrap extraction unit 148 is scrap from different adjacent areas that are connected across the partition walls. Therefore, when the scrap classification unit 151 determines that at least some of the partition walls 2 are included inside the scrap pile S, it creates a boundary line by connecting the partition walls 2. Note that the upper ends of the partition walls 2 may be recognized, straight line components may be extracted from the upper ends of the partition walls 2, and the boundary line may be created by connecting the straight line components, or the junctions between the partition walls 2 at the upper ends of the partition walls 2 may be recognized as corners of the partition walls 2, and the boundary line may be created by connecting the corners with straight lines. Furthermore, if the area information storage unit 130 records that the target area of ​​the best shot image has a partition wall 2 that partitions an area (e.g., an aisle) within which metal scrap does not accumulate, the scrap pile S is not divided even if part of the partition wall 2 is recognized inside the scrap pile S. Note that the scrap sorting unit 151 is not limited to the method of determining whether at least part of the partition wall 2 is included inside the scrap pile S, and may also use a method of determining whether at least part of the scrap is included above the upper end of the partition wall 2. Through the above steps, the three-dimensional shape of the scrap piled up in each area of ​​the scrap yard can be obtained. Note that the scrap classification step S260 is preferably performed after the three-dimensional shape construction step S250, but at least a part of the scrap classification step S260 may be performed before the three-dimensional shape construction step S250.

[0075] Next, the image processing unit 140 uses the target scrap identification unit 152 to identify only the scrap in the target area (target scrap identification step: S270). Specifically, as shown in FIG. 10 , when the constructed 3D shape data includes the 3D shapes of multiple scrap piles S, the positions of one or both of the target area and the scrap are estimated from the position information recorded in association with the image data acquired by the captured image acquisition unit 142, and only the 3D shapes of the scrap in the target area are extracted from the constructed 3D shape data of the scrap pile S. Note that the target scrap identification step S270 is not limited to this. If the position of the target area in the captured image is recorded in advance, only the 3D shapes of the scrap in the target area may be extracted from the constructed 3D shape data of the scrap pile S without using the position information recorded in association with the image data. Note that the scrap classification step S260 and the target scrap identification step S270 may operate simultaneously, or the functions of the scrap classification unit 151 and the target scrap identification unit 152 may be integrated.

[0076] Next, a description will be given of the weight estimation step S300 in which the weight estimation unit 160 estimates the weight of the scrap based on the three-dimensional shape of the scrap acquired by the image processing unit 140. Fig. 11 is a flowchart showing the detailed flow of the weight estimation step.

[0077] First, the weight estimation unit 160 divides the three-dimensional shape of the scrap pile in the target area 4. Fig. 12 is a diagram showing how the three-dimensional shape of the scrap pile is divided. As shown in the figure, the scrap pile is divided vertically into, for example, a 15 cm x 15 cm grid (division step: S310).

[0078] Next, the weight estimation unit 160 estimates the height of each point (each grid) of the three-dimensional shape of the scrap from the bottom of the area 4. For example, the average value of the height coordinates within each point (each grid) may be used as the height of each point. Depending on the size of the grid, the height coordinate at the center of the grid may also be used. (Height estimation step: S320)

[0079] Next, the weight estimation unit 160 estimates the weight of each point using an approximation formula of a variable that takes into account the density relative to the stacking height (weight calculation step: S330). Specifically, when a density variable is used as the variable, the weight of each point is calculated using the following formula. Weight = Base area x Height x Reference weight x Density variable corresponding to height In the above formula, the area of ​​the base is the area of ​​the grid divided in the division step. The height is the height estimated in the height estimation step. The reference weight may be the one acquired in the reference weight acquisition step S110. Furthermore, when weight per unit area is used as a variable, the weight at each point is calculated using the following formula. Weight = Base area x Weight per unit area corresponding to height In the above formula, the area of ​​the base is the area of ​​the grid divided in the division step. In either case, the variable corresponding to the height can be calculated by substituting the height into the approximate expression created in the approximate expression creating step S100.

[0080] Next, the weight estimation unit 160 estimates the weight of the scrap in area 4 (total weight calculation step: S340). Specifically, the weight of each grid is added up to estimate the weight of the scrap pile. In this way, the weight estimation unit 160 estimates the weight of the scrap pile in each area 4 and outputs the result to the display 7 or the like.

[0081] Note that the weight estimation step S300 is not limited to the above method. For example, instead of performing the division step S310, the areas of points with the same height in the three-dimensional shape of the scrap pile may be summed up, and the weight calculation step S330 may be performed for each height. In this case, the weight for each height may be calculated using the following formula: Weight = Total base area x height x reference weight x density variable corresponding to height Next, in the total weight calculation step, the weight of the scrap pile is estimated by adding up the weights at each height.

[0082] According to this embodiment, the following effects are achieved. The image processing method of this embodiment includes a scrap extraction step S240 for extracting scrap from a best shot image captured within the best shot area. This prevents the scrap pile S from being in the blind spot of an obstacle such as the partition wall 2, making it possible to capture an image of the entire scrap pile S and accurately identify the pile of material.

[0083] Furthermore, in the image processing method of this embodiment, the best shot area B is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the camera 8, and the size of area 4. In this way, because the best shot area is set based on the size of the imaging area and the size of area 4, the scrap pile S can be imaged from the imaging position of the camera 8 so that it does not enter the blind spot of obstacles such as the partition wall 2, and the pile can be accurately identified.

[0084] The image processing method of this embodiment also includes an imaging step of capturing image data within the scrap yard 1 while moving the camera 8, a position information generation step S220 of generating the position where the image was captured by the camera 8, and a best shot image extraction step S230 of extracting best shot images captured within the best shot area based on the position information. When capturing images while moving the camera 8, the amount of captured image data becomes enormous, and processing all of the image data places a heavy load on the computer. In contrast, according to this embodiment, image data captured within the best shot area is extracted from the captured image data, thereby reducing the load on the computer. Furthermore, each area 4 within the scrap yard 1 can be captured even with a small number of cameras.

[0085] The image processing method of this embodiment also includes a target scrap identification step S270 for identifying only the scrap pile S in the target area 4 from the scrap piles S extracted in the scrap extraction step S240, and the target scrap identification step S270 extracts only the scrap pile S in the target area 4 in the best shot image based on the position information. This makes it possible to identify only the scrap pile S in the target area 4 based on the position information, and to extract the scrap pile S separately from the scrap piles S in other areas 4.

[0086] The image processing method of this embodiment also includes a scrap classification step S260 in which the scrap pile S extracted in the scrap extraction step S240 is classified into areas 4. As a result, even if an image is captured at an imaging position where multiple areas 4 are included in the image, by classifying the scrap pile S into areas 4, it is possible to prevent scrap piles S piled in different areas 4 from being identified as scrap piles S from the same area 4, and the piles can be accurately identified.

[0087] Furthermore, in the image processing method of this embodiment, in the scrap extraction step S240, the partition walls 2 are extracted from the captured best shot image. As a result, even if adjacent areas 4 exist within the scrap yard 1, the partition walls 2 separating the adjacent areas 4 can be recognized by extracting the partition walls 2 and the scrap, and it is possible to prevent scrap piles S piled up in adjacent areas 4 from being identified as scrap piles S from the same area 4, thereby enabling accurate identification of the piles.

[0088] Furthermore, in the image processing method of this embodiment, in the scrap classification step S260, it is determined whether at least a portion of the partition wall 2 is included inside the scrap pile S, and if it is determined that at least a portion of the partition wall 2 is included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted together in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether a portion of the partition wall 2 is included inside the scrap pile S, and the scrap pile S can be divided into each area 4.

[0089] Furthermore, in the image processing method of this embodiment, in the scrap classification step S260, it is determined whether at least some of the scrap is included above the upper ends of the partition walls 2, and if it is determined that at least some of the scrap is included above the upper ends of the partition walls 2, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted together in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether at least some of the scrap is included above the upper ends of the partition walls 2, and it is possible to separate the scrap piles S for each area 4.

[0090] The image processing method of this embodiment also includes a three-dimensional shape construction step S250 in which a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240 is constructed, and in the scrap classification step S260, the scrap pile S is classified into areas 4 based on the height of the three-dimensional shape of the scrap pile S. In this way, by basing the classification on the height of the three-dimensional shape of the scrap pile S, the difference in height between the scrap piles S in the multiple areas 4 can be utilized, and the scrap pile S can be classified into areas 4.

[0091] The image processing method of this embodiment also includes a three-dimensional shape construction step S250 for constructing a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. This makes it possible to construct the three-dimensional shape of the scrap pile S and estimate the weight of the scrap pile S.

[0092] Furthermore, in the image processing method of this embodiment, in the weight estimation step S300, the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S varies, the scrap pile S will be compressed by its own weight, resulting in different densities. However, according to this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so the weight of the scrap pile S can be estimated more accurately.

[0093] Furthermore, according to this embodiment, the variable corresponding to height is larger as the height increases. As the height of the scrap pile S increases, the density increases because the scrap pile S is compressed by its own weight. However, according to this embodiment, since a variable that increases as the height increases is used, the weight of the scrap pile S can be estimated more accurately.

[0094] Furthermore, according to this embodiment, the variable corresponding to height has a smaller increase in weight per unit height as the height increases. As the height of the scrap pile S increases, the pile is compressed by its own weight, increasing its density, but this increase becomes smaller as the height increases. According to this embodiment, the variable that has a smaller increase in weight per unit height is used as the height increases, allowing for more accurate estimation of the weight of the scrap pile S.

[0095] Furthermore, in the image processing method of this embodiment, in the weight estimation step S300, the three-dimensional shape of the scrap pile S is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby estimating the weight of the scrap pile S. In this way, because the three-dimensional shape of the scrap pile S is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, the weight of the scrap pile S can be estimated more accurately.

[0096] Furthermore, in this embodiment, an image processing method for a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, in which an image captured by a camera 8 includes at least a portion of the plurality of areas 4, comprises an image acquisition step S210 for acquiring an image captured by the camera 8, a scrap extraction step S240 for extracting scrap piles S from the image acquired in the image acquisition step S210, and a scrap classification step S260 for separating the scrap piles S extracted in the scrap extraction step S240 into each of the areas 4. In this way, by extracting scrap piles S from an image captured at an imaging position in which a plurality of areas 4 are included in the image and separating the extracted scrap piles S into each of the areas 4, it is possible to prevent scrap piles S piled up in different areas 4 from being identified as scrap piles S from the same area 4, and to accurately identify the piles.

[0097] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled, an image processing method in which at least some of the plurality of areas 4 are included in an image captured by a camera 8 includes a scrap extraction step S240 in which the partition walls 2 are extracted from the captured best shot image. As a result, even if there are adjacent areas 4 within the scrap yard 1, by extracting the partition walls 2 and the scrap, it is possible to recognize the partition walls 2 separating the adjacent areas 4, and it is possible to prevent scrap piles S piled in adjacent areas 4 from being identified as scrap piles S from the same area 4, thereby enabling accurate identification of the piles.

[0098] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes the following steps: in a scrap sorting step S260, it is determined whether at least a portion of the partition wall 2 is included inside the scrap pile S; and if it is determined that at least a portion of the partition wall 2 is included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of a plurality of areas 4 are extracted as a single unit in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether a portion of the partition wall 2 is included inside the scrap pile S, and the scrap pile S can be divided into each area 4.

[0099] Furthermore, in this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least some of the plurality of areas 4 are included in an image captured by a camera 8 includes the following steps: in a scrap division step S260, it is determined whether at least some of the scrap is included on the upper ends of the partition walls 2; and if it is determined that at least some of the scrap is included on the upper ends of the partition walls 2, the scrap piles S are divided. As a result, even if the scrap piles S of the plurality of areas 4 are extracted together in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether at least some of the scrap is included on the upper ends of the partition walls 2, and it is possible to separate the scrap piles S into each area 4.

[0100] Furthermore, in this embodiment, the image processing method for a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, in which an image captured by a camera 8 includes at least a portion of the plurality of areas 4, further includes a 3D shape construction step S250 for constructing a 3D shape of the scrap pile S extracted in the scrap extraction step S240, and in a scrap sorting step S260, the scrap pile S is divided into areas 4 based on the height of the 3D shape of the scrap pile S. In this way, by using the height of the 3D shape of the scrap pile S as the basis, the difference in height between the scrap piles S in the plurality of areas 4 can be utilized, and the scrap pile S can be divided into areas 4.

[0101] In this embodiment, the image processing method for a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled up, in which at least some of the plurality of areas 4 are included in an image captured by a camera 8, further includes a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. By estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S divided into areas 4, it is possible to estimate the weight according to the type of scrap, and to accurately estimate the weight of the pile.

[0102] In this embodiment, in a scrap yard 1 having a plurality of areas 4 at least some of which are partitioned by partition walls 2 and in which scrap piles S are piled, an image processing method in which at least some of the plurality of areas 4 are included in an image captured by a camera 8 includes a weight estimation step S300 in which the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S differs, the scrap pile S will be compressed by its own weight and will have a different density. However, according to this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so the weight of the scrap pile S can be estimated more accurately.

[0103] In this embodiment, the image processing method for capturing an image of a scrap yard 1, which has an area 4 where scrap piles S are piled, captured by a camera 8 includes an image capturing step S210 for capturing an image captured by the camera 8, a scrap extraction step S240 for extracting the scrap pile S from the image captured in the image capturing step S210, a scrap height information capturing step for capturing the height of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for calculating the weight of the scrap pile S based on the height of the scrap pile S captured in the scrap height information capturing step and a variable corresponding to the height of the scrap pile. Different heights of the scrap pile S result in different densities because the scrap pile S is compressed by its own weight. However, according to this embodiment, by extracting the scrap pile S from the captured image and acquiring the height of the scrap pile S extracted from the captured image, the weight of the scrap pile S can be calculated based on the variable corresponding to the height of the scrap pile S, thereby accurately estimating the weight of the pile.

[0104] Furthermore, in this embodiment, the image processing method for capturing images of the inside of the scrap yard 1 having the area 4 where the scrap pile S is piled up by the camera 8 includes the steps of: constructing a three-dimensional shape of the scrap pile S in the scrap height information acquisition step; and estimating the weight of the scrap pile S by dividing the three-dimensional shape of the scrap pile S into multiple vertical sections and calculating the weight of each section based on the height of each divided section and a variable corresponding to the height of the scrap pile S in the weight estimation step S300. In this way, because the three-dimensional shape of the scrap pile S is divided into multiple vertical sections and the weight of each section is calculated based on the height of each divided section and a variable corresponding to the height of the scrap pile S, the weight of the scrap pile S can be estimated more accurately.

[0105] In this embodiment, in a scrap yard 1 having a plurality of areas 4 where scrap piles S are piled, at least a portion of which are partitioned by partition walls 2, an image captured by a camera 8 includes at least a portion of the plurality of areas 4, or an image processing method for capturing an image of the scrap yard 1 having the area 4 where scrap piles S are piled by a camera 8, a predetermined best shot area is set for the area 4 so that the camera 8 can capture an image of the area 4 without obstructing the area 4, and in the scrap extraction step S240, the scrap pile S is extracted from the best shot image captured within the best shot area. By setting the best shot area in this way, the scrap pile S is not in the blind spot of obstacles such as the partition walls 2, so that the entire scrap pile S can be captured and the pile can be accurately identified. [Explanation of symbols]

[0106] 1:Scrap yard 2: Compartment wall 3: Exterior wall 4: Area 5: 3D shape of the constructed scrap pile 6: Guarda 7: Display 8: Camera 9: Location information acquisition device 10: Image processing system 12: Computer 13: CPU 14: RAM 15:ROM 16: Interface 17: System bus 18: GPU 100: Image processing system 120: Approximation formula creation section 130: Area information storage unit 140: Image processing unit 142: Image acquisition unit 144: Imaging position information acquisition unit 146: Best shot image extraction section 147: Sediment Identification Section 148: Compartment wall and scrap extraction section 150:3D shape construction part 151: Scrap sorting section 152: Target scrap specific part 160: Weight estimation section B: Best shot area S: Scrap pile

Claims

1. A method for processing an image captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and has a plurality of deposition areas in which sediments are deposited, comprising: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being obstructed by an obstacle; An image processing method comprising a deposit extraction step of extracting the deposit in a best shot image captured within the best shot area.

2. the best shot area is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the imaging device, and the size of the deposition area; The image processing method according to claim 1 .

3. Further, an imaging step of capturing image data of the inside of the deposit repository while moving or changing the angle of the imaging device; a position information generating step of generating position information relating to a position or angle at which an image is captured by the imaging device; a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information; The image processing method of claim 2 , comprising:

4. A target deposit identification step is further provided for identifying only the deposits in a target deposit area from the deposits extracted in the deposit extraction step, The target deposit identification step extracts only the deposit in the target deposit area in the best shot image based on the position information. The image processing method according to claim 3 .

5. The method further includes a step of classifying the deposits extracted in the step of extracting the deposits into the respective deposit regions. The image processing method according to claim 1 .

6. the deposit extraction step extracts the partition wall from the captured best shot image; The image processing method according to claim 5 .

7. The step of dividing the pile includes determining whether at least a part of the partition wall is included inside the pile; dividing the pile when it is determined that at least a part of the partition wall is included inside the pile; The image processing method according to claim 6.

8. The step of classifying the deposit includes determining whether at least a portion of the deposit is contained on an upper end of the partition wall; dividing the pile when it is determined that at least a portion of the pile is included on the upper end of the partition wall; The image processing method according to claim 6.

9. A three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step is further provided. The step of classifying the sediment comprises: Dividing the deposit into the deposit regions based on the height of the three-dimensional shape. The image processing method according to claim 5 .

10. Furthermore, a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step; a weight estimation step of estimating a weight of the deposit based on the three-dimensional shape; Equipped with 9. The image processing method according to claim 1.

11. In the weight estimation step, a weight of the pile is calculated based on a variable corresponding to the height of the pile. The image processing method according to claim 10.

12. The variable according to the height is larger as the height is higher. The image processing method according to claim 11.

13. The variable according to the height is such that the higher the height, the smaller the increase per unit height. The image processing method according to claim 11.

14. In the weight estimation step, the three-dimensional shape of the deposit is divided into a plurality of parts in the vertical direction, and a weight of each part is calculated based on a height of each divided part, thereby estimating a weight of the deposit. The image processing method according to claim 11.

15. A system for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and has a plurality of deposition areas in which sediments are deposited, comprising: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being obstructed by an obstacle; An image processing system including a deposit extraction unit that identifies the deposit in a best shot image captured within the best shot area and extracts the deposit.

16. A program for processing an image captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and has a plurality of deposition areas in which sediments are deposited, a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being obstructed by an obstacle; a deposit extraction step of identifying the deposit in a best shot image captured within the best shot area and extracting the deposit; A program that causes a computer to execute the following.

17. 1. A method for processing an image captured by an imaging device in a deposit storage facility having a plurality of deposit areas, at least a portion of which is partitioned by partition walls and in which deposits are deposited, the method comprising: a captured image acquisition step of acquiring an image captured by the imaging device; a deposit extraction step of extracting the deposit from the image acquired in the image acquisition step; a deposit classification step of classifying the deposits extracted in the deposit extraction step into the deposit regions.

18. The deposit extraction step extracts the partition wall from the image acquired in the image acquisition step. The image processing method according to claim 17.

19. The step of dividing the pile includes determining whether at least a part of the partition wall is included inside the pile; The image processing method according to claim 18 , wherein the pile is divided when it is determined that at least a part of the partition wall is included inside the pile.

20. The step of classifying the deposit includes determining whether at least a portion of the deposit is contained on an upper end of the partition wall; dividing the pile when it is determined that at least a portion of the pile is included on the upper end of the partition wall; 19. The image processing method according to claim 18.

21. A three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step is further provided. The step of dividing the deposit comprises dividing the deposit into the deposit regions based on the height of the three-dimensional shape.

21. The image processing method according to claim 17, 19 or 20.

22. The method further includes a weight estimation step of estimating a weight of the sediment classified in the sediment classification step based on the three-dimensional shape of the sediment.

22. The image processing method according to claim 21.

23. the weight estimation step calculates a weight of the pile based on a variable corresponding to the height of the pile; 23. The image processing method according to claim 22.

24. A method for processing an image captured by an imaging device inside a deposit storage facility having a deposit area where deposits are deposited, comprising: a captured image acquisition step of acquiring an image captured by the imaging device; a deposit extraction step of extracting the deposit from the image acquired in the image acquisition step; a deposit height information acquisition step of acquiring the height of the deposit extracted in the deposit extraction step; a weight estimation step of calculating a weight of the deposit based on the height of the deposit acquired in the deposit height information acquisition step and a variable corresponding to the height of the deposit; An image processing method comprising:

25. The deposit height information acquisition step includes constructing a three-dimensional shape of the deposit, The weight estimation step divides the three-dimensional shape of the deposit into a plurality of parts in the vertical direction, calculates a weight of each part based on a height of each divided part and a variable corresponding to the height of the deposit, and estimates the weight of the deposit.

25. The image processing method according to claim 24.

26. a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being obstructed by an obstacle; the deposit extraction step extracts the deposit in a best shot image captured within the best shot area; 25. The image processing method according to claim 17 or 24.

Images