Image processing method, image processing system, and program
The image processing method sets a 'best shot' area to avoid obstacles and uses stereo cameras for accurate scrap identification and separation in partitioned scrap yards, addressing underestimation issues and improving weight estimation accuracy.
Patent Information
- Application Number
- JP2024072220
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-26
AI Technical Summary
In scrap yards with partition walls, existing methods fail to accurately estimate the remaining amount of scrap due to blind spots and misidentification of adjacent scraps as a single unit, leading to underestimation.
An image processing method that sets a 'best shot' area for imaging, extracts deposits within this area, and uses stereo cameras to capture and process images, allowing for accurate identification and separation of deposits by area, while considering the imaging device's size, angle, and height to avoid obstacles.
Enables precise identification and separation of deposits in each area, reducing computational load and ensuring accurate estimation of scrap weight by accounting for density variations based on height.
Smart Images

Figure 0007588442000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing method, an image processing system, and a program. [Background technology]
[0002] In a storehouse (scrap yard) for metal scrap and the like, scrap is piled up and stored. As an apparatus for estimating the weight of scrap piled up in such a storehouse, for example, Patent Document 1 discloses a method for measuring the height of the piles of scrap in each area stored in the scrap yard by utilizing the parallax caused by stereo vision from a pair of two cameras, and estimating the remaining amount of scrap based on the pile heights. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2007-197170 A Summary of the Invention [Problem to be solved by the invention]
[0004] Here, the site of the scrap storage facility may be divided into multiple areas by partition walls. In a scrap yard where such obstacles such as partition walls exist, scrap may be in the blind spot of the obstacle depending on the imaging position of the camera. Also, in a scrap yard where multiple sections are adjacent to each other separated by partition walls, adjacent scrap may be identified as one piece depending on the imaging position of the camera and the stack height of the scrap relative to the height of the partition wall. If the remaining amount of scrap is estimated based on such an image, there is a problem that the scrap cannot be accurately identified and the estimated amount is less than the actual remaining amount.
[0005] The present invention has been developed in consideration of the above-mentioned problems, and its object is to make it possible to accurately identify the sediments deposited in each deposition area in a sediment storage facility having multiple deposition areas. [Means for solving the problem]
[0006] According to one aspect of the present invention, there is provided a method for processing images captured by an imaging device inside a deposit storage facility, at least a portion of which is partitioned by partition walls and has a plurality of deposit areas in which deposits are deposited, the image processing method including a predetermined best shot area set for each deposit area where the imaging device can capture an image of the deposit area without the deposit area being obstructed by an obstacle, and a deposit extraction step for extracting deposits from a best shot image captured within the best shot area. According to the above aspect, by setting the best shot area, the pile will not be in the blind spot of an obstacle, so that the entire pile can be imaged and the pile can be accurately identified.
[0007] According to one aspect of the present invention, the best shot area is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the imaging device, and the size of the deposition area. According to the above aspect, the best shot area is set based on the size of the imaging area and the size of the accumulation area, so that the accumulation can be captured without being in the blind spot of an obstacle from the imaging position, and the accumulation can be accurately identified.
[0008] According to one aspect of the present invention, the method further includes an imaging step of capturing image data inside the deposit storage facility while moving or changing the angle of the imaging device, a position information generation step of generating position information relating to the position or angle captured by the imaging device, and a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information. When capturing images while moving the imaging device, the amount of captured image data becomes huge, and processing all the image data places a heavy load on the computer. In contrast, according to the above-mentioned aspect, image data captured within the best shot area is extracted from the captured image data, so the load on the computer can be reduced. Also, each deposition area in the deposition repository can be captured even with a small number of cameras.
[0009] According to one aspect of the present invention, the method further includes a target deposit identification step for identifying only deposits in a target deposit area from the deposits extracted in the deposit extraction step, and the target deposit identification step extracts only deposits in the target deposit area within the best shot image based on position information. According to the above aspect, it is possible to identify only the deposit in the target deposit area based on the position information, and to extract the deposit separately from the deposit in other deposit areas.
[0010] According to one aspect of the present invention, the method further comprises a deposit classification step of classifying the deposits extracted in the deposit extraction step into deposit regions. According to the above aspect, by separating the deposits by deposition area, even if an image is captured at an imaging position where multiple deposition areas are included in the image, it is possible to prevent deposits deposited in different deposition areas from being identified as deposits from the same deposition area, and the deposits can be accurately identified.
[0011] According to one aspect of the present invention, the deposit extraction step extracts partition walls in the captured best shot image. According to the above aspect, by extracting the partition walls and sediments, even if adjacent deposition areas exist within a deposit storage facility, the partition walls separating the adjacent deposition areas can be recognized, and it is possible to prevent deposits deposited in adjacent deposition areas from being identified as deposits from the same deposition area, thereby enabling the deposits to be accurately identified.
[0012] According to one aspect of the present invention, the pile division step determines whether at least a portion of the partition wall is included inside the pile, and divides the pile if it is determined that at least a portion of the partition wall is included inside the pile. Even if deposits from multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above-described embodiment, the partition walls separating the deposit areas can be recognized by determining whether part of the partition wall is included inside the deposit, and the deposits can be separated by deposit area.
[0013] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the pile is contained on the upper end of the partition wall, and divides the pile if it is determined that at least a portion of the pile is contained on the upper end of the partition wall. Even if deposits from multiple deposit areas are extracted together in the deposit extraction step, according to the above embodiment, the partition walls separating the deposit areas can be recognized by determining whether at least a portion of the deposit is contained on the upper end of the partition wall, and the deposits can be separated by deposit area.
[0014] According to one aspect of the present invention, the method further includes a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit regions based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as a basis, the difference in height of the deposit among a plurality of deposit regions can be utilized, and the deposit can be separated for each deposit region.
[0015] According to one aspect of the present invention, the method further includes a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and a weight estimation step of estimating the weight of the deposit based on the three-dimensional shape. According to the above aspect, a three-dimensional shape of the pile can be constructed, and the weight of the pile can be estimated.
[0016] According to one aspect of the invention, the weight estimation step calculates the weight of the pile based on a variable that depends on the height of the pile. When the height of a pile differs, the pile is compressed by its own weight, resulting in a different density. However, according to the above-described embodiment, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.
[0017] According to one aspect of the invention, the height-dependent variable is greater for higher heights. As the height of a pile increases, the density of the pile increases because the pile is compressed by its own weight. According to the above embodiment, however, the weight of the pile can be estimated more accurately because a larger variable is used as the height increases.
[0018] According to one aspect of the invention, the height dependent variable is such that the higher the height, the smaller the increment per unit height. As the height of the pile increases, the pile is compressed by its own weight, and the density increases, but the increase in density is smaller the higher the height. According to the above embodiment, the weight of the pile can be estimated more accurately because a variable with a smaller increase per height is used as the height increases.
[0019] According to one aspect of the present invention, in the weight estimation step, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, thereby estimating the weight of the deposit. According to the above aspect, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, so that the weight of the deposit can be estimated more accurately.
[0020] According to one aspect of the present invention, there is provided an image processing system for processing images captured by an imaging device inside a deposit storage facility, at least a portion of which is partitioned by partition walls and has a plurality of deposit areas in which deposits are deposited, wherein a predetermined best shot area is set for each deposit area so that the imaging device can image the deposit area without blocking the deposit area by an obstacle, and the image processing system is equipped with a deposit extraction unit that identifies deposits in a best shot image captured within the best shot area and extracts the deposits.
[0021] According to one aspect of the present invention, there is provided a program for processing images captured by an imaging device inside a sediment storage facility, at least a portion of which is partitioned by partition walls and has multiple deposition areas in which sediments are deposited, wherein a predetermined best shot area is set for each deposition area so that the deposition area can be imaged by the imaging device without being blocked by an obstacle, and the program causes a computer to execute a deposit extraction step of identifying deposits in a best shot image captured within the best shot area and extracting the deposits.
[0022] According to one aspect of the present invention, a method for processing images in a sediment storage facility having multiple deposition areas in which sediments are deposited, at least a portion of which is partitioned by partition walls, is provided, in which an image captured by an imaging device includes at least a portion of the multiple deposition areas, and the method includes an image acquisition step for acquiring an image captured by the imaging device, a deposit extraction step for extracting deposits in the image acquired in the image acquisition step, and a deposit classification step for separating the deposits extracted in the deposit extraction step by deposit area. According to the above aspect, by extracting deposits from an image captured at an imaging position where multiple deposit areas are included in the image and separating the extracted deposits by deposit area, it is possible to prevent deposits deposited in different deposit areas from being identified as deposits from the same deposit area, and the deposits can be accurately identified.
[0023] According to one aspect of the present invention, the deposit extraction step extracts partition walls in the images acquired in the image acquisition step. According to the above aspect, by extracting the partition walls and sediments, even if adjacent deposition areas exist within a deposit storage facility, the partition walls separating the adjacent deposition areas can be recognized, and it is possible to prevent deposits deposited in adjacent deposition areas from being identified as deposits from the same deposition area, thereby enabling the deposits to be accurately identified.
[0024] According to one aspect of the present invention, the pile division step determines whether at least a portion of the partition wall is included inside the pile, and divides the pile if it is determined that at least a portion of the partition wall is included inside the pile. Even if deposits from multiple deposit areas are extracted as a single unit in the deposit extraction step, according to the above-described embodiment, the partition walls separating the deposit areas can be recognized by determining whether part of the partition wall is included inside the deposit, and the deposits can be separated by deposit area.
[0025] According to one aspect of the present invention, the pile dividing step determines whether at least a portion of the pile is contained on the upper end of the partition wall, and divides the pile if it is determined that at least a portion of the pile is contained on the upper end of the partition wall. Even if deposits from multiple deposit areas are extracted together in the deposit extraction step, according to the above embodiment, the partition walls separating the deposit areas can be recognized by determining whether at least a portion of the deposit is contained on the upper end of the partition wall, and the deposits can be separated by deposit area.
[0026] According to one aspect of the present invention, the method further includes a three-dimensional shape construction step for constructing a three-dimensional shape of the deposit extracted in the deposit extraction step, and the deposit classification step divides the deposit into deposit regions based on the height of the three-dimensional shape. According to the above aspect, by using the height of the three-dimensional shape of the deposit as a basis, the difference in height of the deposit among a plurality of deposit regions can be utilized, and the deposit can be separated for each deposit region.
[0027] According to one aspect of the present invention, the method further comprises a weight estimation step of estimating the weight of the pile separated in the pile separation step based on the three-dimensional shape of the pile. According to the above embodiment, by estimating the weight of the deposit based on the three-dimensional shape of the deposit divided into each deposition area, it is possible to estimate the weight according to the type of deposit, and thus it is possible to accurately estimate the weight of the deposit.
[0028] According to one aspect of the invention, the weight estimation step calculates the weight of the pile based on a variable that depends on the height of the pile. When the height of a pile differs, the pile is compressed by its own weight, resulting in a different density. However, according to the above-described embodiment, the weight of the pile is calculated based on a variable corresponding to the height, so that the weight of the pile can be estimated more accurately.
[0029] According to one aspect of the present invention, a method for processing images captured by an imaging device inside a deposit storage facility having a deposit area where deposits are deposited comprises: an image acquisition step for acquiring an image captured by the imaging device; a deposit extraction step for extracting deposits in the image acquired in the image acquisition step; a deposit height information acquisition step for acquiring the height of the deposit extracted in the deposit extraction step; and a weight estimation step for calculating the weight of the deposit based on the deposit height acquired in the deposit height information acquisition step and a variable corresponding to the deposit height. When the height of a pile is different, the density differs because the pile is compressed by its own weight. However, according to the above-described embodiment, by extracting the pile from the captured image and obtaining the height of the pile extracted from the captured image, the weight of the pile can be calculated based on a variable corresponding to the height, and the weight of the pile can be accurately estimated.
[0030] According to one aspect of the present invention, the pile height information acquisition step constructs a three-dimensional shape of the pile, and the weight estimation step divides the three-dimensional shape of the pile into multiple parts in the vertical direction, calculates the weight of each part based on the height of each divided part and variables corresponding to the height of the pile, and estimates the weight of the pile. According to the above aspect, the three-dimensional shape of the deposit is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, so that the weight of the deposit can be estimated more accurately.
[0031] According to one aspect of the present invention, a predetermined best shot area is set for the accumulation area in which the accumulation area can be imaged by the imaging device without being obstructed by obstacles, and the accumulation extraction step extracts the accumulation area from a best shot image captured within the best shot area. According to the above aspect, by setting the best shot area, the pile will not be in the blind spot of an obstacle, so that the entire pile can be imaged and the pile can be accurately identified. Effect of the Invention
[0032] According to the present invention, it is possible to accurately identify the sediment deposited in each deposition area in a sediment storage facility having a plurality of deposition areas partitioned by partition walls. [Brief description of the drawings]
[0033] [Figure 1] 1 is a plan view showing the configuration of a scrap yard which is a target of an image processing method according to an embodiment of the present invention; [Diagram 2] 1 is a diagram illustrating a hardware configuration of an image processing system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing a software configuration of the image processing system according to the embodiment of the present invention. [Figure 4] FIG. 13 is a diagram showing map information within a scrap yard and best shot areas in each area. [Figure 5A] FIG. 2 is a diagram for explaining the principle underlying the method for estimating the weight of scrap according to an embodiment of the present invention. [Figure 5B] FIG. 2 is a diagram for explaining the principle underlying the method for estimating the weight of scrap according to an embodiment of the present invention. [Figure 6]4 is a flow chart illustrating a method for estimating scrap weight according to one embodiment of the present invention. [Figure 7] 10 is a flowchart showing a detailed flow of an approximate expression creating step. [Figure 8] 1 is a graph showing an approximation of a density variable. [Figure 9] 10 is a flowchart showing a detailed flow of a three-dimensional shape acquisition step. [Figure 10] 13 is a diagram showing a state in which the constructed three-dimensional shape data includes three-dimensional shapes of a plurality of scrap piles S. FIG. [Figure 11] 10 is a flowchart showing a detailed flow of a weight estimation step. [Figure 12] FIG. 13 is a diagram showing how the three-dimensional shape of a scrap pile is divided. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0034] An image processing method according to an embodiment of the present invention will be described in detail below with reference to the drawings. In the following description, an example will be described in which the weight of scrap piled up in each area of a scrap yard partitioned by partition walls is measured, but the present invention is not limited to this, and can be applied to any case in which the weight of piled up material is measured in a pile storage facility partitioned by partition walls. The piled up material is not limited to scrap, and can be various materials such as product waste, waste material from product processing, waste material from demolition of buildings, powder, etc., and the material is not limited to metal.
[0035] FIG. 1 is a plan view showing the configuration of a scrap yard to which an image processing method according to an embodiment of the present invention is applied. As shown in FIG. 1, the scrap yard 1 is surrounded by an outer wall 3, and the inside of the outer wall 3 is divided into a plurality of areas 4 (4A to 4F) by partition walls 2. The partition walls 2 have a predetermined height, and the upper part of the partition wall 2 is open. Each partition wall 2 extends linearly in a plan view. Each area 4 is rectangular in a plan view, and is not limited to an area in which all sides of the rectangle are divided by the partition walls 2 and the outer wall 3, but may also include an area in which at least one side of the rectangle does not have the partition wall 2 or the outer wall 3. Various types of scrap metal are piled up in each area 4 to form a scrap pile S. The area 4 may be a container, and the side surface constituting the container corresponds to the partition wall 2. The area 4 may have a partition wall 2 that divides an area (for example, a passage) in which scrap metal is not piled up inside or outside the area 4. The outer wall 3 may be a wall that separates an area other than the scrap yard 1 in a building, and may be open at the top. In this case, the outer wall 3 may be treated in the same manner as the partition wall 2 in the process described below.
[0036] In addition, an overhead crane is installed above the scrap yard 1. The overhead crane is equipped with a girder 6, which can move horizontally in the lateral direction of FIG. 1 across the scrap yard 1. The girder 6 is equipped with a plurality of cameras 8 (two cameras 8A and 8B in this embodiment) as imaging devices and a position information acquisition device 9. In this embodiment, the camera 8 used as the imaging device is a stereo camera, and two imaging units (two imaging units 8A1 and 8A2 of the camera 8A and two imaging units 8B1 and 8B2 of the camera 8B) composed of imaging elements such as CCD and CMOS capture two-dimensional images. In addition, the camera 8 only needs to move together with the movement of the girder 6, and an imaging position adjustment jig may be provided between the girder 6 and the camera 8. The position information acquisition device 9 is a device that acquires the position of the image captured by the camera 8 when the camera 8 captures an image. In this embodiment, since the position of the camera 8 in the vertical direction in FIG. 1 is known, the horizontal position in FIG. 1 is acquired. However, depending on the installation location and type of the position information acquisition device 9, the vertical and horizontal positions in FIG. 1 may be acquired. As the position information acquisition device 9, for example, a distance meter that measures the distance to the side wall of the scrap yard 1 may be used, a device that detects the position of the girder 6 of the overhead crane may be used, a GPS may be used, or a sensor or camera that detects a self-position detection marker installed in the scrap yard 1 may be used. In addition, when multiple cameras 8 are used, multiple position information acquisition devices 9 corresponding to each of the multiple cameras 8 may be provided. In addition, when using SLAM technology that creates a map from an image captured by an imaging device and estimates the self-position, the imaging device becomes the position information acquisition device. In addition, when calculating the moving speed of the camera 8, it is sufficient to calculate based on the position and time detected by the position information acquisition device 9.
[0037] In this embodiment, the camera 8 is moved, but the camera 8 may be installed so that its angle can be changed so that its imaging direction can be changed, and the angle of the imaging direction may be detected by the position information acquisition device 9. In addition, if lighting and sunlight within the scrap yard 1 cause areas within the field of view of the image captured by the camera 8 to have large differences in brightness, a shadow-forming member or lighting member may be provided above the camera 8.
[0038] FIG. 2 is a diagram showing a hardware configuration of an image processing system according to an embodiment of the present invention. As shown in FIG. 2, the image processing system 10 has a computer 12 as an information processing device. The computer 12 includes a CPU 13, a GPU 18, a RAM 14, a ROM 15, and an interface 16. The RAM 14, the ROM 15, and the interface 16 are connected to the CPU 13 and the GPU 18 via a system bus 17. The interface 16 is connected to a camera 8, a position information acquisition device 9, and a display 7. In this embodiment, a stereo camera is used as the camera 8 to construct a three-dimensional shape of the scrap pile as described later. However, the present invention is not limited to this. A three-dimensional shape of the scrap pile may be constructed by using a plurality of general cameras 8 that capture two-dimensional images with one imaging unit and utilizing parallax due to stereoscopic vision of the cameras, or a three-dimensional shape of the scrap pile may be constructed by using a camera 8 that captures two-dimensional images with one imaging unit and an imaging distance information acquisition device such as a rangefinder or a TOF imaging device. In order to accurately identify scrap, the image captured by the camera 8 is preferably a color image. In this embodiment, the image processing system 10 is configured by a computer, but is not limited to this, and each function may be distributed using a plurality of terminals.
[0039] Fig. 3 is a diagram showing the software configuration of an image processing system according to an embodiment of the present invention. As shown in Fig. 3, the image processing system 10 includes an approximate equation creation unit 120, an image processing unit 140, a weight estimation unit 160, and an area information storage unit 130. These units are implemented by the CPU 13 and the GPU 18 executing a program recorded in the ROM 15.
[0040] The approximation formula creating unit 120 creates an approximation formula used when estimating the weight of the scrap pile based on the three-dimensional shape of the scrap pile.
[0041] The area information holding unit 130 holds map information of the scrap yard 1, information on each area 4, each partition wall 2, and camera 8. The map information of the scrap yard 1 includes the position and size (for example, the vertical and horizontal widths) of each area 4 and the position and size of the partition wall 2. The information of each area 4 includes the type of scrap to be piled up, the type of area 4 (for example, when a floor area and a container area are mixed), the camera 8 to take the image, and information on the presence or absence of a partition wall 2 that divides an area in which scrap is not piled up in the area 4. The information of the partition wall 2 includes height information. The information of the camera 8 includes the position of the camera 8 in the scrap yard 1, the camera height such as the vertical distance between the camera 8 and the bottom surface of the area 4, the moving speed or the upper limit of the moving speed of the camera 8, and the like. In the case where a plurality of cameras 8 are provided, it is preferable that the area 4 to be imaged is also held. The bottom surface of the area 4 is the floor surface of the area 4, and if the area 4 is a container, it corresponds to the bottom surface of the container. Furthermore, if the area 4 is a container, it is preferable to maintain either the vertical distance between the floor of the scrap yard 1 and the bottom of the container, or the vertical distance between the camera 8 and the bottom of the container.
[0042] In this embodiment, since a plurality of cameras 8 are provided, the areas that the cameras 8 correspond to are set. For example, the camera 8A is set to capture areas 4A, 4B, 4C, and 4D in the scrap yard 1 in FIG. 1, and the camera 8B is set to capture areas 4E and 4F. If the entire area of the area 4F is within the images of both the cameras 8A and 8B, the area 4F is set to be captured by either one of the cameras 8A and 8B. If the entire area of the area 4F is not within the images of either the cameras 8A or 8B, the area 4F may be set to be captured by both the cameras 8A and 8B, and the area 4F may be divided to correspond to each of the cameras 8A and 8B and subjected to image processing described later. Alternatively, after the image is acquired, the image or the 3D construction data of the scrap pile described later may be synthesized.
[0043] The image processor 140 processes the images captured by the camera 8 to obtain the three-dimensional shape of the scrap pile in each area.
[0044] The image processing unit 140 has a captured image acquiring unit 142, an imaging position information acquiring unit 144, a best shot image extracting unit 146, and a deposit identifying unit 147. The deposit identifying unit 147 has a partition wall / scrap extracting unit 148, a 3D shape constructing unit 150, a scrap sorting unit 151, and a target scrap identifying unit 152.
[0045] The captured image acquisition unit 142 can communicate with the camera 8 and acquires image data captured by the camera 8. The image capture position information acquisition unit 144 can communicate with the position information acquisition device 9 and acquires position information of the camera 8 at the time when the camera 8 captures an image. The image data acquired by the captured image acquisition unit 142 is recorded in association with the position information acquired by the image capture position information acquisition unit 144. In this embodiment, the horizontal position in FIG. 1 acquired by the position information acquisition device 9 is recorded in association, but the vertical position in FIG. 1, which is the position of the camera 8 on the girder 6, may also be recorded in association. In addition, corresponding imaging target areas may be set for each of the multiple cameras 8, and thus, when the position information of the image data acquired by the image capture position information acquisition unit 144 is outside the best shot area of all imaging target areas, the image data can be discarded, and the load on the computer of the best shot image extraction unit 146 described later can be reduced.
[0046] The best shot image extraction unit 146 has prerecorded map information in the scrap yard 1 and the best shot area of each area 4. FIG. 4 is a diagram showing map information in the scrap yard 1 and the best shot area of each area. As shown in the figure, a best shot area B is set in each area 4. The best shot area B is an area in which the image data captured by the camera 8 when the area is captured by the camera 8 does not include a blind spot of the scrap pile in the area 4, and the entire area is included in the image. In other words, it is an area in which the scrap pile S in the area 4 is not blocked by obstacles such as the partition wall 2 when the area is captured by the camera 8. The obstacles include not only the partition wall 2 but also the carrying-in and carrying-in jigs, scrap moving pipes or conveyors connected to the carrying-in and carrying-in jigs, carrying-in and carrying-in jigs (such as lifting magnets and arms or buckets having a function of grabbing piles) provided on the crane and extended below the crane, and piles of piles in other areas that are piled up high.
[0047] In addition, the best shot area B is set based on the size of the imaging area determined based on the installation height and viewing angle of the camera 8 and the size of the area 4, and is set so that the scrap pile S in the area 4 is not blocked by the partition wall 2 when the camera 8 captures the image. Furthermore, in this embodiment, the best shot area B of each area is set based on the position of the camera 8 on the girder 6. For example, in the case of areas 4A and 4E where the difference between the width of the image field of view in the moving direction of the camera 8 (the horizontal direction in FIG. 1 in this embodiment) and the width of the area in the image is small, the best shot area B is set near the center of the areas 4A and 4E, and in the case of areas 4B and 4C where the difference between the width of the image field of view in the moving direction of the camera 8 and the width of the area in the image is sufficiently large, the best shot area B is set to a predetermined range larger than the areas 4B and 4C. Note that, as an area where the entire area does not fit within the image field of view, for example, when the area is larger than the image field of view such as area 4D, multiple best shot areas B may be set for the area, or best shot areas B corresponding to each camera 8 may be set to combine images captured by multiple cameras 8.
[0048] In addition, the best shot area B is set so that the scrap pile S is not blocked by obstacles such as a carry-in / out jig between the camera 8 and the area 4, a scrap moving pipe connected to the carry-in / out jig, or a conveyor. In addition, when the camera 8 is installed at an angle such that the imaging unit and the floor are not parallel, the best shot area B is set in consideration of the occurrence of blind spots due to piles in the surrounding area. For example, the best shot area B is set assuming a state in which the scrap pile S piled in the surrounding area of the target area 4 is piled at a predetermined height higher than the height of the partition wall. In addition, the best shot area B may be set in consideration of the lighting and sunlight of the scrap yard 1, or the width of the best shot area B may be set to change with time in consideration of the time when a shadow is likely to appear in the scrap yard 1, or the best shot area B of an area in which the camera 8 does not move directly above at least a part of the area due to the movable area of the girder 6 may be set in a manner different from the setting method of the other best shot areas B.
[0049] If the camera 8 is configured to be movable in the vertical direction of Fig. 1, the best shot area can be set as a rectangular area with a predetermined length in the vertical direction and a predetermined width in the horizontal direction. If the camera 8 is installed so that the angle can be changed to change the imaging direction, the best shot area can be set as an angle.
[0050] The best shot image extraction unit 146 extracts images captured within the best shot region of each area from the image data acquired by the captured image acquisition unit 142. In this embodiment, since the camera 8 is a stereo camera, both two-dimensional images captured by the two imaging units are extracted.
[0051] The partition wall / scrap extraction unit 148 extracts the partition wall 2 and scrap from the image data extracted by the best shot image extraction unit 146. The partition wall / scrap extraction unit 148 is previously trained with training data on the metals constituting the scrap and the partition wall 2. This allows the partition wall 2 and scrap to be extracted from the best shot image. The partition wall / scrap extraction unit 148 can learn the metals constituting the scrap and the partition wall by recognizing the scrap and the partition wall, thereby reducing the possibility of erroneously recognizing the partition wall as scrap. In this embodiment, since the camera 8 is a stereo camera, there are two pieces of image data extracted by the best shot image extraction unit 146, but the partition wall 2 and scrap may be extracted from one or both pieces of image data, or scrap may be extracted from both pieces of image data and the partition wall 2 may be extracted from one piece of image data.
[0052] The three-dimensional shape constructing section 150 constructs three-dimensional shape data of the scrap pile from the image data extracted by the best shot image extracting section 146. In this embodiment, the camera 8 is a stereo camera, so the three-dimensional shape data is constructed from two two-dimensional images. However, when a camera that captures two-dimensional images with one imaging section and an imaging distance information acquisition device are used, the three-dimensional shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. The three-dimensional shape may only be the shape of the upper part of the scrap pile captured in the two-dimensional image. When three-dimensional shape data of the part below the top surface of the scrap pile that cannot be captured by the camera 8 is also constructed, it is sufficient to construct the data based on the vertical distance information between the camera 8 and the bottom surface of the area 4, which is stored in advance in the area information holding section 130. The three-dimensional shape data also includes height information of the scrap pile. In this embodiment, the height of the scrap pile is calculated from the vertical distance from camera 8 to the bottom of area 4 and the vertical distance from camera 8 to the scrap pile, both of which are stored in advance in area information storage unit 130, and is added to the three-dimensional shape data, but this is not limiting, and scrap pile height information may be obtained by an appropriate method without constructing a three-dimensional shape. Note that, when using a best shot image of the area 4 being a container, the height of the scrap pile may be calculated taking into account the height of the bottom of the container.
[0053] The scrap classification unit 151 classifies the scrap pile into areas 4. In this embodiment, the scrap pile is classified into areas 4 based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148 and the three-dimensional data of the scrap pile S constructed by the three-dimensional shape construction unit 150. When multiple scrap piles are extracted by the partition wall / scrap extraction unit 148, the scrap classification unit 151 judges whether the multiple scrap piles are scrap piles in the same area or scrap piles in different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. In addition, in order to detect that scraps in different adjacent areas have been extracted by the partition wall / scrap extraction unit 148 in a connected manner, the scrap classification unit 151 detects a portion in the scrap pile S where the height is significantly different (a portion larger than a predetermined size). In addition, in order for the partition wall / scrap extraction unit 148 to detect that scrap from different adjacent areas is extracted and connected across the partition wall 2, the scrap classification unit 151 may determine whether at least a portion of the partition wall is included inside the scrap pile, or may determine whether at least a portion of the scrap is included on the upper end of the partition wall. Furthermore, when the partition wall / scrap extraction unit 148 detects that scrap from different adjacent areas 4 is extracted and connected across the partition wall 2, the scrap classification unit 151 divides the three-dimensional shape data of the scrap pile S into areas.
[0054] The target scrap identifying unit 152 identifies the three-dimensional shape data of the scrap pile S in the target area 4 in the three-dimensional shape data.
[0055] The weight estimation unit 160 estimates the weight of the scrap pile S for each area 4 based on the three-dimensional shape of the scrap pile S for each area 4 acquired by the image processing unit 140.
[0056] Hereinafter, a method for identifying the scrap piled up in each area, acquiring the three-dimensional shape, and estimating the weight of the scrap using the image processing method of this embodiment will be described. In a conventional method for estimating the weight of scraps based on a three-dimensional shape, the volume of scraps is calculated based on the three-dimensional shape of the scraps, and the weight is calculated by multiplying the volume by a standard bulk density (assuming a uniform density). However, there is a problem that the weight is calculated by assuming a uniform density for the volume in this way, and the error from the actual weight is large. Figures 5A and 5B are diagrams for explaining the basic principle of the method for estimating the weight of scraps according to one embodiment of the present invention. In Figure 5A, the parts with high density are shown in dark colors. In Figure 5B, the parts with high average density when the bottom surface is divided by a predetermined area are shown in dark colors. As shown in Figure 5A, in the part with a high stacking height, the scraps are compressed and deformed by the weight of the scraps, and the density of the scrap pile is high. For this reason, the higher the stacking height, the smaller the gap between the individual scraps, and the larger the error between the weight calculated by multiplying the volume by the standard bulk density and the actual weight. Therefore, in this embodiment, based on the idea that the higher the stack height of the scrap, the higher the average density, the weight is estimated using a variable that increases as the stack height of the scrap increases, as shown in Fig. 5B. Specifically, an approximation formula for the variable considering the density relative to the stack height of the scrap is created, and the weight is estimated based on the reference weight set for each type of scrap and the variable calculated using this approximation formula.
[0057] Fig. 6 is a flow chart showing a method for estimating the weight of scrap according to one embodiment of the present invention. As shown in Fig. 6, the method of this embodiment includes an approximation formula creation step (S100) for creating an approximation formula of variables taking into account the density relative to the stack height of the scrap, a three-dimensional shape acquisition step (S200) for acquiring the three-dimensional shape of the scrap pile S piled in each area 4 of the scrap yard 1, and a weight estimation step (S300) for estimating the weight of the scrap based on the acquired three-dimensional shape of the scrap pile S. Each step will be described in detail below.
[0058] Fig. 7 is a flow chart showing a detailed flow in the approximation equation creation step. As shown in Fig. 7, in the approximation equation creation step S100, first, reference weight data is acquired for each type of scrap stored in the scrap yard (reference weight acquisition step: S110). The reference weight data is obtained by loading the target metal to a height of 1 m in a reference weight measurement container with a bottom dimension of 1 m x 1 m, and measuring the weight. This weight is regarded as the reference weight per unit volume. This reference weight is input to the approximation equation creation unit 120.
[0059] Next, the scrap is scanned with a 3D scanner to obtain shape information (S120). Specifically, for example, each type of scrap to be piled up in the scrap yard is scanned with a 3D scanner to obtain shape information of each individual scrap. When there are multiple different shapes for each individual scrap, or when the shapes are irregular enough to be classified as the same type under the condition that they are smaller than a specified size, shape information for any number of scraps is obtained. This shape information is input to the approximation formula creation unit 120.
[0060] Next, strength information of the scrap unit is acquired (S130). For example, the Young's modulus is calculated by compressing the scrap unit. If the metal constituting the scrap unit is known, the strength information of the scrap unit can use the physical property value of the metal. This strength information is input to the approximation formula creation unit 120.
[0061] Next, the weight of the scrap unit is measured to obtain weight information (S140). This weight information is input to the approximate formula creation unit 120. Note that the weight information of the scrap unit may be calculated from the weight and shape information of the metal per unit volume.
[0062] Next, the approximation formula creation unit 120 simulates the gaps and deformation formed between the scrap units when multiple scrap units are piled up based on the shape information, strength information, and weight information (S150). Specifically, a virtual container (bottom surface 1m x 1m) is simulated, scrap units are randomly placed in it, and based on the strength information and weight information of the scrap units, a simulation is performed to see how the scrap units are piled up in the container, thereby determining the gaps and deformation in the scrap pile. This simulation determines the weight of the scrap pile in the container and the height of the scrap pile. Note that a virtual container (bottom surface 1m x 1m) may be simulated, and the weight of the scrap pile when the height of the scrap pile in the container is 1m may be used as the reference weight.
[0063] The approximate equation creating unit 120 repeats this simulation under various conditions to obtain the relationship between the stack height of the scrap pile and the weight per unit area (S160).
[0064] Next, the approximation formula creation unit 120 obtains the relationship of the weight per unit area for each stacking height of the scrap pile obtained by simulation, and the estimated weight per unit area for each stacking height of the scrap pile without considering the density change from the reference weight. Then, the weight obtained by this simulation is divided by the estimated weight to obtain a density variable. Then, an approximation formula that approximates the relationship between the stacking height and the density variable is created from the value of the density variable for each stacking height (S170). FIG. 8 is a graph showing the approximation formula of the density variable. As shown in the figure, the density variable increases as the height increases, and the increase in the density variable per unit height decreases as the height increases. In addition, the weight per unit area for each stacking height of the scrap pile obtained by simulation may be used as a variable corresponding to each stacking height of the scrap pile, and an approximation formula that approximates the relationship between the stacking height and the weight per unit area may be created. Note that even if the weight per unit area is used as a variable, the weight increases as the height increases, and the increase in density per height decreases as the height increases.
[0065] In the above-mentioned approximate formula creation step S100, the approximate formula is determined by simulation, but the invention is not limited to this. Instead, an approximate formula may be created by repeatedly stacking actual scrap units without acquiring shape information or strength information of the scrap units, and determining the relationship between the stack height of the scrap pile and the weight per unit area. Also, the approximate formula created by the above-mentioned simulation may be corrected by comparing it with the results of an actual experiment.
[0066] Next, the three-dimensional shape acquisition step S200 will be described in detail. Fig. 9 is a flow chart showing a detailed flow of the three-dimensional shape acquisition step. The area information holding unit 130 stores in advance map information of the scrap yard, area information, and camera information. In addition, a best shot area is set for each area of the scrap yard, and the image processing unit 140 stores in advance information about the best shot area of each area.
[0067] In the three-dimensional shape acquisition step, first, the girder 6 and the lifting magnet or other scrap carrying jig extended downward from the camera 8 movably provided on the girder 6 are moved to an image capture start position (image capture preparation step S205). The image capture start position of the girder 6 may be a predetermined position, or, when at least one of the multiple areas 4 is set as a three-dimensional shape acquisition target area, may be a predetermined position around the target area. The image capture start position of the lifting magnet is outside the image capture area of the camera 8. If the lifting magnet is present in the image capture area, an image is captured with the lifting magnet present on the scrap pile S, and the deposits deposited in each deposit area cannot be accurately identified. In the image capture preparation step, the girder 6 and the lifting magnet may be automatically operated, or may be manually operated by the user. When the user manually operates in the image capture preparation step, the image capture start position is displayed on the display 7.
[0068] Next, the camera 8 is moved by moving the girder 6, while the camera 8 is imaged at a predetermined time interval (image capture step), image data is generated, and the image processing unit 140 acquires the generated image data by the image capture acquisition unit 142. (Image capture acquisition step: S210). Note that the image processing unit 140 may be configured to select a camera to capture among the multiple cameras 8 or an image capture target area among the multiple areas 4 when an image capture start instruction is issued from the user or the image processing unit 140, and may determine the camera to capture based on the selected camera or area. In parallel with this, the position information acquisition device 9 generates position information regarding the position where the image was captured by the camera 8 at the time of capturing the image by the camera 8 (position information generation step: S220). The image processing unit 140 acquires the position information generated by the image capture position information acquisition unit 144. The image processing unit 140 records the acquired image data in association with the position information. In this embodiment, the moving speed of the camera 8 is the moving speed of the girder 6, and the upper limit of the moving speed is preferably a speed at which no blurring occurs in the image due to the imaging interval and the performance of the camera 8. The moving speed of the camera 8 can be calculated from the acquisition time of the position information acquired by the imaging position information acquisition unit 144. The operation of moving the girder 6 in the imaging step may be performed automatically or manually by the user. When the user manually operates the girder 6 in the imaging step, the appropriate moving speed of the girder 6 and the current moving speed of the girder 6 may be displayed on the display 7, or when the moving speed of the girder 6 exceeds the upper limit, a deceleration operation instruction or an imaging error may be displayed on the display 7.
[0069] Next, the image processing unit 140, by using the best shot image extraction unit 146, acquires image data (hereinafter referred to as best shot image) whose captured position is included in the best shot area based on the position information by referring to the recorded image data for each area 4 (best shot image extraction step: S230). The number of best shot images acquired for each area 4 is preferably one or more, and multiple images may be acquired so that the width intervals of the best shot areas are equal based on the position information, or multiple images may be acquired so that the intervals between images are equal, or the number of images may be set according to the size and number of the best shot areas. Note that the best shot image extraction step is not limited to this, and image data may be thinned out and extracted from image data captured at a predetermined time interval so that the intervals are longer than the predetermined time interval, and it is determined whether the position information of the thinned out and extracted image data is included in the best shot area of any of the areas 4, and if it corresponds to the best shot area of any of the areas 4, it may be acquired as the best shot image.
[0070] Next, the image processing unit 140 extracts the partition walls 2 and scraps in the best shot image by the partition wall / scrap extraction unit 148 (scrap extraction step: S240). In the scrap extraction step 240, the partition walls 2 separating the areas 4 in the best shot image and the scraps are extracted by a neural network. The partition wall / scrap extraction unit 148 is previously trained with training data on the metals constituting the scraps and the partition walls 2. This makes it possible to extract the partition walls 2 and scraps in the best shot image. If the partition wall / scrap extraction unit 148 does not detect any metal components in all the best shot images of the target area, it determines that the target area is empty of scraps. In addition, before, after, or at the same time as the scrap extraction step, it is preferable that the image processing unit 140 detects a jig for carrying in and out scraps, such as a lifting magnet, in the best shot image. For example, the lifting magnet is detected by a neural network. If a lifting magnet is detected in the best shot image (particularly preferably in the target area), the 3D shape acquisition step for the best shot image or the partition wall 2 and scrap extracted from the best shot image is terminated, and an imaging error is displayed on the display 7.
[0071] Next, the image processing unit 140 constructs a three-dimensional shape of the scrap pile in the best shot image by the three-dimensional shape construction unit 150 (three-dimensional shape construction step: S250). In this embodiment, the camera 8 is a stereo camera, so that three-dimensional shape data is constructed from two two-dimensional images. For example, when a camera that captures two-dimensional images by one imaging unit and an imaging distance information acquisition device are used, the three-dimensional shape data may be constructed based on the image data captured by the camera and the distance to the scrap pile measured by the imaging distance information acquisition device. At this time, the three-dimensional shape data includes height information of the scrap pile. The height of the scrap pile is calculated from the vertical distance from the camera 8 to the bottom surface of the area 4, which is stored in advance, and the vertical distance from the camera 8 to the scrap pile. The three-dimensional shape construction step S250 corresponds to a scrap height information acquisition step.
[0072] Next, the image processing unit 140 divides the scrap piles into areas by the scrap dividing unit 151 (scrap dividing step: S260). Specifically, as shown in Fig. 10, when the constructed 3D shape data includes 3D shapes of multiple scrap piles S, it is determined whether the multiple scrap piles S are scrap piles S in the same area or scrap piles S in different areas based on the partition walls 2 extracted by the partition wall / scrap extraction unit 148. For example, when a partition wall 2 is extracted between multiple scrap piles S, it can be determined that the multiple scrap piles S are scrap piles S in different areas. The method of determining whether multiple scrap piles S are in the same area or different areas based on the partition walls 2 is not limited to this. For example, in the case of an area 4 whose periphery is entirely surrounded by partition walls 2, the upper end of the partition wall 2 may be extracted as a straight line component, a polygon surrounded by the straight line component may be detected, and scrap in the polygon may be determined as scrap piles S in the same area. Furthermore, in the case of multiple areas 4 whose periphery is entirely surrounded by partition walls 2, scrap in an area in which a polygon cannot be formed by the straight line components of the upper end of the partition wall (one side of the polygon cannot be detected) may be determined as scrap outside the target area in the target scrap identification step S270 described later. In this embodiment, whether multiple scrap piles S are in the same area or different areas is determined based on the partition walls 2, but the method is not limited to this, and the determination may be based on the three-dimensional shape of the scrap piles S. In this case, it may be based on any of the height, shape, and distance between the scrap piles S of the three-dimensional shape of the scrap piles S.
[0073] Furthermore, the scrap sorting unit 151 detects locations in the scrap pile S where the height varies greatly (locations larger than a predetermined size) based on the height of the three-dimensional shape of the scrap pile S. When such locations where the height of the scrap pile varies greatly are detected, there is a high possibility that the scrap extracted by the partition wall / scrap extraction unit 148 is a combination of scrap from different adjacent areas. Therefore, when the scrap sorting unit 151 detects locations where the height varies greatly in the scrap pile S, it divides the scrap pile S using the locations where the height varies greatly as boundaries.
[0074] Further, the scrap division unit 151 determines whether at least a part of the partition wall 2 is included inside the scrap pile S. When at least a part of the partition wall is included inside such a scrap pile S, there is a high possibility that the scrap extracted by the partition wall / scrap extraction unit 148 is scrap from different adjacent areas that are connected across the partition wall and extracted. Therefore, when the scrap division unit 151 determines that at least a part of the partition wall 2 is included inside the scrap pile S, the scrap division unit 151 connects the partition wall 2 to create a boundary line and divides the scrap pile S. Note that the upper end of the partition wall 2 may be recognized, a straight line component may be extracted from the upper end of the partition wall 2, and the boundary line may be created by connecting the straight line component, or the junction of the partition walls 2 at the upper end of the partition wall 2 may be recognized as a corner of the partition wall 2, and the corners may be connected with a straight line to create a boundary line. Furthermore, if the area of the best shot image is recorded in the area information storage unit 130 as having a partition wall 2 that partitions an area (e.g., an aisle) where metal scrap is not accumulated inside the area, the scrap pile S is not divided even if a part of the partition wall 2 is recognized inside the scrap pile S. Note that the scrap sorting unit 151 is not limited to the method of determining whether at least a part of the partition wall 2 is included inside the scrap pile S, and may use a method of determining whether at least a part of the scrap is included on the upper end of the partition wall 2. Through the above steps, the three-dimensional shape of the scrap piled up in each area of the scrap yard can be obtained. Note that the scrap classification step S260 is preferably performed after the three-dimensional shape construction step S250, but at least a part of the scrap classification step S260 may be performed before the three-dimensional shape construction step S250.
[0075] Next, the image processing unit 140 uses the target scrap identifying unit 152 to identify only the scraps in the target area (target scrap identifying step: S270). Specifically, as shown in FIG. 10, when the constructed three-dimensional shape data includes three-dimensional shapes of multiple scrap piles S, the positions of one or both of the target area and the scraps are estimated from the position information recorded in association with the image data acquired by the captured image acquiring unit 142, and only the three-dimensional shape of the scraps in the target area is extracted from the scrap pile S of the constructed three-dimensional shape data. Note that the target scrap identifying step S270 is not limited to this, and when the position of the target area in the captured image is recorded in advance, only the three-dimensional shape of the scraps in the target area may be extracted from the scrap pile S of the constructed three-dimensional shape data without using the position information recorded in association with the image data. Note that the scrap classification step S260 and the target scrap identifying step S270 may operate simultaneously, and the functions of the scrap classification unit 151 and the target scrap identifying unit 152 may be integrated.
[0076] Next, a description will be given of the weight estimation step S300 in which the weight estimation unit 160 estimates the weight of the scrap based on the three-dimensional shape of the scrap acquired by the image processing unit 140. Fig. 11 is a flowchart showing a detailed flow of the weight estimation step.
[0077] First, the weight estimation unit 160 divides the three-dimensional shape of the scrap pile in the target area 4. Fig. 12 is a diagram showing how the three-dimensional shape of the scrap pile is divided. As shown in the figure, the scrap pile is divided vertically into, for example, a 15 cm x 15 cm grid (division step: S310).
[0078] Next, the weight estimation unit 160 estimates the height of each point (each grid) of the three-dimensional shape of the scrap from the bottom surface of the area 4. For example, the average value of the height coordinates within each point (each grid) may be used as the height of each point. Depending on the size of the grid, the height coordinate at the center point of the grid may be used. (Height estimation step: S320)
[0079] Next, the weight estimation unit 160 estimates the weight of each point using an approximation equation of a variable that takes into account the density relative to the stacking height (weight calculation step: S330). Specifically, when a density variable is used as the variable, the weight of each point is calculated by the following equation. Weight = Base area x Height x Reference weight x Density variable corresponding to height In the above formula, the area of the base is the area of the grid divided in the division step. The height is the height estimated in the height estimation step. The reference weight may be the one acquired in the reference weight acquisition step S110. Also, when using weight per unit area as a variable, the weight at each point is calculated using the following formula. Weight = Weight per unit area corresponding to base area x height In the above formula, the area of the base is the area of the grid divided in the division step. In either case, the variable corresponding to the height may be calculated by substituting the height into the approximation formula created in the approximation formula creating step S100.
[0080] Next, the weight estimation unit 160 estimates the weight of the scrap in the area 4 (total weight calculation step: S340). Specifically, the weight of each grid is added up to estimate the weight of the scrap pile. In this way, the weight estimation unit 160 estimates the weight of the scrap pile in each area 4, and outputs the result to the display 7 or the like.
[0081] Note that the weight estimation step S300 is not limited to the above method, and for example, the areas of the points with the same height in the three-dimensional shape of the scrap pile may be summed up and the weight calculation step S330 may be performed for each height without performing the division step S310. In this case, the weight of each height is calculated by the following formula. Weight = Total base area x height x base weight x density variable corresponding to height Next, in a total weight calculation step, the weights at each height are added together to estimate the weight of the scrap pile.
[0082] According to this embodiment, the following effects are achieved. The image processing method of this embodiment includes a scrap extraction step S240 for extracting scrap from a best shot image captured within a best shot area. This prevents the scrap pile S from being in a blind spot due to an obstacle such as the partition wall 2, making it possible to capture an image of the entire scrap pile S and accurately identify the pile.
[0083] Furthermore, in the image processing method of this embodiment, the best shot area B is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the camera 8, and the size of the area 4. In this way, since the best shot area is set based on the size of the imaging area and the size of the area 4, it is possible to capture an image of the scrap pile S from the imaging position of the camera 8 without it being in the blind spot of an obstacle such as the partition wall 2, and the pile can be accurately identified.
[0084] The image processing method of this embodiment includes an imaging step of capturing image data in the scrap yard 1 while moving the camera 8, a position information generating step S220 of generating the position captured by the camera 8, and a best shot image extracting step S230 of extracting the best shot image captured in the best shot area based on the position information. When capturing images while moving the camera 8, the captured image data becomes huge, and processing all the image data places a heavy load on the computer. In contrast, according to this embodiment, image data captured in the best shot area is extracted from the captured image data, so the load on the computer can be reduced. Also, each area 4 in the scrap yard 1 can be captured even with a small number of cameras.
[0085] The image processing method of this embodiment further includes a target scrap identification step S270 for identifying only the scrap pile S in the target area 4 from the scrap piles S extracted in the scrap extraction step S240, and the target scrap identification step S270 extracts only the scrap pile S in the target area 4 in the best shot image based on the position information. This makes it possible to identify only the scrap pile S in the target area 4 based on the position information, and to extract the scrap pile S separately from the scrap piles S in other areas 4.
[0086] Moreover, the image processing method of this embodiment further includes a scrap classification step S260 in which the scrap pile S extracted in the scrap extraction step S240 is classified into areas 4. As a result, even if an image is captured at an imaging position where a plurality of areas 4 are included in the image, by classifying the scrap pile S into areas 4, it is possible to prevent scrap piles S piled up in different areas 4 from being identified as scrap piles S in the same area 4, and the piles can be accurately identified.
[0087] In the image processing method of this embodiment, in the scrap extraction step S240, the partition wall 2 is extracted from the captured best shot image. As a result, even if there are adjacent areas 4 in the scrap yard 1, the partition wall 2 separating the adjacent areas 4 can be recognized by extracting the partition wall 2 and the scrap, and it is possible to prevent the scrap piles S piled up in the adjacent areas 4 from being identified as scrap piles S in the same area 4, and the piles can be accurately identified.
[0088] Furthermore, in the image processing method of this embodiment, in the scrap division step S260, it is determined whether at least a portion of the partition wall 2 is included inside the scrap pile S, and if it is determined that at least a portion of the partition wall 2 is included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted as a whole in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether a portion of the partition wall 2 is included inside the scrap pile S, and the scrap pile S can be divided into each area 4.
[0089] Furthermore, in the image processing method of this embodiment, in the scrap division step S260, it is determined whether at least a portion of the scrap is included on the upper end of the partition wall 2, and if it is determined that at least a portion of the scrap is included on the upper end of the partition wall 2, the scrap pile S is divided. As a result, even if the scrap piles S of multiple areas 4 are extracted as a whole in the scrap extraction step S240, it is possible to recognize the partition walls 2 that separate the areas 4 by determining whether at least a portion of the scrap is included on the upper end of the partition wall 2, and the scrap piles S can be divided for each area 4.
[0090] The image processing method of this embodiment also includes a three-dimensional shape construction step S250 in which a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240 is constructed, and in the scrap classification step S260, the scrap pile S is classified into areas 4 based on the height of the three-dimensional shape of the scrap pile S. In this way, by basing the classification on the height of the three-dimensional shape of the scrap pile S, the difference in height between the scrap piles S in multiple areas 4 can be utilized, and the scrap pile S can be classified into areas 4.
[0091] The image processing method of this embodiment further includes a three-dimensional shape construction step S250 for constructing a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. In this way, the three-dimensional shape of the scrap pile S can be constructed, and the weight of the scrap pile S can be estimated.
[0092] Furthermore, in the image processing method of this embodiment, in the weight estimation step S300, the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S differs, the scrap pile S is compressed by its own weight, resulting in a different density. However, according to this embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so that the weight of the scrap pile S can be estimated more accurately.
[0093] Furthermore, according to this embodiment, the variable corresponding to the height is larger as the height increases. As the height of the scrap pile S increases, the scrap pile S is compressed by its own weight, and the density increases. However, according to this embodiment, since a variable that increases as the height increases is used, the weight of the scrap pile S can be estimated more accurately.
[0094] Furthermore, according to this embodiment, the variable corresponding to height has a smaller increase per unit height as the height increases. As the height of the scrap pile S increases, the pile is compressed by its own weight, and the density increases, but the increase becomes smaller as the height increases. According to this embodiment, the variable that has a smaller increase per unit height as the height increases can be used, so that the weight of the scrap pile S can be estimated more accurately.
[0095] Furthermore, in the weight estimation step S300 of the image processing method of this embodiment, the three-dimensional shape of the scrap pile S is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part to estimate the weight of the scrap pile S. In this way, the three-dimensional shape of the scrap pile S is divided into multiple parts in the vertical direction, and the weight of each part is calculated based on the height of each divided part, so that the weight of the scrap pile S can be estimated more accurately.
[0096] In the present embodiment, an image processing method in which at least a portion of the multiple areas 4 are included in an image captured by a camera 8 in a scrap yard 1 having multiple areas 4 at least a portion of which are partitioned by partition walls 2 and in which scrap piles S are piled up, includes an image acquisition step S210 for acquiring an image captured by the camera 8, a scrap extraction step S240 for extracting scrap piles S in the image acquired in the image acquisition step S210, and a scrap classification step S260 for classifying the scrap piles S extracted in the scrap extraction step S240 into areas 4. In this way, by extracting scrap piles S from an image captured at an imaging position in which multiple areas 4 are included in the image and classifying the extracted scrap piles S into areas 4, it is possible to prevent scrap piles S piled up in different areas 4 from being identified as scrap piles S in the same area 4, and to accurately identify piles.
[0097] In the present embodiment, in a scrap yard 1 having a plurality of areas 4 at least a portion of which is partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes a scrap extraction step S240 in which the partition walls 2 in the captured best shot image are extracted. As a result, even if there are adjacent areas 4 in the scrap yard 1, the partition walls 2 separating the adjacent areas 4 can be recognized by extracting the partition walls 2 and the scrap, and it is possible to prevent the scrap piles S piled up in the adjacent areas 4 from being identified as scrap piles S in the same area 4, and the piles can be accurately identified.
[0098] In the present embodiment, in a scrap yard 1 having a plurality of areas 4 at least a portion of which are partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes a scrap division step S260 in which it is determined whether at least a portion of the partition walls 2 is included inside the scrap pile S, and if it is determined that at least a portion of the partition walls 2 is included inside the scrap pile S, the scrap pile S is divided. As a result, even if the scrap piles S of the plurality of areas 4 are extracted as a whole in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether a portion of the partition walls 2 is included inside the scrap pile S, and the scrap piles S can be divided into each area 4.
[0099] In the present embodiment, in a scrap yard 1 having a plurality of areas 4 at least a portion of which are partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes a scrap division step S260 in which it is determined whether at least a portion of the scrap is included on the upper ends of the partition walls 2, and if it is determined that at least a portion of the scrap is included on the upper ends of the partition walls 2, the scrap piles S are divided. As a result, even if the scrap piles S of the plurality of areas 4 are extracted as a whole in the scrap extraction step S240, it is possible to recognize the partition walls 2 separating the areas 4 by determining whether at least a portion of the scrap is included on the upper ends of the partition walls 2, and the scrap piles S can be divided for each area 4.
[0100] In the present embodiment, the image processing method in which at least a portion of the scrap yard 1 having a plurality of areas 4 in which scrap piles S are piled up, at least a portion of which is partitioned by partition walls 2, is included in an image captured by a camera 8 further includes a three-dimensional shape construction step S250 for constructing a three-dimensional shape of the scrap pile S extracted in the scrap extraction step S240, and in the scrap classification step S260, the scrap pile S is divided into areas 4 based on the height of the three-dimensional shape of the scrap pile S. In this way, by using the height of the three-dimensional shape of the scrap pile S as the basis, the difference in height between the scrap piles S in the plurality of areas 4 can be utilized, and the scrap pile S can be divided into areas 4.
[0101] In the present embodiment, the image processing method in which at least a portion of the scrap yard 1 having a plurality of areas 4 in which scrap piles S are piled up, at least a portion of which is partitioned by partition walls 2, is included in an image captured by a camera 8, further includes a weight estimation step S300 for estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S. Thus, by estimating the weight of the scrap pile S based on the three-dimensional shape of the scrap pile S divided into areas 4, it is possible to estimate the weight according to the type of scrap, and to accurately estimate the weight of the pile.
[0102] In the present embodiment, in a scrap yard 1 having a plurality of areas 4 at least a portion of which is partitioned by partition walls 2 and in which scrap piles S are piled up, an image processing method in which at least a portion of the plurality of areas 4 is included in an image captured by a camera 8 includes a weight estimation step S300 in which the weight of the scrap pile S is calculated based on a variable corresponding to the height of the scrap pile S. If the height of the scrap pile S differs, the scrap pile S is compressed by its own weight and thus has a different density. However, according to the present embodiment, the weight of the scrap pile S is calculated based on a variable corresponding to the height, so that the weight of the scrap pile S can be estimated more accurately.
[0103] In this embodiment, the image processing method for capturing an image of the inside of the scrap yard 1 having the area 4 where the scrap pile S is piled up by the camera 8 includes a captured image capturing step S210 for capturing an image captured by the camera 8, a scrap extraction step S240 for extracting the scrap pile S in the image captured in the captured image capturing step S210, a scrap height information capturing step for capturing the height of the scrap pile S extracted in the scrap extraction step S240, and a weight estimation step S300 for calculating the weight of the scrap pile S based on the height of the scrap pile S captured in the scrap height information capturing step and a variable corresponding to the height of the scrap pile. If the height of the scrap pile S differs, the scrap pile S is compressed by its own weight, and therefore the density differs. However, according to this embodiment, the scrap pile S is extracted from the captured image, and the height of the scrap pile S extracted from the captured image is captured, so that the weight of the scrap pile S can be calculated based on the variable corresponding to the height of the scrap pile S, and the weight of the pile can be accurately estimated.
[0104] In addition, in this embodiment, the image processing method in which the inside of the scrap yard 1 having the area 4 where the scrap pile S is piled up is imaged by the camera 8, constructs a three-dimensional shape of the scrap pile S in the scrap height information acquisition step, and estimates the weight of the scrap pile S by dividing the three-dimensional shape of the scrap pile S into a plurality of parts in the vertical direction and calculating the weight of each part based on the height of each divided part and a variable corresponding to the height of the scrap pile S in the weight estimation step S300. In this way, since the three-dimensional shape of the scrap pile S is divided into a plurality of parts in the vertical direction and the weight of each part is calculated based on the height of each divided part and a variable corresponding to the height of the scrap pile S, the weight of the scrap pile S can be estimated more accurately.
[0105] In the present embodiment, in a scrap yard 1 having a plurality of areas 4 in which scrap piles S are piled up, at least a portion of which is partitioned by partition walls 2, an image captured by a camera 8 includes at least a portion of the plurality of areas 4, or an image processing method for capturing an image of a scrap yard 1 having an area 4 in which scrap piles S are piled up by a camera 8, a predetermined best shot area is set for the area 4 in which the camera 8 can capture an image of the area 4 without blocking the area 4 with an obstacle, and in the scrap extraction step S240, the scrap pile S is extracted from the best shot image captured within the best shot area. By setting the best shot area in this way, the scrap pile S is not in the blind spot of obstacles such as the partition walls 2, so that the entire scrap pile S can be captured and the pile can be accurately identified. [Explanation of symbols]
[0106] 1: Scrap yard 2: Partition wall 3: Exterior wall 4: Area 5: 3D shape of the constructed scrap pile 6: Guarda 7: Display 8: Camera 9: Location information acquisition device 10: Image processing system 12: Computer 13: CPU 14: RAM 15: ROM 16: Interface 17: System bus 18: GPU 100: Image processing system 120: Approximation formula creation section 130: Area information storage unit 140: Image processing unit 142: Image acquisition unit 144: Imaging position information acquisition unit 146: Best shot image extraction section 147: Sediment Identification Section 148: Partition wall / scrap extraction section 150:3D shape construction part 151: Scrap sorting section 152: Target scrap specific part 160: Weight estimation section B: Best shot area S: Scrap pile
Claims
1. A method for processing an image captured by an imaging device inside a deposit storage facility, the deposit storage facility having a plurality of deposit areas at least a portion of which is partitioned by partition walls and in which deposits are deposited, comprising the steps of: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being blocked by an obstacle; An imaging step of capturing image data of the inside of the deposit storage facility while moving or changing the angle of the imaging device; a position information generating step of generating position information relating to a position or angle at which an image is captured by the imaging device; a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information; and extracting the deposit from a best shot image captured within the best shot area.
2. The best shot area is set based on the size of the imaging area, which is determined based on the imaging height and viewing angle of the imaging device, and the size of the deposition area. The image processing method according to claim 1 .
3. A target deposit identification step is further provided for identifying only the deposit in a target deposit area from the deposits extracted in the deposit extraction step, The target deposit identification step extracts only the deposit in a target deposit area in the best shot image based on the position information. The image processing method according to claim 1 .
4. The method further includes a step of dividing the deposits extracted in the deposit extraction step into the deposit regions. The image processing method according to claim 1 .
5. The deposit extraction step includes extracting the partition wall from the captured best shot image. The image processing method according to claim 4.
6. A method for processing an image captured by an imaging device inside a deposit storage facility, the deposit storage facility having a plurality of deposit areas at least a portion of which is partitioned by partition walls and in which deposits are deposited, comprising the steps of: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being blocked by an obstacle; a deposit extraction step of extracting the deposit in a best shot image captured within the best shot area; A sediment classification step, The pile classification step includes determining whether at least a part of the partition wall is included inside the pile; When it is determined that at least a part of the partition wall is included inside the pile, the pile is divided. Image processing methods.
7. A method for processing an image captured by an imaging device inside a deposit storage facility, the deposit storage facility having a plurality of deposit areas at least a portion of which is partitioned by partition walls and in which deposits are deposited, comprising the steps of: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being blocked by an obstacle; a deposit extraction step of extracting the deposit in a best shot image captured within the best shot area; A sediment classification step, The step of classifying the deposit includes determining whether at least a portion of the deposit is included on an upper end of the partition wall; dividing the pile when it is determined that at least a portion of the pile is included on the upper end of the partition wall; Image processing methods.
8. A three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step is further provided, The step of dividing the sediment comprises: Dividing the deposit into the deposit regions based on the height of the three-dimensional shape; The image processing method according to claim 4.
9. Further, a three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step; a weight estimation step of estimating a weight of the deposit based on the three-dimensional shape; Equipped with The image processing method according to any one of claims 1 to 7.
10. In the weight estimation step, a weight of the pile is calculated based on a variable corresponding to a height of the pile. The image processing method according to claim 9.
11. A system for processing an image captured by an imaging device inside a deposit storage facility having a plurality of deposit areas, at least a portion of which is partitioned by partition walls, in which deposits are deposited, comprising: The imaging device captures image data of the inside of the deposit storage facility while moving or changing an angle, a position information generating unit that generates position information related to a position or angle captured by the imaging device; a best shot image extraction unit that sets a predetermined best shot area in the pile area where the image can be captured by the image capture device without being blocked by an obstacle, and extracts a best shot image captured within the best shot area based on the position information; an image processing system including a deposit extraction unit that identifies the deposit in a best shot image captured within the best shot area and extracts the deposit;
12. A program for processing images captured by the imaging device of the inside of a sediment storage facility, the inside of which is at least partially partitioned by partition walls and has a plurality of deposition areas in which sediments are deposited, the program comprising: an imaging device which captures image data of the inside of a sediment storage facility while changing its movement or angle; a position information acquisition unit which generates position information relating to the position or angle captured by the imaging device; and a computer capable of communicating with the program, the program comprising: a predetermined best shot area is set for the accumulation area, where the accumulation area can be imaged by the imaging device without being blocked by an obstacle; a best shot image extraction step of extracting a best shot image captured within the best shot area based on the position information; a deposit extraction step of identifying the deposit in a best shot image captured within the best shot area and extracting the deposit; A program that causes a computer to execute the following.
13. A method for processing an image captured by an imaging device in a deposit storage facility having a plurality of deposit areas, at least a portion of which is partitioned by partition walls and in which deposits are deposited, the method comprising the steps of: a captured image acquisition step of acquiring an image captured by the imaging device; a deposit extraction step of extracting the deposit from the image acquired in the image acquisition step; A deposit classification step of classifying the deposits extracted in the deposit extraction step into the deposit regions, The image processing method includes, in the deposit division step, detecting whether the deposits in adjacent different deposit areas are extracted as connected, and dividing the deposits when it is detected that the deposits in adjacent different deposit areas are extracted as connected.
14. The deposit extraction step extracts the partition wall from the image acquired in the image acquisition step. The image processing method according to claim 13.
15. A method for processing an image captured by an imaging device in a deposit storage facility having a plurality of deposit areas, at least a portion of which is partitioned by partition walls and in which deposits are deposited, the method comprising the steps of: a captured image acquisition step of acquiring an image captured by the imaging device; a deposit extraction step of extracting the deposit from the image acquired in the image acquisition step; A deposit classification step of classifying the deposits extracted in the deposit extraction step into the deposit regions, The deposit extraction step includes extracting the partition wall from the image acquired in the image acquisition step, The pile classification step includes determining whether at least a part of the partition wall is included inside the pile; The image processing method further comprises dividing the pile when it is determined that at least a portion of the partition wall is included inside the pile.
16. A method for processing an image captured by an imaging device in a deposit storage facility having a plurality of deposit areas, at least a portion of which is partitioned by partition walls and in which deposits are deposited, the method comprising the steps of: a captured image acquisition step of acquiring an image captured by the imaging device; a deposit extraction step of extracting the deposit from the image acquired in the image acquisition step; A deposit classification step of classifying the deposits extracted in the deposit extraction step into the deposit regions, The deposit extraction step includes extracting the partition wall from the image acquired in the image acquisition step, The step of classifying the deposit includes determining whether at least a portion of the deposit is included on an upper end of the partition wall; dividing the pile when it is determined that at least a portion of the pile is included on the upper end of the partition wall; Image processing methods.
17. A three-dimensional shape construction step of constructing a three-dimensional shape of the deposit extracted in the deposit extraction step is further provided, The step of dividing the pile includes dividing the pile into pile regions based on the height of the three-dimensional shape.
17. The image processing method according to claim 13, 15 or 16.
18. The method further includes a weight estimation step of estimating a weight of the pile divided in the pile dividing step based on the three-dimensional shape of the pile. The image processing method according to claim 17.
19. The weight estimation step calculates a weight of the pile based on a variable corresponding to a height of the pile.
20. The image processing method according to claim 18.
Citation Information
Patent Citations
Image processing device, image processing method and image processing program
JP2006322795A
Scrap remaining amount detection system in scrap yard
JP2007197170A
Range finder
JP2009014444A
Raw material heap measuring system of raw material yard, raw material heap measuring method of raw material yard and computer program
JP2011157187A
Three-dimensional management method of raw material yard
JP2012082033A
