Suspended load image detection device, work machine, and suspended load image detection method

Abstract

The system accurately detects images of suspended loads from video footage captured by the camera. [Solution] The suspended load image detection device includes a controller. The controller extracts all object regions R by performing image processing based on the Segment Anything Model on multiple frame images G. The controller identifies a pair of object regions R at corresponding positions in all adjacent frame image pairs in time series and calculates the inclusion rate of both using equation (1). If the calculated inclusion rate value is greater than or equal to a predetermined value, the controller recognizes that there is an inclusion relationship and identifies the sequence S1 to S3 of object regions R for which the inclusion relationship has been maintained for the longest time. The controller detects the image region of the suspended load based on the multiple object regions R that constitute the sequence S1 to S3 of the identified object regions R. Inclusion rate=Area(A∩B) / min(Area(A),Area(B))…Formula (1)

Description

Technical Field

[0001] The present invention relates to a suspended load image detection device, a work machine, and a suspended load image detection method.

Background Art

[0002] Conventionally, a control device that captures an image of a subject including a suspended load of a crane using a camera and performs safety control of the crane based on the captured image is known (see, for example, Patent Document 1).

[0003] This control device has a controller to which an image captured by a camera is input. The controller recognizes an image of an operator included in the image input from the camera and an image of a hook for suspending a suspended load. The controller executes this recognition process by image processing based on machine learning. After recognizing the images of the operator and the hook, the controller calculates the coordinate position of the operator with respect to the hook as the origin, and calculates the distance from the hook to the operator based on the calculated coordinate position. Then, the controller performs safety control of the crane based on the calculated distance and the maximum standard size of the suspended load.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique disclosed in Patent Document 1, safety control is performed to avoid interference between the suspended load and the operator based on the image of the hook shown in the image. However, in order to further improve safety, it is conceivable to detect an image of the suspended load itself and calculate the distance between the suspended load and the operator.

[0006] However, the shape and dimensions of loads suspended by cranes are not uniform but diverse. Therefore, it is difficult to identify images of suspended loads in video footage simply based on pattern matching or machine learning.

[0007] This invention was made to solve the above-mentioned problems, and aims to accurately detect images of suspended loads from video footage captured by a camera. [Means for solving the problem]

[0008] The first invention relates to a suspended load image detection device which detects an image region of a suspended load of a work machine from an image of a subject including the suspended load of the work machine captured by a camera at a predetermined frame rate, and comprises a controller, the controller performing a frame image extraction process which extracts a plurality of frame images arranged in chronological order from the image captured by the camera, and a SAM (Segment Anything) process for each of the plurality of frame images. The system is configured to perform the following: an object region extraction process that extracts all object regions contained in each frame image by performing image processing based on the Model; an inclusion relationship recognition process that calculates the inclusion rate of one object region to one object region of a pair of object regions located at corresponding positions in all pairs of frame images adjacent to each other in time series using the following formula (1), and recognizes that one object region and the other object region are in an inclusion relationship if the calculated inclusion rate is greater than or equal to a predetermined value; a sequence identification process that identifies the sequence of object regions in which the inclusion relationship is maintained for the longest time between the first frame image and the last frame image in time series, based on the result of the inclusion relationship recognition process; and a detection process that detects the image region of the suspended load based on the sequence of object regions that constitute the sequence of object regions identified by the sequence identification process.

[0009] Coverage = Area(A∩B) / min(Area(A),Area(B)) ... Equation (1) Here, Area A represents one of the pair of object areas, and Area B represents the other of the pair of object areas. Area represents area, where the numerator Area(A∩B) represents the area of ​​the overlapping portion of Area A and Area B when the frame image pair is arranged so that their fields of view match, and the denominator min(Area(A),Area(B)) represents the area of ​​the smaller of Area A and Area B.

[0010] This configuration allows for accurate detection of images of suspended loads from video footage captured by the camera.

[0011] In other words, because the external shapes of crane loads are extremely diverse, it is difficult to learn (detect) them as a single or several classes using existing object detection processes. In contrast, the above configuration uses a Segment Anything Model (SAM) to extract object regions (regions that may contain objects, and are not necessarily regions that indicate objects, as they may be shadow regions, for example) from the entire frame image, regardless of the object class. From the extracted object regions, images that satisfy the conditions for a crane load can be detected as crane load images.

[0012] However, in object extraction using SAM, a single object may be extracted as multiple object regions (over-fractionation extraction), or conversely, a region covering multiple objects (or things that look like objects, such as shadows) may be extracted as a single object region, or a single object may be extracted as multiple object regions with overlapping parts (overlap extraction). In addition, when extracting object regions with SAM, there is a problem that the extraction results can change even in images of the same scene due to subtle changes in the image. Specifically, for example, an object region that was extracted in one frame image may not be extracted in the next frame image. Therefore, in order to detect suspended loads using SAM, it is necessary to deal with such unstable extraction results.

[0013] In contrast, the above configuration calculates the inclusion rate of a pair of object regions at corresponding positions for all adjacent frame image pairs in multiple frame images, rather than a single frame image, based on equation (1). If the calculated inclusion rate is a predetermined value, it is recognized that an inclusion relationship exists. The sequence of object regions that maintains this inclusion relationship for the longest time is identified, and the image region of the suspended load is detected based on the object regions belonging to this identified sequence. This avoids the problem of instability in object region extraction mentioned above, compared to detecting the image region of the suspended load based only on object regions extracted from a single frame image.

[0014] Here, a larger coverage ratio indicates a larger overlapping area when a pair of object regions are superimposed, suggesting a higher probability of representing the same object. However, instead of this coverage ratio, it is also possible to use IoU (Intersection over Union), shown in equation (2) below.

[0015] IoU(A,B)=Area(A∩B) / Area(A∪B)……Formula (2) Here, Area(A∩B) represents the area of ​​the overlap between region A and region B, and Area(A∪B) represents the area of ​​the union of A and B.

[0016] However, while IoU is a valid indicator for measuring the overlap between regions, it is unsuitable as an indicator for determining whether region A and region B represent the same object. This is because if region A is output as a region representing the entire suspended load, and region B is output as a smaller region representing a part of the suspended load, the value of IoU(A,B) will be small. In contrast, the coverage ratio of equation (1) does not become small even in such cases, so it functions as an effective indicator for determining whether region A and region B represent the same object. With the above configuration, the image region of the suspended load can be detected with high accuracy by detecting the image region of the suspended load based on the series of object regions determined to have an coverage relationship based on the coverage ratio of equation (1), that is, the series of object regions estimated to represent the same object.

[0017] The second invention is preferable in the first invention, where the detection process calculates the area of ​​the logical OR region, which is the logical OR of the object regions included in each of the multiple frame images that span the series, for the multiple object regions that constitute the series of object regions identified by the series identification process, and detects the logical OR region having the area of ​​the median or the logical OR region closest to the area of ​​the median as the image region of the suspended load.

[0018] This configuration prevents the image area of ​​a suspended load from being detected as extremely small or extremely large due to variations in the area of ​​the logical OR region in each frame image, which result from the suspended load being over-extracted or overlappingly extracted by multiple object regions. If the number of object regions is odd, the object region with the median area is detected as the image area of ​​the suspended load; if the number of object regions is even, the logical OR region close to the median area is detected as the image area of ​​the suspended load.

[0019] The third invention is that, in the first invention, the controller is further capable of detecting a person's image from the video input from the camera and, based on the detected person's image, identifying an image region showing the person's walking trajectory between the first frame image and the last frame image as a walking trajectory region, and when executing the sequence identification process, the controller is configured to exclude sequences to which the object region to which at least a part overlaps with the walking trajectory region identified in the walking trajectory identification process belongs.

[0020] This configuration minimizes the misidentification of image areas such as the floor or shadows as image areas of suspended loads. Specifically, the inventors considered that people may walk on the floor or shadows, but not on suspended loads. Based on this idea, when performing the sequence identification process, they excluded (i.e., did not extract) sequences to which object regions overlapping with human walking trajectories belong. This ultimately minimizes the misidentification of the floor or shadows as image areas of suspended loads.

[0021] The fourth invention is configured such that, in the first invention, the controller detects feature points representing the same stationary point in real space in each of the plurality of frame images extracted by the frame image extraction process, and performs a correction process by multiplying the other frame images by a transformation matrix so that the coordinate positions from the image origin of the feature points of one of the plurality of frame images set in advance and the feature points of the other frame images coincide, and it is preferable that the correction process is performed before the execution of the object region extraction process, or after the execution of the object region extraction process and before the execution of the inclusion relationship recognition process.

[0022] With this configuration, even if the camera's position moves due to factors such as wind, the coordinate positions of identical feature points in each frame image relative to the image origin do not change. Therefore, when the inclusion relationship recognition process described above extracts a pair of object regions at corresponding positions between different frame images and calculates the inclusion ratio, it is possible to prevent the calculation of the inclusion ratio from being performed for object regions representing objects at completely different positions in real space. Thus, the inclusion relationship of object regions between different frame images can be accurately obtained.

[0023] The work machine according to the fifth invention comprises a suspended load image detection device described in any one of the first to fourth inventions, a work machine body, and the camera attached to the work machine body.

[0024] According to this configuration, since the working machine is equipped with a suspended load image detection device, the safety during the operation of the working machine can be improved by using the detection result of the suspended load image detection device.

[0025] The suspended load image detection method according to the sixth invention is a suspended load image detection method for detecting an image area of the suspended load from an image of a subject including the suspended load of a working machine captured by a camera at a predetermined frame rate, and the controller extracts a plurality of frame images arranged in time series from the image captured by the camera in a frame image extraction step, and the controller executes image processing based on SAM (Segment Anything Model) for each of the plurality of frame images to extract a plurality of object areas included in each frame image in an object area extraction step, and the controller, in all pairs of adjacent frame images in the time series among the plurality of frame images, identifies a pair of object areas located at corresponding positions between the plurality of object areas included in one frame image and the plurality of object areas included in the other frame image, and calculates the inclusion rate of the other object area with respect to one of the identified pair of object areas by the following formula (1), and when the calculated inclusion rate value is a predetermined value or more, recognizes that the one object area and the other object area are in an inclusion relationship in an inclusion relationship recognition step, and the controller, based on the recognition result in the inclusion relationship recognition step, identifies a series of object areas in which the inclusion relationship is maintained the longest between the first frame image and the last frame image when viewed in time series in a series identification step, and the controller includes a detection step of detecting the image area of the suspended load based on the plurality of object areas constituting the series of object areas identified in the series identification step.

[0026] Inclusion rate = Area(A∩B) / min(Area(A), Area(B))... Formula (1) Here, region A represents one of the pair of object regions, and region B represents the other of the pair of object regions. Area represents the area. The numerator Area(A∩B) represents the area of the overlapping portion between region A and region B when the pair of frame images are arranged such that their viewing angles coincide. The denominator min(Area(A), Area(B)) represents the smaller of the areas of region A and region B.

[0027] According to this method, the same operational effects as those of the first invention can be obtained.

Effects of the Invention

[0028] According to the present invention, an image of a suspended load can be accurately detected from an image captured by a camera.

Brief Description of the Drawings

[0029] [Figure 1] It is a schematic diagram showing a crane (an example of a working machine) equipped with a suspended load image detection device according to an embodiment of the present invention. [Figure 2] It is a block diagram showing the configuration of a control system including a controller. [Figure 3] It is an example of a frame image captured by a camera. [Figure 4] It is a schematic diagram showing an example in which an object region R is extracted from a frame image by image processing based on SAM. [Figure 5] It is an explanatory diagram for explaining an example of over-segmentation extraction, misclassification extraction, and overlap extraction. [Figure 6] It is a diagram showing an example of an inclusion graph. [Figure 7] It is a diagram in which candidates for the image region of the suspended load are narrowed down from the inclusion graph of FIG. 6. [Figure 8] It is a diagram in which a walking trajectory region is superimposed and displayed on each frame image shown in FIG. 7. [Figure 9] It is a diagram in which candidates for the image region of the suspended load are further narrowed down from FIG. 6. [Figure 10]This figure shows the logical OR region in each frame image. [Figure 11] This is an explanatory diagram showing an example of defining a logical OR domain. [Figure 12] This figure shows the result of performing position correction processing on each frame image in the modified example. [Modes for carrying out the invention]

[0030] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0031] Figure 1 is a schematic diagram showing a crane 1 (an example of a work machine) equipped with a suspended load image detection device 200 according to an embodiment of the present invention.

[0032] This crane 1 comprises a crane body 10 (an example of a work machine body), a camera 30 attached to the crane body 10 that captures images of a subject including the suspended load 50, and a controller 100 that constitutes a suspended load image detection device 200.

[0033] The crane body 10 comprises an upper slewing body 11, a lower traveling body 12 that supports the upper slewing body 11 so that it can rotate, a work attachment 15 including a boom 13 and a jib 14, and a mast 16 which is a boom luffing member.

[0034] The lower traveling body 12 is configured to travel on the ground by the rotation of its left and right crawlers. The upper slewing body 11 is supported by the lower traveling body 12 so as to be able to slewing around a central axis CL that extends vertically relative to the lower traveling body 12. A cab 17 is provided at the front end of the upper slewing body 11. The cab 17 corresponds to the driver's seat of the crane 1.

[0035] Furthermore, the work attachment 15 includes a base end that is rotatably supported on the upper slewing body 11 in the luffing direction and a tip end opposite to the base end, and is detachably attached to the upper slewing body 11. As described above, in this embodiment, the work attachment 15 includes the boom 13 and the jib 14.

[0036] The boom 13 is of the so-called lattice type and is pivotally supported on the upper slewing body 11 around a rotation axis that extends in the left-right direction, with a boom foot pin (not shown) at its lower end as the pivot point.

[0037] The jib 14 is rotatably connected (pivoted) to the tip of the boom 13, and the axis of rotation of the jib 14 is a horizontal axis parallel to the axis of rotation of the boom 13 relative to the upper slewing body 11.

[0038] The crane 1 further includes a rear strut 21 and a front strut 22 rotatably supported at the tip of the boom 13, a jib guideline 23 stretched between the tip of the front strut 22 and the tip of the jib 14, a rear strut guideline 24 stretched between the tip of the rear strut 21 and the boom 13, a boom guideline 25 stretched between the tip of the boom 13 and the tip of the mast 16, a jib luffing rope 26 wrapped between the tip of the front strut 22 and the tip of the rear strut 21, a boom luffing rope 27 wrapped between the tip of the mast 16 and the upper slewing body 11, a main winding rope 28 for suspending loads, and a hook 29 suspended from the lower end of the main winding rope 28.

[0039] The main hoisting rope 28 is wound around a point sheave (not shown) provided at the tip of the jib 14 (hereinafter referred to as the jib tip) 14A. The main hoisting rope 28 has a hanging portion 28a extending downward from the point sheave and a connecting portion (not shown) that extends further toward the crane body 10 than the point sheave and connects to the main hoisting winch 18 provided near the base end of the boom 13. In Figure 1, the hanging portion 28a of the main hoisting rope 28 is simplified and shown as a single straight line, but in reality, it hangs down from both sides of the point sheave, so it appears as multiple straight lines externally.

[0040] Furthermore, a camera 30 capable of capturing images of the suspended load 50 is provided at the tip 14A of the jib. The camera 30 is mounted on one side of the jib tip 14A in the thickness direction (the side facing the viewer in Figure 1) so as not to interfere with the point sheave. The camera 30 captures images of the subject, including the suspended load 50, at a predetermined sampling rate and transmits them to the controller 100. The camera 30 is composed of, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) camera or a CCD (Charge-Coupled Device Camera) camera, but is not limited to these.

[0041] The hook 29 is attached to the lower end of the hanging portion 28a of the main winding rope 28. A load 50 is suspended from the hook 29 using a lifting device 51 such as a rope or chain.

[0042] Crane 1 further includes a main hoisting winch 18 that raises and lowers the hook 29 by winding in and unwinding the main hoisting rope 28, a jib luffing winch (not shown) that raises and lowers the jib 14 with its base end as a pivot point by winding in and unwinding the jib luffing rope 26, and a boom luffing winch (not shown) that raises and lowers the boom 13 with its base end as a pivot point by winding in and unwinding the boom luffing rope 27. The operation of these various winches is controlled by a controller 100 in accordance with the operations performed by the operator on an operating device 40 (Figure 2) located inside the cab 17.

[0043] [Configuration of the control system including the controller] Figure 2 is a block diagram showing the configuration of the control system including the controller 100. The controller 100 consists of a CPU (Central Processing Unit), ROM (Read Only Memory) for storing control programs, RAM (Random Access Memory) used as the CPU's workspace, and the like.

[0044] In addition to the camera 30, the controller 100 is connected to the operating device 40, the display 31, and the crane drive unit 45, etc., so that it can send and receive signals.

[0045] The operating device 40 is located inside the cab 17 and receives operations performed by the operator to drive the crane 1, and inputs operation signals corresponding to those operations to the controller 100.

[0046] The display 31 is located inside the cab 17 and displays images captured by the camera 30 and information necessary for operating the crane 1.

[0047] The crane drive unit 45 includes a slewing drive unit for slewing the upper slewing body 11 of the crane 1, a boom drive unit for raising and lowering the boom 13, and a winch drive unit for driving various winches, including the main hoisting winch 18. Each drive unit is composed of a hydraulic circuit including a hydraulic pump driven by an engine.

[0048] The controller 100 functions by having the CPU execute a control program stored in ROM, thereby comprising a drive control unit 101, an image processing unit 102, and a display control unit 103.

[0049] The drive control unit 101 inputs a command signal to the crane drive unit 45 corresponding to the operation (for example, the direction and amount of operation) received by the operating device 40. The drive control unit 101 then causes the crane 1 to perform a predetermined operation corresponding to the operation of the operating device 40.

[0050] The image processing unit 102 receives video input from the camera 30 and detects images of the suspended load 50 contained in the received video. Here, the video input from the camera 30 consists of a large number of frame images captured by the camera 30 at a predetermined frame rate. The image processing unit 102 detects images of the suspended load 50 based on each frame image.

[0051] The display control unit 103 receives video input from the camera 30 and displays the received video on a display 31 located inside the cab 17. At that time, the display control unit 103 displays the image of the suspended load 50 detected by the image processing unit 102 on the display 31 in a way that makes it distinguishable from other object images, such as by surrounding it with a frame. As a result, the operator of the crane 1 can operate the crane 1 while accurately recognizing the position of the suspended load 50, thereby improving safety during operation.

[0052] By the way, since the suspended load 50 of crane 1 has a wide variety of external shapes, it is difficult to accurately detect the image of the suspended load 50 using conventional image processing methods that rely solely on pattern matching or machine learning.

[0053] Therefore, in this embodiment, the image processing unit 102 extracts all object regions R (regions that may contain objects, and are not necessarily regions that indicate objects, as they may be shadow regions, for example) contained in the frame image by performing image processing based on the known method SAM (Segment Anything Model) on the frame image, and from among the extracted object regions R, it detects the object regions R that satisfy predetermined conditions based on the inclusion graph described later as the image regions of the suspended load 50. In this embodiment, the detection process of the image regions of the suspended load 50 based on SAM is performed when the suspended load 50 is placed on the ground and stationary (i.e., when the lifting and lowering movement of the suspended load 50 by the crane 1 has stopped), but is not limited to this.

[0054] Figure 3 shows an example of a frame image captured by camera 30.

[0055] As shown in Figure 3, the frame image shows not only the suspended load 50, but also other objects 52-56 and shadows 57-59. Note that in Figure 3, in addition to hook 29, another hook 32 capable of suspending the load is visible, but for simplification, this hook 32 is omitted from the illustration in Figure 1.

[0056] Figure 4 is a schematic diagram showing an example of extracting object regions R from a frame image using SAM-based image processing. In this figure, the extracted object regions R are enclosed in thick lines for clarity. The quality and number of extracted object regions R vary depending on the feature extraction model and the number of coordinate prompts used when running SAM. This feature extraction model constitutes part of SAM, and in this example, as an example of a feature extraction model, we use ViT-H (Vision Transformer-Huge) (paper: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.), and perform region division of the entire frame image by placing 1024 coordinate prompts at equal intervals across the entire image.

[0057] In the frame image, the central part of the image region r of the suspended load 50, which overlaps with the hook 29 of crane 1 in the center, is extracted as two object regions R. In this way, one object may be divided and extracted into multiple object regions R (hereinafter referred to as over-divided extraction). Conversely, there are cases where a region covering multiple objects (or things that look like objects, such as shadows) is extracted as a single object region R (hereinafter referred to as confused extraction), and cases where one object is extracted as multiple object regions R with overlapping portions (hereinafter referred to as overlapping extraction).

[0058] Figure 5 is an explanatory diagram illustrating examples of over-segmentation, confusion, and overlapping extraction.

[0059] The upper part of Figure 5 shows the imaging status of object 60 by camera 30, and the lower part shows examples of the extraction results of object regions R obtained through over-segmentation, confusion extraction, and overlapping extraction. In each figure, the thick lines indicate object regions R extracted by SAM, and the dashed lines indicate the outline of the object. In the example of over-segmentation in Figure 5, one object 60 is extracted as three object regions R along the three outlines of the object surface. In the example of confusion extraction in Figure 5, the object 60 itself and its shadow region are extracted together as a single object region R. In the example of overlapping extraction in Figure 5, one object 60 is extracted as two overlapping object regions R along the outlines of the object surface.

[0060] Furthermore, when extracting object regions using SAM, there is a problem (the problem of extraction instability) in which the extraction results can change even in images of the same scene due to subtle changes in the image. Specifically, for example, an object region R that was extracted in one frame image may not be extracted in the frame image at the next time point. Therefore, in order to detect the suspended load 50 using SAM, it is necessary to deal with such unstable extraction results.

[0061] Therefore, in this embodiment, the image processing unit 102 performs a frame image extraction process that extracts multiple frame images G (see Figure 6, described later) arranged in a time series, rather than a single frame image, and performs SAM-based image processing on each extracted frame image G. The image processing unit 102 then generates an inclusion graph, described later, based on the multiple frame images G, and identifies the object region R that is most stably extracted among the multiple frame images as a candidate image region for the suspended load 50 based on the generated inclusion graph. This avoids the problem of instability in the extraction of the object region R described above.

[0062] [Inclusion Graph] Figure 6 shows an example of an inclusion graph. An inclusion graph is a graph that associates object regions R that are presumed to represent the same object with each other, across multiple (three in the example of Figure 6) frame images G arranged in a time series, by connecting them with edges.

[0063] In this correspondence process, the inclusion rate K of a pair of object regions R located at corresponding positions in adjacent frame image pairs in time series is calculated, and the object regions R are matched only if the inclusion rate K is greater than or equal to a predetermined value. The inclusion rate K is an index that indicates the degree to which two regions A and B represent the same object.

[0064] The inclusion rate K is calculated by the following equation (1).

[0065] K=Area(A∩B) / min(Area(A),Area(B))……Formula (1) Here, region A represents one of a pair of object regions R, and region B represents the other of the pair of object regions R. Area represents area, where the numerator, Area(A∩B), represents the area of ​​the overlapping portion of region A and region B when the frame image pair is arranged so that their fields of view match, and the denominator, min(Area(A),Area(B)), represents the area of ​​the smaller of region A and region B.

[0066] The coverage ratio K in equation (1) is the value obtained by dividing the area of ​​the overlap between region A and region B by the smaller of the two areas. Here, since the numerator, Area(A∩B), is smaller than both Area(A) and Area(B), the coverage ratio K takes a value between 0 and 1, and a larger coverage ratio K indicates a greater degree of overlap. Furthermore, the coverage ratio K is particularly large when region A and region B overlap by roughly the same size. By using such a coverage ratio K to associate object regions R between multiple frame images G, an coverage graph is generated.

[0067] In the inclusion graph, each object region R is treated as a node 300, and in the time series, object regions R that are inclusion-related with the object region R between adjacent frame images G are connected by edges 301. Connecting with edges 301 does not necessarily mean that the object regions R are connected by straight lines on the graph; it is sufficient if the object regions R can be associated with each other.

[0068] Specifically, the image processing unit 102 acquires multiple frame images G at predetermined time intervals (every second in this example) based on the video input from the camera 30, and extracts (enumerates) all object regions R contained in each frame image G by performing SAM-based image processing on each of the acquired frame images G. The image processing unit 102 then calculates the inclusion rate K of object regions R located at corresponding positions in adjacent frame images G.

[0069] Here, object regions R located at corresponding positions are object regions R whose coordinate positions in adjacent frame images G correspond to each other. In other words, they are object regions R that have overlapping portions when adjacent frame images G are superimposed with matching field angles.

[0070] The image processing unit 102 then calculates the inclusion rate K of one region A to the other region B using equation (1), assuming that a pair of object regions R located at corresponding positions in adjacent frame images G are regions A and B. If the calculated inclusion rate K exceeds a threshold, it performs an inclusion relationship recognition process to recognize that one region A and the other region B are in an inclusion relationship. If the image processing unit 102 recognizes that regions A and B are in an inclusion relationship, it connects both regions A and B with an edge 301 to establish a correspondence.

[0071] In the inclusion graph of Figure 6, the image processing unit 102 connects three frame images G acquired every second based on the video from the camera 30 with edges 301, where object regions R with an inclusion rate of 0.5 or higher are considered to have an inclusion relationship. These three frame images G are arranged sequentially from left to right in chronological order. In the following explanation, the three frame images G will be distinguished as frame images g1, g2, and g3 from left to right. Note that the number of frame images G is not limited to three; it may be two. However, from the viewpoint of resolving the instability problem of object region extraction by SAM described above, it is preferable to have at least three or more.

[0072] This inclusion graph displays three object region series S1 to S3, each consisting of an object region R representing the same object. In Figure 6, the line type of edge 301 (solid line, dashed line, double-dotted line) and the shape of node 300 (circle, triangle, square) are different for each object region series S1 to S3. In the following explanation, when it is necessary to distinguish between the three object region series S1 to S3, the object region series shown with circular node 300 and solid edge 301 will be referred to as the first object region series S1, the object region series shown with triangular node 300 and dashed edge 301 will be referred to as the second object region series S2, and the object region series shown with square node 300 and dashed edge 301 will be referred to as the third object region series S3.

[0073] In the aforementioned inclusion graph, the object region R belonging to the first object region series S1, indicated by the circular node 300 and the solid edge 301, corresponds to the object region series of the suspended load 50. In this inclusion graph, for example, in the first frame image g1 from the left, the suspended load 50 is extracted as multiple object regions R, but in the second frame image g2 from the left, the entire suspended load is extracted as a single object region R. Since each object region R of the suspended load 50 in the first frame image g1 and the object region R of the suspended load 50 in the second frame image g2 are in an inclusion relationship based on the above equation (1), it can be determined that they are the same object.

[0074] After generating the inclusion graph, the image processing unit 102 identifies the object region R that is most stably extracted across the multiple frame images G that constitute the inclusion graph. In other words, the image processing unit 102 identifies the object region series S1 to S3 in which the inclusion relationship is maintained for the longest time series across the multiple frame images G. Specifically, the image processing unit 102 calculates the maximum edge continuity length for each object region series S1 to S3 that represents the same object included in the inclusion graph. Here, the maximum edge continuity length is the temporal length from the start node to the end node of the part where the edges are continuously connected for the longest period of time. Then, the image processing unit 102 compares the maximum edge continuity lengths of each object region series S1 to S3 and performs a series identification process to identify the object region series with the largest maximum edge continuity length.

[0075] In the example shown in Figure 6, the maximum continuous edge length of the first object region sequence S1 is equivalent to three frame images in time (3 seconds in this example), the maximum continuous edge length of the second object region sequence S2 is equivalent to three frame images in time (3 seconds in this example), and the maximum continuous edge length of the third object region sequence S3 is equivalent to two frame images in time (2 seconds in this example). Therefore, the image processing unit 102 identifies the first object region sequence S1 and the second object region sequence S2 as the object region sequences with the largest maximum edge lengths. The image processing unit 102 then narrows down the object regions R belonging to the first object region sequence S1 and the second object region sequence S2 identified by the sequence identification process as candidates for image regions of the suspended load 50. Figure 7 shows the candidates for image regions of the suspended load 50 extracted by the narrowing down process.

[0076] The image processing unit 102 performs a walking area elimination process to further narrow down the candidate image areas of the suspended load 50 shown in Figure 7, by eliminating object areas R that overlap with the worker's walking trajectory area W (see Figure 8, described later).

[0077] Specifically, the image processing unit 102 detects human images in each frame image G by performing image processing based on pattern matching and machine learning, and identifies the human walking trajectory region W by combining the human images detected in each frame image. Here, "human" includes, for example, workers performing lifting operations around suspended loads.

[0078] Figure 8 shows the identified walking trajectory region W superimposed on the candidate image regions of the suspended load 50 in each frame image G shown in Figure 7. The image processing unit 102 narrows down the candidates by excluding object regions R that overlap at least partially with the worker's walking trajectory region W in Figure 7 from the candidate image regions of the suspended load 50. Figure 9 shows the candidate image regions of the suspended load 50 extracted by this narrowing down process. In the example shown in Figure 9, the object region R belonging to the first object region series S1 ultimately remains as a candidate image region of the suspended load 50.

[0079] The image processing unit 102 then performs a detection process to detect the image region of the suspended load 50 based on a plurality of object regions R that are candidates for the image region of the suspended load 50 shown in each frame image G in Figure 9. Specifically, in this detection process, a logical OR region L is defined, which is the logical OR of the object regions R that are candidates for the image region of the suspended load 50 in each frame image G, and its area is calculated. The logical OR region L with the area of ​​the median is then detected as the object region of the suspended load 50. If the number of frame images is even, the median value is calculated as the average of the areas of the two central images, and the logical OR region with the area closest to this median value is detected as the image region of the suspended load 50.

[0080] Figure 10 shows the OR region L in each frame image G with diagonal lines. In this example, the area of ​​the OR region L in the second frame image g2 from the left (the region consisting of one object region in Figure 10) is the median, so the image processing unit 102 detects this OR region L as the image region of the suspended load 50. Figure 11 is an explanatory diagram showing examples of OR region L definition. The left diagram shows an example of OR region L definition when the object region R is over-divided and extracted, the center diagram shows an example of OR region L definition when the object region R corresponding to one object is correctly extracted, and the right diagram shows an example of OR region L definition when the object regions R overlap and are extracted.

[0081] After detecting the image region of the suspended load 50, the image processing unit 102 performs detection (tracking) using Non-Patent Literature 1 (Paper title: Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022). This makes it possible to detect (track) the image region of the suspended load 50 in real time. In other words, since SAM-based image processing is computationally burdensome, it is difficult to process the video input from the camera 30 in real time. However, after detecting the image region of the suspended load 50 using SAM, it becomes possible to perform real-time detection of the image region of the suspended load 50 by executing other detection processes.

[0082] When the display control unit 103 displays the video from the camera 30 on the display 31, it displays the image area of ​​the suspended load 50 detected by the image processing unit 102 in real time on the display 31, such as by surrounding it with a frame.

[0083] [Effects and Effects] According to the suspended load image detection device 200 of this embodiment, the controller 100 performs a frame image extraction process to extract a plurality of frame images G arranged in chronological order from the video captured by the camera 30, and performs SAM (Segment Anything) on ​​each of the plurality of frame images G. The system is configured to perform the following: an object region extraction process that extracts all object regions R contained in each frame image G by performing image processing based on the Model; an inclusion relationship recognition process that calculates the inclusion rate of one object region R to the other of a pair of object regions R located at corresponding positions in all pairs of frame images G that are adjacent to each other in time series, using formula (1), and recognizes that one object region R and the other object region R are in an inclusion relationship if the calculated inclusion rate is greater than or equal to a predetermined value; a sequence identification process that identifies the object region sequence S1 to S3 (first object region sequence S1 in the example of Figure 6) in which the inclusion relationship is maintained for the longest time between the first frame image G (frame image g1 in Figure 6) and the last frame image G (frame image g3 in Figure 6) in time series; and a detection process that detects the image region of the suspended load 50 based on the multiple object regions R that constitute the object region sequence S1 to S3 identified by the sequence identification process.

[0084] With this configuration, the image of the suspended load 50 can be accurately detected from the video captured by the camera 30.

[0085] In other words, with the above configuration, by using SAM (Segment Anything Model), it is possible to extract object regions R from the entire frame image G regardless of the object class, and from the extracted object regions R, images that satisfy the conditions for a suspended load 50 can be detected as suspended load 50 images.

[0086] Furthermore, in the above configuration, instead of using a single frame image G, the object region series S1 to S3 (in the example in Figure 6, the first object region series S1) in which the above-mentioned inclusion relationship is maintained for the longest time series is identified across multiple frame images G, and the image region of the suspended load 50 is detected based on the object regions R belonging to this identified series S1 to S3. This avoids the problem of instability in the extraction of object regions R based on SAM described above, compared to the case where the image region of the suspended load 50 is detected based only on object regions R extracted from a single frame image G.

[0087] Furthermore, by adopting the coverage ratio of equation (1) as an indicator of identical objects, it is possible to identify object region sequences S1 to S3 that represent identical objects with greater accuracy compared to, for example, the case where IoU (Intersection over Union) is used.

[0088] Furthermore, in this embodiment, the detection process involves calculating the area of ​​the logical OR region L, which is the logical OR of the object regions R contained in each of the multiple frame images g1 to g3 that span the object region series S1 to S3 identified by the series identification process, and detecting the logical OR region L having the area of ​​the median or the logical OR region L closest to the area of ​​the median as the image region of the suspended load 50.

[0089] This configuration prevents the image area of ​​the suspended load 50 from being detected as extremely small or extremely large due to variations in the area of ​​the logical OR region L in each frame image g1 to g3, which result from the suspended load 50 being over-extracted or overlappingly extracted by multiple object regions R.

[0090] Furthermore, in this embodiment, the controller 100 is capable of detecting images of people from the video input from the camera 30 and further executing a walking trajectory identification process that identifies an image region showing the walking trajectory of a person between the first frame image g1 and the last frame image g3 as a walking trajectory region W based on the detected images of people, and is configured to exclude the series to which an object region R that overlaps at least a part with the walking trajectory region identified by the walking trajectory identification process belongs when executing the series identification process.

[0091] This configuration minimizes the possibility of image areas such as the floor or shadows being mistakenly detected as image areas of the suspended load 50. Specifically, the inventors considered that while a person may walk on the floor or shadows, they would not walk on the suspended load 50. Based on this idea, when executing the sequence identification process, they excluded sequences to which object regions R that overlap with the human walking trajectory region W belonged. This minimizes the possibility of the floor or shadows being mistakenly detected as image areas of the suspended load 50.

[0092] In this embodiment, the crane 1 comprises a suspended load image detection device 200, a crane body 10, and a camera 30 attached to the crane body 10.

[0093] With this configuration, since the crane 1 is equipped with a suspended load image detection device 200, the safety of the crane 1 during operation can be improved by utilizing the detection results from the suspended load image detection device 200.

[0094] (modified version) Next, a modified version of the above embodiment will be described. In this modified version, the image processing unit 102 of the controller 100 corrects the position of the image in each frame image G when performing SAM-based image processing on multiple frame images G, which is different from the above embodiment. In this modified version, the image processing unit 102 performs this position correction processing before performing the object region extraction processing. Note that the content of the image processing performed by the image processing unit 102 in this modified version is the same as in the above embodiment except for the execution of this position correction processing, so only the position correction processing will be described below.

[0095] Specifically, this position correction process suppresses the issue where, when the position of camera 30 changes due to wind or other factors, the object region R extracted by SAM in each frame image G will have different coordinate positions relative to the image origin, even though they represent the same object. The image origin is usually set to the upper left corner of each frame image, but is not limited to this.

[0096] Specifically, the position correction process uses the AKAZE feature vector (Accelerated Keypoints and Zero-crossings of the Laplacian of Gaussian) used by Das et al. and a high-speed method using RANSAC (Random Sample Consensus) (Paper title: Dipankar Das and Jun Miura. Camera motion compensation and person detection in construction site using yolo-bayes model. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 4566-4572. IEEE, 2022.).

[0097] Figure 12 shows the result of the position correction process on the frame image G. In this figure, the three frame images G are shown in chronological order from left to right. As shown in this figure, the position correction process converts the coordinate systems of the other two frame images g2 and g3 (excluding the initial frame image g1, the leftmost frame image in Figure 12) to the coordinate system of the initial frame image g1. Here, the initial frame image g1 corresponds to a pre-set reference image among the three frame images g1 to g3. Note that the reference image does not necessarily have to be the initial frame image g1; it could be the second frame image g2 or the third frame image g3.

[0098] Specifically, the image processing unit 102 detects feature points in each frame image g1 to g3 that represent the same stationary point in real space, and multiplies the coordinate position of the feature point detected in the initial frame image g1 within the field of view with the coordinate position of the feature point detected in the other two frame images g2 and g3 within the field of view, respectively, by a transformation matrix.

[0099] In the example in Figure 12, although the camera 30 is actually shaking up and down as we move to the right from the initial frame image g1, the image position of the suspended load 50 relative to the origin (upper left corner in this example) of each frame image g1, g2, and g3 is kept constant as shown by the dashed line in the figure due to the transformation matrix described above. Instead, the upper edge of the second frame image from the left, g2, is cut off, and the lower edge of the third frame image from the left, g3, is cut off.

[0100] [Effects and Effects] According to the modified suspended load image detection device 200, even if the camera 30 moves due to wind or other factors, the coordinate positions of the same feature points captured in each frame image g1 to g3 relative to the image origin do not change. Therefore, when generating an inclusion graph based on inclusion relationship recognition processing, it is possible to prevent the inclusion ratio from being calculated for object regions of objects that are in completely different positions in real space when extracting a pair of object regions R located at corresponding positions between different frame images g1 to g3. Thus, the inclusion relationship between object regions R between different frame images g1 to g3 can be accurately obtained. Furthermore, when executing the walking trajectory acquisition processing, it is possible to prevent the camera 30 from shaking, which would prevent the accurate acquisition of the walking trajectory of a person (worker).

[0101] (Other embodiments) Although the suspended load image detection device 200 according to an embodiment of the present invention has been described above, the present invention is not limited thereto, and for example, the following modified embodiments can be adopted.

[0102] (1) In the above embodiment and its modifications, the function of the image processing unit 102 in the controller 100 does not necessarily have to be realized by the controller 100 mounted on the crane 1, but may be realized by a higher-level controller located in a different location away from the crane 1, for example.

[0103] (2) In the above embodiment and modified examples, the suspended load image detection device 200 is configured to detect an image of the suspended load 50 from an image captured by a camera 30 provided on the crane 1. However, the camera 30 does not necessarily have to be provided on the crane 1 and may be attached to other fixed objects, etc.

[0104] (3) In the above embodiment and modified examples, the image processing unit 102 is configured to detect, when finally narrowing down the image of the suspended load 50, the object region R having the median area or the object region R closest to the median area among the multiple object regions R constituting the object region series S1 to S3 identified by the series identification process as the image region of the suspended load 50. However, it is not limited to this, and for example, it may be configured to detect, for example, the object region R having the largest area as the image region of the suspended load 50.

[0105] (4) In the above embodiments and modifications, a crane 1 was given as an example of a work machine equipped with a suspended load image detection device 200, but it is not limited to this, and the work machine may be configured in any way, for example, a demolition machine, an aerial work platform, etc. When the work machine is configured as a crane 1, the crane 1 does not necessarily have to have a jib 14 and struts 21, 22, and may be configured to have only a boom 13 (so-called crane specification), or it may be configured to have only one strut and the jib 14 does not luff (so-called fixed jib specification). Also, the boom 13 and jib 14 are not limited to a lattice structure, and a telescopic structure that can be extended and retracted by hydraulics, etc. may be adopted. Also, the lower traveling body 12 of the crane 1 does not necessarily have to travel by the rotation of crawlers, for example, it may travel by the rotation of wheels (tires). Also, a fixed lower body may be provided instead of the lower traveling body 12 of the crane 1.

[0106] (5) In the above embodiments and modified examples, the safety of the operator during operation is improved by displaying an image of the suspended load 50 detected by the controller 100 on the display 31. However, the invention is not limited to this, and for example, safety control such as load swing control may be performed based on the image detected by the controller 100.

[0107] (6) The present invention includes a method for detecting suspended load images, which is realized by the processing performed by the controller 100 in the above embodiments and modified examples.

[0108] (7) The present invention includes any combination of the embodiments and modifications described above. [Explanation of Symbols]

[0109] 1: Crane (working machine) 10: Crane body (working machine body) 30: Camera 50: Hanging load 100: Controller 200: Suspended load image detection device A: One of the object regions B: The other object region G: Frame image g1: Frame image g2: Frame image g3: Frame image K: Inclusion rate L: Logical sum area R: Object region S1: First object region series (object region series) S2: Second object region series (object region series) S3: Third object domain sequence (sequence of object domains) W: Walking trajectory area

Claims

1. A suspended load image detection device for detecting an image region of a suspended load from a video of a subject including a suspended load of a work machine, which is captured by a camera at a predetermined frame rate, Equipped with a controller, The aforementioned controller, A frame image extraction process that extracts multiple frame images arranged in chronological order from the video captured by the aforementioned camera, An object region extraction process is performed on each of the aforementioned multiple frame images to extract all object regions contained in each frame image by performing image processing based on SAM (Segment Anything Model), In the aforementioned plurality of frame images, for all pairs of frame images that are adjacent to each other in time series, the inclusion ratio of one object region to the other object region of a pair of object regions located at corresponding positions is calculated using the following formula (1), and if the calculated inclusion ratio is greater than or equal to a predetermined value, the inclusion relationship recognition process recognizes that the one object region and the other object region are in an inclusion relationship. Based on the results of the inclusion relationship recognition process, a sequence identification process is performed to identify the sequence of object regions in which the inclusion relationship is maintained for the longest period between the first frame image and the last frame image in a time series. A suspended load image detection device configured to perform a detection process that detects the image region of the suspended load based on a plurality of object regions that constitute the series of object regions identified by the series identification process. Coverage rate = Area (A∩B) / min (Area (A), Area (B))...Equation (1) Here, region A represents one of the pair of object regions, and region B represents the other of the pair of object regions. Area represents area, where the numerator, Area(A∩B), represents the area of ​​the overlapping portion of region A and region B when the frame image pair is arranged so that their fields of view match, and the denominator, min(Area(A), Area(B)), represents the area of ​​the smaller of region A and region B.

2. In the suspended load image detection device according to claim 1, The aforementioned detection process is a process in which, for a plurality of object regions constituting the sequence of object regions identified by the sequence identification process, the area of ​​the OR region, which is the OR of the object regions included in each of a plurality of frame images that span the sequence, is calculated, and the OR region having the area of ​​the median or the OR region closest to the area of ​​the median is detected as the image region of the suspended load, in a suspended load image detection device.

3. In the suspended load image detection device according to claim 1, The controller is capable of detecting images of people from video input from the camera, and further performing a walking trajectory identification process that identifies an image region showing the walking trajectory of a person between the first frame image and the last frame image based on the detected images of people, and is configured to exclude the series to which the object region to which at least a part overlaps with the walking trajectory region identified by the walking trajectory identification process belongs when performing the series identification process, the suspended load image detection device.

4. In the suspended load image detection device according to claim 1, The controller is configured to detect feature points representing the same stationary point in real space in each of the plurality of frame images extracted by the frame image extraction process, and to perform a correction process by multiplying the other frame images by a transformation matrix so that the coordinate positions from the image origin of the feature points of one of the plurality of frame images set in advance match those of the feature points of the other frame images, and the correction process is configured to be performed before the execution of the object region extraction process, or after the execution of the object region extraction process and before the execution of the inclusion relationship recognition process, in a suspended load image detection device.

5. A suspended load image detection device according to any one of claims 1 to 4, The main body of the work machine, A work machine comprising the camera attached to the main body of the work machine.

6. A method for detecting a suspended load image from an image of a subject including a suspended load of a work machine, which is captured by a camera at a predetermined frame rate, wherein the image region of the suspended load is detected from the image of the suspended load. The controller performs a frame image extraction step in which it extracts multiple frame images arranged in chronological order from the video captured by the camera, The controller performs an object region extraction step in which it extracts multiple object regions contained in each of the multiple frame images by performing image processing based on SAM (Segment Anything Model) for each of the multiple frame images, The controller identifies a pair of object regions that are in a corresponding position between the multiple object regions in one frame image and the multiple object regions in the other frame image in all adjacent frame image pairs in the time series among the multiple frame images, calculates the inclusion rate of one of the identified pair of object regions with respect to the other object region using the following formula (1), and recognizes that the one object region and the other object region are in an inclusion relationship if the calculated inclusion rate is greater than or equal to a predetermined value, and The controller performs a sequence identification step, based on the recognition results in the inclusion relationship recognition step, to identify the sequence of object regions in which the inclusion relationship is maintained for the longest period between the first frame image and the last frame image in a time series. A method for detecting a suspended load image, comprising: a detection step in which the controller detects an image region of the suspended load based on a plurality of object regions that constitute the series of object regions identified in the series identification step. Coverage rate = Area (A∩B) / min (Area (A), Area (B))...Equation (1) Here, region A represents one of the pair of object regions, and region B represents the other of the pair of object regions. Area represents area, where the numerator, Area(A∩B), represents the area of ​​the overlapping portion of region A and region B when the frame image pair is arranged so that their fields of view match, and the denominator, min(Area(A), Area(B)), represents the area of ​​the smaller of region A and region B.

Images