Information processing device, information processing system, information processing method, and program

Abstract

To provide a technique capable of appropriately detecting a detection object, even when there is a factor that obstructs detection of the detection object.SOLUTION: An information processing device (1) comprises: a first detection unit (21) which is attached to an upper turning body of a conveyance machine, and acquires an image from a first imaging device for imaging a lower side to detect a detection object; a second detection unit (22) which is attached to a position of the conveyance machine different from the first imaging device, and acquires an image from a second imaging device for imaging a lateral side to detect the detection object; and use image determination unit (113) which extracts a region where the second detection unit (22) is unable to detect the detection object, on the basis of images from both of the first and the second imaging devices or images from only the second imaging device, and determines whether to detect the detection object using a detection result by the first detection unit (21) and a detection result by the second detection unit (22), or detect the detection object using only a detection result by the second detection unit (22), in accordance with the region.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] At construction sites where cranes are used for work, the crane operator visually checks the safety of workers on the ground to prevent accidents such as falling loads. However, workers may be in the crane operator's blind spot, and it is also possible that the crane operator may overlook something. For this reason, there is room for improvement in safety management that relies solely on visual checks.

[0003] Here, an example of a conventional technique for safety management of crane work is the technique described in the following Patent Document 1. Patent Document 1 discloses a system that checks safety by synthesizing and displaying images taken by a tip camera attached near the tip of the crane's boom and a peripheral camera attached to the body or swivel base of the crane. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2011-151742 Summary of the Invention [Problem to be solved by the invention]

[0005] The above-mentioned Patent Document 1 aims to manage the safety of cranes using multiple cameras, but does not automatically detect workers working near the cranes. On the other hand, when the same object (worker) is photographed using multiple cameras, the viewpoints (e.g., shooting angles) of each camera are different, so the detection and false detection rates are different for each camera. Therefore, there is a possibility that the detection of the worker, who is the detection target, cannot be performed appropriately.

[0006] One aspect of the present disclosure has been made in consideration of the above-mentioned conventional problems, and aims to provide a technology that can properly detect a detection target even when there is a factor that hinders detection of the detection target. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention comprises a first detection unit that is attached to an upper rotating body of a conveying machine and detects a detection target by acquiring an image from a first photographing device that photographs the downward side, a second detection unit that is attached to a position on the conveying machine different from the first photographing device and detects the detection target by acquiring an image from a second photographing device that photographs the lateral side, and a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on both images from the first photographing device and the second photographing device, or only images from the second photographing device, and determines whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

[0008] In addition, in order to solve the above-mentioned problems, an information processing system according to one embodiment of the present invention comprises a first photographing device attached to an upper rotating body of a conveying machine and photographing the downward side, a second photographing device attached to a position on the conveying machine different from the first photographing device and photographing the lateral side, a first detection unit that acquires an image from the first photographing device and detects a detection target, a second detection unit that acquires an image from the second photographing device and detects the detection target, and a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on both images from the first photographing device and the second photographing device, or only images from the second photographing device, and determines whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

[0009] In addition, in order to solve the above-mentioned problems, an information processing method according to one embodiment of the present invention includes a step of detecting a detection target by acquiring an image from a first photographing device that is attached to an upper rotating body of a conveying machine and photographs the downward side; a step of detecting the detection target by acquiring an image from a second photographing device that is attached to a position on the conveying machine different from the first photographing device and photographs the lateral side; and a step of extracting an area in which the second detection unit cannot detect the detection target based on both images from the first photographing device and the second photographing device, or only the image from the second photographing device, and determining whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

[0010] In addition, in order to solve the above-mentioned problems, a program according to one embodiment of the present invention causes a computer to execute the following processes: a process of detecting a detection target by acquiring an image from a first photographing device that is attached to an upper rotating body of a conveying machine and photographs the downward side; a process of detecting the detection target by acquiring an image from a second photographing device that is attached to a position on the conveying machine different from the first photographing device and photographs the lateral side; and a process of extracting an area in which the second detection unit cannot detect the detection target based on both images from the first photographing device and the second photographing device, or only the image from the second photographing device, and determining whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area. Effect of the Invention

[0011] According to one aspect of the present invention, even if there is a factor that hinders detection of a detection target, the detection target can be appropriately detected. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system including an information processing device according to a first embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment of the present invention. [Diagram 3] 13 is a diagram showing an example of a danger area set in an image captured by a second photographing device. FIG. [Figure 4] FIG. 13 is a diagram showing an example of a danger area set in an image captured by a first image capturing device. [Diagram 5] 5 is a flowchart showing an example of a processing procedure of the information processing device according to the first embodiment of the present invention. [Figure 6] 11 is a diagram for explaining the relationship between the imaging range of a side image captured by the second imaging device and the imaging range of a lower image captured by the first imaging device. FIG. [Figure 7] 11 is a diagram showing an example of a case where a worker enters a danger area in a lower image captured by the first camera device. FIG. [Figure 8] FIG. 2 is a diagram for explaining a frame of a crane. [Figure 9] FIG. 13 is a diagram showing a case where the frame of a crane is included in the side image captured by the second camera device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Embodiment 1) <Example of information processing system configuration> Fig. 1 is a diagram showing a configuration example of an information processing system including an information processing device according to a first embodiment of the present invention. Fig. 1 shows a configuration example of a crane system 100 as an example of an information processing system, but the system can also be applied to transport machines other than cranes, such as hydraulic excavators, backhoes, and backhoes. Examples of cranes include crawler cranes, crawler cranes, tower cranes, and cab-back crane vehicles.

[0014] As shown in Fig. 1, the crane system 100 includes an information processing device 1, a crane CR, a first photographing device 5 which is an imaging camera, an inclinometer 7, and a second photographing device 9 which is also an imaging camera. The imaging camera of the second photographing device 9 may be a wide-angle camera. The first photographing device 5 and the second photographing device 9 may be sensors or the like that can capture images, and are not limited to cameras. An example of a sensor is a laser sensor such as LiDAR (Light Detection And Ranging).

[0015] As described below, the detection rate of workers differs between images captured by the first photographing device 5 and images captured by the second photographing device 9. Since the second photographing device 9 photographs the worker from an oblique side, the person appears relatively clear, the detection rate is high, and the false detection rate is low. On the other hand, since the first photographing device 5 photographs the worker from directly above, the person appears small, the detection rate is low, and the false detection rate is high.

[0016] The crane CR is a machine that lifts and moves heavy objects. The crane CR includes a carriage CR1, a base CR2, a boom CR3, a cable CR4, a hook CR5, and a crane room CR6. The base CR2 and the boom CR3 are also called the upper rotating body.

[0017] The cart section CR1 is configured to be movable in the depth direction of Fig. 1 along rails. A base section CR2 is placed on the cart section CR1. This allows the entire crane CR to move in the depth direction of Fig. 1 along the rails. The crane CR is not limited to a mobile type, and may be a fixed type.

[0018] The base CR2 is a structure that supports the boom CR3 and the like. The base CR2 and the carriage CR1 are connected so that the base CR2 can rotate on the carriage CR1. This allows the crane CR to transport a load located 360 degrees around it. The rotation range may be less than 360 degrees. The crane CR may also be one that cannot rotate within a horizontal plane.

[0019] The boom CR3 is a long structure also called a jib. The crane CR is a jib crane because it is equipped with the boom CR3. One end of the boom CR3 is rotatably connected to the base CR2. As shown by the dashed line in FIG. 1, the boom CR3 is configured to rotate around the connection point with the base CR2 as a fulcrum, so that the inclination angle of the boom CR3 can be changed. In addition, a pulley (not shown) is provided at the other end of the boom CR3, and a cable CR4 hangs down from the pulley.

[0020] The cable CR4 is a rope for lifting a load, and a hook CR5 for hanging the load is attached to the end of the cable CR4. When a load is to be moved by the crane CR, the load is sling-hooked onto the hook CR5 and lifted up.

[0021] The crane room CR6 is a space housing various devices for operating the crane CR, and the operator of the crane CR (hereinafter simply referred to as the operator) operates the crane CR from the crane room CR6. For example, the operator can adjust the inclination angle of the boom CR3, raise and lower the hook CR5 (suspended load) by winding up and down the cable CR4, and rotate the base CR2 and the structure above it.

[0022] Further, an inclinometer 7 for measuring the inclination angle of the boom CR3 is attached to the boom CR3 of the crane CR. A first camera 5 is attached near the tip of the boom CR3.

[0023] The first camera 5 is for monitoring people entering an area (hereinafter referred to as a danger area) of a predetermined size in actual dimensions below the load suspended by the crane CR, and is attached facing vertically downward. As a result, the hook CR5 located vertically below the first camera 5 and the danger area below the hook CR5 are photographed by the first camera 5. The danger area in the image photographed by the first camera 5 is also referred to as a first predetermined area.

[0024] Here, when the inclination angle of the boom CR3 is changed, the size of various objects (e.g., a worker) captured in the image captured by the first camera 5 also changes, and the proportion of the danger area in the first camera 5 also changes. Specifically, the larger the inclination angle of the boom CR, the farther the distance from the first camera 5 to the danger area becomes, so the size of various objects captured in the captured image becomes smaller, and the proportion of the danger area in the captured image also becomes smaller. On the other hand, the smaller the inclination angle of the boom CR, the closer the distance from the first camera 5 to the danger area becomes, so the size of various objects captured in the captured image becomes larger, and the proportion of the danger area in the captured image also becomes larger.

[0025] For this reason, if a person is detected from the image captured by the first camera 5 without considering the tilt angle of the boom CR3, the detection accuracy may be low. Therefore, the information processing device 1 applies a scale according to the tilt angle measured by the inclinometer 7 to set a danger area of ​​a predetermined size in actual dimensions in the captured image, and detects various objects within the danger area.

[0026] The images captured by the first photographing device 5 may be stored in a recording medium, for example, and in this case, the information processing device 1 may read out the images from the recording medium. However, since the information processing device 1 needs to quickly detect when a person enters a dangerous area, it is preferable for the information processing device 1 to receive the images directly from the first photographing device 5, not via a recording medium. The same applies to the measurement results of the inclinometer 7.

[0027] The second camera 9 is mounted facing diagonally downward so as to capture a side image. It is preferable that the second camera 9 rotates together with the crane CR so as to continue capturing an image of the area below the hook CR5 even when the crane CR rotates.

[0028] The information processing device 1 may be installed at the site where the crane CR is installed, or may be installed in, for example, a remote control center, etc. When installed in a remote location, the information processing device 1 may acquire captured images and measurement results of the inclinometer 7 by communication via a network.

[0029] Also, a part of the processing performed by the information processing device 1 may be configured to be performed by another computer system. For example, a calculation using a detection model described later is performed to detect various objects, and this calculation may be configured to be performed by another computer system. In this case, the information processing device 1 transmits information required for the calculation to the other computer system by communication via a network. Then, the information processing device 1 receives the calculation result transmitted from the other computer system.

[0030] <Configuration of information processing device 1> Fig. 2 is a block diagram showing an example of a main configuration of the information processing device 1. As shown in Fig. 2, the information processing device 1 includes a control unit 10, a memory 11, a large-capacity storage device 12, an image capture device IF 13, an inclinometer IF 14, a display device IF 15, a sound output device IF 16, and a network IF 17. The information processing device 1 may be, for example, a personal computer, a server, or a workstation.

[0031] As the mass storage device 12, for example, a static random access memory (SRAM), a hard disk drive (HDD), a solid-state drive (SSD), a secure digital (SD) card, an embedded multi-media controller (eMMC), or a combination of these can be applied. A processor (not shown) loads an information processing program stored in the mass storage device 12 into the memory 11 and executes it, thereby realizing the functions of each unit included in the control unit 10.

[0032] The information processing device 1 also includes, as interfaces (IFs) with external devices, an imaging device IF 13, an inclinometer IF 14, a display device IF 15, a sound output device IF 16, and a network IF 17. Any interface may be used as these interfaces, and for example, a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI), or the like may be used.

[0033] The photographing device IF13 is an interface for acquiring the above-mentioned photographed images from the first photographing device 5 and the second photographing device 9. The inclinometer IF14 is an interface for acquiring the measured value of the tilt angle from the inclinometer 7. The photographed images and the measured value are both analog data. The process of converting the analog data into digital data may be performed before input to the information processing device 1, or may be performed by the information processing device 1. The photographed images and the measured value of the tilt angle may be acquired via an external storage device or the like of the information processing device 1.

[0034] The display device IF15 is an interface used when the information processing device 1 displays an image on a display device (not shown). For example, the information processing device 1 can display, via the display device IF15, an image captured by the first image capturing device 5 or the second image capturing device 9, a measurement value of the inclinometer 7, or the like on the display device.

[0035] The sound output device IF16 is an interface used when the information processing device 1 causes a sound output device (not shown) to output sound. The sound output device is, for example, a speaker. For example, when the information processing device 1 detects that a person has entered a dangerous area, the information processing device 1 can cause the sound output device to output a warning message or a warning sound via the sound output device IF16.

[0036] The network IF 17 is an interface for the information processing device 1 to communicate with various devices on a network such as the Internet, and may be, for example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks. Note that the information processing device 1 may also be provided with an interface for accessing, for example, an external storage device (e.g., a hard disk) in addition to the above-mentioned interfaces.

[0037] The control unit 10 includes a first detection unit 21, a second detection unit 22, a utilization image determination unit 113, and an alarm unit 114. The first detection unit 21 includes an image acquisition unit 101, an inclination angle identification unit 102, a scale identification unit 103, an altitude determination unit 104, an area setting unit 105, an input data generation unit 106, an object detection unit 107, and an intrusion frequency counting unit 108. The second detection unit 22 includes a side image acquisition unit 110, a side image area setting unit 111, and a side image object detection unit 112.

[0038] The image acquisition unit 101 acquires an image of the downward side captured by the first camera 5 attached to the boom CR3 of the crane CR. The method of acquiring the captured image is not particularly limited, and for example, the image acquisition unit 101 may acquire the captured image from the first camera 5 via the camera IF13.

[0039] The inclination angle identifying unit 102 identifies the inclination angle of the boom CR3 of the crane CR. The method of identifying the inclination angle is not particularly limited, and for example, the inclination angle identifying unit 102 may acquire the measurement value of the inclinometer 7 via the inclinometer IF14.

[0040] The scale specifying unit 103 specifies a scale according to the tilt angle specified by the tilt angle specifying unit 102. Specifically, the scale specifying unit 103 specifies the scale using correlation information derived using the size of a reference object with known dimensions in each of reference images obtained by photographing a reference object with the boom CR at different tilt angles. The correlation information is information indicating the correlation between the tilt angle and the scale. Details of the correlation information and the specification of the scale using the correlation information will be described later.

[0041] The altitude determination unit 104 determines the altitude of the position corresponding to the dangerous area. When the altitude determination is performed, the area setting unit 105 sets the dangerous area in the captured image by applying a scale according to the altitude determined by the altitude determination unit 104 and the tilt angle identified by the tilt angle identification unit 102. Details of these processes will be described later with reference to FIG.

[0042] The area setting unit 105 applies a scale according to the inclination angle identified by the inclination angle identification unit 102 to set an area (danger area) having a predetermined size in actual dimensions in the captured image of the lower side captured by the first image capturing device 5. The scale is identified by the scale identification unit 103 as described above. The area setting method will be described in detail later. The danger area in the image captured by the first image capturing device 5 is also referred to as a first predetermined area.

[0043] The input data generating unit 106 generates input data to be input to a detection model used by the object detecting unit 107 to detect an object. More specifically, the input data generating unit 106 cuts out the danger area set by the area setting unit 105 from the captured image, adjusts it to a predetermined size, and generates an adjusted image. This adjusted image becomes input data to the detection model. Although details will be described later, this detection model is generated by machine learning using a learning model such as a neural network.

[0044] The object detection unit 107 detects a predetermined detection object appearing in the captured image. In this embodiment, an example will be described in which the object is a person and a hook, that is, an example in which the object detection unit 107 detects a person and a hook CR5 within a dangerous area appearing in the captured image. These detections are performed based on the output values ​​of the detection model described above.

[0045] The detection target is not limited to these examples. For example, the target detection unit 107 may detect an item (such as a helmet) worn by a person instead of a person, or may detect a suspended load instead of the hook CR5. When detecting multiple targets, the target detection unit 107 may detect each target from the output value of one detection model capable of detecting the multiple targets, or may detect each target from the output value of a detection model prepared for each target.

[0046] The intrusion counting unit 108 counts the number of times that a detection target (typically a person) of the target detection unit 107 intrudes into the area, i.e., the dangerous area, set by the area setting unit 105. The method of counting the number of intrusions will be described in detail later.

[0047] As described above, the information processing device 1 includes an inclination angle identification unit 102 that identifies the inclination angle of the boom CR3 of the crane CR. The information processing device 1 also includes an area setting unit 105 that applies a scale according to the inclination angle identified by the inclination angle identification unit 102 and sets an area of ​​a predetermined size in actual dimensions in an image captured by the first camera 5 attached to the boom CR3.

[0048] According to the above configuration, the tilt angle of the boom CR3 is identified and a scale corresponding to the tilt angle is applied to set the area, so that even if an object of known dimensions does not appear in the captured image, an area of ​​a predetermined size in actual dimensions can be set. Then, the object detection unit 107 detects a predetermined detection object from the set area (danger area), so it is possible to perform highly accurate detection with a smaller amount of calculation compared to when detection is performed from the entire captured image.

[0049] Fig. 3 is a diagram showing an example of a danger area set in an image captured by the second camera 9. In the left diagram of Fig. 3, a circular danger area ar5 is set with its center located directly below the hook CR5 included in the side image captured by the second camera 9.

[0050] Here, in the left diagram of Fig. 3, a load is attached to the hook CR5. If there is a person directly under the suspended load, the whole or part of the person will be in the blind spot of the first image capture device 5, making it difficult for the information processing device 1 to detect the person. Of course, there is no problem if the person can be detected before entering the blind spot, but in manufacturing sites for large products or buildings, there are cases where openings or the like are provided near the ground, and such openings may be in the blind spot of the first image capture device 5. In such cases, it is difficult to detect a person coming out of the opening.

[0051] Therefore, the information processing device 1 of this embodiment sets a danger area in the side image captured by the second image capturing device 9, and detects a person who enters the set danger area. This makes it possible to compensate for the blind spot of the first image capturing device 5 and ensure safety management.

[0052] The right diagram in Fig. 3 shows a side image IMG3 captured by the second imaging device 9. The side image IMG3 shows the hook CR5 and the suspended load, and a danger area ar6 is set. If coordinate axes are set in the image area of ​​this side image IMG3, with the left-right direction being the x-direction and the up-down direction being the y-direction, the coordinates of the center position of the danger area ar6 are expressed as (x1, y1). As will be explained below, when setting the danger area ar6, first the coordinates (x1, y1) are found, and the danger area ar6 is set based on these coordinates.

[0053] Specifically, first, the side image acquisition unit 110 acquires a side image captured by the second imaging device 9. Then, the side image region setting unit 111 sets a danger region ar6 in the image region of the side image acquired by the side image acquisition unit 110.

[0054] More specifically, the side image region setting unit 111 specifies the x coordinate of the position where the hook CR5 is detected in the image region of the side image. The value of this x coordinate is x1 of the coordinate of the center position. Note that the hook CR5 may be detected by any method, and may be detected by using, for example, a detection model generated by machine learning.

[0055] Next, the side image area setting unit 111 calculates the central position coordinate y1. In calculating the central position coordinate y1, the side image area setting unit 111 first calculates the horizontal distance d2 from the second camera 9 to the hook CR5 from the inclination angle identified by the inclination angle identification unit 102. For example, as shown in the left diagram of Fig. 3, if the inclination angle of the boom CR3 is θ and the length of the boom CR3 is L, then d2 = L cos θ is calculated.

[0056] Next, the side image region setting unit 111 specifies a position in the side image IMG3 at which a point at a horizontal distance d2=L·cosθ from the second imaging device 9 appears. This position can be expressed by a function indicating at what position in the side image IMG3 a point at a predetermined horizontal distance from the second imaging device 9 appears.

[0057] For example, in the side image IMG3 shown on the right side of Fig. 3, the positions at which points at distances d1 to d4 from the second image capturing device 9 are captured are indicated by dashed lines. In this way, it is uniquely determined at which position in the side image IMG3 a point at a predetermined horizontal distance from the second image capturing device 9 is captured, so it is possible to specify a function indicating the position at which a point at a horizontal distance d2 = L cos θ from the second image capturing device 9 is captured.

[0058] 3, the lateral image region setting unit 111 may specify a function that indicates the broken line of d2. Then, the lateral image region setting unit 111 assigns x1 to this function to calculate a value that is set as y1.

[0059] The above process determines the central position coordinates (x1, y1) of the danger area ar6, and the side image area setting unit 111 sets the danger area ar6 based on these coordinates. For example, in the example of Fig. 3, the side image area setting unit 111 sets the danger area ar6 between the dashed lines d1 and d3 with the coordinates (x1, y1) as the center. The danger area in the image captured by the second image capturing device 9 is also referred to as a second predetermined area.

[0060] After the danger area ar6 is set, the lateral image object detection unit 112 detects people within the danger area ar6. The detection method can be the same as that used for detecting people from the image captured by the first image capturing device 5.

[0061] For example, the input data generating unit 106 may cut out the portion of the danger area ar6 from the side image and adjust the size to generate an adjusted image. The side image object detecting unit 112 may input the adjusted image to a person detection model and determine whether or not a person has been detected based on the output value. Note that the detection model for detecting a person from the side image may be generated in advance by machine learning using the side image as training data.

[0062] As described above, the information processing device 1 of this embodiment includes a side image area setting unit 111 that sets a danger area in the image area of ​​the side image, and a side image object detection unit 112 that detects a detection object (typically a person) in the set danger area. The side image area setting unit 111 sets a danger area in the image area of ​​the side image below the position where the hook CR5 is detected and at a position corresponding to the horizontal distance from the second imaging device 9 to the hook CR5 obtained from the inclination angle identified by the inclination angle identification unit 102.

[0063] According to this configuration, a danger zone is set in the image area of ​​the side image captured by the second camera 9 that captures the image of the lower side of the hook CR5 from the side, and a detection target within the danger zone is detected. As a result, even if the detection target enters a position where the situation cannot be grasped from the image captured by the first camera 5 due to a blind spot of the hook CR5 or the suspended load, the detection target can be detected from the side image.

[0064] Furthermore, according to the above configuration, a danger area is set in the image area of ​​the side image below the position where the hook CR5 is detected and at a position corresponding to the horizontal distance from the second camera 9 to the hook CR5, which is calculated from the inclination angle identified by the inclination angle identification unit 102. This makes it possible to set a danger area in an area of ​​the image area of ​​the side image that corresponds to a position that is in the blind spot of the first camera 5.

[0065] It is to be noted that when the hook CR5 is in a high position, it is assumed that the hook CR5 will not appear in the side image. In such a case, the hook CR5 may be detected from the image captured by the first camera 5, and the value of x1 in the side image may be obtained from the position of the hook CR5 in the image. In this case, the correspondence between the position of the hook CR5 in the image captured by the first camera 5 and the value of x1 may be formulated in advance.

[0066] Next, the used image determination unit 113 determines whether a detection target (typically a person) has been detected within the danger area based on information from the side image area setting unit 111, the side image object detection unit 112, and the object detection unit 107. The determination method will be described later.

[0067] The alarm unit 114 issues an alarm when a detection target (typically a person) of the target detection unit 107 is detected within the danger area. The alarm may be issued by any method, such as sound, image, or a combination thereof, as long as it can make a predetermined target recognize that a detection target has been detected. The target of the alarm may be a worker working around the crane CR. However, the worker may enter the danger area, for example, when slinging a load on the hook CR5, and an alarm is not necessary when entering in this manner. Therefore, the target of the alarm may be, for example, a watchman or supervisor at the construction site, an operator of the information processing device 1, or the like. In these cases, when the target of the alarm judges it necessary, an alarm may be issued to the worker by the alarm unit 114 or other means.

[0068] <Example of setting danger zone> Fig. 4 is a diagram showing an example of a danger area set in an image from the first image capturing device 5. More specifically, Fig. 4 shows an example of setting the danger area when the inclination angle of the boom CR3 is steep, and also shows an example of setting the danger area when the inclination angle of the boom CR3 is gentle. Note that in Fig. 4, the boom CR3 with a steep inclination angle is shown by a dashed line.

[0069] 4 illustrates an example in which a square area with an actual side length of a and centered at a position directly below the hook CR 5 is set as the danger area. The actual size of the danger area is constant regardless of the inclination angle of the boom CR, but since the shooting range of the first camera 5 changes depending on the inclination angle of the boom CR, the size of the danger area captured in the captured image differs.

[0070] In the example of FIG. 4, when the inclination angle of the boom CR is steep, the width of the photographing range AR1 of the first camera 5 is A1. The image photographed in this state is the photographed image IMG1. On the other hand, when the inclination angle of the boom CR is gentler, the first camera 5 is in a lower position and the width of the photographing range AR2 becomes A2 (A1>A2). The image photographed in this state is the photographed image IMG2. Both photographed images IMG1 and IMG2 were photographed with the same photographing settings and have the same image size and resolution. The width of both of these photographed images is B (pixels).

[0071] The area setting unit 105 sets a danger area with an actual side length of a for each captured image thus captured. In the example of Fig. 4, a danger area with an actual side length of b1 (pixels) is set for the captured image IMG1, and a danger area with an actual side length of b2 (pixels) is set for the captured image IMG2.

[0072] As can be seen from FIG. 4, b1 < b2. This reflects the difference in the distance from the first imaging device 5 to the danger area due to the difference in the tilt angle of the boom CR3. That is, in the captured images IMG1 and IMG2, the greater the tilt angle of the boom CR3, the farther the distance from the first imaging device 5 to the danger area, and the size of the danger area shown in the captured image is set to reflect that it becomes smaller.

[0073] The size of the danger area in the captured images IMG1 and IMG2 is determined based on the scale specified by the scale specifying unit 103. As described above, the scale is derived using the size of the reference object in each of the reference images obtained by imaging a reference object with known dimensions with the boom CR3 at different tilt angles.

[0074] For example, with the tilt angle of the boom CR3 maximized, a reference object of a predetermined length is placed directly below the first imaging device 5 and imaged. Note that the reference object only needs to have a known length, and may be, for example, a line segment drawn on the ground surface. Then, the length (for example, the number of pixels) of the reference object shown in the captured image (reference image) is measured. Thereby, the correspondence between the actual dimension when the tilt angle of the boom CR3 is maximum and the size of the reference object on the reference image can be specified.

[0075] Similarly, with the tilt angle of the boom CR3 minimized, a reference object of a predetermined length is placed directly below the first imaging device 5 and imaged. Then, the length (for example, the number of pixels) of the reference object shown in the captured image (reference image) is measured. Thereby, the correspondence between the actual dimension when the tilt angle of the boom CR3 is minimum and the size of the reference object on the reference image can be specified.

[0076] Then, from the correspondence relationship between the actual dimension and the size of the reference object on the reference image when the inclination angle of the boom CR3 is maximum, and the correspondence relationship between the actual dimension and the size of the reference object on the reference image when the inclination angle of the boom CR3 is minimum, the correspondence relationship at other inclination angles is also derived. In other words, a function for calculating the scale from the inclination angle of the boom CR3 can be derived. If this function is stored in the large-capacity storage device 12 or the like as correlation information, the scale specification unit 103 can specify the scale using this function.

[0077] If such a function is stored as correlation information in the mass storage device 12 or the like, the scale specification unit 103 can specify the scale using this function. Then, by using the scale specified in this manner, the area setting unit 105 can set a danger area having a predetermined size in actual dimensions for each of the captured images IMG1 and IMG2.

[0078] For example, in the example of Fig. 4, the scale (number of pixels in the captured image / actual dimensions) when the inclination angle of the boom CR3 is steep is specified as X1, and the scale when the inclination angle of the boom CR3 is gentle is specified as X2. In this case, the area setting unit 105 multiplies a, which is the actual dimension width of the danger area, by the scale X1 to calculate the width b1 of the danger area in IMG1 (a x X1 = b1). In addition, the area setting unit 105 multiplies a, which is the actual dimension width of the danger area, by the scale X2 to calculate the width b2 of the danger area in IMG2 (a x X2 = b2).

[0079] As described above, the scale specifying unit 103 may specify a scale according to the tilt angle using correlation information derived using the size of a reference object with known dimensions in each of the reference images obtained by photographing a reference object with known dimensions at different tilt angles of the boom CR3. The area setting unit 105 may then set a danger area by applying the scale specified by the scale specifying unit 103. According to this configuration, the correlation information derived using the reference images obtained by actually photographing a reference object with known dimensions is used, making it possible to specify an appropriate scale.

[0080] After the danger area is set in the captured image IMG1 as described above, detection of a person from the danger area is performed. As described above, a detection model is used for detecting a person, so the input data generating unit 106 generates input data for this detection model.

[0081] Specifically, as shown in Fig. 4, the input data generating unit 106 cuts out the danger area set by the area setting unit 105 from the captured image IMG1 to generate an image IMG1a. Then, the generated image IMG1a is adjusted to a predetermined size (a square with one side having a length of C pixels in the example of Fig. 4) to generate an adjusted image IMG1b. This adjusted image IMG1b becomes input data for the detection model.

[0082] The input data generating unit 106 may cut out an area wider than the danger area set by the area setting unit 105. This makes it possible to detect a worker approaching the danger area, and also makes it possible to detect a person within a range that covers the actual danger area even if the reference position in the captured image IMG2 for setting the danger area is shifted due to the swinging of the hook CR5 or the like.

[0083] The method of generating input data from the captured image IMG2 is also similar. That is, the input data generating unit 106 cuts out the dangerous area set by the area setting unit 105 from the captured image IMG2 to generate an image IMG2a. Then, the generated image IMG2a is adjusted to a predetermined size (a square with one side having a length of C pixels in the example of FIG. 4) to generate an adjusted image IMG2b. This adjusted image IMG2b becomes input data for the detection model.

[0084] The generation of an adjusted image used when generating a detection model using machine learning is also the same as that described above. That is, in machine learning, a learning target area is cut out from a captured image, and the cut-out image is adjusted to a predetermined size to generate an adjusted image. The size of this adjusted image is the same as the adjusted image generated by the input data generation unit 106.

[0085] <Monitoring process flow> The flow of the risk area monitoring process by the information processing device 1 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the processing procedure of the monitoring process by the information processing device 1 according to the first embodiment of the present invention. This monitoring process includes a method of determining a usable image by the information processing device 1.

[0086] First, the image acquisition unit 101 acquires a photographed image (a downward image) photographed by the first photographing device 5. In addition, the side image acquisition unit 110 acquires a photographed image (a side image) photographed by the second photographing device 9 (S21).

[0087] Next, the inclination angle identification unit 102 identifies the inclination angle of the boom CR3 measured by the inclinometer 7 (S22). Then, the scale identification unit 103 identifies a scale according to the inclination angle for the captured images acquired by the image acquisition unit 101 and the side image acquisition unit 110 (S23). The method of identifying the scale from the inclination angle is as described above, and therefore detailed description will not be repeated here.

[0088] Next, the object detection unit 107 detects the hook CR5 from the captured image acquired by the image acquisition unit 101 (S24). For the detection of the hook CR5, a learning model (detection model) generated using machine learning such as a neural network can be used. Furthermore, this detection model may be one that detects only the hook CR5, or one that detects both the hook CR5 and a person (worker). The process of step S24 may be performed prior to the processes of steps S22 to S23, or may be performed in parallel with these processes.

[0089] Next, the region setting unit 105 applies the scale specified by the scale specifying unit 103 to set a danger region of a predetermined size in actual dimensions in the captured images acquired by the image acquiring unit 101 and the lateral image acquiring unit 110 (S25). In this process, the danger region is set based on the position of the hook CR5 detected by the target detecting unit 107. Also, for example, the region setting unit 105 may set a square with one side measuring 5 meters in actual dimensions and centered on the position of the hook CR5 as the danger region.

[0090] Next, the input data generation unit 106 cuts out the danger area specified by the area setting unit 105 from the captured image acquired by the image acquisition unit 101, adjusts the size, and generates an adjusted image (S26). Then, the target detection unit 107 detects workers by inputting the adjusted image generated by the input data generation unit 106 into the detection model for detecting people generated by the above-mentioned machine learning (S27).

[0091] Similarly, the lateral image region setting unit 111 cuts out a danger area from the captured image acquired by the lateral image acquisition unit 110, adjusts the size, and generates an adjusted image (S26). Then, the lateral image object detection unit 112 detects workers by inputting the adjusted image generated by the lateral image region setting unit 111 into the detection model for detecting people generated by the above-mentioned machine learning (S27).

[0092] Next, the image-to-be-used determination unit 113 determines which image is to be used to detect a person (worker) (S28). That is, the image-to-be-used determination unit 113 determines whether to detect a person using only the image from the second image capturing device 9, or whether to detect a person using both the image from the second image capturing device 9 and the image from the first image capturing device 5.

[0093] This is because the detection rate of the worker differs between the image captured by the first image capture device 5 and the image captured by the second image capture device 9. The second image capture device 9 captures the worker from an oblique side, so the person is captured relatively clearly, resulting in a high detection rate and a low false detection rate. On the other hand, the first image capture device 5 captures the worker from directly above, so the person is captured small, resulting in a low detection rate and a high false detection rate. In particular, since round objects are easily erroneously determined to be helmets, it is better to detect using only the image from the second image capture device 9 if possible, but if there is a possibility that the worker is not captured in this image, the worker is detected using images from both the second image capture device 9 and the first image capture device 5.

[0094] In other words, the mounting positions of the first photographing device 5 and the second photographing device 9 are determined so that the second detection unit 22 has a higher detection rate of the detection target and a lower rate of erroneous detection of the detection target compared to the first detection unit 21.

[0095] 6 is a diagram for explaining an example of a determination method of the used image determination unit 113 of the information processing device 1 according to this embodiment. The range photographed by the first camera 5 changes depending on the boom angle (for example, AR1, AR2), whereas the photographing range AS of the second camera 9 is fixed. In the example shown in FIG. 6, when the boom CR3 is at the lowest position, part of the danger area ar2 is outside the photographing range AS of the second camera 9.

[0096] Therefore, since the detection rates of workers differ between the first and second camera devices 5 and 9 as described above, if the worker is completely within the danger area ar2 in the image of the second camera device 9, or if, for example, more than 80% of the danger area ar2 is within the image of the second camera device 9, the worker is detected using only the image of the second camera device 9; otherwise, the worker is detected using images from both the second camera device 9 and the first camera device 5.

[0097] Specifically, the side image region setting unit 111 sets the danger region in the side image as described above, and at this time, for example, calculates the ratio of the following formula (Formula 1). Note that the area may be, for example, the number of pixels in the region.

[0098] Ratio of overlapping area of ​​risk area = overlapping area with risk area in side image / area of ​​risk area (Equation 1) Furthermore, the side image region setting unit 111 may calculate the depth ratio of the danger region in the side image seen from the second image capturing device 9 by using the following formula (Formula 2).

[0099] Depth ratio of the danger area in the side image = Depth of the danger area in the side image / Depth of the danger area (Equation 2) Then, for example, if the ratio calculated using the above (Equation 1) or (Equation 2) is 80% or more, the image usage determination unit 113 determines the presence or absence of a worker using the image from the second camera 9 only, and if not, determines the presence or absence of a worker using images from both the second camera 9 and the first camera 5.

[0100] The area in which the detection target (worker) can be detected is the area of ​​the danger area that overlaps with the area photographed by the second imaging device 9. The area in which the detection target (worker) cannot be detected is the area of ​​the danger area that does not overlap with the area photographed by the second imaging device 9.

[0101] Furthermore, determining the ratio (degree) of the area that overlaps with the area photographed by the second photographing device 9 within the risk area has the same meaning as determining the ratio (degree) of the area that does not overlap.

[0102] Returning to the description of the flowchart shown in Fig. 5, next, when it is determined that the presence or absence of a worker is to be determined using images from only the second photographing device 9, the used image determination unit 113 determines whether or not a person (worker) has been detected using only the detection result by the lateral image object detection unit 112. When it is determined that the presence or absence of a worker is to be determined using images from both the second photographing device 9 and the first photographing device 5, the used image determination unit 113 determines whether or not a person (worker) has been detected using the detection result by the lateral image object detection unit 112 and the detection result by the object detection unit 107 (S29).

[0103] In addition, when making a judgment using both the detection results by the lateral image object detection unit 112 and the detection results by the object detection unit 107, if a person (worker) is detected by either the lateral image object detection unit 112 or the object detection unit 107, the used image judgment unit 113 may determine that a person (worker) has been detected.

[0104] If the used image determination unit 113 determines that a person (worker) has not been detected (S29, No), the process returns to step S21 and the subsequent processes are repeated. If the used image determination unit 113 determines that a person (worker) has been detected (S29, Yes), it determines whether or not a person (worker) has been detected in a predetermined number of images immediately before (S30).

[0105] If a person (worker) is detected in the immediately preceding predetermined number of images (S30, No), it is assumed that an alarm has already been issued, and the process returns to step S21 and the subsequent processes are repeated. If a person (worker) is not detected in the immediately preceding predetermined number of images (S30, Yes), the alarm unit 114 issues an alarm (S31).

[0106] Finally, the intrusion counting unit 108 increments the intrusion count by one to indicate the number of times a person (worker) has been detected (S32), and the process returns to step S21 to repeat the subsequent steps. Note that the processes of S31 and S32 may be performed in parallel, or the process of S32 may be performed first.

[0107] When using the information processing device 1 to perform monitoring and warning to prevent accidents involving the dropping of a load suspended by a crane CR, it is preferable to focus on monitoring areas where there is a high need. Therefore, according to the above configuration, a danger area is set based on the position of the hook CR5 detected from the captured image. This allows an appropriate warning to be issued even when the hook CR5 swings and changes its position, or when the shooting direction of the first camera 5 shifts. Furthermore, according to the above configuration, the range in which the detection target is detected can be limited to within the danger area, making it possible to detect the detection target more efficiently and accurately than when the detection target is detected from the entire captured image.

[0108] As described above, the information processing device 1 includes an intrusion counting unit that counts the number of times a detection target (a person in the above example) has intruded into the danger area. When a detection target is detected in a danger area in one of a plurality of captured images captured in time series, the intrusion counting unit 108 increments the intrusion count by one on the condition that the detection target is not detected in the danger area in a predetermined number of captured images captured immediately before the captured image. Details will be described based on Fig. 9, but this configuration allows the number of intrusions into the danger area to be counted correctly.

[0109] In step S31, if there is no suspended load or if the suspended load is light and the risk of a falling accident is low, no alarm may be issued. There are no particular limitations on the method of detecting the presence or absence of a suspended load and the weight of the suspended load. For example, if a load cell for measuring the weight of the suspended load is installed on the crane CR, the measurement value of this load cell may be input to the information processing device 1. Then, the alarm unit 114 may issue an alarm only if the input measurement value is equal to or greater than a threshold value (e.g., 1 t).

[0110] <Example of warning> Fig. 7 is a diagram showing an example in which an alarm is issued by the process of Fig. 5. In this example, a person to be detected passes through a danger area shown by a circle in the figure. As in this example, the shape of the danger area may be circular. Of course, the shape of the danger area may be set arbitrarily, and may be, for example, a rectangle, a polygon, or an ellipse.

[0111] In the example of Fig. 7, a person moves from position P1 to P9, and the person is photographed at each position. However, the person is not successfully detected from the images taken at all positions, but is only successfully detected at the positions shown in the dashed frame in the figure. Also, suppose that the "predetermined number" in step S30 of Fig. 5 is, for example, 3.

[0112] In this case, when the person's position becomes P3, it is determined in step S29 of Fig. 5 that a person has been detected. Then, since it is determined that a person has not been detected in positions P1-P2, it is determined in the following step S30 that a person has not been detected in the immediately preceding predetermined number of images. As a result, when the person's position becomes P3, the process of step S31 is performed to initiate an alarm, and the process of step S32 increments the number of intrusions into the danger zone by one.

[0113] Next, when the person's position becomes P5, it is also determined in S29 that a person has been detected, but because the person has been detected in the image two images prior (the image taken when the person is at position P3), it is determined in S30 that the person has been detected in the specified number of images immediately prior. As a result, the number of intrusions is not incremented.

[0114] Similarly, when the person's position becomes P8, it is determined in step S29 that a person has been detected, but since the person has been detected in the three images taken before (the image taken when the person is at position P5), it is determined in step S30 that the person has been detected in the predetermined number of images immediately before. As a result, the number of intrusions is not incremented here. In this way, the process of FIG. 5 makes it possible to correctly count the number of intrusions into the danger zone.

[0115] The alarm termination conditions are not particularly limited. For example, a configuration may be adopted in which a person is detected from an area slightly larger than the danger area, and the alarm may be terminated when a person is detected outside the danger area. This allows the alarm to be terminated at the timing when the person's position reaches P9 in FIG. 7. In addition to this, for example, the alarm may be terminated when it is determined in step S29 that a person has not been detected a predetermined number of times in succession, or when a predetermined time has elapsed after the alarm was started.

[0116] The intrusion counting unit 108 may further include a function for tallying and outputting the counted number of intrusions. Similarly, the alarm unit 114 may further include a function for tallying and outputting the number of alarms.

[0117] According to these configurations, the number of intrusions into the danger zone and the number of warnings can be made known to the supervisor or worker of the work using the crane CR, and the supervisor can be prompted to take measures to prevent accidents and the worker can be made more aware of the danger zone. The output mode is not particularly limited, and for example, the number of intrusions and the number of warnings for each predetermined period may be output in the form of a graph.

[0118] <Advantages of the information processing device 1 according to the first embodiment> As described above, according to the information processing device 1 according to the first embodiment, when the ratio of the area where the second detection unit 22 cannot detect the detection target to the dangerous area (first predetermined area) is equal to or less than a predetermined value, the utilization image determination unit 113 detects the detection target using only the detection result by the second detection unit 22, and when the ratio is greater than the predetermined value, detects the detection target using the detection result by the first detection unit 21 and the detection result by the second detection unit 22. Therefore, even if there is a factor that hinders the detection of the detection target, the detection target can be appropriately detected. Also, for example, when the detection rate of the second detection unit 22 is high and the false detection rate is low, the second detection unit 22 can be given priority to detect the detection target better when there is no factor that hinders detection.

[0119] In addition, the utilized image determination unit 113 can obtain a similar effect by detecting the detection target using only the detection result by the second detection unit 22 when the proportion of the area in which the second detection unit 22 can detect the detection target to the danger area (first specified area) is equal to or greater than a second specified value, and detecting the detection target using the detection result by the first detection unit 21 and the detection result by the second detection unit 22 when the proportion is smaller than the second specified value.

[0120] The first detection unit 21 also includes an inclination angle identification unit 102 that identifies the inclination angle of the boom with respect to the horizontal direction, an area setting unit 105 that applies a scale according to the inclination angle to set a danger area (first predetermined area) centered on the hook in the downward image captured by the first camera device 5, and an object detection unit 107 that detects a detection object within the danger area (first predetermined area). Therefore, it is possible to appropriately detect a detection object within the danger area (first predetermined area) centered on the hook.

[0121] The second detection unit 22 includes a side image area setting unit 111 that sets a danger area (second predetermined area) below the hook detected in the side image captured by the second imaging device 9 at a position corresponding to the horizontal distance from the second imaging device 9 to the hook calculated from the inclination angle, and a side image object detection unit 112 that detects a detection target within the danger area (second predetermined area). Therefore, the detection target can be appropriately detected in the danger area (second predetermined area) that is set at a position corresponding to the horizontal distance to the hook.

[0122] (Embodiment 2) The information processing device according to the second embodiment of the present invention differs from the information processing device 1 according to the first embodiment only in that the functions of the side image object detection unit 112 and the used image determination unit 113 are different. Therefore, the information processing device according to the second embodiment will be described with the reference number 1', and the side image object detection unit and the used image determination unit according to the second embodiment will be described with the reference numbers 112' and 113', respectively. Note that detailed descriptions of other configurations and functions that overlap with the first embodiment will not be repeated.

[0123] Fig. 8 is a diagram for explaining the frame of the crane CR. In Fig. 8, the cart section CR1 of the crane CR is indicated by a solid line, and the base section CR2 is indicated by a dotted line. The cart section CR1 is further provided with a frame (skeleton) CR1a in order to support the base section CR2. In addition, a second imaging device 9 is attached to the base section CR2.

[0124] The base CR2 rotates, but the cart CR1 and its frame (skeleton) CR1a do not rotate. That is, the second imaging device 9 attached to the base CR2 rotates, but the frame CR1a does not rotate, so the frame CR1a may be photographed by the second imaging device 9. Furthermore, since the frame CR1a is relatively close to the second imaging device 9, the frame CR1a may occupy a large part of the danger area in the image.

[0125] Fig. 9 is a diagram showing a case where the frame CR1a of the crane CR is included in the side image captured by the second imaging device 9. As shown in Fig. 9, the frame CR1a occupies a large portion of the danger area in the side image, making it difficult to detect the detection target. Note that in the second embodiment, the area in which the frame CR1a appears in the side image is an area in which the detection target cannot be detected.

[0126] Therefore, as described above, since the detection rates of workers differ between the first photographing device 5 and the second photographing device 9, if the frame CR1a is not within the danger zone of the second photographing device 9, the worker is detected only using the image of the second photographing device 9, and if this is not the case, detection is performed using both the second photographing device 9 and the first photographing device 5.

[0127] Specifically, the lateral image object detection unit 112' detects the frame CR1a, and if the frame CR1a is detected and its area (e.g., the number of pixels) is equal to or greater than a predetermined value (e.g., 25%) of the danger area, the utilization image determination unit 113' determines whether or not a worker is present using the images captured by the second imaging device 9 and the first imaging device 5, and if not, performs the determination using only the second imaging device 9. Alternatively, the determination may be made based on whether or not the area of ​​the frame CR1a is equal to or greater than a predetermined value.

[0128] The detection of the frame CR1a may be performed by machine learning such as a neural network, or may be performed by image processing. In the case of image processing, for example, when the color of the frame CR1a is determined, the lateral image object detection unit 112' may determine that the frame CR1a is an area in which the number of pixels having color information within a predetermined range is included at a predetermined rate or more in the dangerous area. The reason for using "color information within a predetermined range" is that the color of a part of the frame CR1a changes over time, or the color changes as a result of shooting on sunny or cloudy days. The range of the color information may be, for example, a color in the RGB (red, green, blue) format, a hue in the HSV (hue, saturation, brightness) format, or a color difference in the YUV format.

[0129] In order to reduce the amount of calculation, the side image may be reduced before the above processing is performed, rather than using the risk area in the side image as is. Alternatively, in order to improve the detection accuracy, the image may be subjected to noise removal processing, etc.

[0130] <Advantages of the information processing device 1' according to the second embodiment> As described above, according to the information processing device 1 of the first embodiment, when the area of ​​the frame (obstacle) or the ratio of the area of ​​the frame (obstacle) in the side image is equal to or smaller than a predetermined value, the utilization image determination unit 113′ detects the detection target using only the detection result by the second detection unit 22, and when the area of ​​the frame (obstacle) or the ratio of the area of ​​the frame (obstacle) in the side image is greater than the predetermined value, the utilization image determination unit 113′ detects the detection target using the detection result by the first detection unit 21 and the detection result by the second detection unit 22. Therefore, when the ratio of the obstacle in the image captured by the second imaging device 9 is large, the detection target can be appropriately detected by detecting the detection target using the detection result by the first detection unit 21 and the detection result by the second detection unit 22.

[0131] Furthermore, the lateral image object detection unit 112' determines, as a frame (obstacle), an area in which the number of pixels having color information within a predetermined range is included at a predetermined rate or more in the lateral image captured by the second image capturing device 9. Therefore, the lateral image object detection unit 112' can detect a frame by image processing.

[0132] <Example of software implementation> The functions of the information processing device 1 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block (particularly, the control unit 10) of the device.

[0133] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0134] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0135] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0136] 〔summary〕 An information processing device according to a first aspect of the present invention includes: a first detection unit that is attached to an upper rotating body of the conveying machine and that detects a detection target by acquiring an image from a first photographing device that photographs a lower side; a second detection unit that is attached to a position of the conveying machine different from that of the first photographing device and that detects the detection target by acquiring an image from a second photographing device that photographs a lateral side; and a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on images from both the first and second imaging devices, or images only from the second imaging device, and determines whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

[0137] An information processing device according to a second aspect of the present invention is the information processing device according to the first aspect, The first detection unit sets a first predetermined area in the lower image captured by the first imaging device, The region in which the detection target cannot be detected is a region in the first predetermined region that does not overlap with a region photographed by the second photographing device, The determination unit is When a ratio of an area in which the detection target cannot be detected to a first predetermined area is equal to or less than a predetermined value, the detection target is detected using only a detection result by the second detection unit; When the ratio is greater than the predetermined value, the detection target is detected using the detection result by the first detection unit and the detection result by the second detection unit.

[0138] An information processing device according to a third aspect of the present invention is the information processing device according to the second aspect, the transport machine is a crane, the first photographing device is attached near the tip of the boom of the crane and photographs a lower side of the hook of the boom from above, the second photographing device is attached at a position for photographing a lower side of the hook of the boom from the side, The first detection unit is an inclination angle identification unit that identifies an inclination angle of the boom with respect to a horizontal direction; a region setting unit that sets the first predetermined region centered on the hook in the lower image captured by the first photographing device by applying a scale corresponding to the tilt angle; and an object detection unit that detects the detection object within the first predetermined area.

[0139] An information processing device according to a fourth aspect of the present invention is the information processing device according to the third aspect, The second detection unit is a side image area setting unit that sets a second predetermined area below the hook detected in the side image captured by the second photographing device, at a position corresponding to a horizontal distance from the second photographing device to the hook calculated from the inclination angle; and a lateral image object detection unit that detects the detection object within the second predetermined area.

[0140] An information processing device according to a fifth aspect of the present invention is the information processing device according to the first aspect, The region in which the detection target cannot be detected is a region in the side image captured by the second imaging device where an obstacle overlaps when detecting the detection target, The second detection unit detects the obstacle when detecting the detection target from the side image captured by the second imaging device, The determination unit is When the area of ​​the obstacle or the ratio of the area of ​​the obstacle in the side image to a predetermined value is equal to or smaller than a predetermined value, the detection target is detected using only the detection result by the second detection unit; When the area of ​​the obstacle or the proportion of the area of ​​the obstacle in the side image is greater than the predetermined value, the detection result by the first detection unit and the detection result by the second detection unit are used to detect the detection target.

[0141] An information processing device according to a sixth aspect of the present invention is the information processing device according to the fifth aspect, the transport machine is a crane, the first photographing device is attached near the tip of the boom of the crane and photographs a lower side of the hook of the boom from above, the second photographing device is attached at a position for photographing a lower side of the hook of the boom from the side, The first detection unit is an inclination angle identification unit that identifies an inclination angle of the boom with respect to a horizontal direction; a region setting unit that sets the first predetermined region centered on the hook in the lower image captured by the first photographing device by applying a scale corresponding to the tilt angle; and an object detection unit that detects the detection object within the first predetermined area.

[0142] An information processing device according to a seventh aspect of the present invention is the information processing device according to the sixth aspect, The second imaging device is attached to the upper rotating body of the crane, The second detection unit is a side image area setting unit that sets a second predetermined area below the hook detected in the side image captured by the second photographing device, at a position corresponding to a horizontal distance from the second photographing device to the hook calculated from the inclination angle; The system includes a side image object detection unit that detects a frame constituting the crane as the obstacle from the side image captured by the second photographing device, and detects the detection object within the second specified area.

[0143] An information processing device according to an eighth aspect of the present invention is the information processing device according to the seventh aspect, The lateral image object detection unit determines, as the frame, an area in the lateral image captured by the second imaging device in which a number of pixels having color information within a predetermined range is included at a predetermined rate or more.

[0144] An information processing device according to a ninth aspect of the present invention is the information processing device according to any one of the first to eighth aspects, The mounting positions of the first photographing device and the second photographing device are determined so that the second detection unit has a higher detection rate of the detection object and a lower false detection rate of the detection object compared to the first detection unit.

[0145] An information processing system according to a tenth aspect of the present invention includes: A first photographing device that is attached to an upper rotating body of the conveying machine and photographs the lower side; A second photographing device is attached to a position of the conveying machine different from that of the first photographing device and photographs a lateral side of the conveying machine; a first detection unit that acquires an image from the first image capture device and detects a detection target; a second detection unit that acquires an image from the second image capture device and detects the detection target; and a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on images from both the first and second imaging devices, or images only from the second imaging device, and determines whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

[0146] An information processing method according to an eleventh aspect of the present invention includes the steps of: a step of detecting a detection target by acquiring an image from a first image capturing device attached to an upper rotating body of the conveying machine and capturing an image of a lower side; a step of detecting the detection target by acquiring an image from a second image capturing device that is attached to a position of the conveying machine different from that of the first image capturing device and captures an image of a lateral side; The method includes a step of extracting an area in which the detection target cannot be detected based on images from both the first and second photographing devices, or images only from the second photographing device, and determining whether to detect the detection target using the detection results from the images from the first photographing device and the detection results from the images from the second photographing device depending on the area, or to detect the detection target using only the detection results from the images from the second photographing device.

[0147] A program according to a twelfth aspect of the present invention includes: On the computer, A process of acquiring an image from a first image capturing device attached to an upper rotating body of the transport machine and capturing an image of a lower side, and detecting a detection target; A process of detecting the detection target by acquiring an image from a second image capturing device that is attached to a position of the conveying machine different from that of the first image capturing device and captures an image of a lateral side of the conveying machine; Based on images from both the first and second photographing devices, or only the image from the second photographing device, the second detection unit extracts an area in which the detection target cannot be detected, and determines whether to detect the detection target using the detection result from the image from the first photographing device and the detection result from the image from the second photographing device, or to detect the detection target using only the detection result from the image from the second photographing device, depending on the area.

[0148] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0149] 1. Information processing device 5. First Camera 7 Inclinometer 9 Second Camera 10 Control section 11. Memory 12 Mass storage 13 Imaging device IF 14 Inclinometer IF 15 Display device IF 16 Sound output device IF 17 Network Interface 21 First detection unit 22 Second detection section 100 Information Processing Systems 101 Image acquisition unit 102 Inclination angle identification part 103 Scale Specific Section 104 Altitude judgment section 105 Area setting section 106 Input data generation unit 107 Target detection unit 108 Intrusion Count Unit 110 Lateral image acquisition unit 111 Side image area setting unit 112 Lateral image object detection unit 113 Usage image determination unit 114 Alarm section

Claims

1. a first detection unit that is attached to an upper rotating body of the conveying machine and that detects a detection target by acquiring an image from a first photographing device that photographs a lower side; a second detection unit that is attached to a position of the conveying machine different from that of the first imaging device and that detects the detection target by acquiring an image from a second imaging device that captures an image of a lateral side; an information processing device comprising: a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on images from both the first imaging device and the second imaging device, or images only from the second imaging device, and determines whether to detect the detection target using a detection result by the first detection unit and a detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

2. The first detection unit sets a first predetermined area in the lower image captured by the first imaging device, The region in which the detection target cannot be detected is a region in the first predetermined region that does not overlap with a region photographed by the second photographing device, The determination unit is When a ratio of an area in which the detection target cannot be detected to a first predetermined area is equal to or less than a predetermined value, the detection target is detected using only a detection result by the second detection unit; When the ratio is greater than the predetermined value, the detection target is detected using a detection result by the first detection unit and a detection result by the second detection unit. The information processing device according to claim 1 .

3. the transport machine is a crane, the first photographing device is attached near a tip of a boom of the crane and photographs a lower side of a hook of the boom from above, the second photographing device is attached at a position for photographing a lower side of the hook of the boom from a side, The first detection unit is an inclination angle identification unit that identifies an inclination angle of the boom with respect to a horizontal direction; a region setting unit that sets the first predetermined region centered on the hook in the lower image captured by the first photographing device by applying a scale corresponding to the tilt angle; an object detection unit that detects the detection object within the first predetermined area, The information processing device according to claim 2 .

4. The second detection unit is a side image area setting unit that sets a second predetermined area below the hook detected in the side image captured by the second photographing device, at a position corresponding to a horizontal distance from the second photographing device to the hook calculated from the inclination angle; a lateral image object detection unit that detects the detection object within the second predetermined area, The information processing device according to claim 3 .

5. The region in which the detection target cannot be detected is a region in which an obstacle overlaps when detecting the detection target in the side image captured by the second imaging device, The second detection unit detects the obstacle when detecting the detection target from the side image captured by the second imaging device, The determination unit is When an area of ​​the obstacle or a ratio of an area of ​​the obstacle in the side image is equal to or smaller than a predetermined value, the detection target is detected using only a detection result by the second detection unit; when an area of ​​the obstacle or a ratio of an area of ​​the obstacle in the side image is larger than the predetermined value, the detection target is detected using a detection result by the first detection unit and a detection result by the second detection unit. The information processing device according to claim 1 .

6. the transport machine is a crane, the first photographing device is attached near a tip of a boom of the crane and photographs a lower side of a hook of the boom from above, the second photographing device is attached at a position for photographing a lower side of the hook of the boom from a side, The first detection unit is an inclination angle identification unit that identifies an inclination angle of the boom with respect to a horizontal direction; a region setting unit that sets a first predetermined region centered on the hook in the lower image captured by the first photographing device by applying a scale corresponding to the tilt angle; an object detection unit that detects the detection object within the first predetermined area, The information processing device according to claim 5 .

7. The second imaging device is attached to the upper rotating body of the crane, The second detection unit is a side image area setting unit that sets a second predetermined area below the hook detected in the side image captured by the second photographing device, at a position corresponding to a horizontal distance from the second photographing device to the hook calculated from the inclination angle; a side image object detection unit that detects a frame constituting the crane as the obstacle from the side image captured by the second imaging device and detects the detection object within the second predetermined area, The information processing device according to claim 6.

8. the lateral image object detection unit determines, in the lateral image captured by the second image capture device, an area in which a number of pixels having color information within a predetermined range is included at a predetermined rate or more as the frame; The information processing device according to claim 7.

9. The mounting positions of the first photographing device and the second photographing device are determined so that the second detection unit has a higher detection rate of the detection object and a lower erroneous detection rate of the detection object compared to the first detection unit. The information processing device according to any one of claims 1 to 8.

10. A first photographing device that is attached to an upper rotating body of the conveying machine and photographs an image of a lower side; A second photographing device that is attached to a position of the conveying machine different from that of the first photographing device and photographs a lateral side of the conveying machine; a first detection unit that acquires an image from the first image capture device and detects a detection target; a second detection unit that acquires an image from the second image capture device and detects the detection target; an information processing system comprising: a determination unit that extracts an area in which the second detection unit cannot detect the detection target based on images from both the first imaging device and the second imaging device, or images only from the second imaging device, and determines whether to detect the detection target using the detection result by the first detection unit and the detection result by the second detection unit, or to detect the detection target using only the detection result by the second detection unit, depending on the area.

11. a step of detecting a detection target by acquiring an image from a first image capturing device attached to an upper rotating body of the conveying machine and capturing an image of a lower side; a step of detecting the detection target by acquiring an image from a second imaging device that is attached to a position of the conveying machine different from that of the first imaging device and captures an image of a lateral side; extracting an area in which the detection target cannot be detected based on images from both the first and second imaging devices, or images only from the second imaging device, and determining whether to detect the detection target using detection results from the images from the first imaging device and the images from the second imaging device depending on the area, or to detect the detection target using only the detection results from the images from the second imaging device.

12. On the computer, A process of acquiring an image from a first image capturing device attached to an upper rotating body of the conveying machine and capturing an image of a lower side, and detecting a detection target; A process of detecting the detection target by acquiring an image from a second image capturing device that is attached to a position of the conveying machine different from that of the first image capturing device and captures an image of a lateral side of the conveying machine; extracting an area in which the detection target cannot be detected based on images from both the first and second photographing devices, or images only from the second photographing device, and determining whether to detect the detection target using the detection results from the images from the first photographing device and the detection results from the images from the second photographing device depending on the area, or to detect the detection target using only the detection results from the images from the second photographing device.

Images