Monitoring device, program and notification method
The monitoring device uses three-dimensional position information to convert and superimpose threshold values accurately on images, addressing the inaccuracy in conventional methods and enhancing safety monitoring.
Patent Information
- Application Number
- JP2023079546
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
AI Technical Summary
Conventional methods fail to accurately represent threshold values, such as dangerous areas, on a photographed image due to changes in distance from the suspended load or worker to the imaging device, leading to inaccurate representation in two-dimensional space.
A monitoring device that utilizes captured images and distance information to determine the three-dimensional position of objects, converting these positions to two-dimensional coordinates, and superimposes threshold values accurately on the image based on three-dimensional position information.
Enables precise representation of threshold values, like dangerous areas, on the photographed image, allowing for accurate safety monitoring and risk determination in real space.
Smart Images

Figure 2024163705000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a monitoring device, a program, and a notification method. [Background technology]
[0002] Patent Document 1 discloses a construction support system that detects the worker and the suspended load by analyzing image data of the work area captured by an imaging device mounted facing downward on the tip of the boom of a work machine, obtains height information from position information acquired by an antenna, and outputs an alarm based on the positional relationship in three-dimensional space. According to the construction support system of Patent Document 1, the detection results of the worker and the suspended load are displayed superimposed on the image together with a set threshold value in real space (for example, a danger area). Summary of the Invention [Problem to be solved by the invention]
[0003] However, with conventional technology, when the detection results of the worker and the suspended load are displayed superimposed on an image together with a set threshold value in real space (e.g., a danger area), there is a problem that the threshold value (e.g., a danger area) cannot be accurately grasped because the information in real space is not reflected.
[0004] For example, when an image is captured by an imaging device attached to the tip of a boom of a work machine facing downward, the distance of the suspended load or worker to the imaging device will change depending on the scene. In this case, since the size changes on the image, if a position in real space is set as a threshold (for example, a danger area), it cannot be expressed well in a two-dimensional image.
[0005] The present invention has been made in view of the above, and has an object to accurately represent a set threshold value (for example, a danger area) in real space on a captured image. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides a monitoring device that monitors the working condition of a work machine, comprising an object detection unit that detects three-dimensional position information of an object that is a monitoring target using a captured image and distance information, a work content determination unit that determines whether or not work performed by the work machine is safe based on the three-dimensional position information of the object acquired by the object detection unit, a two-dimensional conversion unit that calculates the two-dimensional coordinates within the captured image of a set threshold value in real space based on the three-dimensional position information of the object acquired by the object detection unit, and a notification unit that superimposes the threshold value in real space on the captured image in accordance with the two-dimensional coordinates and notifies the result determined by the work content determination unit. Effect of the Invention
[0007] According to the present invention, it is possible to accurately represent a set threshold value (for example, a danger area) in real space on a captured image. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is an overall configuration diagram of a crane system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of the monitoring device. [Diagram 3] FIG. 3 is a functional block diagram showing functions realized by the monitoring device. [Figure 4] FIG. 4 is a diagram showing the relationship between luggage in a captured image and a danger area, which is a threshold value in real space. [Diagram 5] FIG. 5 is a flowchart showing an outline of the flow of processing for determining the contents of a processing job in the monitoring device. [Figure 6] FIG. 6 is a diagram showing a specific example of the work content determined by the work content determination unit, as viewed from directly above the work. [Figure 7] FIG. 7 is an enlarged view of the target area of the determination example of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a monitoring device, a program, and a notification method will be described in detail with reference to the accompanying drawings.
[0010] 1 is an overall configuration diagram of a crane system 1 according to an embodiment. The crane system 1 of the present embodiment is constructed by a monitoring device 9, a communication terminal 10, a crane control device 30, and a management server 50.
[0011] The monitoring device 9, the communication terminal 10, the crane control device 30, and the management server 50 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. The communication network 100 may include not only wired communication but also networks using wireless communication such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution).
[0012] The monitoring device 9 and the crane control device 30 are provided at the work site of the crane 210. The monitoring device 9 and the crane control device 30 may each be configured integrally with the crane 210. An operator at the site can check the work content at the site using the crane control device 30. Also, a manager at a remote location can check the work content at the site using the communication terminal 10.
[0013] The monitoring device 9 includes an imaging device 7 such as a camera, and a distance sensor 8. The imaging device 7 is provided on the crane 210. The imaging device 7 captures images of subjects such as the hook 221 of the crane 210, the baggage P, and surrounding objects and the worker W, and acquires captured images (two-dimensional data) such as image data, color, and brightness information. The distance sensor 8 is provided on the crane 210. The distance sensor 8 measures the distance to the subject captured by the imaging device 7, and acquires distance information (three-dimensional data) such as point cloud data.
[0014] The image capturing device 7 may capture time lapse images or videos, in which case the distance sensor 8 acquires distance information in synchronization with the frame rate of the image capturing device 7.
[0015] The distance sensor 8 is a ToF (Time of Flight) sensor. The distance sensor 8 measures the distance from a light source to an object by irradiating the object with laser light from a light source and measuring the scattered or reflected light.
[0016] Specifically, the distance sensor 8 is a LiDAR (Light Detection and Ranging) sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of the ToF sensor, the distance may be measured by a phase difference detection method. In the phase difference detection method, a laser beam that is amplitude-modulated at a fundamental frequency is irradiated onto a measurement range, and the phase difference between the irradiated light and the reflected light is measured. The time obtained by the phase difference detection method is multiplied by the speed of light to calculate the distance.
[0017] In this embodiment, the monitoring device 9 is provided with the imaging device 7 and the distance sensor 8, but is not limited to this. For example, the monitoring device 9 may be provided with a stereo camera capable of three-dimensional measurement using two or more cameras, instead of the imaging device 7 and the distance sensor 8, and the captured image and distance information may be obtained by the stereo camera.
[0018] The monitoring device 9 detects the object (e.g., worker W, hook 221 of crane 210, and luggage (hanging load) P suspended from hook 221 of crane 210) based on the captured image and distance information, and acquires three-dimensional position information of the object. More specifically, the monitoring device 9 sets a reference position from the three-dimensional position information of the detected object. The monitoring device 9 determines the working status of the crane 210 based on the object and the three-dimensional position information of the object. The monitoring device 9 communicates the determined working status of the crane 210 to the crane control device 30.
[0019] In addition, the monitoring device 9 determines dangerous areas in the environment for the worker W from the three-dimensional position information of the worker W, reverse-calculates the coordinates in the captured image, and draws them on the display device 925 (see Figure 2), thereby accurately representing the threshold in the real space set on the display device 925 on the image.
[0020] The crane control device 30 is a device that controls the crane 210 in accordance with the work situation transmitted from the monitoring device 9. For example, if a dangerous situation occurs in which the worker W gets too close to the suspended load, the crane control device 30 performs safety control such as stopping the operation of the crane 210.
[0021] Next, the hardware configuration of the monitoring device 9 will be described.
[0022] Fig. 2 is a block diagram showing an example of a hardware configuration of the monitoring device 9. As shown in Fig. 2, the monitoring device 9 includes a controller 900 that controls the processing or operation of the monitoring device 9, as well as the imaging device 7 and distance sensor 8 as shown in Fig. 1.
[0023] The controller 900 includes an imaging device I / F (Interface) 901, a sensor device I / F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 915, a HDD (Hard Disk Drive) controller 914, a network I / F 916, a DVD-RW (Digital Versatile Disc Rewritable) drive 918, a media I / F 922, an external device connection I / F 923, a timer 924 and a display device 925.
[0024] Of these, the photographing device I / F 901 is an interface for transmitting and receiving various data or information to and from the photographing device 7. The sensor device I / F 902 is an interface for transmitting and receiving various data or information to and from the distance sensor 8. The bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 911 shown in FIG.
[0025] Furthermore, the CPU 911 controls the overall operation of the monitoring device 9. The ROM 912 stores programs used to drive the CPU 911, such as the IPL. The RAM 913 is used as a work area for the CPU 911. The HD 915 stores various data such as programs. The HDD controller 914 controls reading and writing of various data from and to the HD 915 under the control of the CPU 911. The network I / F 916 is an interface for data communication using the communication network 100.
[0026] The DVD-RW drive 918 controls reading and writing of various data from and to a DVD-RW 917 as an example of a removable recording medium. The medium is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc. The media I / F 922 controls reading and writing (storing) of data from and to a recording medium 921 such as a flash memory. The external device connection I / F 923 is an interface for connecting an external device such as an external PC 930. The timer 924 is a measuring device having a time measurement function. The timer 924 may be a software timer implemented by a computer.
[0027] The programs executed by the monitoring device 9 of this embodiment are provided as files in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD.
[0028] The program executed by the monitoring device 9 of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the monitoring device 9 of the present embodiment may be provided or distributed via a network such as the Internet. The program executed by the monitoring device 9 of the present embodiment may be provided by being pre-installed in a ROM or the like.
[0029] The display device 925 is a device that displays drawing contents (for example, see FIG. 6 described later) from the monitoring device 9. The display device 925 may be a monitor device, or an external PC 930 such as a tablet terminal or a PC (Personal Computer).
[0030] Next, the functions realized by the monitoring device 9 will be described.
[0031] Here, Fig. 3 is a functional block diagram showing functions realized by the monitoring device 9. As shown in Fig. 3, the monitoring device 9 has a data receiving unit 91, an object detecting unit 92, a work content determining unit 93, a two-dimensional converting unit 94, and a display control unit 95 functioning as a notification unit. Each of these units is a function realized by any of the components shown in Fig. 2 operating in response to an instruction from the CPU 911 in accordance with a program loaded from the HD 915 onto the RAM 913, or a means for performing the function. The monitoring device 9 notifies the crane control device 30 of the work status and draws on the display device 925 by each function fulfilling the following roles.
[0032] The data receiving unit 91 receives the captured image captured by the imaging device 7 and receives the distance information measured by the distance sensor 8 .
[0033] The object detection unit 92 receives the captured image and distance information from the data receiving unit 91. Using the captured image and distance information, the object detection unit 92 detects an object in the image that is a target of monitoring (for example, a worker W, the hook 221 of the crane 210, and a load (hanging load) P suspended from the hook 221 of the crane 210). As a method for detecting an object, the object detection unit 92 may use a detection algorithm that is deductively designed using a feature amount of the object in the image, or may use a learning algorithm that is characterized by detecting the object based on similarity with a model created in advance.
[0034] Furthermore, the object detection unit 92 detects the three-dimensional position information of the object to be monitored from the median and average of the three-dimensional position information within the detected range. Detecting the three-dimensional position information of the object enables safety monitoring in real space. For example, the most dangerous scene in a lifting operation is a situation in which a load P is being lifted by the hook 221 of the crane 210. When the load P is being lifted by the hook 221 of the crane 210, there is a risk that the load P will move and fall in the worst case scenario.
[0035] The work content determination unit 93 determines whether the work by the crane 210 is currently safe (whether it is in a safe state or a dangerous state) based on the three-dimensional position information of the object acquired by the object detection unit 92. Specifically, the work content determination unit 93 determines whether the load P is suspended by grasping the relative three-dimensional position information of the hook 221 of the crane 210 and the load P (for example, when the hook 221 and the load P are within 2 m in each of x, y, and z). The work content determination unit 93 can detect truly dangerous work by determining that it is dangerous when the worker W is present within a predetermined range of the suspended load (suspended load) P (for example, within a 3 m radius of the suspended load P).
[0036] The two-dimensional conversion unit 94 calculates the two-dimensional coordinates within the captured image of the danger area, which is the set threshold value in real space, based on the three-dimensional position information of the luggage P detected from among the targets obtained by the object detection unit 92.
[0037] For example, in the case of an imaging device 7 that has been corrected to an ideal pinhole image, the two-dimensional conversion unit 94 converts the size [mm] in real space into the two-dimensional size [pix] on the captured image using the following formula (Formula 1). I=(S*f) / (Z*P)…(Formula 1) I: Two-dimensional pixel size of the object on the captured image [pix] S: Size of the object in real space [mm] P: Pixel pitch of the captured image [mm] Z: Distance between the imaging device and the work position [mm] f: focal length of the image capture device [mm]
[0038] For example, under the conditions below, if the size of an object in real space is 2,000 mm, the two-dimensional pixel size of the object in the captured image will be as follows: Two-dimensional pixel size [pix] = 2,000 / (0.006*10,000 / 2) = approx. 66.7 [pix] Focal length of camera 7: 2mm Pixel pitch of the imaging device 7: 0.006 mm Distance between the imaging device 7 and the work position: 10,000 mm
[0039] In other words, under the above conditions, if you want to draw a length of 2,000 mm (= 2 m) in real space at the work position, 66.7 pix corresponds to the two-dimensional size on the captured image.
[0040] 4 is a diagram showing the relationship between the baggage P in the captured image and the danger area, which is a threshold in real space. Since the position and three-dimensional coordinates of the suspended baggage P in the captured image are known by the object detection unit 92, the two-dimensional conversion unit 94 can calculate the real size of the baggage P by the following formula, assuming that the three-dimensional coordinates are (x, y, z) = (real_x, real_y, real_z), the width of the baggage P in the captured image is bbox.width[pix], and the height of the baggage P is bbox.height[pix]. (package width [mm]) = bbox.width * 0.006 * real_z / 2 (Luggage vertical width [mm]) = bbox.height * 0.006 * real_z / 2
[0041] If the actual size of the package P is known, the two-dimensional conversion unit 94 can obtain the two-dimensional pixel size of the package P at the work position from the above-mentioned (Equation 1).
[0042] The central coordinates of the package P are (x, y) = (bbox.x, bbox.y). Since bbox.x and bbox.y can be regarded as the amount of deviation from the center position of the image, the two-dimensional conversion unit 94 needs to convert this amount of deviation from real_z to the two-dimensional pixel size at the work position. The conversion formula at this time is as follows: (Horizontal deviation from center [mm])=bbox.x*0.006*real_z / 2 (Vertical deviation from center [mm])=bbox.y*0.006*real_z / 2
[0043] The two-dimensional conversion unit 94 obtains the x, y coordinates at the work position based on the above-mentioned amount of deviation. (x coordinate at working position) = (lateral deviation from center [mm]) / (0.006 * 10,000 / 2) (y coordinate at working position) = (vertical deviation from center [mm]) / (0.006 * 10,000 / 2)
[0044] By these calculations in the two-dimensional conversion unit 94, the set threshold value (danger area) in real space can be drawn as shown below (for example, see FIG. 6 described later) with the center of the captured image of the baggage P as the starting point. By performing this calculation, the set threshold value (danger area) in real space can be accurately expressed on the captured image.
[0045] The display control unit 95 superimposes the set threshold value in real space (danger area) on the captured image and displays it on the display device 925, and also displays the result of the determination by the work content determination unit 93 on the display device 925. In other words, the display control unit 95 is a notification unit that superimposes the set threshold value in real space (danger area) on the captured image and notifies the result of the determination by the work content determination unit 93.
[0046] Next, the process of determining the processing work content in the monitoring device 9 will be described.
[0047] Here, FIG. 5 is a flow chart showing a flow of a process of determining the contents of a processing operation in the monitoring device 9. As shown in FIG. 5, first, the object detection unit 92 receives the photographed image (two-dimensional data) photographed by the photographing device 7 and the distance information (three-dimensional data) acquired by the distance sensor 8 from the data receiving unit 91, and detects objects (e.g., the worker W, the hook 221 of the crane 210, and the baggage (hanging load) P suspended from the hook 221 of the crane 210) on the photographed image (step S1). More specifically, the object detection unit 92 detects subjects such as the hook 221, the baggage P, and the worker W based on color and brightness information of the photographed image photographed by the photographing device 7. The detection method may use a detection algorithm deductively designed using the feature amount of the subject on the image, or may use a learning algorithm characterized by detection based on similarity with a model created in advance.
[0048] Next, the object detection unit 92 detects the three-dimensional position information of the target object from the median and average values of the three-dimensional position information within the detected range (step S2).
[0049] Next, the task content determination unit 93 executes a tracking process using a previously detected object D3 that was previously detected (step S3). More specifically, the task content determination unit 93 associates the previously detected object D3 with the object on the image detected in step S1 and the three-dimensional position information of the object detected in step S2. The task content determination unit 93 adds time information such as speed and acceleration to the associated information, and generates a detection object D4 as a result of executing the tracking process.
[0050] Next, the work content determination unit 93 determines whether the work by the crane 210 is in a safe state or a dangerous state (steps S4 to S11).
[0051] More specifically, the work content determination unit 93 calculates the distance in three-dimensional position information between the hook 221 and the luggage P based on the detection object D4, and determines whether the distance is within a preset threshold range (step S4).
[0052] If the difference is not within the threshold range (No in step S4), the work content determination unit 93 determines that the work is before the work (step S7).
[0053] On the other hand, if it is within the threshold range (Yes in step S4), the work content judgment unit 93 judges whether the amount of movement of the three-dimensional position information of the same object between frames is within a predetermined threshold range based on the detected object D4 (step S5).
[0054] If the load is not within the threshold range (No in step S5), the work content determination unit 93 determines that the load P is shaking (step S8).
[0055] On the other hand, if it is within the threshold range (Yes in step S5), the work content determination unit 93 determines that work is in progress and determines the work content from the position of the detected object based on the detected object D4 (step S6). More specifically, if the position of the detected object is within 30 cm from the ground, the work content determination unit 93 generates the work content D5 as "ground cutting work". Also, if the position of the detected object is 30 cm or more from the ground, the work content determination unit 93 generates the work content D5 as "suspension work".
[0056] Next, the work content determination unit 93 calculates the distance between the worker W and the baggage P based on the three-dimensional position information, and determines whether the distance is within a preset threshold range (step S9).
[0057] If the difference is not within the threshold range (No in step S9), the work content determination unit 93 determines that work is in progress and that the work state is safe (step S11).
[0058] On the other hand, if it is within the threshold range (Yes in step S9), the work content determination unit 93 determines that work is in progress and the work state is dangerous, and causes the display control unit 95 to display a warning (step S10).
[0059] Here, a specific example of the determination of the work content by the work content determination unit 93 will be described.
[0060] Fig. 6 is a diagram showing a specific example of the determination of the work content by the work content determination unit 93, looking down on the work from directly above. The entire Fig. 6 is an example in which a set threshold value (danger area) in real space is superimposed on a photographed image (two-dimensional data) captured by the photographing device 7.
[0061] The object detection unit 92 uses the three-dimensional position information of the object in the photographed image detected in step S1 and the object detected in step S2 to determine the position in the photographed image and the position in three-dimensional space. The position in the image refers to the area surrounded by the frame labeled "baggage", "worker", and "hook", and the position in three-dimensional space refers to "x, y, z" written at the top of the label. In the example shown in FIG. 6, the center of the photographed image is x=0m, the left side is expressed as negative and the right side as positive. The y coordinate is expressed as the center of the image is y=0m, the upper side is expressed as negative and the lower side is expressed as positive. The z coordinate is expressed as a positive value relative to the distance from the camera 7.
[0062] It should be noted that the accuracy of detecting an object is poor if it is detected by a detection algorithm designed deductively using the feature amount of the object in the captured image or by a learning algorithm that detects the object based on similarity to a model created in advance. Therefore, the task content determination unit 93 detects the object D3 captured in succession at time using the previously detected object D3 (step S3 in FIG. 5). This is a tracking process that is generally called tracking.
[0063] The work content determination unit 93 determines whether the positions of the hook 221 and the load (suspended load) P are within range based on the detection object D4 detected in step S3 in Fig. 5 (step S4 in Fig. 5). In the example shown in Fig. 6, the three-dimensional position information of the hook 221 is (x, y, z) = (0.5 m, 0.7 m, 1.0 m), and the three-dimensional position information of the load P is (x, y, z) = (0.7 m, 0.8 m, 7.0 m). The distance between the two points is calculated based on this information as follows: SQRT((0.5-0.7) 2 +(0.7-0.8) 2 +(1-7.0) 2 ) = approx. 6.0m For example, if the preset threshold range in this case is 7 m, the work content determination unit 93 determines that the task is "in progress."
[0064] Step S5 in Fig. 5 is to judge whether or not the cargo is swaying, and the work content judgment unit 93 judges that the cargo is swaying when, for example, the cargo P or the hook 221 moves beyond a preset threshold range between different frames. Note that, in the example shown in Fig. 6, the explanation will be continued assuming that there is no cargo swaying.
[0065] The work content determination unit 93 determines the work content from the position of the detected object (step S6 in FIG. 5). For example, in the example shown in FIG. 6, the work content determination unit 93 determines that the suspended load is floating 30 cm or more above the ground from the difference between the detected z coordinates (9.0 m, 9.1 m) of the worker W and the z coordinate (7.0 m) of the load P, and therefore determines the work content to be "load suspension work."
[0066] Next, the work content determination unit 93 determines whether the positions of the worker W and the baggage P are within range (step S9 in FIG. 5). In the example shown in FIG. 6, the work content determination unit 93 sets an area surrounded by a further 6 m outside the detection range of the baggage P with the baggage P at the center as the set threshold value (danger area) in the real space. The danger area is an area that is determined to be "dangerous" if the worker W enters it.
[0067] 6 is drawn according to the processing of the above-mentioned two-dimensional conversion unit 94. First, the two-dimensional conversion unit 94 calculates the two-dimensional pixel size of the object from the following conditions in accordance with the above-mentioned (Equation 1). Focal length of camera 7: 2mm Pixel pitch of the imaging device 7: 0.006 mm Distance between the imaging device 7 and the work position: 10,000 mm Actual size of the object: 6,000mm
[0068] The distance between the camera 7 and the work position may be a preset distance, may be a distance detected by marking, or may be calculated from information on the worker W and the suspended load P during work. When the above formula (Formula 1) is calculated, the pixel size of the target object is 200 [pix].
[0069] Here, Fig. 7 is an enlarged view of the target location in the determination example of Fig. 6. As shown in Fig. 7, the center coordinates of the package P on the captured image detected by the object detection unit 92 are (x, y) = (-265, 128), the width is 96 [pix], and the height is 55 [pix]. Furthermore, since the detected three-dimensional z coordinate is 7,000 [mm], the two-dimensional conversion unit 94 calculates the size of the package P in real space by the following formula. Actual size of object [mm] = (pixel size of object [pix]) * (pixel pitch [mm] * (installation height of image capture device 7 [mm] / focal length [mm])) That is, Package width [mm] = 96 * 0.006 * 7,000 / 2 = 2,016 [mm] Length of package [mm] = 55 * 0.006 * 7,000 / 2 = 1,155 [mm] This size will be the following pixel size at the working position: Pixel size of width of package P at work position [pix] = 2,016 / (0.006*10,000 / 2) = approx. 67 [pix] Vertical pixel size of package P at work position [pix] = 1,155 / (0.006*10,000 / 2) = approx. 39 [pix]
[0070] In other words, as shown in Figure 4, the horizontal width of the danger area is (pixel size of the danger area x 2) + (pixel size of the horizontal width of the suspended load P at the work position) = 200 x 2 + 67 = 467 [pix]. Similarly, as shown in Figure 4, the vertical width of the danger area is (pixel size of the danger area x 2) + (pixel size of the vertical width of the suspended load P at the work position) is 439 [pix].
[0071] Further, the two-dimensional conversion unit 94 obtains the amount of deviation of the central coordinates of the package P from the center of the image by the following formula. Horizontal deviation from center [mm] = -265*0.006*7,000 / 2 = -5,565 [mm] Vertical deviation from center [mm] = 128 * 0.006 * 7,000 / 2 = 2,688 [mm]
[0072] The two-dimensional conversion unit 94 determines the x, y coordinates of the work position based on this deviation amount as follows. (x coordinate at working position) = -5,565 / (0.006*10,000 / 2) = approx. -186 [pix] (y coordinate at working position) = 2,688 / (0.006*10,000 / 2) = approx. 90 [pix]
[0073] As a result, as shown in FIG. 7, the display control unit 95 can draw the frame of the danger area as a rectangle of 439×467 [pix] in length and width and centered at (x, y)=(−186, 90).
[0074] Regarding the judgment of the work contents by the work content judgment unit 93, in the example shown in FIG. 6, there are workers W (W1, W2) at positions (x, y, z) = (10.3 m, 5.2 m, 9.0 m) and (x, y, z) = (-0.6 m, 4.7 m, 9.1 m), respectively. The worker W (W2) at (x, y, z) = (10.3 m, 5.2 m, 9.0 m) is outside the danger area and therefore in a safe position, but the worker W (W1) at (x, y, z) = (-0.6 m, 4.7 m, 9.1 m) is inside the danger area and therefore in a dangerous position. The work content judgment unit 93 judges that the current work state is dangerous because one or more workers W (W1) are in a dangerous position. The work content judgment unit 93 notifies the crane control device 30 of the judgment result.
[0075] By executing the above-mentioned processing, the display control unit 95 becomes able to draw danger areas based on the work position, so that the operator of the crane 210 can accurately grasp on a monitor or the like whether or not the worker W is in a danger position.
[0076] In this way, according to this embodiment, the two-dimensional coordinates of the set threshold value (danger area) in real space in the captured image are calculated based on the three-dimensional position information of the object, so that the set threshold value (danger area) in real space can be accurately displayed on the captured image. For example, if an area with a radius of 3 m around the suspended load is set as dangerous when a suspended load is detected, it is possible to display on the image which areas are dangerous areas in the environment of the worker working under the suspended load.
[0077] Furthermore, according to this embodiment, once a dangerous area can be represented on a captured image, subsequent risk determination can be performed on an image basis (two-dimensionally).
[0078] In this embodiment, the display control unit 95 functioning as the notification unit superimposes a set threshold value (danger area) in real space on the captured image and displays it on the display device 925, and also displays the result determined by the work content determination unit 93 on the display device 925, but this is not limited to this. For example, the display control unit 95 may store the generated image on the cloud and notify the user by allowing the user to check it on a browser, instead of displaying it on hardware such as the display device 925.
[0079] In this embodiment, the two-dimensional conversion unit 94 calculates the two-dimensional coordinates of the threshold (danger area) in real space based on the work position of the worker W in the captured image by using height information of the worker W within a predetermined range centered on the worker W or the hook 221, but this is not limited to the above. For example, the two-dimensional conversion unit 94 may calculate the two-dimensional coordinates of the threshold (danger area) in real space in the captured image by using distance information (for example, distance information to the ground) other than the worker W, the hook 221, and the baggage P within a predetermined range centered on the worker W or the hook 221.
[0080] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to execute each function described above.
[0081] Although the embodiment of the present invention has been described above, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiments are included within the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0082] 9 Monitoring device 91 Data receiving section 92 Object detection unit 93 Work content judgment section 94 Two-dimensional conversion section 95 Notification Department [Prior art documents] [Patent documents]
[0083] [Patent Document 1] JP 2022-175364 A
Claims
1. A monitoring device for monitoring a working state of a work machine, an object detection unit that detects three-dimensional position information of an object that is a target of monitoring by using the captured image and distance information; a work content determination unit that determines whether or not work to be performed by the work machine is safe based on the three-dimensional position information of the object acquired by the object detection unit; a two-dimensional conversion unit that calculates two-dimensional coordinates of a set threshold value in real space in the captured image based on the three-dimensional position information of the object obtained by the object detection unit; a notification unit that superimposes a threshold value in the real space on the captured image in accordance with the two-dimensional coordinates and notifies a result of the determination made by the work content determination unit; A monitoring device comprising:
2. a data receiving unit for receiving the captured image and the distance information used to detect three-dimensional position information of the object; 2. The monitoring device according to claim 1.
3. The work machine is a crane, the object is a worker, a hook of the crane, and a load suspended from the hook of the crane, the object detection unit calculates three-dimensional position information of the target object from a height of the worker present within a predetermined range on an XY plane when the position (X, Y) of the hook or the baggage is taken as the center; The two-dimensional conversion unit calculates two-dimensional coordinates in the captured image of a threshold value in real space based on a working position of the worker by using height information of the worker within a predetermined range centered on the worker or the hook.
2. The monitoring device according to claim 1.
4. The work machine is a crane, the object is a worker, a hook of the crane, and a load suspended from the hook of the crane, the object detection unit calculates three-dimensional position information of the target object from distance information of the object other than the worker, the hook, and the luggage that are present within a predetermined range on an XY plane when the position (X, Y) of the hook or the luggage is taken as the center; the two-dimensional conversion unit calculates two-dimensional coordinates in the captured image of a threshold value in real space by using distance information of the worker, the hook, and the baggage within a predetermined range centered on the worker or the hook; 2. The monitoring device according to claim 1.
5. The two-dimensional conversion unit converts the size in real space into the two-dimensional size on the captured image using the following formula: I=(S*f) / (Z*P) I: pixel size of object [pix] S: Actual size of the object [mm] P: pixel pitch of the captured image [mm] Z: Distance between the photographing device and the work position [mm] f: focal length of the photographing device that captures the photographed image [mm] 2. The monitoring device according to claim 1.
6. A computer that controls a monitoring device that monitors the working state of a work machine, an object detection unit that detects three-dimensional position information of an object that is a target of monitoring by using the captured image and distance information; a work content determination unit that determines whether or not work to be performed by the work machine is safe based on the three-dimensional position information of the object acquired by the object detection unit; a two-dimensional conversion unit that calculates two-dimensional coordinates of a set threshold value in real space within the captured image based on the three-dimensional position information of the object obtained by the object detection unit; a notification unit that superimposes a threshold value in the real space on the captured image in accordance with the two-dimensional coordinates and notifies a result of the determination made by the work content determination unit; A program to function as a
7. A notification method for a monitoring device that monitors a work state of a work machine, comprising: an object detection step of detecting three-dimensional position information of an object to be monitored using the captured image and distance information; a work content determination step of determining whether or not work to be performed by the work machine is safe based on the three-dimensional position information of the object acquired in the object detection step; a two-dimensional conversion process for calculating two-dimensional coordinates of a set threshold value in real space in the captured image based on the three-dimensional position information of the object acquired in the object detection process; a notification step of superimposing a threshold value in the real space on the captured image in accordance with the two-dimensional coordinates and notifying a result of the determination made in the task content determination step; A notification method comprising:
Citation Information
Patent Citations
Work support device, and work support method
JP2018080054A
Information processor, imaging apparatus, equipment control system, information processing method and program
JP2018091652A
Approach warning system
JP2022076884A
Construction assistance system, construction assistance method, height calculation method, and construction assistance program
JP2022175364A
