Work machine image display system, work machine image display method, and work machine image display program

Abstract

To provide a working machine image display system, a working machine image display method, and a program for working machine image display that enable a work manager to efficiently acquire information on an object approaching a working machine.SOLUTION: An image display system 1S comprises: an object detection unit 75 that detects intrusion of an object into a detection range which is set around a working machine 1 and that outputs a detection signal; an imaging unit 70 capable of imaging an area around the working machine 1; a browsing unit 104 capable of displaying a display image corresponding to an input display command signal; an extraction unit 103 that extracts, from a plurality of images imaged by the imaging unit 70, a specific image that is an image corresponding to a time point at which the detection signal is output and meets an extraction condition which is set for a browsing person to view the object at the browsing unit 104; and an image input unit that inputs the display command signal including the specific image to the browsing unit (being a display unit).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a work machine image display system, a work machine image display method, and a work machine image display program. [Background technology]

[0002] Conventionally, there is known technology that detects objects such as people around a work machine and slows down or stops the work machine depending on the detection results. Patent Document 1 discloses a technology that sends information about the work machine when such control is activated to a server and generates a report to help the user take safety measures. Furthermore, Patent Document 2 discloses a technology that saves video when an obstacle is detected and sends the video to a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 084161 [Patent Document 2] International Publication No. 2021 / 241033 Summary of the Invention [Problem to be solved by the invention]

[0004] The technologies described in Patent Documents 1 and 2 have the problem that the large amount of information sent to the server may contain a lot of useless information that does not contribute to identifying the target object, and therefore it takes a lot of time and effort for the work manager to extract and identify the target object from the information.

[0005] An object of the present invention is to provide a work machine image display system, a work machine image display method, and a work machine image display program that enable a work manager to efficiently obtain information about objects approaching a work machine. [Means for solving the problem]

[0006] The present invention provides an image display system for a work machine. The image display system includes a detection unit, an imaging unit, a display unit, an image extraction unit, and an image input unit. The detection unit detects that an object has entered a detection range set around the work machine and outputs a detection signal. The imaging unit is capable of capturing an image of the area around the work machine including the detection range. The display unit is capable of displaying a display image in accordance with an input display command signal. The image extraction unit extracts, from multiple images captured by the imaging unit, a specific image that corresponds to the time the detection signal was output and that satisfies extraction conditions set so that a viewer can visually recognize the object on the display unit. The image input unit inputs the display command signal including the specific image to the display unit.

[0007] According to this configuration, a specific image corresponding to the time when the object was detected from the images captured by the imaging unit and extracted based on the extraction conditions can be displayed on the display unit. Therefore, compared to when all images captured by the imaging unit or all images corresponding to the detection signal are displayed on the display unit, viewers such as work managers can efficiently obtain information about objects approaching the work machine.

[0008] In the above configuration, the detection signal may include related information relating to the object, and the image input unit may input the display command signal including each of the specific image and the corresponding related information to the display unit.

[0009] According to this configuration, related information relating to the object can be displayed on the display unit along with a specific image, allowing viewers such as work managers to more efficiently obtain information about objects approaching the work machine.

[0010] In the above configuration, the image extraction unit extracts, as the multiple images, multiple videos corresponding respectively to multiple detection periods, each of the multiple detection periods being a period during which the detection unit continuously detects the object, and the extraction condition may include a condition that the specific image is the video corresponding to a relatively long detection period within the population of the videos.

[0011] According to this configuration, it is possible to display on the display unit, among the specific images, those with long detection periods, so that the viewer can efficiently check images with high risk.

[0012] In the above configuration, the image input unit may input the display command signal to the display unit so that the moving image corresponding to the relatively long detection period among the population of moving images is preferentially displayed on the display unit.

[0013] According to this configuration, specific images with longer detection periods can be displayed preferentially, allowing the viewer to check images with a higher risk level more efficiently.

[0014] In the above configuration, the image extraction unit extracts, as the multiple images, multiple videos corresponding respectively to multiple detection periods, each of the multiple detection periods being a period during which the detection unit continuously detects the object, and the extraction condition may include a condition that the specific image is the video corresponding to the detection period longer than a predetermined threshold value.

[0015] According to this configuration, by displaying an image corresponding to a detection period longer than a predetermined threshold as a specific image on the display unit, images with short detection periods are not displayed on the display unit, and viewers can efficiently check images with high urgency.

[0016] In the above configuration, the related information may include information regarding the distance between the work machine and the object, and the extraction condition may include a condition that the specific image is an image corresponding to a relatively small distance within the population of images.

[0017] According to this configuration, an image in which the distance between the work machine and the object is small can be displayed on the display unit, allowing the viewer to efficiently check images that pose a high risk.

[0018] In the above configuration, the image input unit may input the display command signal to the display unit so that the specific image corresponding to the relatively small distance among the population of the specific image is preferentially displayed on the display unit.

[0019] According to this configuration, specific images that are closer in distance can be displayed preferentially, allowing the viewer to check images with a higher risk level more efficiently.

[0020] In the above configuration, the device may further include a memory unit capable of storing the image captured by the imaging unit together with time information and further storing the detection signal together with the time information, and the image extraction unit may extract, from the images stored in the memory unit, the image that corresponds to the time information accompanying the detection signal as the specific image.

[0021] According to this configuration, since the image and the detection signal are stored together with time information, it is possible to easily and quickly identify the specific image corresponding to the detection signal.

[0022] The above configuration may further include an image storage unit capable of storing the image captured by the imaging unit together with time information, and a signal storage unit capable of storing the detection signal output from the detection unit together with time information, and the image extraction unit may identify, from among the images stored in the image storage unit, the image that corresponds to the time information accompanying the detection signal stored in the signal storage unit as the specific image.

[0023] According to this configuration, since the image and the detection signal are each stored together with time information, it is possible to easily and quickly identify the specific image corresponding to the detection signal. Furthermore, since the image and the detection signal are stored in separate storage units, it is easy to display each piece of information independently on the display unit, or to combine the information and display it on the display unit.

[0024] In the above configuration, the device may further include a conversion unit capable of compressing and converting at least one of the image captured by the imaging unit and the detection signal output from the detection unit into a predetermined format data, and a transmission unit that transmits the format data generated by the conversion unit.

[0025] According to this configuration, by compressing and transmitting images and detection signals, it is possible to increase the amount of information stored in the memory unit, etc., and shorten the time it takes for the transmitter to transmit information. Furthermore, if the transmitter is provided on the work machine, shortening the information transmission time can reduce the load on the work machine's power supply.

[0026] In the above configuration, the work machine has an actuator and a locking unit capable of locking the movement of the actuator, and the related information further includes additional information which is information relating to at least one of the locked state of the actuator by the locking unit, the detectable state of the object by the detection unit, and the operating speed of the work machine, and the image input unit may input the display command signal to the display unit so that the additional information is displayed on the display unit together with the specific image.

[0027] According to this configuration, additional information related to the danger between the object and the work machine can be displayed on the display unit, allowing the viewer to efficiently check specific images that pose a higher risk.

[0028] In the above configuration, the image extraction unit extracts a video corresponding to a detection period, the detection period being a period during which the detection unit continuously detects the object, the video being composed of the plurality of images, the image extraction unit extracts the specific image that satisfies the extraction conditions from the plurality of images as a thumbnail image representative of the video, and the image input unit may input the display command signal corresponding to the video including the thumbnail image to the display unit.

[0029] According to this configuration, a specific image extracted based on extraction conditions from among multiple images that make up a video can be displayed as a thumbnail image that represents the video, allowing the viewer to easily grasp the outline of the detected object from the thumbnail image and efficiently begin viewing the video, the details of which they wish to check.

[0030] In the above configuration, the related information includes relative position information of the object with respect to the work machine, and the image extraction unit is capable of further specifying the position of the object in the image based on the position information, and the extraction condition may include a condition that the thumbnail image is the image in which the object is located most centrally in a predetermined direction among the plurality of images.

[0031] According to this configuration, the image in which the object is located most centrally in a specified direction among multiple images is extracted as a thumbnail image, allowing the viewer to easily grasp the outline of the detected object from the thumbnail image.

[0032] In the above configuration, the angle of view of the imaging unit may be set so that the contact surface of the work machine is included in the image captured by the imaging unit, the direction away from the work machine is the up-down direction in the image, and the predetermined direction may be the left-right direction in the image that is perpendicular to the up-down direction.

[0033] With this configuration, it is possible to easily and accurately check in the left-right center of the thumbnail image whether an object is located near the work machine.

[0034] In the above configuration, the image extraction section may be capable of saving or outputting images including at least the thumbnail image in a predetermined file format.

[0035] This configuration makes it possible to save images, including thumbnail images that capture the object, in file formats such as PDF, and to output them by data transfer or printing. This allows viewers to use this information in meetings with workers involved with work machinery, etc., to raise awareness.

[0036] In the above configuration, the image extraction unit can determine whether the object included in the image is a person and identify the person by multiple body parts including at least the face, and the extraction condition may include a condition that the thumbnail image is an image in which the face of the identified person is located most centrally in a specified direction among the multiple images.

[0037] According to this configuration, the image in which a person's face is located most centrally in a specified direction among multiple images is extracted as a thumbnail image, allowing the viewer to easily grasp the outline of the detected person from the thumbnail image.

[0038] In the above configuration, the image extraction unit may be capable of determining whether the object included in the image is a person, and the extraction condition may include a condition that the thumbnail image is an image in which the area occupied by the person is relatively large among the plurality of images.

[0039] According to this configuration, an image in which a person is visible in a large area can be given priority as a thumbnail image, so that a viewer can easily grasp the outline of the detected person from the thumbnail image.

[0040] In the above configuration, the image extraction unit can determine whether the object included in the image is a person and identify the direction of the person's face, and the extraction condition may include a condition that the thumbnail image is an image among the plurality of images in which the face of the identified person is facing in a direction different from the work machine.

[0041] With this configuration, it is possible to efficiently extract images of people who are not facing the work machine and who may be working without being aware of the presence of the work machine, and to take corrective measures against those people.

[0042] In the above configuration, when the detection unit detects the object during each of the multiple detection periods, the image input unit may input the display command signal to the display unit so that, among the multiple videos, the video in which the identified person's face spends a relatively long time facing a direction away from the work machine is displayed preferentially on the display unit.

[0043] According to this configuration, images of people who spend more time not facing the work machine can be efficiently extracted, and corrective measures can be taken quickly.

[0044] The present invention provides a method for displaying images of a work machine. The image display method includes an image acquisition step, a detection step, an image extraction step, and a display step. In the image acquisition step, an area including a detection range set around the work machine is captured and an image is acquired. In the detection step, an intrusion of an object into the detection range is detected and a detection signal is output. In the image extraction step, a specific image is extracted from the multiple captured images that corresponds to the time when the detection signal was output and that satisfies extraction conditions set so that a viewer can visually recognize the object through a display unit. In the display step, the specific image is displayed on the display unit.

[0045] According to this method, a specific image that corresponds to the time when an object was detected from the captured images and that is extracted based on extraction conditions can be displayed on the display unit.This makes it possible for viewers such as work managers to efficiently obtain information about objects that have approached the work machine, compared to when all of the captured images or all of the images corresponding to the detection signal are displayed on the display unit.

[0046] In the above method, the image extraction process may include extracting a video corresponding to a detection period, the detection period being a period during which the object is continuously detected in the detection process, the video being composed of the plurality of images, and the image extraction process may include extracting the specific image that satisfies the extraction conditions from the plurality of images that compose the video as a thumbnail image that represents the video.

[0047] According to this method, a specific image extracted based on extraction conditions from among multiple images that make up a video can be displayed as a thumbnail image that represents the video, allowing the viewer to easily grasp the outline of the detected object from the thumbnail image and efficiently begin viewing the video, the details of which they wish to check.

[0048] The present invention also provides a program for displaying images of a work machine that is executed on a computer connected to an imaging unit capable of capturing images of an area including a detection range set around the work machine, and a display unit capable of displaying a display image in accordance with an input display command signal. The program causes the computer to execute a storage step, a detection step, an image extraction step, and an image input step. The storage step stores images captured by the imaging unit. The detection step detects that an object has entered the detection range based on the images captured by the imaging unit, and outputs a detection signal. The image extraction step extracts a specific image from multiple images captured by the imaging unit that corresponds to the time the detection signal was output and satisfies extraction conditions set so that a viewer can visually recognize the object through the display unit. The image input step inputs the display command signal including the specific image to the display unit, causing the specific image to be displayed.

[0049] According to this program, the program can be executed on a computer so that a specific image corresponding to the time when an object was detected from among the images captured by the imaging unit, and further extracted based on extraction conditions, is displayed on the display unit.This makes it possible for viewers such as work managers to efficiently obtain information about objects approaching the work machine, compared to when all of the images captured by the imaging unit or all of the images corresponding to the detection signal are displayed on the display unit.

[0050] In the above configuration, the imaging unit may be capable of capturing a video composed of a plurality of images, and the image extraction step may further include extracting a specific image from the plurality of images based on the extraction conditions as a thumbnail image representative of the video corresponding to a detection period during which the object is continuously detected, and the display step may further include displaying a thumbnail of the extracted specific image to represent the video, and when a viewer viewing the display unit selects one of the specific images displayed on the display unit, enlarging and displaying the video including the one specific image on the display unit.

[0051] According to this program, a specific image extracted based on extraction conditions from among multiple images that make up a video can be displayed as a thumbnail image that represents the video, allowing viewers to easily grasp the outline of the detected object from the thumbnail image and efficiently begin viewing the video in detail.

[0052] In the above configuration, the display step may further include displaying a plurality of the specific images corresponding to a plurality of the videos in the form of thumbnails on the display unit, and displaying a selection guide that can assist the viewer in selecting a specific image with a high degree of risk from the plurality of the specific images.

[0053] According to this program, the viewer can easily select a specific image with a high risk level based on the selection guide and view that video. [Effects of the Invention]

[0054] According to the present invention, it is possible to provide a work machine image display system, a work machine image display method, and a work machine image display program that enable a work manager to efficiently obtain information about objects approaching a work machine. [Brief explanation of the drawings]

[0055] [Figure 1] 1 is a side view showing a work machine according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram of an image display system according to a first embodiment of the present invention. [Figure 3] 1 is a plan view showing a peripheral detection area of ​​a work machine according to a first embodiment of the present invention. [Figure 4] 3 is a flowchart of an image display process executed by the image display system according to the first embodiment of the present invention. [Figure 5] 3 is an example of a screen displayed on a display unit of the image display system according to the first embodiment of the present invention. [Figure 6] 10 is an example of a screen displayed on a display unit of an image display system according to a modified example of the first embodiment of the present invention. [Figure 7] 10 is an example of a screen displayed on a display unit of an image display system according to a second embodiment of the present invention. [Figure 8] 10 is an example of an image displayed on a display unit of an image display system according to a second embodiment of the present invention. [Figure 9] 10 is an example of an image displayed on a display unit of an image display system according to a second embodiment of the present invention. [Figure 10] 10 is an example of an image displayed on a display unit of an image display system according to a second embodiment of the present invention. [Figure 11] 10 is an example of a screen displayed on a display unit of an image display system according to a modified example of the second embodiment of the present invention. [Figure 12] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 13] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 14] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 15]10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 16] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 17] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. [Figure 18] 10 is an example showing how a person in an image is recognized in the image display system according to the second embodiment of the present invention and its modified example. DETAILED DESCRIPTION OF THE INVENTION

[0056] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0057] First Embodiment FIG. 1 shows a work machine 1 equipped with an image display system 1S (FIG. 2) according to a first embodiment of the present invention. In this embodiment, the work machine 1 is a hydraulic excavator. The work machine 1 includes a crawler-type undercarriage 10 (main body) capable of traveling on the ground G (traveling surface), an upper rotating body 12 mounted on the undercarriage 10 so as to be rotatable about a rotation center axis perpendicular to the ground G, and a work attachment 3 mounted on the upper rotating body 12. The undercarriage 10 includes a left crawler 11L and a right crawler 11R (FIG. 3). The work attachment 3 includes a boom 4 supported on the upper rotating body 12 so as to be able to rise and fall, an arm 5 rotatably connected to the tip of the boom 4, and a bucket 6 rotatably connected to the tip of the arm 5. The upper rotating body 12 has a rotating frame 16, a counterweight 17, and a cab 18.

[0058] The work machine 1 is equipped with a boom cylinder 7 that operates to raise and lower the boom 4 relative to the upper rotating body 12, an arm cylinder 8 that operates to rotate the arm 5 relative to the boom 4, and a bucket cylinder 9 that operates to rotate the bucket 6 relative to the arm 5.

[0059] FIG. 2 is a block diagram of an image display system 1S according to this embodiment. FIG. 3 is a plan view showing the peripheral detection area of ​​a work machine 1. Work site managers desire to understand when, who (what) detected an object that entered the detection range, and under what circumstances it was detected by a work machine 1 equipped with a peripheral detection function as described below, and to use this information to improve safety activities at the work site. In this embodiment, the image display system 1S includes an external terminal 100 ( FIG. 2 ) located at a location different from the work machine 1, and image information about the surroundings of the work machine 1 is displayed on the external terminal 100. For example, the external terminal 100 is a personal computer located in a management office at the work site. The external terminal 100 may also be located in a remote management facility away from the work site. The external terminal 100 may also be a tablet terminal held by a worker at the work site or the like. In addition to the external terminal 100, a portion of the work machine 1 described below also constitutes part of the image display system 1S according to this embodiment.

[0060] As shown in FIG. 2, the work machine 1 has an imaging unit 70, a display image generating unit 71, a monitor 72, an image storage unit 73 (memory unit), a conversion unit 74, an object detection unit 75 (detection unit), a detection signal storage unit 76 (memory unit), a communication device 77, an operating lever 80, a surrounding detection function disable switch 81, a control unit 90, an operation control unit 91, a lever lock 92, and a drive unit 95.

[0061] The imaging unit 70 is capable of capturing images of an area including a detection range set around the work machine 1. In this embodiment, the imaging unit 70 includes four cameras. One of these cameras is disposed inside the cab 18, and the remaining three cameras include a left camera 60L, a rear camera 60B, and a right camera 60R disposed in the rear portion of the upper rotating body 12, as shown in FIG. 3. The images captured by each camera of the imaging unit 70 include still images and video (video). In this case, the video can be considered as a collection of multiple still images (multiple images).

[0062] The display image generation unit 71 generates a display image to be displayed on the monitor 72. As an example, the display image generation unit 71 can generate one display image so that images captured by four cameras are displayed on the monitor 72 simultaneously.

[0063] The monitor 72 is disposed inside the cab 18, and displays images captured by the imaging unit 70, and in particular, display images generated by the display image generation unit 71. The operator of the work machine 1 can check an image of the surroundings of the work machine 1 through the monitor 72. The operator can switch the image displayed on the monitor 72 using a switch (not shown).

[0064] The image storage unit 73 is disposed inside the cab 18 and stores the images captured by the imaging unit 70 .

[0065] The conversion unit 74 can convert images captured by the imaging unit 70, compress gradation, and change the frame rate of images stored in the image storage unit 73, as well as convert multiple images into a single image. As a result, the load of transmitting images from the work machine 1 to the external terminal 100 is reduced.

[0066] The object detection unit 75 detects that an object has entered a detection range set around the work machine 1, and outputs a detection signal including relevant information related to the object. In this embodiment, the object detection unit 75 detects the intrusion of an object by performing image processing on the image captured by the imaging unit 70. The relevant information includes information such as the position of the object, such as a person, and the distance between the object and the work machine 1. The detection signal may be output at the same time as the object is detected, or may be output so as to include a portion of a predetermined time before or after. As an example, if the intrusion of an object is detected at time t, a detection signal may be output at time t, a detection signal may be output at time t+α (α is a constant), or detection signals may be output continuously at predetermined intervals between time t and time t+α.

[0067] As shown in Figure 3, right side areas AR1 and AR2, rear areas AB1 and AB2, and left side areas AL1 and AL2 are set around the work machine 1 (rear and to the left and right sides) according to the detection ranges of each camera 60R, 60L, and 60B, respectively, as indicated by dashed double-dashed lines.The object detection unit 75 detects in which area the object is present, and determines the operation (turning operation or traveling operation) that should limit the speed of the work machine 1 (or force it to stop).

[0068] The detection signal storage unit 76 stores the detection signal generated by the object detection unit 75 .

[0069] The communication device 77 transmits and receives various information and signals to and from the external terminal 100 by wireless or wired communication.

[0070] The operating levers 80 are disposed inside the cab 18 and are operated by the operator. The operating levers 80 include a plurality of levers for operating the traveling operation of the undercarriage 10, the swinging operation of the upper rotating body 12, and the rotational operations of the boom 4, arm 5, and bucket 6 of the work attachment 3, respectively.

[0071] The perimeter detection function disable switch 81 is located in the cab 18 and switches ON / OFF the perimeter detection function performed by the object detection unit 75. Normally, when the work machine 1 is in operation, the detection function is ON, but when the perimeter detection function disable switch 81 is operated at the operator's discretion, the detection function can be turned OFF.

[0072] The control unit 90 comprehensively controls the operation of the work machine 1. The control unit 90 is made up of a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores control programs, a RAM (Random Access Memory) used as a work area for the CPU, and the like.

[0073] The operation control unit 91 limits the traveling speed of the lower traveling body 10 and the rotation speed of the upper rotating body 12 according to the detection result of the object by the object detection unit 75. The operation control unit 91 constitutes a part of the control unit 90. The control unit 90 and the operation control unit 91 may have different configurations.

[0074] Table 1 is a table summarizing the above restrictions executed by the operation control unit 91. Normal operation indicates a state in which the above restrictions are not executed. Deceleration control indicates a state in which deceleration control is applied based on a predetermined calculation formula in response to the amount of operation input to the operation lever 80. Furthermore, stop control indicates a state in which the travel (forward or backward) of the lower traveling body 10, the rotation (right or left rotation) of the upper rotating body 12, or the movement (work) of the work attachment 3 is prevented, regardless of the amount of operation input to the operation lever 80. [Table 1]

[0075] The drive unit 95 performs the travel operation of the lower traveling structure 10, the swing operation of the upper rotating structure 12, and the drive operation of each member of the work attachment 3 (boom 4, arm 5, bucket 6) in the work machine 1. The drive unit 95 includes a hydraulic circuit (not shown), etc. The hydraulic circuit includes a hydraulic pump that operates by receiving driving force from the engine, an actuator such as a swing motor or a hydraulic cylinder, and a control valve (flow control valve) interposed between the hydraulic pump and the actuator. The control unit 90 adjusts the travel speed of the lower traveling structure 10, the swing speed of the upper rotating structure 12, the hoisting speed of the boom 4, etc. by adjusting the pilot pressure input to the control valve as the control input.

[0076] The lever lock 92 (lock unit) is disposed inside the cab 18 and is operated by the operator. The lever lock 92 can be switched between a locked state and an unlocked state. When the lever lock 92 is in the locked state, the travel speed of the lower traveling structure 10, the swing speed of the upper rotating structure 12, the hoisting speed of the boom 4, etc. are blocked (the actuators are locked) regardless of the operation input to the operating lever 80. On the other hand, when the lever lock 92 is in the unlocked state, the travel speed of the lower traveling structure 10, the swing speed of the upper rotating structure 12, the hoisting speed of the boom 4, etc. are permitted. The lever lock 92 may be of a lever lock solenoid structure, or may be of a structure in which each actuator is stopped by an inverse proportional valve.

[0077] As shown in FIG. 2, the external terminal 100 includes a communication device 101, an external storage unit 102 (storage unit), an extraction unit 103 (image extraction unit, image input unit), and a viewing unit 104 (display unit).

[0078] The communication device 101 transmits and receives various information and signals to and from the communication device 77 of the work machine 1 via wireless or wired communication.

[0079] The external storage unit 102 is capable of storing images captured by the imaging unit 70 and the like.

[0080] The extraction unit 103 has a function of extracting information to be displayed on the viewing unit 104 from various types of information transmitted from the work machine 1 to the external terminal 100. For example, a program or application installed on the external terminal 100 is executed on the external terminal 100, causing the external terminal 100 to function as the extraction unit 103. In particular, in this embodiment, the extraction unit 103 extracts a specific image from the images captured by the imaging unit 70, which is an image that corresponds to the time when the detection signal was output and which satisfies extraction conditions set so that the viewer can visually recognize the object on the viewing unit 104. Furthermore, the extraction unit 103 inputs the display command signal including each of the specific image and the related information corresponding to it to the viewing unit 104.

[0081] The viewing unit 104 is capable of displaying a display image in accordance with an input display command signal. In this embodiment, the viewing unit 104 is configured as a display or monitor of a personal computer. A viewer such as a site manager in the management office of the work site can view the viewing unit 104 to check the surrounding conditions of the work machine 1.

[0082] Next, the image display processing executed by the image display system 1S according to this embodiment will be described in detail. FIG. 4 is a flowchart of the image display processing. For example, when the engine (not shown) of the work machine 1 starts and the image display system 1S starts the flow of FIG. 4, the imaging unit 70 starts acquiring video of the surroundings of the work machine 1 (step S1). As also shown in FIG. 2, the video acquired by the imaging unit 70 is distributed to three units: the display image generation unit 71, the video storage unit 73, and the object detection unit 75 (step S2). The video storage unit 73 acquires time information corresponding to the acquired video (step S3), and the time information and the video (video) are linked (associated) (step S4). As a result, the video is stored in the video storage unit 73 together with the time information (step S5). Next, the video stored in the video storage unit 73 is compressed and converted by the conversion unit 74 (FIG. 2), and its frame rate is changed (step S6). At this time, the images from the four cameras may be converted into one image. By combining the images from the four cameras into one image, the work manager viewing the viewing unit 104 described below can check areas other than the direction in which the object was detected, and can therefore understand the situation around the work machine at the time of detection, such as what work was being done and whether the situation was such that it was not possible to operate the machine to move away from the object. Furthermore, by performing the conversion process described above, the image stored in the image storage unit 73 becomes ready to be sent to the external terminal 100.

[0083] Meanwhile, in the image distributed to the object detection unit 75 in step S2, a known image processing program executed by the object detection unit 75 determines whether or not a person is present in the image (person detection). At this time, the distance between the work machine 1 (more specifically, each camera) and the person is calculated (step S7). Next, the calculated distance information is sent to the control unit 90 (step S8). Based on the distance information, the control unit 90 determines whether or not the person is present within any of the detection ranges AL1, AL2, AB1, AB2, AR1, and AR2 in FIG. 3 (step S9). Here, if a person is present within any of the above ranges, a predetermined control is executed on the work machine 1 in accordance with Table 1 above (step S10). When this control is executed, the object detection unit 75 generates a detection signal and stores it in the detection signal storage unit 76 together with machine information (machine number, etc.) of the work machine 1 (step S11). Note that the detection signal is generated continuously at predetermined intervals as long as a person is present in any of the above areas. As a result of the above, the information stored in the detection signal storage unit 76 becomes available for transmission to the external terminal 100. On the other hand, if no person is present within any of the above ranges in step S9, the normal state is assumed and the work machine 1 is controlled in accordance with the operation received by the operating lever 80 (step S12).

[0084] Furthermore, the image distributed to the display image generating unit 71 in step S2 is displayed on the monitor 72 installed in the cab 18 (step S13). At this time, if a predetermined control (deceleration, stop) is executed in step S10, that information is also displayed on the monitor 72 (step S14). As a result, the operator can recognize the current situation of the work machine 1 and its surroundings.

[0085] The video information converted by the conversion unit 74 in step S6 and the various pieces of information stored in the detection signal storage unit 76 in step S11 are transmitted to the external terminal 100 via the communication device 77 (step S15). When the communication device 101 of the external terminal 100 receives the above information, it is stored as is in the external storage unit 102. At this time, all of the videos captured by the imaging unit 70 and all of the detection signals generated by the object detection unit 75 may be stored in the external storage unit 102. Next, the extraction unit 103 extracts information to be displayed on the viewing unit 104 from this information. Specifically, the extraction unit 103 synchronizes (links) the videos and detection signals stored in the external storage unit 102 based on time information (step S16). As a result, each detection signal is assigned according to time in the video information captured by the imaging unit 70. Note that the time information may be acquired by installing a GPS in the work machine 1, and the same time information may be associated with the video and the detection signals. In this case, the video and the detection signals can be synchronized with high precision.

[0086] Next, the extraction unit 103 extracts (cuts out) partial images at the time of detection from the images stored in the external storage unit 102 in accordance with each detection signal (step S17). At this time, one image (video) is generated corresponding to the period (one detection period) during which the detection signal is continuously output (generated). In other words, if a person is detected at different time periods while the work machine 1 is in use, images corresponding to the respective detection periods are generated. The extraction unit 103 stores each of the generated images in the external storage unit 102 (step S18). The images are also saved together with detection signals containing relevant information such as the distance between the work machine 1 and the person. The extraction unit 103 then preferentially displays the generated images on the viewing unit 104 in accordance with predetermined conditions (extraction conditions) (step S19).

[0087] FIG. 5 is an example of the display screen of the viewing unit 104 in the image display system 1S according to this embodiment. In the figure, three days' worth of video information is displayed in the top, middle, and bottom rows, with different videos displayed from left to right on each date. Also, the number of corresponding videos (number of videos) is displayed below each date. Note that in FIG. 5, only the top left video shows a video in which four camera videos have been combined into one. Of these, the camera video on the bottom right corresponds to the camera inside the cab 18. Note that details of the other videos are omitted in FIG. 5, but similar videos are displayed for each. Note that video from a single camera may also be displayed.

[0088] Each video is accompanied by the time the video was captured, i.e., the time the object (person) was detected and the duration (length, continuous detection time). Also, directly below each video, buttons for operations such as play, pause, stop, fast forward, and rewind are provided. A viewer (site manager) viewing the viewing unit 104 can view and watch each video by operating these buttons with the cursor of a personal computer. When the play button is pressed, the corresponding video may be enlarged to fill the entire screen and playback may begin.

[0089] Also, in Figure 5, machine information such as the model, machine number, and hour meter (working hours) of the work machine 1 is displayed in the upper left corner of the screen of the personal computer in the management office. Also, to the right of this, display priority menus 1 to 3 are shown as examples of extraction conditions for displaying video. Here, continuous detection time is set as display priority 1, detection distance is set as display priority 2, and surrounding detection function ON is set as display priority 3. In other words, among multiple videos from the same date, a specific image with a long continuous detection time is displayed preferentially from the left. When there are two videos with the same continuous detection time, the one with the shorter detection distance in display priority 2 is displayed preferentially on the left. Similarly, videos with the surrounding detection function ON are displayed preferentially over those with the function OFF.

[0090] Display priority 1 is set so that the site manager can view, with priority, high-risk footage in which a person has entered the detection area for a long period of time. Display priority 2 is set so that similarly high-risk footage in which the distance between the work machine 1 and the person is short is viewed. Display priority 3 is set so that the perimeter detection function is ON and work is affected by the work machine 1 slowing down or stopping.

[0091] In another embodiment, the display priority may be determined by using the approach speed as an index to extract video to be displayed with priority. Specifically, the extraction unit 103 calculates for each video whether the approach speed between the work machine 1 and the worker is fast or not from the measured distance value and time. This enables the viewer to confirm situations where the work machine 1 is moving fast even though the worker is close to it, or situations where the worker is approaching even though the work machine 1 is moving, and further, when the work machine 1 and the worker are approaching each other and the approach speed is fast, this can be used as material to encourage the worker to improve their work.

[0092] As described above, in this embodiment, the extraction unit 103 can extract a specific image corresponding to the time when an object was detected from the images captured by the imaging unit 70, and further based on the extraction conditions, and display the specific image on the viewing unit 104. Therefore, compared to when all images captured by the imaging unit 70 or all images corresponding to the detection signal are displayed on the viewing unit 104, viewers such as work managers can efficiently obtain information about objects approaching the work machine 1.

[0093] Furthermore, the detection signal includes related information relating to the object, and the extraction unit 103 inputs the display command signal including each of the extracted image and the related information corresponding thereto to the viewing unit 104. With this configuration, related information relating to the object (such as the detection time in FIG. 15 ) can be displayed on the viewing unit 104 together with the specific image, allowing a viewer such as a work manager to obtain information about the object more efficiently.

[0094] In particular, in this embodiment, the extraction unit 103 extracts, as the plurality of images, a plurality of videos corresponding to a plurality of detection periods, respectively. Each of the plurality of detection periods is a period during which the object detection unit 75 continuously detects an object. The extraction condition includes a condition that the specific image is a video corresponding to a relatively long detection period within the population of videos. Video with a long detection period is assumed to indicate a situation in which a worker does not evacuate from near the work machine 1 or the work machine 1 does not take evasive action, and is a target for which the work manager should take corrective action against the worker. Therefore, with the above-described configuration, specific images with long detection periods can be displayed on the viewing unit 104, allowing the viewer to efficiently check images with a high level of danger or urgency.

[0095] Additionally, in this embodiment, the extraction unit 103 inputs the display command signal to the viewing unit 104 so that the video corresponding to the relatively long detection period among the population of videos is preferentially displayed on the viewing unit 104. This allows specific images with longer detection periods to be preferentially displayed, allowing the viewer to more efficiently check images with a high degree of risk. Note that, as described above, the display order of images may be set based on other conditions (display priority).

[0096] FIG. 6 is an example of a screen displayed on the viewing unit 104 of an image display system 1S according to a modified example of this embodiment (first embodiment). In this modified example, images for one day are displayed in order from the left. At this time, images with a continuous detection time of 1 second or more are selected as the display target using a pull-down menu. In this case, if the update period for detecting an object is 100 msec, images in which an object has been detected 10 or more times in succession are displayed on the viewing unit 104, so images containing false detections can be excluded.

[0097] Furthermore, a time chart T is displayed below each image, showing the status of the "intrusion level," "operation control," "lever lock," and "perimeter detection function" for each time period. The "intrusion level" indicates whether the object (person) has entered the stop zone or deceleration zone. The "operation control" indicates whether the operation of the work machine 1 is subject to stop control or deceleration control due to the intrusion into the above zone. The "lever lock" indicates whether the lever lock 92 inside the cab 18 is in the unlocked or locked state. The "perimeter detection function" indicates whether the detection function is turned on or off by the operator operating the perimeter detection function disable switch 81 (Figure 2). By displaying such information (related information) together, the manager can grasp the circumstances under which the detection occurred in more detail, thereby efficiently narrowing down the images that should be prioritized for review.

[0098] For example, even if a person enters the stop zone when the lever lock 92 is in the locked state, the work machine 1 will not move, so the manager can determine that the confirmation priority is low. Also, if a person enters the stop zone when the perimeter detection function is turned off, it can be determined that guidance is needed for the worker, making it possible to utilize this information in safety activities at the work site.

[0099] Also, directly below the time chart T, there is a display lever A (seek bar) that allows the viewer to select which time period of the day the video will be displayed. The time period selected by the display lever is indicated by a bold frame B in the time chart T. Of the four videos shown in FIG. 6, the video at 10:15 in the bold frame B corresponds to a particularly dangerous situation in which a person has entered a stopping area. For this reason, the extraction unit 103 displays the video on the viewing unit 104 so that it is surrounded by a bold frame C, thereby attracting the viewer's attention and making it easier for them to check the video. Furthermore, as shown in FIG. 6, it is more desirable to also display the words "Stopping area detected."

[0100] In addition, multiple selection menus D are arranged directly below the display lever A. When the viewer arbitrarily checks each menu, the extraction unit 103 (FIG. 2) displays images that satisfy the menu conditions on the viewing unit 104. As shown in FIG. 6, in addition to the currently selected menu, the engine speed, the operating pressure when the operator operates the control lever 80, the current work mode of the work machine 1, and the like are prepared. The operating pressure can be referenced to determine whether the operator is aware of a dangerous state. For example, if an object has entered the detection area but the operator is operating the work machine 1 with a high engine speed or a large operating pressure, the viewer of the viewing unit 104 can infer that the operator is working at a high operating speed without noticing the object.

[0101] As described above, by displaying on the viewing unit 104, together with each video, information related to the work machine 1 contained in the detection signal, such as the status of the lever lock 92, the operating status (detection enabled status) of the perimeter detection function of the object detection unit 75, and information (additional information) related to the operation speed (operation speed) of the work machine 1, such as deceleration or stoppage by the operation control unit 91, the viewer can efficiently check images with a higher degree of danger (specific images). Furthermore, by using this information as a determination condition for preferentially displaying on the viewing unit 104 by the extraction unit 103, the site manager can prioritize checking dangerous situations. As a result, for example, it becomes possible to check whether anyone has entered the detection area when the perimeter detection function of the object detection unit 75 is not operating. The viewing unit 104 may extract images to display by combining the video in which the object is closest to the center of the screen with a determination condition for each time series.

[0102] Furthermore, in this embodiment, the images and detection signals transmitted from the work machine 1 are each stored in the external storage unit 102 of the external terminal 100. Then, the extraction unit 103 extracts, from the images stored in the external storage unit 102, the image that corresponds to the time information accompanying the detection signal as the specific image. In this way, since the images and detection signals are each stored in the external storage unit 102 together with their time information, it is possible to easily and quickly identify the specific image that corresponds to the detection signal.

[0103] In addition, in this embodiment, as shown in Figures 2 and 3, by adding time information to each of the video and detection signals, it becomes possible to transmit each piece of information individually to the external terminal 100 and then synchronize them.

[0104] Furthermore, in this embodiment, the image display system 1S has a conversion unit 74 that can compress and convert images captured by the imaging unit 70 and detection signals output from the object detection unit 75 into data in a predetermined format, and a communication device 77 (transmission unit) that transmits the format data generated by the conversion unit 74. By compressing and transmitting images and detection signals in this way, it is possible to increase the amount of information stored in the external storage unit 102 (storage unit) and shorten the information transmission time by the communication devices 77, 101. Furthermore, as shown in Figure 2, when the conversion unit 74 and communication device 77 are provided in the work machine 1, the load on the battery (power source) of the work machine 1 can be reduced by shortening the information transmission time.

[0105] Second Embodiment Next, a second embodiment of the present invention will be described. Fig. 7 shows an example of a screen displayed on the viewing unit 104 of the image display system 1S according to this embodiment. This embodiment differs from the first embodiment in the extraction of an image (specific image) that the extraction unit 103 displays on the viewing unit 104. In this embodiment, a thumbnail image is selected from a plurality of images constituting each video based on predetermined extraction conditions so that the viewer can efficiently grasp the content of each video. Then, the extraction unit 103 inputs the display command signal corresponding to the video including the thumbnail image to the viewing unit 104.

[0106] 8, 9, and 10 are examples of images displayed on the viewing unit 104. Assume that the video (video) displayed in the upper left of FIG. 7 includes three images (still images) shown in FIGS. 8 to 10. In FIG. 8, within the field of view of the rear camera 60B (FIG. 3), a worker is present in the horizontal center, close to the work machine 1. The object detection unit 75 or the extraction unit 103 identifies the worker, and the worker in the image is surrounded by a frame H. On the other hand, in FIG. 9, the worker is present at the right end within the same field of view. In FIG. 9, the worker is further away from the work machine 1 than in FIG. 8. For example, FIG. 9 corresponds to an image obtained when the worker is first included in the field of view of the camera when the upper rotating body 12 rotates while the worker is stationary. Furthermore, in FIG. 10, the worker is present in the horizontal center within the field of view, closer to the work machine 1 than in FIG. 8. The worker is identified by the object detection unit 75 or the extraction unit 103, and the head of the worker in the image is further surrounded by a frame HA. The frame display on the viewing unit 104 may be optionally switchable to improve the viewer's visibility.

[0107] Here, it is easier for a viewer viewing the viewing unit 104 to identify the worker when the worker is positioned in the horizontal center as in FIGS. 8 and 10 rather than as in FIG. 9. For this reason, FIGS. 8 and 10 are preferable as the thumbnail images. Furthermore, it is easier to recognize the worker's face in FIG. 8 than in FIG. 10, so it is more preferable as the thumbnail image. As a result, the extraction unit 103 selects the image in FIG. 8 as the thumbnail image representing the video, and the image is displayed as a thumbnail in the corresponding portion as in FIG. 7.

[0108] As described above, in this embodiment, the extraction unit 103 can display an image (specific image) extracted from multiple images constituting a video based on predetermined extraction conditions as a thumbnail image representing the video. Therefore, the viewer can easily grasp the outline of the detected person (object) from the thumbnail image without playing the video, and can efficiently begin viewing the video for further details. In other words, the viewer can check images corresponding to dangerous situations without having to review many videos captured by the imaging unit 70 or all videos corresponding to the detection signal. The extraction unit 103 can also extract, as the thumbnail image, an image from a camera that best satisfies the extraction conditions from among images captured by multiple cameras. The extraction unit 103 may also extract, as the thumbnail image, an image that best satisfies the extraction conditions from among images from a camera preselected by the viewer.

[0109] In particular, in this embodiment, the related information included in the detection signal includes relative position information (distance) of the object with respect to the work machine 1. The extraction unit 103 is then able to further identify the position of the object in the image based on the position information, and the extraction conditions include a condition that the thumbnail image is the image in which the object is located most centrally in a predetermined direction among the plurality of images.

[0110] With this configuration, the image in which the object is located most centrally in a specified direction among multiple images is extracted as a thumbnail image, allowing the viewer to easily grasp the outline of the detected object from the thumbnail image.

[0111] 8 to 10, the angle of view of each camera of the imaging unit 70 may be set so that the contact surface of the work machine 1 is included in the image captured by the imaging unit 70, and the direction away from the work machine 1 is the up-down direction in the image. In this case, the predetermined direction is the left-right direction in the image, which is perpendicular to the up-down direction. With this configuration, the viewer can easily and accurately confirm in the thumbnail image that the target object is located near the work machine 1 by looking at the center of the left-right direction in the image.

[0112] The extraction unit 103 may be capable of saving or outputting images including at least the thumbnail image in a predetermined file format. In this case, images including thumbnail images in which objects are reflected can be saved in a file format such as PDF, or output by data transfer or printout, allowing viewers to use this information in meetings with workers involved with the work machine 1, for example, to raise their awareness.

[0113] FIG. 11 is an example of a screen displayed on the viewing unit 104 of an image display system 1S according to a modified example of this embodiment (second embodiment). FIG. 11 has specifications similar to those of FIG. 6 of the previous embodiment. In this embodiment, images for one day are displayed in two rows, one above the other. In this modified example, too, the time chart can be used to analyze the time periods during which dangerous work occurs. Furthermore, for example, in images in which a person is detected in a stop zone, each image is surrounded by a thick frame (selection guide), thereby assisting the viewer in their confirmation work.

[0114] 12 to 18 show an example of how a person in an image is recognized in the image display system 1S according to the second embodiment of the present invention and its modified example. When the extraction unit 103 extracts thumbnail images, the person may be extracted based on the coordinates of the person's feet, as indicated by the dots in FIG. 12, and the entire person may be displayed surrounded by a frame H. However, in this case, the person may be positioned higher in the image and smaller in size. Therefore, by displaying the detected person's face (HA) in the center of the image, as shown in FIG. 13, the viewer can more easily grasp the situation at the work site in the video. In this case, as shown in FIG. 14, it is desirable for the object detection unit 75 (FIG. 2) to detect the person in advance by body part, for example, by each range: face (HA), shoulders and above (HB), upper body (HC), knees and above (HD), and whole body (H). By detecting each body part in this way, even if a part of the body or the lower body is hidden, detection is possible, leading to an improved detection rate.

[0115] Furthermore, as described above, if the extraction unit 103 is capable of determining whether an object included in the image is a person and identifying the person by a plurality of body parts including at least a face, the extraction condition may include a condition that the thumbnail image is an image in which the identified person's face is located most centrally in a predetermined direction among the plurality of images. With this configuration, the image in which the person's face is located most centrally in a predetermined direction among the plurality of images is extracted as a thumbnail image, allowing the viewer to easily grasp the outline of the detected person from the thumbnail image.

[0116] Furthermore, the extraction unit 103 may preferentially select, as thumbnail images, images in which a large range of body parts is detected in the video, in other words, images in which a person occupies a relatively large area. In this case, even if a person's body is hidden behind an object as in FIGS. 15 and 16, or even if a person is holding an object and part of their body is hidden by the object as in FIGS. 17 and 18, the extraction unit 103 extracts an image in which the object (person) is easily recognizable, allowing the viewer to easily grasp the outline of the detected person from the thumbnail image. For example, compared to FIGS. 15 and 17, the range of the detected body is larger (wider) in FIGS. 16 and 18, and therefore they are preferentially extracted as thumbnail images.

[0117] <Image display method> The image display system 1S described above can execute the following image display method. The image display method includes an image acquisition step of capturing an image of an area including a detection range set around the work machine 1, a detection step of detecting an object entering the detection range and outputting a detection signal, an image extraction step of extracting a specific image from the captured images that corresponds to the time when the detection signal was output and that satisfies extraction conditions set for a viewer to view the object through the viewing unit 104, and a display step of displaying the specific image on the display unit.

[0118] According to this method, a specific image that corresponds to the time when the object was detected from the captured images and that is extracted based on the extraction conditions can be displayed on the viewing unit 104. This makes it possible for a viewer such as a work manager to efficiently obtain information about objects that have approached the work machine 1, compared to when all of the captured images or all of the images corresponding to the detection signal are displayed on the viewing unit 104.

[0119] The image extraction process also includes extracting a video corresponding to a detection period, the detection period being a period during which the object is continuously detected in the detection process, the video being composed of the plurality of images, and the image extraction process includes extracting a specific image that satisfies the extraction conditions from the plurality of images that compose the video as a thumbnail image that represents the video.

[0120] According to this method, a specific image extracted based on extraction conditions from among multiple images that make up a video can be displayed as a thumbnail image that represents the video, allowing the viewer to easily grasp the outline of the detected object from the thumbnail image and efficiently begin viewing the video, the details of which they wish to check.

[0121] The contents included in the description of the image display system 1S above can constitute a part of the image display method.

[0122] <Image display program> The image display system 1S described above includes the following image display program that causes a computer, exemplified by the external terminal 100, to perform predetermined operations. This program is an image display program for the work machine 1 that is executed on computers connected to an imaging unit 70 that can capture images of an area including a detection range set around the work machine 1, and a viewing unit 104 that can display a display image in accordance with an input display command signal. The program causes the computer to execute the following steps: a storage step of storing images captured by the imaging unit 70; a detection step of detecting that an object has entered the detection range based on the images captured by the imaging unit 70 and outputting a detection signal; an image extraction step of extracting a specific image from a plurality of images captured by the imaging unit 70 that corresponds to the time the detection signal was output and that satisfies extraction conditions set so that the viewer can visually recognize the object through the viewing unit 104; and a display step of inputting the display command signal including the specific image to the display unit and displaying the specific image.

[0123] According to this program, the program can be executed on a computer so that a specific image corresponding to the time when an object was detected from among the images captured by the imaging unit 70, which is further extracted based on extraction conditions, is displayed on the viewing unit 104. This makes it possible for a viewer, such as a work manager, to efficiently obtain information about objects approaching the work machine 1, compared to when all of the images captured by the imaging unit or all of the images corresponding to the detection signal are displayed on the viewing unit 104.

[0124] In addition, the imaging unit 70 is capable of capturing a video composed of a plurality of images, and the image extraction step further includes extracting the specific image from the plurality of images based on the extraction conditions as a thumbnail image representative of the video corresponding to the detection period during which the object is continuously detected, and the display step further includes displaying a thumbnail of the extracted specific image to represent the video, and when a viewer viewing the viewing unit 104 selects one of the specific images displayed on the viewing unit 104, enlarging and displaying the video including the one specific image on the viewing unit 104.

[0125] According to this program, a specific image extracted based on extraction conditions from among multiple images that make up a video can be displayed as a thumbnail image that represents the video, allowing viewers to easily grasp the outline of the detected object from the thumbnail image and efficiently begin viewing the video in detail.

[0126] The display step further includes displaying a plurality of the specific images corresponding to the plurality of videos as thumbnails on the viewing unit 104, and displaying a selection guide that can assist the viewer in selecting a specific image with a high risk level from the plurality of specific images. The selection guide includes the time chart T, the bold frame B, the bold frame C, the menu D, etc. in FIG. 6.

[0127] According to this program, the viewer can easily select a specific image with a high risk level based on the selection guide and view that video.

[0128] The contents included in the description of the image display system 1S can constitute a part of the image display program.

[0129] The image display system 1S according to the present invention and the work machine 1 equipped with the same have been described above, but the present invention is not limited to this and can take on modified embodiments such as those described below.

[0130] (1) The techniques disclosed in the first embodiment and its modified examples, and the second embodiment and its modified examples may be combined with each other. The methods for extracting thumbnail images described in Figures 8 to 10 and 12 to 18 may be applied when extracting video to be displayed on the viewing unit 104 in the first embodiment.

[0131] (2) In the above embodiments, the extraction unit 103 extracts, as a thumbnail image, an image in which a person's face or body part is more easily visible, but the present invention is not limited to this. The extraction unit 103 can determine whether an object included in an image is a person and identify the orientation of the person's face, and the extraction condition may include a condition that the thumbnail image is an image in which the identified person's face is facing in a direction different from the work machine 1 among the multiple images.

[0132] With this configuration, images of people who are not facing the work machine 1 and who may be working without noticing the presence of the work machine 1 can be efficiently extracted, and the site manager viewing the viewing unit 104 can take corrective measures against the people.

[0133] Furthermore, when the object detection unit 75 detects an object during each of multiple detection periods, the extraction unit 103 (image input unit) may input the display command signal to the viewing unit 104 so that, among the multiple videos, a video in which the identified person's face spends a relatively long time facing a direction different from the work machine 1 is preferentially displayed on the viewing unit 104.

[0134] As mentioned above, nearby workers who are not facing the direction of the work machine 1 may be working without noticing the work machine 1. Therefore, by having the extraction unit 103 extract images as described above, it becomes possible to efficiently extract images of workers who are not facing the work machine 1 for a longer period of time and are in dangerous situations, allowing the site manager to quickly take corrective measures against the worker.

[0135] (3) In another embodiment, the extraction conditions used by extraction unit 103 to extract videos to display on viewing unit 104 may include a condition that the specific image is the video corresponding to the detection period longer than a preset threshold. According to this configuration, by displaying an image corresponding to a detection period longer than a preset threshold as a specific image on viewing unit 104, images with short detection periods are not displayed on viewing unit 104, allowing the viewer to efficiently check images with high urgency. Furthermore, by not displaying such videos with short detection periods, the priority of videos momentarily erroneously detected by object detection unit 75 is lowered, thereby eliminating the need to check unnecessary videos.

[0136] On the other hand, if the detection signal is interrupted for a moment and then immediately detected again, it may be determined that the target object is being detected continuously. For example, if the update period for detecting the target object is 100 msec, even if the target object is not detected for three periods, the detection signal is determined to be continuous, i.e., the setting does not take into account the 300 msec signal-free period.

[0137] (4) In another embodiment, the related information may include information relating to the distance between the work machine 1 and an object, and the extraction conditions used by the extraction unit 103 to extract videos to display on the viewing unit 104 may include a condition that the specific image is an image corresponding to a relatively small distance within the population of images. With this configuration, images in which the distance between the work machine 1 and the object is small can be displayed on the viewing unit 104, allowing the viewer to efficiently check images that pose a high risk.

[0138] In this case, too, the extraction unit 103 may input the display command signal to the viewing unit 104 so that the specific image corresponding to the relatively small distance from within the population of the specific images is preferentially displayed on the viewing unit 104. With this configuration, specific images with smaller distances can be displayed preferentially, allowing the viewer to more efficiently check images with a high degree of danger. For example, between a video image detected for 5 seconds at a point 2 m from the work machine 1 and a video image detected for 5 seconds at a point 1 m from the work machine 1, the video image detected at the point 1 m is set to have a higher priority.

[0139] (5) In the above embodiment, as shown in FIG. 2, the image and the detection signal are each stored in the external storage unit 102 of the external terminal 100. However, the image display system 1S may further include an image storage unit (not shown) capable of storing the image captured by the imaging unit 70 together with time information, and a signal storage unit (not shown) capable of storing the detection signal output from the object detection unit 75 together with time information. In this case, the extraction unit 103 may identify, as the specific image, the image stored in the image storage unit that corresponds to the time information associated with the detection signal stored in the signal storage unit. Even in this configuration, since the image and the detection signal are each stored together with time information, the specific image corresponding to the detection signal can be easily and quickly identified. Furthermore, to improve the visibility of the viewing unit 104, it is possible to hide one piece of information from the other. Furthermore, since the image and the detection signal are stored in separate storage units, it is easy to display each piece of information independently on the viewing unit 104 or to combine and display the information on the viewing unit 104.

[0140] (6) In each of the above embodiments, the imaging unit 70 is described as being attached to the work machine 1, but the present invention is not limited to this. The imaging unit 70 may be installed in part of the work site, or may be mounted on an aircraft capable of flying above the work site. Furthermore, the imaging unit 70 may be a detection device other than a camera, such as a LiDAR (Light Detection and Ranging), or may be a combination of a camera and a LiDAR.

[0141] For example, when using point cloud data acquired by LiDAR, color information (RGB) can be extracted from the camera's color image and assigned as the color of each point, which can then be displayed on the viewing unit 104. Furthermore, when LiDAR is used, it is possible to determine which part of the object's coordinates is the closest point to the work machine 1, and therefore it is possible to display this closest point in red on the viewing unit 104 or to increase the size of the displayed point. Furthermore, when point cloud data such as the above is acquired, unlike ordinary camera images, it is possible to move the viewpoint and display it on the viewing unit 104, which has the effect of making it easier for the viewer to visually grasp the distance between the work machine 1 and the object. In this case, by displaying a pre-specified area in red (a special color), the position of the area can be more easily visually recognized as the viewpoint moves. [Explanation of symbols]

[0142] 1. Work machinery 10 Undercarriage 100 External Terminal 101 Communication equipment 102 External storage section (storage section) 103 Extraction unit (image extraction unit, image input unit) 104 Viewing section (display section) 12 Upper rotating body 16 Swivel frame 17 Counterweight 18 Cab 1S image display system 3 Work attachments 4. Boom 5 Arm 6 buckets 7 Boom cylinder 70 Imaging unit 71 Display image generation unit 72 monitors 73 Video Storage Department 74 Conversion Unit 75 Object detection unit (detection unit) 76 Detection signal storage unit 77 Communication device (transmitter) 8 Arm Cylinder 80 Operating lever 81 Periphery detection function disable switch 9 Bucket cylinder 90 Control Unit 91 Motion control section 92 Lever lock (lock part) 95 Drive unit G ground

Claims

1. a detection unit that detects when an object enters a detection range set around the work machine and outputs a detection signal; an imaging unit capable of imaging an area around the work machine including the detection range; a display unit capable of displaying a display image in accordance with an input display command signal; an image extraction unit that extracts, from among a plurality of images captured by the imaging unit, a specific image that corresponds to the time when the detection signal is output and that satisfies extraction conditions set for a viewer to visually recognize the object on the display unit; and an image input unit that inputs the display command signal including the specific image to the display unit; Equipped with the image extraction unit extracts, as the plurality of images, a plurality of moving images corresponding to a plurality of detection periods, each of the plurality of detection periods being a period during which the detection unit continuously detects the object; the extraction conditions include a condition that the specific image is the video corresponding to the detection period that is relatively long in the population of the video; the detection signal includes relevant information related to the object, the relevant information including the detection period; The image input unit inputs the display command signal including the specific image and the associated information corresponding to the specific image to the display unit.

2. 2. The image display system for a work machine according to claim 1, wherein the image input unit inputs the display command signal to the display unit so that the moving image corresponding to the relatively long detection period within the population of moving images is preferentially displayed on the display unit.

3. An image display system for a work machine as described in claim 1, wherein the extraction conditions include a condition that the specific image is the video corresponding to the detection period longer than a predetermined threshold value.

4. the associated information includes information regarding a distance between the work machine and the object; The image display system for a work machine according to claim 1 , wherein the extraction conditions include a condition that the specific image is the image corresponding to the distance that is relatively small within the population of images.

5. 5. The image display system for a work machine according to claim 4, wherein the image input unit inputs the display command signal to the display unit so that the specific image corresponding to the relatively small distance within the population of the specific image is preferentially displayed on the display unit.

6. a storage unit capable of storing the image captured by the imaging unit together with time information, and further capable of storing the detection signal together with the time information; 2. The image display system for a work machine according to claim 1, wherein the image extraction unit extracts, from the images stored in the storage unit, the image that corresponds to the time information accompanying the detection signal as the specific image.

7. an image storage unit capable of storing the image captured by the imaging unit together with time information; a signal storage unit capable of storing the detection signal output from the detection unit together with time information; Further provided with 2. The image display system for a work machine according to claim 1, wherein the image extraction unit identifies, from among the images stored in the image storage unit, the image that corresponds to the time information associated with the detection signal stored in the signal storage unit as the specific image.

8. a conversion unit capable of compressing and converting at least one of the image captured by the imaging unit and the detection signal output from the detection unit into data in a predetermined format; a transmitting unit that transmits the format data generated by the converting unit; The work machine visual display system of claim 1 further comprising:

9. The work machine has an actuator and a locking unit capable of locking movement of the actuator, the related information further includes additional information that is information regarding at least one of a lock state of the actuator by the lock unit, a detectable state of the object by the detection unit, and an operating speed of the work machine, 2. The image display system for a work machine according to claim 1, wherein the image input unit inputs the display command signal to the display unit so that the additional information is displayed on the display unit together with the specific image.

10. The video is composed of the plurality of images, the image extraction unit extracts the specific image that satisfies the extraction condition from the plurality of images as a thumbnail image that represents the moving image; The image display system for a work machine according to claim 1 , wherein the image input unit inputs the display command signal corresponding to the moving image including the thumbnail image to the display unit.

11. the related information includes relative position information of the object with respect to the work machine; the image extraction unit is capable of further identifying a position of the object in the image based on the position information; The image display system for a work machine according to claim 10 , wherein the extraction conditions include a condition that the thumbnail image is an image in which the object is located most centrally in a predetermined direction among the plurality of images.

12. the angle of view of the imaging unit is set so that the ground contact surface of the work machine is included in the image captured by the imaging unit, and the direction away from the work machine is the up-down direction in the image; The image display system for a work machine according to claim 11, wherein the predetermined direction is a left-right direction in the image that is perpendicular to the up-down direction.

13. The image display system for a work machine according to claim 10, wherein the image extraction unit is capable of saving or outputting images including at least the thumbnail images in a predetermined file format.

14. A detection unit that detects when an object enters a detection range set around the work machine and outputs a detection signal; an imaging unit capable of imaging an area around the work machine including the detection range; a display unit capable of displaying a display image in accordance with an input display command signal; an image extraction unit that extracts, from among a plurality of images captured by the imaging unit, a specific image that corresponds to the time when the detection signal is output and that satisfies extraction conditions set for a viewer to visually recognize the object on the display unit; and an image input unit that inputs the display command signal including the specific image to the display unit; Equipped with the image extraction unit extracts a video corresponding to a detection period, the detection period being a period during which the detection unit continuously detects the object; the video is composed of the plurality of images, the image extraction unit extracts the specific image that satisfies the extraction condition from the plurality of images as a thumbnail image that represents the moving image; the image input unit inputs the display command signal corresponding to the moving image including the thumbnail image to the display unit; the image extraction unit is capable of determining whether the object included in the image is a person and identifying the person by a plurality of body parts including at least a face; The extraction conditions include a condition that the thumbnail image is an image in which the identified person's face is located most centrally in a predetermined direction among the plurality of images.

15. A detection unit that detects when an object enters a detection range set around the work machine and outputs a detection signal; an imaging unit capable of imaging an area around the work machine including the detection range; a display unit capable of displaying a display image in accordance with an input display command signal; an image extraction unit that extracts, from among a plurality of images captured by the imaging unit, a specific image that corresponds to the time when the detection signal is output and that satisfies extraction conditions set for a viewer to visually recognize the object on the display unit; and an image input unit that inputs the display command signal including the specific image to the display unit; Equipped with the image extraction unit extracts a video corresponding to a detection period, the detection period being a period during which the detection unit continuously detects the object; the video is composed of the plurality of images, the image extraction unit extracts the specific image that satisfies the extraction condition from the plurality of images as a thumbnail image that represents the moving image; the image input unit inputs the display command signal corresponding to the moving image including the thumbnail image to the display unit; the image extraction unit is capable of determining whether the object included in the image is a person; The extraction conditions include a condition that the thumbnail image is an image in which the area occupied by the person is relatively large among the plurality of images.

16. A detection unit that detects when an object enters a detection range set around the work machine and outputs a detection signal; an imaging unit capable of imaging an area around the work machine including the detection range; a display unit capable of displaying a display image in accordance with an input display command signal; an image extraction unit that extracts, from among a plurality of images captured by the imaging unit, a specific image that corresponds to the time when the detection signal is output and that satisfies extraction conditions set for a viewer to visually recognize the object on the display unit; and an image input unit that inputs the display command signal including the specific image to the display unit; Equipped with the image extraction unit extracts a video corresponding to a detection period, the detection period being a period during which the detection unit continuously detects the object; the video is composed of the plurality of images, the image extraction unit extracts the specific image that satisfies the extraction condition from the plurality of images as a thumbnail image that represents the moving image; the image input unit inputs the display command signal corresponding to the moving image including the thumbnail image to the display unit; the image extraction unit is capable of determining whether the object included in the image is a person and identifying the direction of the person's face; The extraction conditions include a condition that the thumbnail image is an image among the plurality of images in which the face of the identified person is facing in a direction different from the work machine.

17. 17. The image display system for a work machine according to claim 16, wherein, when the detection unit detects the object during each of the plurality of detection periods, the image input unit inputs the display command signal to the display unit so that, of the plurality of videos, a video in which the identified person's face spends a relatively long time facing a direction away from the work machine is preferentially displayed on the display unit.

18. an image acquisition step of capturing an image of an area including a detection range set around the work machine; a detection step of detecting an object entering the detection range and outputting a detection signal; an image extraction step of extracting, from the plurality of captured images, a specific image that corresponds to the time when the detection signal was output and that satisfies extraction conditions set for a viewer to visually recognize the object through a display unit; a display step of displaying the specific image on the display unit; Equipped with the image extraction step includes extracting, as the plurality of images, a plurality of moving images corresponding to a plurality of detection periods, each of the plurality of detection periods being a period during which the object is continuously detected; the extraction conditions include a condition that the specific image is the video corresponding to the detection period that is relatively long in the population of the video; the detection signal includes relevant information related to the object, the relevant information including the detection period; The display step is a work machine image display method in which the specific image and the associated information corresponding to the specific image are displayed on the display unit.

19. The video is composed of the plurality of images, 19. The image display method for a work machine according to claim 18, wherein the image extraction step includes extracting the specific image that satisfies the extraction condition from the plurality of images that make up the video as a thumbnail image that represents the video.

20. A work machine image display program executed on a computer connected to an imaging unit capable of imaging an area including a detection range set around the work machine, and a display unit capable of displaying a display image according to an input display command signal, The computer, a storage step of storing the image captured by the imaging unit; a detection step of detecting that an object has entered the detection range based on the image captured by the imaging unit and outputting a detection signal; an image extraction step of extracting a specific image from a plurality of images captured by the imaging unit, the specific image corresponding to the time when the detection signal was output, and satisfying extraction conditions set for a viewer to visually recognize the object through the display unit; a display step of inputting the display command signal including the specific image to the display unit to display the specific image; Execute the image extraction step includes extracting, as the plurality of images, a plurality of moving images corresponding to a plurality of detection periods, each of the plurality of detection periods being a period during which the object is continuously detected; the extraction conditions include a condition that the specific image is the video corresponding to the detection period that is relatively long in the population of the video; the detection signal includes relevant information related to the object, the relevant information including the detection period; The display step includes inputting the display command signal including the specific image and the associated information corresponding thereto to the display unit.

21. the imaging unit is capable of capturing a moving image made up of a plurality of images, the image extraction step further includes extracting the specific image from the plurality of images based on the extraction conditions as a thumbnail image representative of the moving image corresponding to a detection period during which the object is continuously detected; The display step includes: displaying a thumbnail of the extracted specific image so as to represent the moving image; When a viewer viewing the display unit selects one of the specific images displayed on the display unit, the moving image including the selected specific image is enlarged and displayed on the display unit; 21. The work machine image display program according to claim 20, further comprising:

22. The display step includes: displaying a plurality of the specific images corresponding to a plurality of the moving images side by side as thumbnails on the display unit; displaying a selection guide that can assist the viewer in selecting a specific image with a high risk level from among the plurality of specific images; 22. The computer program product for displaying images of a work machine according to claim 21, further comprising:

Images