Work machine and operating system for work machine

Abstract

To provide a work machine capable of notifying a problem in detecting humans with an object detection system.SOLUTION: The work machine includes: a lower running body; an upper rotating body; an operator cab placed on the upper rotating body; an external sound pickup unit M1 placed outside the operator cab; an object detection unit that includes an imaging device S6 for detecting people around the upper rotating body; and a controller 30 as the control unit. The controller 30 is configured to notify a problem in human detection when there is a difference between the direction of a person detected by the object detection unit, i.e., the person detection direction and the sound direction as the direction from which the sound is emitted, as captured by the external sound pickup unit M1.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a work machine and an operation system for a work machine. [Background technology]

[0002] Conventionally, there has been known an excavator having a perimeter monitoring device that monitors the perimeter of the work machine (see Patent Document 1 below). The excavator described in Patent Document 1 superimposes a frame corresponding to the object on an image displayed on a display unit when the distance between the excavator and an object detected by the perimeter monitoring device is less than a predetermined value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-004484 Summary of the Invention [Problem to be solved by the invention]

[0004] In the excavator described in Patent Document 1, depending on the conditions under which the cameras and stereo cameras constituting the periphery monitoring device capture images, people around the work machine may not be detected, and frames corresponding to people may not be superimposed on the image displayed on the display unit.

[0005] The present disclosure provides a work machine and a work machine operation system that can notify of inadequate human detection by an object detection device. [Means for solving the problem]

[0006] One aspect of the present disclosure provides a work machine comprising: a lower running body; an upper rotating body rotatably mounted on the lower running body; a driver's cab mounted on the upper rotating body; an external sound collection device arranged outside the driver's cab; an object detection device that detects people around the upper rotating body; and a control device that notifies of a deficiency in person detection when a person detection direction, which is the direction of the person detected by the object detection device, differs from a sound direction, which is the direction from which sound collected by the external sound collection device is emitted.

[0007] Another aspect of the present disclosure provides an operation system for a work machine, the system comprising: a work machine including a lower running body, an upper rotating body rotatably mounted on the lower running body, a driver's cab provided on the upper rotating body, an external sound collection device arranged outside the driver's cab, and an object detection device that detects people around the upper rotating body; and a control device that notifies of a deficiency in person detection when a person detection direction, which is the direction of the person detected by the object detection device, differs from a sound direction, which is the direction from which sound collected by the external sound collection device is emitted. [Effects of the Invention]

[0008] According to the above aspects of the present disclosure, it is possible to provide a work machine and a work machine operation system that can notify of inadequate human detection by an object detection device. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a side view of a work machine according to an embodiment of the present disclosure; FIG. [Figure 2] FIG. 2 is a top view of the work machine shown in FIG. [Figure 3] 2 is a schematic diagram of an external sound collector attached to the work machine shown in FIG. 1. [Figure 4] FIG. 2 is a block diagram showing a schematic configuration of the work machine shown in FIG. [Figure 5] FIG. 2 is a plan view showing the configuration of a cab in the work machine shown in FIG. [Figure 6] FIG. 2 is a conceptual diagram of a two-way conversation between the operator of the work machine shown in FIG. 1 and people nearby. [Figure 7] 5 is a flow chart illustrating the operation of the control device for the work machine shown in FIG. 4. [Figure 8] 2 is a plan view showing the positional relationship between the work machine shown in FIG. 1 and people around it. FIG. [Figure 9] 5 is an image diagram showing an image displayed on a display device of the work machine shown in FIG. 4. FIG. [Figure 10] FIG. 2 is a plan view showing a modified example of the work machine shown in FIG. [Figure 11] 1. FIG. 4 is a top view of another example of the configuration of the work machine shown in FIG. [Figure 12] 1 is a schematic diagram of an operation system for a work machine according to an embodiment of the present disclosure. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The embodiments described below are illustrative and do not limit the invention. Not all features and combinations thereof in the embodiments of the present disclosure are necessarily essential to the invention. In addition, identical or corresponding components in each drawing are designated by identical or corresponding reference numerals, and redundant description may be omitted.

[0011] First, an overview of a work machine 100 will be described with reference to Figures 1 and 2. Figure 1 is a top view of the work machine 100, and Figure 2 is a side view of the work machine 100.

[0012] The work machine 100 according to the embodiment of the present disclosure is a shovel. The work machine 100 may be a machine other than a shovel, such as a crane, an asphalt finisher, or a forklift. In the illustrated example, the shovel serving as the work machine 100 is an excavator equipped with a bucket 6 as an end attachment, but the work machine 100 may also be an applied machine such as a forestry machine equipped with an end attachment other than the bucket 6.

[0013] +X in Fig. 1 represents one direction of the X axis that constitutes a three-dimensional Cartesian coordinate system, and -X represents the other direction of the X axis. In Fig. 2, +Y represents one direction of the Y axis that constitutes a three-dimensional Cartesian coordinate system, and -Y represents the other direction of the Y axis. In Fig. 1, +Z represents one direction of the Z axis that constitutes a three-dimensional Cartesian coordinate system, and -Z represents the other direction of the Z axis. In Fig. 1, the +X side of the work machine 100 corresponds to the front side of the work machine 100, and the -X side of the work machine 100 corresponds to the rear side of the work machine 100. Furthermore, the +Y side of the work machine 100 corresponds to the left side of the work machine 100, and the -Y side of the work machine 100 corresponds to the right side of the work machine 100. Furthermore, the +Z side of the work machine 100 corresponds to the top side of the work machine 100, and the -Z side of the work machine 100 corresponds to the bottom side of the work machine 100. The same applies to the other figures.

[0014] The work machine 100 comprises a lower traveling body 1, an upper rotating body 3 that is mounted on the lower traveling body 1 so as to be rotatable via a rotating mechanism 2, an attachment AT for performing various tasks, and a driver's cab 10. The driver's cab 10 is also called a cabin or cab. The front side of the work machine 100 (upper rotating body 3) corresponds to the side of the upper rotating body 3 to which the attachment AT is attached, when the work machine 100 is viewed from directly above along the rotation axis of the upper rotating body 3. Furthermore, the left, right, and rear sides of the work machine 100 (upper rotating body 3) correspond to the left, right, and rear sides, respectively, as viewed from the perspective of an operator seated in the driver's seat in the driver's cab 10.

[0015] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the crawlers 1C include a left crawler 1CL and a right crawler 1CR. The left crawler 1CL is driven by a left traveling hydraulic motor 2ML, and the right crawler 1CR is driven by a right traveling hydraulic motor 2MR. The left traveling hydraulic motor 2ML is a traveling drive unit that drives the left crawler 1CL as a driven part, and can rotate the left crawler 1CL. The right traveling hydraulic motor 2MR is a traveling drive unit that drives the right crawler 1CR as a driven part, and can rotate the right crawler 1CR. Note that the traveling drive units may be electric motors.

[0016] A boom 4 is rotatably attached to the center of the front of the upper rotating body 3, an arm 5 is rotatably attached to the tip of the boom 4, and a bucket 6 is rotatably attached to the tip of the arm 5. In the illustrated example, the boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of an attachment AT. The boom 4, arm 5, and bucket 6 are driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

[0017] The bucket 6 is an example of a work tool (end attachment). The bucket 6 is used, for example, for excavation work. Instead of the bucket 6, another work tool may be attached to the end of the arm 5 depending on the type of work, etc. The other work tool may be, for example, another type of bucket, such as a large bucket, a slope bucket, or a dredging bucket. The other work tool may also be a type of work tool other than a bucket, such as an agitator, a breaker, a grapple, or a lifting magnet.

[0018] The swing hydraulic motor 2A, the left traveling hydraulic motor 2ML, the right traveling hydraulic motor 2MR, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 are hydraulic actuators driven by hydraulic oil discharged from a hydraulic pump.

[0019] Note that in the work machine 100, all or some of the driven parts, such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6, may be electrically driven. In other words, the work machine 100 may be a hybrid excavator, an electric excavator, or the like, in which all or some of the driven parts are driven by electric actuators.

[0020] Additionally, the work machine 100 is equipped with an information transmission device G1, an external sound collection device M1, an imaging device S6, and an external sound output device SP1.

[0021] The imaging device S6 is provided on the upper rotating body 3 or the cab 10, and captures images of the periphery of the work machine 100 to obtain image information showing the periphery of the work machine 100. In the illustrated example, the imaging device S6 includes a front camera S6F, a left camera S6L, a right camera S6R, and a rear camera S6B.

[0022] The front camera S6F is a camera that captures images in front of the work machine 100, and is attached to the outside of the cab 10, such as on the roof of the cab 10 or on the side of the boom 4. The front camera S6F may also be attached to the inside of the cab 10, for example, on the ceiling of the cab 10. The left camera S6L is a camera that captures images to the left of the work machine 100, the right camera S6R is a camera that captures images to the right of the work machine 100, and the rear camera S6B is a camera that captures images to the right of the work machine 100. Specifically, the front camera S6F, left camera S6L, right camera S6R, and rear camera S6B are all monocular wide-angle cameras equipped with an imaging element such as a CCD or CMOS, and output the captured images to the display device D1 (see FIGS. 4 and 5). Information about the images captured by the imaging device S6 is taken into the controller 30.

[0023] In the illustrated example, the front camera S6F is attached to the roof of the driver's cab 10, the left camera S6L is attached to the left end of the upper surface of the upper rotating body 3, the right camera S6R is attached to the right end of the upper surface of the upper rotating body 3, and the rear camera S6B is attached to the rear end of the upper surface of the upper rotating body 3.

[0024] The imaging device S6 may constitute an object detection device that detects objects around the work machine 100. Examples of objects include people, animals, vehicles, construction machinery, buildings, and holes. The object detection device may be configured to be able to distinguish between people and non-human objects when detecting them. In other words, the object detection device may be configured to function as a person detection device. The object detection device may be constituted by a device other than a camera. For example, the object detection device may be a LiDAR. A LiDAR is a device that can measure the distance between the LiDAR (laser source) and a point cloud of, for example, one million or more points within a monitoring range. The object detection device may also be another device that can measure the distance to an object, such as a stereo camera, a range imaging camera, or a millimeter-wave radar. When a millimeter-wave radar or the like is used as the object detection device, the object detection device may emit a number of signals (such as laser light) toward the object and receive the reflected signals to derive the distance and direction of the object. Alternatively, the object detection device may be a combination of two or more types of devices. For example, the object detection device may be a combination of an imaging device and a LiDAR, a combination of an imaging device and a millimeter wave radar, or a combination of an imaging device and a stereo camera.

[0025] The external sound collection device M1 is disposed outside the cab 10, and generates a signal from sounds collected outside the cab 10 around the work machine 100, and outputs the signal to the controller 30. In other words, the external sound collection device M1 is a device that collects external sounds, and is also called a microphone. In the illustrated example, the external sound collection device M1 is provided on the upper rotating body 3 or the cab 10, and converts sounds (air vibrations) generated around the work machine 100 into mechanical vibrations, and then converts the mechanical vibrations into electrical signals. Specifically, the external sound collection device M1 is configured to be able to pick up voices uttered by workers around the work machine 100, and includes multiple microphones, such as a front microphone M1F, a left microphone M1L, a right microphone M1R, and a rear microphone M1B.

[0026] The front microphone M1F is a microphone that collects sounds generated in front of the work machine 100, and is attached to the outside of the cab 10, such as on the roof of the cab 10 or on the side of the boom 4. The front microphone M1F may also be attached to the inside of the cab 10, for example, on the ceiling of the cab 10. The left microphone M1L is a microphone that collects sounds generated to the left of the work machine 100, the right microphone M1R is a microphone that collects sounds generated to the right of the work machine 100, and the rear microphone M1B is a microphone that collects sounds generated behind the work machine 100. The electrical signals generated by the front microphone M1F, left microphone M1L, right microphone M1R, and rear microphone M1B are taken into the controller 30.

[0027] In the illustrated example, the front microphone M1F is attached to the roof of the cab 10, the left microphone M1L is attached to the left end of the upper surface of the upper rotating body 3, the right microphone M1R is attached to the right end of the upper surface of the upper rotating body 3, and the rear microphone M1B is attached to the rear end of the upper surface of the upper rotating body 3. In this way, the four external sound collection devices M1 (the front microphone M1F, the left microphone M1L, the right microphone M1R, and the rear microphone M1B) are provided at different positions on the upper rotating body 3. Therefore, the controller 30 can detect the direction of the sound source based on the difference in the sounds collected by each of the four external sound collection devices M1 (e.g., difference in volume). Furthermore, when an array microphone is used as the external sound collection device M1, the direction of the sound source can be detected based on, for example, a phase shift or difference in volume.

[0028] In the illustrated example, the four external sound collection devices M1 and the four image capture devices S6 are arranged to correspond to one another. Specifically, the front microphone M1F is arranged adjacent to the front camera S6F, the left microphone M1L is arranged adjacent to the left camera S6L, the right microphone M1R is arranged adjacent to the right camera S6R, and the rear microphone M1B is arranged adjacent to the rear camera S6B.

[0029] The external sound output device SP1 is a device that outputs sound toward the periphery of the work machine 100. The external sound output device SP1 is arranged, for example, outside the cab 10, and receives a signal generated from sound collected by the internal sound collection device M2 via the controller 30, and outputs sound based on that signal toward workers and other people positioned around the work machine 100. In the illustrated example, the external sound output device SP1 is an omnidirectional speaker that is configured to output sound uniformly in all directions. However, the external sound output device SP1 may also be a directional speaker that outputs sound toward a specific direction, such as forward.

[0030] The information transmission device G1 is a device for communicating the status of the work machine 100 to someone outside the work machine 100. In the illustrated example, the information transmission device G1 is provided on the upper rotating body 3 or the operator's cab 10, and is configured to be able to communicate the status of the work machine 100 to an operator around the work machine 100. Specifically, the information transmission device G1 is a light-emitting device, and includes a front light bar G1F, a left light bar G1L, a right light bar G1R, and a rear light bar G1B.

[0031] The front light bar G1F is a light-emitting device that can visually convey information to workers and the like in front of the work machine 100, and is attached to the outside of the cab 10, such as on the roof of the cab 10 or on the side of the boom 4. The front light bar G1F may also be attached to the inside of the cab 10, for example, on the ceiling of the cab 10. The left light bar G1L is a light-emitting device that can visually convey information to workers and the like on the left side of the work machine 100, the right light bar G1R is a light-emitting device that can visually convey information to workers and the like on the right side of the work machine 100, and the rear light bar G1B is a light-emitting device that can visually convey information to workers and the like behind the work machine 100. The front light bar G1F, left light bar G1L, right light bar G1R, and rear light bar G1B each emit light in response to an electrical signal from the controller 30. In the illustrated example, the light-emitting devices are LED lights, but they may also be other light-emitting devices, such as halogen lamps. Furthermore, the light emitting device is of a multicolor emission type, but may be of a monochromatic emission type.

[0032] In the illustrated example, the front light bar G1F is attached to the roof of the operator's cab 10, the left light bar G1L is attached to the left end of the upper surface of the upper rotating body 3, the right light bar G1R is attached to the right end of the upper surface of the upper rotating body 3, and the rear light bar G1B is attached to the rear end of the upper surface of the upper rotating body 3. In this way, the four information transmission devices G1 (front light bar G1F, left light bar G1L, right light bar G1R, and rear light bar G1B) are provided in different positions on the upper rotating body 3. Therefore, the controller 30 can communicate the status of the work machine 100 to workers and the like who are respectively located in front of, on the left, right, and rear of the work machine 100 by separately operating each of the four information transmission devices G1.

[0033] In the illustrated example, the four information transmission devices G1 and the four external sound collection devices M1 are arranged to correspond to each other. Specifically, the front light bar G1F is arranged adjacent to the front microphone M1F, the left light bar G1L is arranged adjacent to the left microphone M1L, the right light bar G1R is arranged adjacent to the right microphone M1R, and the rear light bar G1B is arranged adjacent to the rear microphone M1B.

[0034] Fig. 3 is a diagram showing an example configuration of an external sound collection device M1 and an information transmission device G1 attached to a work machine 100. Specifically, Fig. 3 is a perspective view of a left microphone M1L and a left light bar G1L attached to a housing having a substantially rectangular parallelepiped shape. Note that the following description with reference to Fig. 3 relates to the combination of the left microphone M1L and the left light bar G1L, but also applies similarly to the combination of the front microphone M1F and the front light bar G1F, the combination of the right microphone M1R and the right light bar G1R, and the combination of the rear microphone M1B and the rear light bar G1B.

[0035] As shown in FIG. 3, the left microphone M1L and left light bar G1L are arranged on the left side of a substantially rectangular parallelepiped housing so as to face to the left of the work machine 100. With this arrangement, the left microphone M1L can efficiently collect sounds generated to the left of the work machine 100, and the left light bar G1L can efficiently communicate the status of the work machine 100 to an operator on the left of the work machine 100. For example, the left microphone M1L can pick up the voice uttered by an operator on the left of the work machine 100, and the left light bar G1L can notify the operator that the left microphone M1L has picked up the operator's voice by emitting light in a predetermined color. In this case, an operator on the left of the work machine 100 who speaks into the left microphone M1L can confirm that their voice has reached the left microphone M1L (i.e., the operator of the work machine 100) by seeing the left light bar G1L emitting light in a predetermined color.

[0036] The information transmission device G1 may be provided at the upper part of each of the four side surfaces of the cab 10 (see FIG. 6). For example, the information transmission device G1 may be configured so that a front light bar G1F is attached to the upper part of the front surface of the cab 10, a left light bar G1L is attached to the upper part of the left surface of the cab 10, a right light bar G1R is attached to the upper part of the right surface of the cab 10, and a rear light bar G1B is attached to the upper part of the rear surface of the cab 10. The information transmission device G1 may also be a single rotating light such as a Nico Torch attached to the upper surface of the cab 10, or a display device such as a liquid crystal display or an organic EL display.

[0037] The controller 30 is an example of a control device, and is configured, for example, by a computer including a CPU, a volatile storage device, a non-volatile storage device, and various input / output interfaces. The controller 30 then realizes various functions, for example, by reading a program from the non-volatile storage device, loading it into the volatile storage device, and having the CPU execute the program. In the illustrated example, the controller 30 is configured to realize various functions to control the work machine 100. The various functions include, for example, a machine guidance function that guides the operator in manually operating the work machine 100. The various functions may also include a contact avoidance function that automatically or autonomously operates or stops the work machine 100 to avoid contact between the work machine 100 and an object that is present within a monitoring range around the work machine 100.

[0038] The boom angle sensor S1 detects the boom angle, which is the rotation angle of the boom 4 relative to the upper rotating body 3. The arm angle sensor S2 detects the arm angle, which is the rotation angle of the arm 5 relative to the boom 4. The bucket angle sensor S3 detects the bucket angle, which is the rotation angle of the bucket 6 relative to the arm 5.

[0039] Each of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may be, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder, etc.

[0040] The controller 30 receives a detection signal corresponding to the boom angle from the boom angle sensor S1, a detection signal corresponding to the arm angle from the arm angle sensor S2, and a detection signal corresponding to the bucket angle from the bucket angle sensor S3.

[0041] The machine body tilt sensor S4 detects the tilt state of the machine body (undercarriage 1 or upper rotating body 3) relative to the horizontal plane. The machine body tilt sensor S4 is attached to, for example, the upper rotating body 3, and detects the tilt angle of the work machine 100 (i.e., the upper rotating body 3) around two axes in the forward / backward and left / right directions. The machine body tilt sensor S4 may be, for example, an acceleration sensor, a six-axis sensor, an IMU, or the like. A detection signal corresponding to the tilt angle detected by the machine body tilt sensor S4 is input to the controller 30.

[0042] The rotation sensor S5 outputs information related to the rotation of the upper rotating body 3. The rotation sensor S5 detects, for example, the rotation angular velocity of the upper rotating body 3 relative to the lower traveling body 1. The rotation sensor S5 may also detect the rotation angle. The rotation sensor S5 may be, for example, a gyro sensor, a resolver, or a rotary encoder. A detection signal corresponding to the rotation angle or rotation angular velocity of the upper rotating body 3 detected by the rotation sensor S5 is input to the controller 30.

[0043] The positioning device PS measures the position of the upper rotating body 3. The positioning device PS is, for example, a Global Navigation Satellite System (GNSS) compass, and detects the position and orientation of the upper rotating body 3. A detection signal corresponding to the position and orientation of the upper rotating body 3 is input to the controller 30. The function of detecting the orientation of the upper rotating body 3 may be realized by a direction sensor attached to the upper rotating body 3.

[0044] The operator's cab 10 is a compartment space in which an operator rides, and is provided on the front left side of the upper rotating body 3. However, when the work machine 100 is remotely controlled or when the work machine 100 operates in a fully automatic manner, the operator's cab 10 may be omitted.

[0045] The communication device T1 communicates with an external device through a communication network including a mobile communication network, a satellite communication network, the Internet, etc. The communication device T1 is, for example, a mobile communication module compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), a communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

[0046] In response to operations by an operator seated in the cab 10, the work machine 100 operates actuators to drive driven parts such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6.

[0047] Alternatively, the work machine 100 may be configured so that it can be remotely operated from outside the work machine 100. When the work machine 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0048] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operation performed by the operator. This allows the work machine 100 to realize a function of automatically operating at least some of the driven parts, such as the lower traveling structure 1, upper rotating structure 3, boom 4, arm 5, and bucket 6, i.e., a so-called "machine control function."

[0049] Figure 4 is a diagram that schematically shows an example of the configuration of a work machine 100. In Figure 4, the mechanical power transmission system, hydraulic oil lines, pilot lines, and electrical control system are indicated by double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0050] The drive system of the work machine 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. The hydraulic drive system of the work machine 100 also includes hydraulic actuators such as a swing hydraulic motor 2A, a left traveling hydraulic motor 2ML, a right traveling hydraulic motor 2MR, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

[0051] The engine 11 is an example of a power source for the work machine 100, and is mounted, for example, on the rear of the upper rotating body 3. The power source for the work machine 100 may be a combination of a power source such as a battery or a fuel cell and an electric motor. Specifically, the engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by the controller 30, and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 may also be a gasoline engine, a hydrogen engine, or the like.

[0052] The regulator 13 controls the discharge amount of the main pump 14. For example, the regulator 13 controls the discharge amount of the main pump 14 by adjusting the angle (tilting angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0053] The main pump 14 is mounted on the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic oil to the control valve unit 17 through a hydraulic oil line. In the illustrated example, the main pump 14 is a variable displacement hydraulic pump.

[0054] The control valve unit 17 is one of the hydraulic control devices that controls the hydraulic system of the work machine 100. In the illustrated example, the control valve unit 17 includes control valves 171-176. The control valve unit 17 is configured to selectively supply hydraulic oil discharged by the main pump 14 to one or more hydraulic actuators via the control valves 171-176. The control valves 171-176 control the flow rate of hydraulic oil flowing from the main pump 14 to the hydraulic actuators, and the flow rate of hydraulic oil flowing from the hydraulic actuators to a hydraulic oil tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left traveling hydraulic motor 2ML, a right traveling hydraulic motor 2MR, and a swing hydraulic motor 2A. Specifically, the control valve 171 corresponds to the left traveling hydraulic motor 2ML, the control valve 172 corresponds to the right traveling hydraulic motor 2MR, and the control valve 173 corresponds to the swing hydraulic motor 2A. Furthermore, the control valve 174 corresponds to the bucket cylinder 9 , the control valve 175 corresponds to the boom cylinder 7 , and the control valve 176 corresponds to the arm cylinder 8 .

[0055] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to supply hydraulic oil to hydraulic control devices via a pilot line. In the illustrated example, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generating device may also be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to various hydraulic control devices via pilot lines, in addition to a function of supplying hydraulic oil to the control valve unit 17 via a hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0056] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the illustrated example, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0057] The operating device 26 is a device used by an operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator may be a hydraulic actuator or an electric actuator.

[0058] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each actuator and outputs the detected value to the controller 30. In the illustrated example, the controller 30 can control the opening area of ​​the valve 31 in accordance with the output of the operation sensor 29. The controller 30 then supplies the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17. The pressure of the hydraulic oil supplied to each pilot port (pilot pressure) is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to supply the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17.

[0059] Valve 31, which functions as a control valve for machine control, is disposed in a pipe connecting pilot pump 15 and a pilot port of a control valve in control valve unit 17, and is configured so that the flow path area of ​​the pipe can be changed. In the illustrated example, valve 31 is a solenoid valve that operates in response to a control command output by controller 30. Therefore, controller 30 can use valve 31 to adjust the pilot pressure acting on the pilot port of the control valve, regardless of the operation of operating device 26 by the operator.

[0060] With this configuration, the controller 30 can operate the hydraulic actuator corresponding to a specific operating device 26 even when the specific operating device 26 is not being operated.

[0061] As shown in FIG. 4, the control system of the work machine 100 includes a controller 30, a display device D1, an input device D2, a horn button HS, a talk button KS, an external sound collection device M1, an internal sound collection device M2, an external sound output device SP1, an internal sound output device SP2, an external volume dial DL1, an internal volume dial DL2, a switch SW, and a communication device T1.

[0062] The controller 30 is configured to output a control command to the regulator 13 as necessary, thereby changing the discharge rate of the main pump 14.

[0063] Furthermore, the controller 30 may be configured to perform control relating to a machine guidance function that guides (provides guidance for) the manual operation of the work machine 100 by the operator via the operation device 26. Furthermore, the controller 30 may be configured to perform control relating to a machine control function that automatically assists the manual operation of the work machine 100 by the operator via the operation device 26.

[0064] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers. For example, the machine guidance function and the machine control function may be realized by a dedicated controller (control device).

[0065] The interior of the cab 10 will now be described with reference to Figure 5. Figure 5 is a top view of the interior of the cab 10. The work machine 100 is equipped with a driver's seat 50, an operating device 26, a display device D1, and the like, which are arranged inside the cab 10. A door for getting on and off is provided on the left side of the driver's seat 50. The operator can enter the cab 10 by opening the door for getting on and off.

[0066] The driver's seat 50 is located in the center of the driver's cab 10 in a top view. The driver's seat 50 includes a seat portion 51 on which the operator sits and a backrest 52. The driver's seat 50 is a reclining seat, and the tilt angle of the backrest 52 is adjustable. A left armrest 53L is located on the left side of the driver's seat 50, and a right armrest 53R is located on the right side. The left armrest 53L and the right armrest 53R are rotatably supported by the backrest 52.

[0067] A left console 54L is disposed on the left side of the driver's seat 50, and a right console 54R is disposed on the right side. The left console 54L and the right console 54R extend in the front-to-rear direction. The driver's seat 50 is slidable in the front-to-rear direction. The driver's seat 50 may be configured to be slidable in the front-to-rear direction together with the left console 54L and the right console 54R.

[0068] The left armrest 53L is disposed on top of the left console 54L. The right armrest 53R is disposed on top of the right console 54R. The left armrest 53L is disposed so as to cover a portion of the left console 54L in a top view. The right armrest 53R is disposed so as to cover a portion of the right console 54R in a top view.

[0069] The operating device 26 includes a left operating lever 26L, a right operating lever 26R, a left travel pedal 26PL, a right travel pedal 26PR, a left travel lever 26DL, and a right travel lever 26DR.

[0070] The left operation lever 26L is provided in front of the left console 54L. Similarly, the right operation lever 26R is provided in front of the right console 54R. An operator seated in the driver's seat 50 can operate the left operation lever 26L while holding the left operation lever 26L with his left hand, and can operate the right operation lever 26R while holding the right operation lever 26R with his right hand. An operator seated in the driver's seat 50 can operate the left operation lever 26L with his left hand to drive the arm cylinder 8 and the swing hydraulic motor 2A. In addition, an operator seated in the driver's seat 50 can operate the right operation lever 26R with his right hand to drive the boom cylinder 7 and the bucket cylinder 9. The bases of the left operation lever 26L and the right operation lever 26R are covered with lever boots 27.

[0071] The left travel pedal 26PL and the right travel pedal 26PR are located on the floor in front of the driver's seat 50. An operator seated in the driver's seat 50 can operate the left travel pedal 26PL with his left foot to drive the left travel hydraulic motor 2ML. Also, an operator seated in the driver's seat 50 can operate the right travel pedal 26PR with his right foot to drive the right travel hydraulic motor 2MR.

[0072] The left travel lever 26DL and the right travel lever 26DR are located between the left travel pedal 26PL and the right travel pedal 26PR in a top view. The left travel lever 26DL and the right travel lever 26DR extend upward from the floor in front of the driver's seat 50. An operator seated in the driver's seat 50 can drive the left travel hydraulic motor 2ML by gripping and operating the left travel lever 26DL with their left hand, similar to operating the left travel pedal 26PL. Furthermore, an operator seated in the driver's seat 50 can drive the right travel hydraulic motor 2MR by gripping and operating the right travel lever 26DR with their right hand, similar to operating the right travel pedal 26PR. Furthermore, the left travel lever 26DL and the right travel lever 26DR are positioned so that the operator can operate the left travel lever 26DL and the right travel lever 26DR simultaneously with one hand.

[0073] The display device D1 is disposed inside the cab 10 in a location that is easily visible to the operator seated in the cab 10, and displays various information images under the control of the controller 30. In the illustrated example, the display device D1 is disposed on the front right side of the driver's seat 50, and is connected to the controller 30 via a dedicated line. The display device D1 displays various types of image information. The display device D1 includes a display screen that displays information such as the working conditions or operating state of the work machine 100. The operator seated in the driver's seat 50 can perform work using the work machine 100 while checking the various types of information displayed on the display device D1. The display device D1 may be provided with an input device D2.

[0074] The input device D2 is provided within reach of the operator seated on the left side of the driver's cab 10 in the driver's seat 50, and receives various operational inputs from the operator and outputs signals corresponding to the operational inputs to the controller 30. The input device D2 includes a touch panel mounted on the display of the display device D1 that displays various information images, a knob switch provided at the tip of one or more lever portions of a plurality of operating levers included in the operating device 26, or a button switch, lever, toggle switch, rotary dial, or the like provided around the display device D1. A signal corresponding to the content of an operation performed on the input device D2 is input to the controller 30.

[0075] A gate bar 55 is attached to the front surface of the front end of the left console 54L. The gate bar 55 operates in conjunction with the operation of a gate lock lever GL provided on the left console 54L. The gate bar 55 is attached to a frame inside the left console 54L so that it can be raised and lowered around an axis at the top end that extends in the left-right direction.

[0076] The gate lock lever GL is a mechanical input operation unit for switching between a state in which the work machine 100 can be operated by the operation device 26 (operable state) and a state in which the work machine 100 cannot be operated by the operation device 26 (inoperable state). In the illustrated example, the gate lock lever GL is configured so that the operator can switch between a first operation position that realizes the inoperable state and a second operation position that realizes the operable state. The controller 30 switches between the operable state and the inoperable state depending on the operation state of the gate lock lever GL. In the illustrated example, the controller 30 switches between the operable state and the inoperable state of the work machine 100 by electrically switching between a connected state and a disconnected state of the pilot line depending on the operation state of the gate lock lever GL.

[0077] When the gate lock lever GL is in the second operating position, the gate bar 55 is raised forward (passing prohibited state) so as to prevent the operator from passing through the boarding / exiting door, as shown in Fig. 5. On the other hand, when the gate lock lever GL is in the first operating position, the gate bar 55 is housed inside the left console 54L (passing permitted state) so as not to prevent the operator from passing through the boarding / exiting door.

[0078] With this configuration, the operator cannot operate the work machine 100 unless he or she sets the gate lock lever GL to the second operating position and puts the gate bar 55 in a passage-prohibiting state. Therefore, this configuration can prevent the work machine 100 from moving unintentionally, even if the operator inadvertently touches the operating device 26 when getting on or off the machine. Therefore, this configuration can improve the safety of the work machine 100.

[0079] Furthermore, the work machine 100 may be configured so that it can accept a predetermined operation to start the engine 11 only when the gate lock lever GL is in the second operating position and the gate bar 55 is in a pass-prohibiting state. In other words, the work machine 100 may be configured so that it cannot start the engine 11 when the gate lock lever GL is in the first operating position and the gate bar 55 is in a pass-permitting state.

[0080] A switch SW is installed on the right console 54R. A window console 56 is installed on the right side of the right console 54R. The window console 56 extends over the entire length of the driver's cab 10 in the fore-and-aft direction and is arranged parallel to the right console 54R. A display device D1 is installed in front of the window console 56. An external volume dial DL1, an internal volume dial DL2, an internal sound collector M2, a radio tuner, etc. are installed on the window console 56. The radio tuner, etc. may be installed on the left console 54L or the right console 54R.

[0081] The internal sound collector M2 is a device that is disposed inside the cab 10 and collects sounds generated within the cab 10. In the illustrated example, the internal sound collector M2 is an indoor microphone that is configured to be able to pick up the voice of an operator within the cab 10. The internal sound collector M2 outputs a signal generated from the sounds collected within the cab 10 to the controller 30.

[0082] The horn button HS is a button that is operated by the operator of the work machine 100 to sound the horn. In the illustrated example, the horn button HS is a knob switch provided at the tip of the left operation lever 26L.

[0083] The speech button KS is a button that the operator of the work machine 100 operates when he or she wants to speak to a worker around the work machine 100. In the illustrated example, the speech button KS is a knob switch provided at the tip of the right operating lever 26R.

[0084] The talk button KS is an operation unit connected to the controller 30, which is a control device. Based on the operation state of the talk button KS, the controller 30 switches between a state in which the operator of the work machine 100 speaks to a target person around the work machine 100 and a state in which the target person speaks to the operator of the work machine 100. The state in which the operator speaks to the target person is a state in which sound based on a signal generated from sound collected by the internal sound collection device M2 is output from the external sound output device SP1. The state in which the target person speaks to the operator is a state in which sound based on a signal generated from sound collected by the external sound collection device M1 is output from the internal sound output device SP2.

[0085] The internal sound output device SP2 is provided, for example, inside the cab 10 and outputs sound based on a signal generated from the sound collected by the external sound collection device M1. That is, the internal sound output device SP2 is a device that outputs sound to an operator in the cab 10 and is provided inside the cab 10. The internal sound output device SP2 converts an electrical signal input from the controller 30 into a physical sound (air vibration) and outputs the physical sound. The internal sound output device SP2 may be provided in any position, for example, near the display device D1, near the input device D2, or near the boarding / exiting door of the cab 10. In the illustrated example, the internal sound output device SP2 includes a left interior speaker SP2L attached to the upper left corner of the rear wall of the cab 10 and a right interior speaker SP2R attached to the upper right corner of the rear wall of the cab 10. The internal sound output device SP2 may be headphones or earphones worn by the operator. In this case, the headphones or earphones are connected to the controller 30 so as to be able to communicate with the controller 30 via, for example, Bluetooth (registered trademark) or the like.

[0086] The external volume dial DL1 is configured to adjust the volume of the sound output by the external sound output device SP1. The volume of the sound output by each external sound output device SP1 may be additionally adjusted using a device other than the external volume dial DL1, such as a touch panel attached to the display device D1. The external volume dial DL1 may be configured to be infinitely rotatable in both the clockwise and counterclockwise directions. This is to accommodate cases where volume adjustment using the external volume dial DL1 and volume adjustment using a device other than the external volume dial DL1 are used together.

[0087] The internal volume dial DL2 is configured to adjust the volume of the sound output by the internal sound output device SP2. The volume of the sound output by each internal sound output device SP2 may be additionally adjusted using a device other than the internal volume dial DL2, such as a touch panel attached to the display device D1. The internal volume dial DL2 may be configured to be infinitely rotatable in both the clockwise and counterclockwise directions. This is to accommodate cases where volume adjustment using the internal volume dial DL2 and volume adjustment using a device other than the internal volume dial DL2 are used together.

[0088] The switch SW is an example of an operating device for switching the operating state of the conversation function. In the illustrated example, the switch SW is provided on the top surface of the right console 54R. However, the switch SW may be one of the input devices D2, may be realized by a touch panel provided on the display device D1, or may be a knob switch.

[0089] The conversation function is a function for enabling a conversation between an operator OP of the work machine 100 and a worker WK located around the work machine 100, as shown in FIG. 6. FIG. 6 is a perspective view of the work machine 100 on which the operator OP is riding and the worker WK located at the front left of the work machine 100. FIG. 6 shows the operator OP's voice being collected by the internal sound collection device M2 and output from the external sound output device SP1, and the worker WK's voice being collected by the external sound collection device M1 and output from the internal sound output device SP2. In the work machine 100 shown in FIG. 6, an information transmission device G1 is provided at the top of each of the four side surfaces of the cab 10. The front light bar G1F provided at the top of the front of the cab 10 emits green light, and the left light bar G1L provided at the top of the left side of the cab 10 emits white light. In FIG. 6, the front light bar G1F emitting green light has a dot pattern attached. When the worker WK sees the front light bar G1F emitting green light, he or she can recognize that his or her voice is being detected by the front microphone M1F. Note that, for clarity, other devices such as the imaging device S6 are not shown in Figure 6.

[0090] The operating state of the conversation function includes an ON state (the state shown in FIG. 6) in which conversation between the operator OP and the worker WK is possible, and an OFF state in which conversation between the operator OP and the worker WK is not possible. However, the operating state of the conversation function may additionally include at least one of an audible state (with respect to the operator OP) in which the operator OP can hear the voice of the worker WK but the worker WK cannot hear the voice of the operator OP, and an utterance enabled state (with respect to the operator OP) in which the worker WK can hear the voice of the operator OP but the operator OP cannot hear the voice of the worker WK.

[0091] Specifically, when the switch SW is operated to switch the operation state of the conversation function to the ON state, the external sound collection device M1, the external sound output device SP1, the internal sound collection device M2, and the internal sound output device SP2 become available. Conversely, when the switch SW is operated to switch the operation state of the conversation function to the OFF state, the external sound collection device M1, the external sound output device SP1, the internal sound collection device M2, and the internal sound output device SP2 become unavailable. Furthermore, when the switch SW is operated to switch the operation state of the conversation function to the audible state, the external sound collection device M1 and the internal sound output device SP2 become available. Furthermore, when the switch SW is operated to switch the operation state of the conversation function to the speech-enabled state, the external sound output device SP1 and the internal sound collection device M2 become available. In the illustrated example, the operator OP can use the external sound output device SP1 to talk to the worker WK by pressing the speech button KS and speaking when the internal sound collection device M2 is available.

[0092] Next, the operation of the controller 30 of the work machine 100 according to this embodiment will be described with reference to Figures 4 to 9. Figure 7 is a flow diagram illustrating an example of the operation of the controller 30 of the work machine 100 shown in Figure 4. Figure 8 is a plan view showing the positional relationship between the work machine 100 and people HU1, HU2 such as workers WK around it. Figure 9 is an image diagram showing an example of an image 85 displayed on the display device D1 of the work machine 100 of Figure 4.

[0093] In the work machine 100 of this embodiment, the controller 30 shown in Fig. 4 also functions as a control device that notifies, for example, of a deficiency in human detection. Specifically, the controller 30 notifies of a deficiency in human detection when, for example, the human detection direction D1c, which is the direction of the person HU1 detected by the image capture device S6 that constitutes the object detection device shown in Fig. 8, differs from the sound directions D1m, D2m, which are the directions from which the sound collected by the external sound collection device M1 is emitted. Note that a case where the human detection direction D1c and the sound directions D1m, D2m differ includes a case where the object detection device does not detect the person HU2 in the sound direction D2m, and there is no human detection direction that corresponds to the sound direction D2m.

[0094] The object detection device includes, for example, a front camera S6F, a left camera S6L, a right camera S6R, and a rear camera S6B that constitute the imaging device S6. The external sound collection device M1 also includes, for example, a plurality of microphones, such as a front microphone M1F, a left microphone M1L, a right microphone M1R, and a rear microphone M1B. The controller 30 can, for example, notify the user of any deficiencies in human detection by the object detection device via the display device D1.

[0095] When the object detection device is unable to detect a person around the work machine 100, and as a result, no human detection result is reported even though a person is present around the work machine 100, this is sometimes referred to as a "failed report." In contrast, when the object detection device detects an object other than a person as a person, and reports an erroneous human detection result based on the detection result of that object, this is sometimes referred to as a "false report." The work machine 100 of this embodiment, for example, suppresses the occurrence of "failed reports" such as those described above.

[0096] An example of a normal image 85 displayed on the image display unit 42 of the display device D1 will be described below, followed by an example of a notification regarding a human detection defect displayed on the image display unit 42. As shown in Fig. 9, the image 85 that an output control unit 305 (see Fig. 4) of the controller 30, which will be described later, causes to be displayed on the image display unit 42 constituting the display device D1 includes, for example, a plurality of images. The image display unit 42 includes, for example, a date and time display area 42a, a driving mode display area 42b, an attachment display area 42c, a fuel efficiency display area 42d, and an engine control status display area 42e.

[0097] The date and time display area 42a is an area that displays an image showing the current date and time. The driving mode display area 42b is an area that displays an image showing the current driving mode. The attachment display area 42c is an area that displays an image showing the currently attached attachment. The fuel efficiency display area 42d is an area that displays an image showing fuel efficiency information calculated by the controller 30. The fuel efficiency display area 42d includes, for example, an average fuel efficiency display area 42d1 that displays an image showing lifetime average fuel efficiency or section average fuel efficiency, and an instantaneous fuel efficiency display area 42d2 that displays an image showing instantaneous fuel efficiency. The engine control status display area 42e is an area that displays an image showing the control status of the engine 11.

[0098] The image display unit 42 also includes, for example, a coolant temperature display area 42g, a remaining fuel amount display area 42h, a rotation speed level display area 42i, a urea water remaining amount display area 42j, and a hydraulic oil temperature display area 42k.

[0099] The coolant temperature display area 42g is an area that displays an image that indicates the current temperature state of the engine coolant. The remaining fuel amount display area 42h is an area that displays an image that indicates the remaining amount of fuel stored in the fuel tank. The rotation speed level display area 42i is an area that displays, as an image, the current level set by the dial. The rotation speed level display area 42i displays a number that indicates the selected level. The urea water remaining amount display area 42j is an area that displays, as an image, the remaining amount of urea water stored in the urea water tank. The hydraulic oil temperature display area 42k is an area that displays an image that indicates the temperature state of the hydraulic oil in the hydraulic oil tank.

[0100] Furthermore, the image display unit 42 includes, for example, a status display area 421, a first image display area 422, and a second image display area 423. The status display area 421 is an area that displays an image that shows the positional relationship between the work machine 100 and people HU1 and HU2 detected in the vicinity of the work machine 100. The first image display area 422 and the second image display area 423 are areas that display image information captured by the imaging device S6.

[0101] The image displayed in the status display area 421 includes, for example, a direction display icon 421a, a work machine icon 421b, a first circular area 421c, a second circular area 421d, and human detection icons 421e, 421f, and 421g. Note that the status display area 421 may display, for example, an overhead image generated by processing an image from the imaging device S6. The overhead image is an image of the work machine 100 and its surrounding objects viewed from above. The work machine icon may be displayed in the center of the overhead image, or the circular area and human detection icon may be superimposed on the overhead image.

[0102] The direction display icon 421a indicates the direction in which the work machine 100 can travel. The work machine icon 421b is an icon that represents the work machine 100. The first circular area 421c is an image that shows the range in which the attachment AT will rotate when the upper rotating body 3 is rotated while maintaining the current attitude of the attachment AT. The second circular area 421d is an image that shows the range in which the attachment AT will rotate when the upper rotating body 3 is rotated with the attachment AT fully extended in the horizontal direction. The person detection icons 421e, 421f, and 421g are images that display people that have been detected around the work machine 100.

[0103] In the first image display area 422, for example, a right image is displayed. The right image is an image that shows the space to the right of the work machine 100, and includes an image 422c of the right end of the top surface of the upper rotating body 3. The right image is a real viewpoint image generated by the controller 30, and is generated based on an image acquired by the camera S6R. The first image display area 422 is displayed to the right of the status display area 421.

[0104] 9, the right image displayed in the first image display area 422 includes an image 422a of a person HU1 who is the target of a two-way conversation with the operator OP, a person detection frame 422b, and an image 422c of the right end of the upper surface of the upper rotating body 3. The person detection frame 422b is an image generated by the output control unit 305 based on the person detection result by the object detection device or the detection unit 302, for example, and is a rectangular frame displayed to surround the image 422a of the detected person.

[0105] In the second image display area 423, for example, a rear image is displayed. The rear image is an image that shows the space behind the work machine 100, and includes an image 423c of the counterweight. The rear image is a real viewpoint image generated by the controller 30, and is generated based on an image acquired by the camera S6B. The second image display area 423 is displayed below the status display area 421 as a reference.

[0106] 9, the rear image displayed in the second image display area 423 includes an image 423c of the counterweight mounted on the rear of the upper rotating body 3, but does not include a human detection frame because the object detection device was unable to detect humans due to conditions such as sunlight and color. However, the rear image includes an emphasis image 423b that emphasizes the sound direction D2m, which is the direction from which the sound collected by the external sound collection device M1 is emitted.

[0107] Note that, if the external sound collection device M1 can identify not only the direction from which the sound is emitted but also the position from which the sound is emitted, the rear image displayed in the second image display area 423 may include an emphasis image 423b that emphasizes the position from which the sound is emitted. In the example shown in Fig. 9, the emphasis image 423b is, for example, a rectangular frame that emphasizes the sound direction D2m from which the sound is emitted or the position from which the sound is emitted within the rear image displayed in the second image display area 423.

[0108] The highlighted image 423b is displayed in a different display mode from the human detection frame 422b that is displayed when the object detection device detects the person HU1. Specifically, the human detection frame 422b is a solid-line frame, and the highlighted image 423b is a dashed-line frame.

[0109] Note that differentiating the display manner of the emphasized image 423b and the human detection frame 422b is not limited to changing the line type. For example, the color or thickness of the emphasized image 423b and the human detection frame 422b may be changed, the emphasized image 423b may be made to blink, or a sound or light may be used to notify the user. Furthermore, the emphasized image 423b may be an image that emphasizes the first image display area 422 or the second image display area 423, such as a frame that surrounds the outer edge of the first image display area 422 or the second image display area 423, or an icon displayed in the first image display area 422 or the second image display area 423.

[0110] The image display unit 42 also includes, for example, a notification icon 42m that notifies of a defect in human detection. In the example shown in Fig. 9, the notification icon 42m is, for example, an exclamation mark icon "!" and is displayed in the upper left of the image display unit 42. As shown in Fig. 5, the display device D1 is disposed, for example, on the right side of the driver's seat 50. The upper left of the image display unit 42 of the display device D1 where the notification icon 42m is displayed is close to the center of the field of view of the operator OP who is seated in the driver's seat 50 and looking forward of the upper rotating body 3, and is a position that is more easily visible to the operator OP.

[0111] The image display unit 42 may also include, for example, an information display area 42n that displays information related to defects in human detection. The information display area 42n is a pop-up window that is displayed when the operator OP selects the notification icon 42m via an input device D2 such as a touch panel or a mouse, and is displayed in a position that does not overlap the status display area 421, the first image display area 422, and the second image display area 423. The information display area 42n includes, for example, a function ON / OFF icon 42n1, a message display area 42n2, and a close icon 42n3.

[0112] The function ON / OFF icon 42n1 is an icon that the operator OP selects via the input device D2 to switch on and off the auxiliary function of human detection by the external sound collection device M1. The message display area 42n2 is an area that displays text information regarding defects in human detection. The close icon 42n3 is an icon that the operator OP selects via the input device D2 to close the information display area 42n.

[0113] Next, a description will be given of the configuration of the controller 30 provided in the work machine 100 of this embodiment. The controller 30 also functions as a control device that notifies of an inadequate person detection when the person detection direction D1c, which is the direction of people HU1, HU2 detected by the object detection device, differs from the sound directions D1m, D2m, which are the directions from which sounds collected by the external sound collection device M1 are emitted.

[0114] 4, the controller 30 includes, for example, an acquisition unit 301, a detection unit 302, an estimation unit 303, a determination unit 304, and an output control unit 305. These units of the controller 30 represent, for example, various functions of the controller 30. Specifically, each function of the controller 30 is realized, for example, by the CPU of the controller 30 reading out a program stored in a non-volatile storage device, loading the program into a volatile storage device, and executing the program.

[0115] The controller 30 starts the processing flow shown in Fig. 7, for example, when the operator OP starts the work machine 100. The controller 30 first executes processing P1 to detect people HU1, HU2 around the work machine 100. In processing P1, the acquisition unit 301 of the controller 30 acquires, for example, the results of object detection by an object detection device. Furthermore, the detection unit 302 of the controller 30 detects the people HU1, HU2 from the object detection results acquired by the acquisition unit 301.

[0116] The object detection result by the object detection device includes, for example, images from the front camera S6F, left camera S6L, right camera S6R, and rear camera S6B that configure the imaging device S6. The method by which the detection unit 302 detects people is not particularly limited, and any known method can be used. The detection unit 302 may use, for example, a trained model as a method for detecting people.

[0117] The trained model is mainly composed of, for example, a neural network. The neural network of the trained model may be a so-called deep neural network that has one or more intermediate layers (hidden layers) between an input layer and an output layer. In the neural network, weighting parameters that represent the connection strength with lower layers are defined for each of the multiple neurons that make up each intermediate layer.

[0118] The neural network is configured such that neurons in each layer output the sum of values ​​obtained by multiplying each input value from multiple neurons in the upper layer by a weighting parameter defined for each neuron in the upper layer through a threshold function to neurons in the lower layer. Then, machine learning, specifically deep learning, is performed on the trained model, and as a result, the weighting parameters of the neural network are optimized.

[0119] The training data used in machine learning includes, for example, image information and information indicating whether or not a person is captured in the image information. By performing machine learning using the training data, when image information is input, the trained model outputs information indicating whether or not a person is captured in the image information and information indicating the area in which the person is captured (for example, position coordinates and size in the image information).

[0120] Furthermore, instead of using a trained model for the person detection process, a person may be detected from image information by, for example, comparing feature amounts extracted from image information with predetermined feature amounts indicating a person and determining whether the feature amounts are similar based on a predetermined threshold. For example, when a person is detected from image information, the detection unit 302 outputs information indicating the area in the image information in which the person is captured (e.g., position coordinates). Note that the object detection device used for person detection in process P1 may be only the image capture device S6, may include the image capture device S6 and other devices (e.g., LiDAR, millimeter-wave radar), or may be only devices other than the image capture device S6.

[0121] Furthermore, in this process P1, the output control unit 305 causes the display device D1 to display a person detection frame 422b based on the position coordinates and the like of the person HU1 detected by the detection unit 302. In the example shown in Figures 8 and 9, an image of the person HU1 is captured on the right side of the upper rotating body 3 by the right camera S6R that constitutes the object detection device, and the person HU1 is detected by the detection unit 302. As a result, the output control unit 305 causes the first image display region 422 of the image display unit 42 of the display device D1 to display a person detection frame 422b that surrounds the image 422a of the person HU1.

[0122] 7 and 8, the controller 30 then executes process P2 to detect the voices of people HU1 and HU2 present around the work machine 100. In this process P102, the acquisition unit 301 of the controller 30 acquires, for example, signals generated from sounds collected by each of the front microphone M1F, left microphone M1L, right microphone M1R, and rear microphone M1B that configure the external sound collection device M1.

[0123] Furthermore, the detection unit 302 of the controller 30 recognizes the speech of people HU1 and HU2, for example, by calculating features from the signals generated by each microphone and comparing them with phonemes. A phoneme refers to a phonological unit that is the smallest unit of speech. The method by which the detection unit 302 recognizes speech is not particularly limited, and any known method can be used. The detection unit 302 may use, for example, the trained model described above as a speech recognition method. The training data used for machine learning of the trained model includes, for example, information obtained by statistically processing features calculated from the speech of many people. By performing machine learning using the training data, the trained model, for example, when a speech feature is input, outputs a sentence corresponding to the speech.

[0124] Furthermore, in this process P2, the controller 30 estimates sound directions D1m and D2m based on signals generated from the voices of people HU1 and HU2 collected by the multiple microphones that make up the external sound collection device M1. In the example shown in Fig. 8, the multiple microphones that make up the external sound collection device M1 are installed so that one microphone corresponds to one direction. Specifically, a front microphone M1F that collects sounds arriving from the front of the upper rotating body 3, a left microphone M1L that collects sounds arriving from the left of the upper rotating body 3, a right microphone M1R that collects sounds arriving from the right of the upper rotating body 3, and a rear microphone M1B that collects sounds arriving from the rear of the upper rotating body 3 are installed.

[0125] In this case, it is assumed that the detection unit 302 recognizes a voice from the signal generated by the right microphone M1R and the signal generated by the rear microphone M1B. Then, the estimation unit 303 of the controller 30 estimates, for example, the right direction of the upper rotating body 3 corresponding to the installation direction of the right microphone M1R and the rear direction of the upper rotating body 3 corresponding to the installation direction of the rear microphone M1B as the voice direction D1m of the first person HU1 and the voice direction D2m of the second person HU2, respectively. Note that the number of microphones constituting the external sound collection device M1 and the directions corresponding to each microphone are merely examples and are not particularly limited.

[0126] Next, the controller 30 executes a process P3 for determining whether or not there is a defect in human detection, as shown in Fig. 7. In this process P3, the determination unit 304 of the controller 30 determines that there is a defect in human detection when the human detection direction D1c, which is the direction of the person HU1 detected by the object detection device, differs from the sound directions D1m, D2m, which are the directions from which the sound collected by the external sound collection device M1 is emitted.

[0127] 8, the person detection direction D1c, which is the direction of the person HU1 detected by the right camera S6R that constitutes the object detection device, is to the right of the upper revolving body 3. In addition, the sound direction D1m, which is the direction from which the sound collected by the right microphone M1R that constitutes the external sound collection device M1 is emitted, is also to the right of the upper revolving body 3. Therefore, the determination unit 304 determines that the person detection direction D1c and the sound direction D1m for the person HU1 located to the right of the work machine 100 match.

[0128] 8, the rear camera S6B constituting the object detection device is unable to detect the person HU2 located behind the work machine 100 due to, for example, the influence of the direction or brightness of the light, or the similarity between the color of the person HU2 and the color of the surroundings. Meanwhile, the estimation unit 303 estimates the rear direction of the work machine 100 as the sound direction D2m, which is the direction from which the sound collected by the rear microphone M1B constituting the external sound collection device M1 is emitted. In this case, the determination unit 304 determines that the person detection direction for the person HU2 located behind the work machine 100 and the sound direction D2m are different.

[0129] In process P3, if the human detection direction and the voice direction D2m differ in at least one direction, the determination unit 304 determines that there is a human detection defect (YES). On the other hand, if the human detection direction D1c and the voice direction D1m match in all directions, the determination unit 304 determines that there is no human detection defect (NO).

[0130] In this process P3, if the determination unit 304 determines that there is no defect in human detection (NO), the controller 30 ends the process flow shown in Fig. 7. In this way, when the human detection direction D1c and the voice direction D1m match, the controller 30 may notify that a human has been detected. Specifically, the output control unit 305 of the controller 30 does not display the notification icon 42m on the image display unit 42 of the display device D1, but displays a human detection frame 422b notifying that a human HU1 has been detected. Furthermore, the output control unit 305 may, for example, display a message indicating that there is no defect in human detection in the information display area 42n.

[0131] On the other hand, if the determination unit 304 determines in process P3 that there is a human detection deficiency (YES), the controller 30 executes process P4, which causes the object detection device to redetect the human HU2 in the sound direction D2m. In process P4 for redetecting the human HU2, the detection unit 302 of the controller 30 changes, for example, the human detection conditions of the object detection device. Specifically, the detection unit 302 changes, for example, the conditions for extracting feature amounts from images of the image capture device S6 that constitutes the object detection device, or the algorithm for extracting feature amounts from images, in order to detect the human HU2 in the rear direction of the upper rotating body 3, whose human detection direction and sound direction D2m do not match.

[0132] By changing the human detection algorithm in process P4 as described above, it is possible to suppress false alarms in human detection in process P1 described above. Specifically, if the same human detection algorithm as process P4 is used in process P1 described above, objects other than humans are detected as humans, and false alarms in which erroneous human detection results are notified based on those detection results will increase, causing stress for the operator OP. Therefore, by partially using the algorithm changed as described above only in process P4, which re-detects humans, it is possible to suppress false alarms when human detection is performed normally in process P1 described above or in situations where no people are present around the work machine 100.

[0133] Thereafter, the controller 30 executes process P5 to determine whether there is a human detection deficiency, similar to process P3 described above. In the previous process P4, when the object detection device detects the human HU2, a human detection frame surrounding the image 423a of the human HU2 is displayed in the rear image in the second image display area 423 of the display device D1. In this case, the determination unit 304 determines in process P5 that there is no human detection deficiency (NO). On the other hand, suppose that the object detection device did not detect the human HU2 in the previous process P4, and the human detection direction and the sound direction D2m differ in at least one direction. In this case, in process P5, the determination unit 304 again determines that there is a human detection deficiency (YES).

[0134] In this process P5, if the determination unit 304 determines that there is no defect in human detection (NO), the controller 30 ends the process flow shown in Fig. 7. On the other hand, in this process P5, if the determination unit 304 determines that there is a defect in human detection (YES), the controller 30 executes process P6 to emphasize the voice direction D2m that does not match the human detection direction.

[0135] In this process P6, the controller 30 displays an emphasis image 423b that emphasizes the sound direction D2m on the display device D1, for example, as shown in Fig. 9. Note that, although details will be described later, if the position from which the sound is emitted can be identified by the external sound collection device M1, the controller 30 may display an emphasis image 423b that emphasizes the position from which the sound is emitted.

[0136] The output control unit 305 of the controller 30, for example, causes the display device D1 to display the highlighted image 423b in a display mode different from the human detection frame 422b serving as a human detection image notifying that a human has been detected. More specifically, the output control unit 305, for example, as described above, changes the line type, color, or thickness of the highlighted image 423b and the human detection frame 422b, flashes the highlighted image 423b, or notifies by sound output from the internal sound output device SP2 or by the light of a warning lamp. Furthermore, the output control unit 305 may, for example, cause the display device D1 to display a highlighted image such as a frame surrounding the outer edge of the second image display area 423 or an icon displayed in the second image display area 423, as described above.

[0137] Furthermore, as described above, the output control unit 305 may cause the image display unit 42 of the display device D1 to display a notification icon 42m notifying of a defect in human detection, or to display an information display area 42n. The information display area 42n, which is displayed when the operator OP selects the notification icon 42m, includes a message display area 42n2 that displays text information regarding the defect in human detection, as described above. The information display area 42n is an example of an emphasized image that includes information regarding the defect in human detection. After the end of process P6, the controller 30 ends the process flow shown in FIG. 7.

[0138] An operator OP in the cab 10 of the work machine 100 can speak to people HU1, HU2 around the work machine 100, for example, via an internal sound collector M2 inside the cab 10 and an external sound output device SP1 outside the cab 10. In addition, the people HU1, HU2 around the work machine 100 can speak to the operator OP in the cab 10 of the work machine 100, for example, via an external sound collector M1 outside the cab 10 and an internal sound output device SP2 inside the cab 10.

[0139] Specifically, when an operator OP in the cab 10 of the work machine 100 wants to speak to people HU1, HU2 around the work machine 100, he or she operates the speech button KS to speak. The internal sound collection device M2 then collects the voice of the operator OP, and a signal generated from the collected voice is input to the controller 30. The controller 30, for example, causes an external sound output device SP1 installed outside the cab 10 to output a sound based on a signal generated from the sound collected by the internal sound collection device M2.

[0140] Furthermore, when the operator OP releases the talk button KS, the voices of people HU1, HU2 around the work machine 100 are collected by the external sound collection device M1, and a signal corresponding to the voices is input from the external sound collection device M1 to the controller 30. The controller 30, for example, causes an internal sound output device SP2 installed inside the cab 10 to output a sound based on a signal generated from the sound collected by the external sound collection device M1. This enables two-way conversation between the operator OP of the work machine 100 and people HU1, HU2, such as workers WK, present around the work machine 100.

[0141] The operation of the work machine 100 of this embodiment will now be described.

[0142] As described above, the work machine 100 of this embodiment includes a lower traveling body 1, an upper rotating body 3 rotatably provided on the lower traveling body 1, and a cab 10 provided on the upper rotating body 3. The work machine 100 also includes an external sound collection device M1 arranged outside the cab 10, an object detection device that detects people HU1, HU2 around the upper rotating body 3, and a controller 30. The controller 30 is a control device that notifies of a deficiency in person detection when a person detection direction D1c, which is the direction of people HU1, HU2 detected by the object detection device, differs from sound directions D1m, D2m, which are the directions from which sounds collected by the external sound collection device M1 are emitted.

[0143] With this configuration, the work machine 100 of this embodiment can use the external sound collection device M1 to collect sounds emitted by people HU2 around the work machine 100 that could not be detected by the object detection device, and can notify of deficiencies in person detection based on the voice direction D2m of the person HU2. As a result, deficiencies in person detection by the object detection device can be compensated for by the sound detection results of the external sound collection device M1, and the operator OP of the work machine 100 can be more reliably notified of the presence of people HU2 around the work machine 100. Therefore, the work machine 100 of this embodiment can improve safety when the work machine 100 is traveling or working compared to conventional methods.

[0144] Furthermore, in the work machine 100 of this embodiment, the controller 30, which is a control device, notifies that a person has been detected when the person detection direction D1c and the voice direction D1m match. With this configuration, the operator OP can recognize, based on the notification from the controller 30, that there has been no inadequate person detection (missed report) by the object detection device in the direction where the person detection direction D1c and the voice direction D1m match.

[0145] The work machine 100 of this embodiment is further equipped with a display device D1 arranged inside the cab 10. The controller 30, which is a control device, causes the display device D1 to display an emphasis image 423b that emphasizes the voice direction D2m or the position from which the voice is emitted by the person HU2.

[0146] With this configuration, the operator OP of the work machine 100 can visually recognize the highlighted image 423b displayed on the display device D1. This allows the operator OP to recognize the sound direction D2m, which is the direction in which the sound emitted by the person HU2 that could not be detected by the object detection device reached the external sound collection device M1, or the rough position of the person HU2, from the highlighted image 423b. Therefore, the operator OP can direct his or her attention to the sound direction D2m or position of the person HU2, thereby improving safety when the work machine 100 is traveling or working.

[0147] Furthermore, in the work machine 100 of this embodiment, the controller 30, which is a control device, causes the display device D1 to display the highlighted image 423b in a display mode different from the human detection frame 422b, which is a human detection image that notifies that a person has been detected. With this configuration, the work machine 100 of this embodiment can prevent the operator OP, who is looking at the display device D1, from confusing the human detection frame 422b with the highlighted image 423b.

[0148] Furthermore, in the work machine 100 of this embodiment, the controller 30, which is a control device, causes the display device D1 to display an emphasized image including information relating to deficiencies in human detection by the object detection device, such as in the information display area 42n. With this configuration, the work machine 100 of this embodiment allows the operator OP, who views the emphasized image in the information display area 42n or the like displayed on the display device D1, to grasp more detailed information relating to deficiencies in human detection.

[0149] Furthermore, in the work machine 100 of this embodiment, the controller 30, which is a control device, causes the object detection device to redetect the person HU2 in the voice direction D2m when the person detection direction and the voice direction D2m do not match.

[0150] With this configuration, according to the work machine 100 of this embodiment, even if the object detection device is unable to detect the person HU2 the first time it detects a person, it is possible for the object detection device to detect the person HU2 in the sound direction D2m. This allows the external sound collection device M1 to assist the object detection device in detecting the people HU1, HU2, thereby improving the detection accuracy of the object detection device for the people HU1, HU2.

[0151] Furthermore, in the work machine 100 of this embodiment, the external sound collection device M1 includes multiple microphones. Furthermore, the controller 30, which is a control device, estimates the sound directions D1m and D2m based on signals generated from sounds collected by the multiple microphones.

[0152] With this configuration, the controller 30 can store in advance in a nonvolatile storage device the directions in which each of the multiple microphones constituting the external sound collection device M1, such as the front microphone M1F, the left microphone M1L, the right microphone M1R, and the rear microphone M1B, is installed. Then, the controller 30 can estimate the sound directions D1m and D2m based on the signals generated by each microphone and the directions in which each microphone is installed.

[0153] The work machine 100 of this embodiment further includes an internal sound collection device M2 and an internal sound output device SP2 arranged inside the cab 10, and an external sound output device SP1 arranged outside the cab 10. The controller 30, which is a control device, causes the internal sound output device SP2 to output a sound based on a signal generated from the sound collected by the external sound collection device M1, and causes the external sound output device SP1 to output a sound based on a signal generated from the sound collected by the internal sound collection device M2.

[0154] With this configuration, the voices uttered by people HU1, HU2 around the work machine 100 can be collected by the external sound collection device M1, and the voice based on a signal generated from that voice by the external sound collection device M1 can be output from the internal sound output device SP2 located inside the cab 10. Furthermore, the voice uttered by the operator OP can be collected by the internal sound collection device M2, and the voice based on a signal generated from that voice by the internal sound collection device M2 can be output from the external sound output device SP1 located outside the cab 10. This makes it possible to have a two-way conversation between the operator OP of the work machine 100 in the cab 10 and the people HU1, HU2 outside the cab 10, using only equipment installed in the work machine 100.

[0155] The work machine 100 of this embodiment also includes a talk button KS as an operation unit connected to the controller 30, which is a control device. The controller 30 switches between the following two states based on the operation state of the talk button KS, which is an operation unit. One state is a state in which sound based on a signal generated from sound collected by the internal sound collection device M2 is output from the external sound output device SP1. The other state is a state in which sound based on a signal generated from sound collected by the external sound collection device M1 is output from the internal sound output device SP2.

[0156] With this configuration, the operator OP of the work machine 100 can switch the speech button KS to deliver the voice he or she speaks inside the cab 10 to the people HU1, HU2 around the work machine 100 via the internal sound collection device M2 and the external sound output device SP1. The operator OP of the work machine 100 can also switch the speech button KS to hear the voices spoken by the people HU1, HU2 around the work machine 100 via the external sound collection device M1 and the internal sound output device SP2. This allows smooth two-way conversation between the operator OP inside the cab 10 and the people HU1, HU2 outside the cab 10 without using a communication device or the like.

[0157] Furthermore, in the work machine 100 of this embodiment, the controller 30, which is a control device, changes the human detection conditions of the object detection device when the object detection device redetects the human HU2. With this configuration, the work machine 100 of this embodiment can more reliably detect a human HU2 that was not detected during the first human detection by the object detection device when the object detection device redetects the human HU2.

[0158] As described above, according to this embodiment, it is possible to provide a work machine 100 that is capable of notifying of deficiencies in human detection by an object detection device. Note that the work machine according to the present disclosure is not limited to the configuration of the work machine 100 in the embodiment described above.

[0159] For example, in the above embodiment, the multiple microphones constituting the external sound collection device M1 of the work machine 100 are described as an omnidirectional front microphone M1F, left microphone M1L, right microphone M1R, and rear microphone M1B. However, the external sound collection device M1 may include one or more array microphones. An array microphone, also known as a microphone array, has multiple microphones that are arranged three-dimensionally and integrally configured. An array microphone can detect the direction of a sound source, for example, based on the phase shift or volume difference of the sounds collected by each microphone.

[0160] Figure 10 is a plan view showing a modified example of the work machine 100 shown in Figure 1. The work machine 100 according to this modified example differs from the work machine 100 according to the previously described embodiment in that the front microphone M1FA, left microphone M1LA, right microphone M1RA, and rear microphone M1BA that make up the external sound collection device M1 are each array microphones. The rest of the configuration of the work machine 100 according to this modified example is the same as that of the work machine 100 according to the previously described embodiment, and therefore similar parts are given the same reference numerals and description thereof will be omitted.

[0161] The controller 30 of the work machine 100 according to the modified example shown in Fig. 10 executes the processing flow shown in Fig. 7, similar to the controller 30 of the work machine 100 according to the embodiment described above. As shown in Fig. 10, in processing P1, it is assumed that the object detection device does not detect person HU1 to the right rear of the upper rotating body 3, but only detects person HU2 to the left rear of the upper rotating body 3. In this case, the detection unit 302 of the controller 30 detects only person detection direction D2c corresponding to person HU2, and does not detect person detection direction corresponding to person HU1.

[0162] In process P2, the estimation unit 303 of the controller 30 estimates the positions of the people HU1 and HU2 who made the voices, based on signals generated from the voices of the people HU1 and HU2 collected by the rear microphone M1BA, which is an array microphone. Furthermore, the estimation unit 303 estimates voice directions D1m and D2m, which are the directions from which the voices reached the rear microphone M1BA, based on the estimated positions of the people HU1 and HU2.

[0163] Thereafter, in process P3, the determination unit 304 of the controller 30 determines that there is a deficiency in human detection (YES) because the human detection direction of the person HU1 to the rear right of the upper revolving body 3 does not match the voice direction D1m. Thereafter, the controller 30 executes processes P4 to P6, similar to the controller 30 in the work machine 100 of the previously described embodiment. Therefore, in this modified example as well, it is possible to provide a work machine 100 that is capable of notifying of deficiencies in human detection by the object detection device, similar to the previously described embodiment.

[0164] Furthermore, in the work machine 100 of this modified example, the external sound collection device M1 includes a rear microphone M1BA, which is an array microphone, and therefore the controller 30 can estimate not only the sound directions D1m, D2m, but also the position from which the voices of people HU1, HU2 are emitted. As a result, the controller 30 can cause the display device D1 to display an emphasis image 423b that emphasizes the position from which the voices of people HU1, HU2 are emitted, further improving safety while the work machine 100 is traveling or working. Note that in the work machine 100 of this modified example, the external sound collection device M1 does not necessarily need to include multiple array microphones, and it is sufficient that it includes at least one array microphone.

[0165] Next, another example configuration of the work machine 100 will be described with reference to Fig. 11. Fig. 11 is a top view of another example configuration of the work machine 100. The work machine 100 shown in Fig. 11 differs from the work machine 100 shown in Fig. 1 in that the external sound output device SP1 is made up of four speakers (front speaker SP1F, left speaker SP1L, right speaker SP1R, and rear speaker SP1B). In the work machine 100 shown in Fig. 1, the external sound output device SP1 is made up of a single omnidirectional speaker provided above the cab 10.

[0166] With this configuration, the work machine 100 shown in FIG. 11 can output sound to the worker WK in front of the work machine 100 without outputting sound to the workers WK on the left, right, and rear of the work machine 100, for example, by turning on the front speaker SP1F (a state in which sound can be output) and turning off the left speaker SP1L, the right speaker SP1R, and the rear speaker SP1B (a state in which sound cannot be output).

[0167] 11, a front camera S6F and a front microphone M1F are provided adjacent to the front speaker SP1F, and a front light bar G1F is provided on the housing of the front microphone M1F. A left camera S6L and a left microphone M1L are provided adjacent to the left speaker SP1L, and a left light bar G1L is provided on the housing of the left microphone M1L. A right camera S6R and a right microphone M1R are provided adjacent to the right speaker SP1R, and a right light bar G1R is provided on the housing of the right microphone M1R. A rear camera S6B and a rear microphone M1B are provided adjacent to the rear speaker SP1B, and a rear light bar G1B is provided on the housing of the rear microphone M1B.

[0168] With this configuration, the work machine 100 can turn on (light-emitting state) the light bar corresponding to the speaker that is turned on, and turn off (light-emitting state) the light bar corresponding to the speaker that is turned off.

[0169] The external sound output device SP1 may be configured with one or more parametric speakers. A parametric speaker is a speaker that uses ultrasonic waves and can selectively transmit sound to people within a specific narrow range. A parametric speaker can transmit sound toward any position.

[0170] In the work machine 100 shown in Fig. 11, the controller 30 may detect a worker WK around the work machine 100 based on images captured by the imaging device S6, and identify the position of the worker WK. Furthermore, if there are multiple workers WK around the work machine 100, the controller 30 may distinguish between a conversation target person (a worker WK who is speaking) and a non-conversation target person (a worker WK who is not speaking) based on the output of the four external sound collection devices M1. The controller 30 may also distinguish between a conversation target person (a worker WK facing the work machine 100) and a non-conversation target person (a worker WK who is not facing the work machine 100) based on images captured by the imaging device S6. The controller 30 may then turn on the speaker and light bar facing the worker WK. For example, if there is a worker WK (speaking) behind the work machine 100, the controller 30 may turn on the rear speaker SP1B while keeping the front speaker SP1F, left speaker SP1L, and right speaker SP1R off. In this case, the controller 30 may turn on the rear light bar G1B while keeping the front light bar G1F, left light bar G1L, and right light bar G1R off. Note that such a function may be implemented in the work machine 100 shown in Figures 1 to 5.

[0171] With this configuration, the controller 30 can output sound in the direction where the worker WK is located without outputting sound in the direction where the worker WK is not present, so the worker WK can easily recognize whether he or she is considered a conversation target or non-target.

[0172] Next, an example configuration of the operation system SYS according to an embodiment of the present disclosure will be described with reference to Fig. 12. Fig. 12 is a schematic diagram showing an example configuration of the operation system SYS. As shown in Fig. 12, the operation system SYS includes a work machine 100, a remote control room RC, and a management center MC. Note that the detailed configuration of the work machine 100 is omitted from Fig. 12. This is because the work machine 100 shown in Fig. 12 has the same configuration as the work machine 100 shown in Fig. 1 or Fig. 10.

[0173] The work machine 100, remote control room RC, and management center MC are connected to one another so that they can send and receive data via a communication network NW. Note that the work machine 100, remote control room RC, and management center MC may also be connected to one another so that they can send and receive data directly to one another without going through the communication network NW. In the illustrated example, the work machine 100 transmits information about the work site to the remote control room RC. This allows the remote operator RO in the remote control room RC to understand the situation at the work site based on the information from the work machine 100.

[0174] The work machine 100 is equipped with a sensor that can three-dimensionally recognize the position and shape of objects present at the work site. For example, the work machine 100 is equipped with a spatial recognition device. Therefore, the work machine 100 can transmit the results of three-dimensionally measuring the work site to the remote control room RC.

[0175] The spatial recognition device is a device for recognizing the space around the work machine 100. In the illustrated example, the spatial recognition device is a LiDAR. The LiDAR measures the distance between the LiDAR and each of one million or more points within a monitoring range. The spatial recognition device may be any device that can measure the distance to an object. For example, the spatial recognition device may be a stereo camera, or may be a combination of the image capture device S6 and a distance measuring device such as millimeter wave radar.

[0176] The operation system SYS may include one or more work machines 100. When multiple work machines 100 are included, the remote operator RO of a specific work machine 100 can obtain information about the work site obtained by that specific work machine 100, as well as information about the work site obtained by one or more other work machines 100.

[0177] The remote control room RC is equipped with a communication device T2, a remote controller 30E, an operation device 26E, an operation sensor 29E, a display device D1E, an internal sound collection device M2E, and an internal sound output device SP2E. The remote control room RC also is equipped with an operation seat DS where a remote operator RO who remotely operates the work machine 100 sits.

[0178] The communication device T2 is configured to be able to communicate with the communication device T1 attached to the work machine 100.

[0179] The remote controller 30E is a computing device that executes various calculations. In this embodiment, the remote controller 30E is configured as a microcomputer including a CPU and memory. The various functions of the remote controller 30E are realized by the CPU executing programs stored in the memory.

[0180] The display device D1E is a device capable of displaying various types of information. The display device D1E displays images based on information transmitted from the work machine 100 so that the remote operator RO in the remote control room RC can visually recognize the surroundings of the work machine 100. In the illustrated example, the display device D1E is a liquid crystal display that displays images captured by an imaging device S6 mounted on the work machine 100. Note that the display device D1E may be a display or projector that realizes naked-eye stereoscopic vision, or may be VR goggles or the like.

[0181] The internal sound output device SP2E is a device capable of outputting various types of sound information. The internal sound output device SP2E outputs sound based on information transmitted from the work machine 100 so that the remote operator RO in the remote control room RC can hear sounds generated at the work site. The internal sound output device SP2E may be configured, for example, to output sound captured by an external sound collector M1 attached to the outside of the cab 10, or may be configured to output sound captured by an internal sound collector M2 attached to the inside of the cab 10. In this case, the internal sound collector M2 may be provided at a position corresponding to the ear position of the operator seated in the cab 10. The internal sound output device SP2E may be a stationary device such as a speaker, or may be a wearable device such as earphones or headphones. The speaker may be a monaural speaker, a stereo speaker, or a surround speaker. The speaker may also be an omnidirectional speaker or a directional speaker. The wearable device may have a noise canceling function, a spatial audio function (stereophonic sound function), or a bone conduction function.

[0182] An operation sensor 29E is installed on the operation device 26E to detect the operation content of the operation device 26E. The operation sensor 29E is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor 29E may be configured with other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29E outputs information related to the detected operation content of the operation device 26E to the remote controller 30E. The remote controller 30E generates an operation signal based on the received information and transmits the generated operation signal to the work machine 100. The operation sensor 29E may be configured to generate the operation signal. In this case, the operation sensor 29E may output the operation signal to the communication device T2 without passing through the remote controller 30E. With this configuration, the remote operator RO can remotely operate the work machine 100 from the remote control room RC.

[0183] The management center MC is a facility where various devices are installed to manage the work machine 100 at the work site or the remote operation of the work machine 100 by the remote operator RO in the remote control room RC. In the illustrated example, the management center MC is installed in a location away from both the work site of the work machine 100 and the remote control room RC. The management center MC is also equipped with a management device 200, an internal sound collection device M2C, and an internal sound output device SP2C.

[0184] The management device 200 is an example of a control device, and is, for example, a server computer (a so-called cloud server) or an edge server. The management device 200 is typically a fixed terminal device, but may also be a portable terminal device (for example, a laptop computer, a tablet, or a smartphone).

[0185] With this configuration, a manager at the management center MC can, for example, use a sound collection device (external sound collection device M1 or internal sound collection device M2) attached to the work machine 100 and the internal sound output device SP2C to listen to sounds emitted at the work site. The manager at the management center MC can also, for example, use an internal sound collection device M2E and internal sound output device SP2C provided in the remote control room RC to listen to sounds emitted in the remote control room RC. The manager at the management center MC can, for example, use the internal sound collection device M2C and the external sound output device SP1 attached to the work machine 100 to communicate the voice he or she makes to a worker WK who is around the work machine 100. The manager at the management center MC can, for example, use the internal sound collection device M2C and the internal sound output device SP2 attached to the work machine 100 to communicate the voice he or she makes to an operator OP of the work machine 100. An administrator at the management center MC can, for example, use an internal sound collection device M2C and an internal sound output device SP2E installed in the remote control room RC to transmit the voice he or she makes to the remote operator RO in the remote control room RC.

[0186] The operation of the working machine operation system SYS according to this embodiment will be described below.

[0187] The operation system SYS for a work machine of this embodiment includes, for example, a work machine 100 and a remote controller 30E installed in a remote control room RC. The work machine 100 includes a lower traveling body 1, an upper rotating body 3 rotatably mounted on the lower traveling body 1, a cab 10 mounted on the upper rotating body 3, an external sound collection device M1 disposed outside the cab 10, and an object detection device that detects people HU1, HU2 around the upper rotating body 3. The remote controller 30E also functions as a control device that notifies of an inadequate person detection when person detection directions D1c, D2c, which are the directions of people HU1, HU2 detected by the object detection device, differ from sound directions D1m, D2m, which are the directions from which sound collected by the external sound collection device M1 is emitted.

[0188] With this configuration, the work machine operation system SYS of this embodiment can use the external sound collection device M1 to collect the voices of people HU1, HU2 around the work machine 100 that could not be detected by the object detection device, and can notify of deficiencies in person detection based on the voice directions D1m, D2m of person HU2. As a result, deficiencies in person detection by the object detection device can be compensated for by the sound detection results of the external sound collection device M1, and the remote operator RO of the work machine 100 can be more reliably notified of the presence of person HU2 around the work machine 100. Therefore, the work machine 100 of this embodiment can improve safety when the work machine 100 is traveling or working compared to conventional methods.

[0189] The preferred embodiments of the present disclosure have been described above. However, the invention according to the present disclosure is not limited to the above-described embodiments. Various modifications, substitutions, etc. may be applied to the above-described embodiments without departing from the scope of the invention according to the present disclosure. Furthermore, each of the features described with reference to the above-described embodiments may be combined as appropriate unless technically inconsistent. [Explanation of symbols]

[0190] 1 Undercarriage 3 Upper rotating body 10 Driver's cab 30 Controller (control device) 30E Remote controller (control device) 42n Information display area (highlighted image) 100 Work Machinery 422b Human detection frame 423b Enhanced image D1 display device D1c Person detection direction D1m Audio direction D2c Person Detection Direction D2m Audio Direction HU1 person HU2 people KS speech button (operation part) M1 External sound collection device M1B Rear Microphone M1BA Rear Mic (Multiple Microphones) M1F Front microphone M1FA Front Microphone (Multiple Microphones) M1L left microphone (microphone) M1LA Left Mic (Multiple Microphones) M1R Right Microphone M1RA Right Mic (Multiple Microphones) M2 internal sound collector S6 Imaging device (object detection device) SP1 External Sound Output Device SP2 internal sound output device SYS Work machine operating system

Claims

1. a lower running body; an upper rotating body rotatably provided on the lower traveling body; an operator's cab provided on the upper rotating body; An external sound collecting device arranged outside the driver's cab; an object detection device that detects people around the upper rotating body; and a control device that notifies of a defect in human detection when a human detection direction, which is the direction of a person detected by the object detection device, differs from a sound direction, which is the direction from which sound collected by the external sound collection device is emitted. Work machinery.

2. the control device notifies that a person has been detected when the person detection direction and the sound direction match; 2. The work machine according to claim 1.

3. Further comprising a display device disposed inside the cab; the control device causes the display device to display an emphasis image that emphasizes the direction of the sound or the position from which the sound is emitted; A work machine according to claim 1 or 2.

4. the control device causes the display device to display the highlighted image in a display mode different from a human detection image that notifies the user that a human has been detected.

4. The work machine according to claim 3.

5. the control device causes the display device to display the highlighted image including information about the defect in the human detection.

4. The work machine according to claim 3.

6. the control device causes the object detection device to re-detect a person in the direction of the sound; 2. The work machine according to claim 1.

7. the external sound collecting device includes a plurality of microphones; The control device estimates the sound direction based on signals generated from the sound collected by the plurality of microphones.

2. The work machine according to claim 1.

8. an internal sound collecting device and an internal sound output device arranged inside the driver's cab; an external sound output device disposed outside the driver's cab, the control device causes the internal sound output device to output a sound based on a signal generated from the sound collected by the external sound collection device, and causes the external sound output device to output a sound based on a signal generated from the sound collected by the internal sound collection device.

2. The work machine according to claim 1.

9. Further, an operation unit connected to the control device is provided, The control device switches between a state in which a sound based on a signal generated from the sound collected by the internal sound collection device is output from the external sound output device and a state in which a sound based on a signal generated from the sound collected by the external sound collection device is output from the internal sound output device, based on an operation state of the operation unit.

9. A work machine according to claim 8.

10. the control device changes the person detection condition of the object detection device during the re-detection.

7. The work machine according to claim 6.

11. a work machine including a lower traveling body, an upper rotating body rotatably provided on the lower traveling body, a driver's cab provided on the upper rotating body, an external sound collecting device arranged outside the driver's cab, and an object detecting device that detects people around the upper rotating body; and a control device that notifies of a defect in human detection when a human detection direction, which is the direction of a person detected by the object detection device, differs from a sound direction, which is the direction from which sound collected by the external sound collection device is emitted. Work machine operating systems.

Images