Display unit of work machine, work machine, and remote control system of work machine

Abstract

To provide a display unit of a work machine capable of improving safety.SOLUTION: A display unit of a work machine in an aspect of an embodiment includes a control unit that is configured to acquire image information picked up by an imaging unit provided to the work machine, acquire information relevant to sound signals obtained by each of multiple sound collection units provided to the work machine, and display the sound generation situation around the work machine based on information about the sound signals acquired from each of the multiple sound collection units on a display unit while superimposing the same over the image information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display device for a work machine, a work machine, and a remote control system for a work machine. [Background technology]

[0002] Conventionally, there has been a demand for providing operators of work machines with appropriate information on the surrounding environment in which the work machine is working. For example, a technology has been proposed that calculates the degree of danger for each position outside the machine and extracts and outputs audio from positions with a high degree of danger (see Patent Document 1). This enables the operator to instantly avoid danger. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-58314 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the system described in Patent Document 1 does not take into consideration displaying the source of the sound. In other words, it is believed that safety can be further improved if the location of a sound source in the vicinity can be visually recognized when image information showing the vicinity of a work machine is displayed.

[0005] In view of the above, safety is improved by displaying the state of sound generation around the work machine superimposed on image information captured by an imaging device. [Means for solving the problem]

[0006] A display device for a work machine according to one embodiment of the present invention includes a control unit configured to acquire image information captured by an imaging device provided on the work machine, acquire information regarding sound signals acquired by each of a plurality of sound collection devices provided on the work machine, and display on the display unit the sound generation situation around the work machine based on the information regarding the sound signals acquired from each of the plurality of sound collection devices, superimposed on the image information. Effect of the Invention

[0007] According to one aspect of the present invention, the state of sound generation around the work machine can be recognized, thereby improving safety. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a remote control system according to the first embodiment. [Diagram 2] FIG. 2 is a side view showing the shovel (excavator) according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of the arrangement of microphones included in a microphone array provided on the shovel according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of a drive control system of the shovel according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a configuration of a hydraulic system of the excavator according to the first embodiment. [Figure 6] FIG. 6 is a functional block diagram illustrating an example of the configuration of the remote control system according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of the layout of the remote control room RC according to the first embodiment. [Figure 8] FIG. 8 is a conceptual diagram showing the surroundings of the shovel according to the first embodiment. [Figure 9] FIG. 9 is a conceptual diagram showing generation of a sound heat map by sound signals of a microphone array in a map generating unit according to the first embodiment. [Figure 10]FIG. 10 is a diagram showing an example of a display screen displayed on the display device according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a display screen displayed on the display device according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a screen displayed on a display device in a remote control room according to a modified example. [Figure 13] FIG. 13 is a conceptual diagram showing the surroundings of the shovel according to the second embodiment. [Figure 14] FIG. 14 is a conceptual diagram showing a configuration up to outputting sound in a remote control room in the second embodiment. [Figure 15] FIG. 15 is a side view of a shovel as a work machine according to the third embodiment. [Figure 16] FIG. 16 is a side view of a shovel as a work machine according to the fourth embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of the configuration of an image display unit and an operation unit of a display device of a shovel according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example, not a limitation of the invention, and all features and combinations described in the embodiment are not necessarily essential to the invention. In addition, the same or corresponding components in each drawing are denoted by the same or corresponding reference numerals, and the description may be omitted.

[0010] (First embodiment) First, an overview of the remote control system SYS according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of the remote control system SYS according to the first embodiment.

[0011] <Devices that make up the remote control system> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes a shovel 100 and a remote operation room RC.

[0012] The excavator 100 and the remote control room RC are connected to each other via a communication line NW so as to be able to send and receive data.

[0013] Wireless communication is possible for the shovel 100. The shovel 100 is then able to transmit and receive data to and from a device (for example, a remote control room RC) connected to the communication line NW.

[0014] The shovel 100 can then transmit information about the work site to the remote control room RC. This allows the remote control room RC to check the work site in accordance with the information from the shovel 100. Note that in this embodiment, the device that measures the work site is not limited to the shovel 100, and may be another type of device, such as a drone that flies over the work site, a fixed camera, or an imaging device that can be carried by a user.

[0015] For example, an imaging device S6 is provided in the shovel 100. The shovel 100 transmits imaging information indicating the imaging results of the work site captured by the imaging device S6 to the remote control room RC.

[0016] The remote operation system SYS may include one or more shovels 100. This allows the remote operation system SYS to provide information about the work site to the remote operation room RC through the multiple shovels 100.

[0017] <Example of remote control room configuration> The remote control room RC is equipped with a communication device T2, a remote controller 40, an operation device 42, an operation sensor 43, a display device D1, and a speaker A2. Also, the remote control room RC is provided with an operation seat DS where an operator OP who remotely operates the excavator 100 sits.

[0018] The communication device T2 is configured to control communication with a communication device T1 (see FIG. 2) attached to the excavator 100.

[0019] The remote controller 40 is an information processing device that executes various calculations. In this embodiment, the remote controller 40 is configured with a microcomputer including a CPU and a memory. Various functions of the remote controller 40 are realized by the CPU executing a program stored in the memory.

[0020] The display device D1 displays a screen based on information transmitted from the shovel 100 so that the operator OP in the remote control room RC can visually confirm the surroundings of the shovel 100. The display device D1 enables the operator to confirm the status of the work site including the surroundings of the shovel 100 even though he is in the remote control room RC.

[0021] An operation sensor 43 for detecting the operation content of the operation device 42 is installed in the operation device 42 (one example of an operation unit). The operation sensor 43 is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor 43 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 43 outputs information on the detected operation content of the operation device 42 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the excavator 100. The operation sensor 43 may be configured to generate an operation signal. In this case, the operation sensor 43 may output the operation signal to the communication device T2 without passing through the remote controller 40. This makes it possible to remotely operate the excavator 100 from the remote operation room RC.

[0022] The speaker A2 outputs sound information transmitted from the shovel 100 so that the operator OP in the remote control room RC can recognize sounds occurring around the shovel 100.

[0023] <Excavator configuration example> Next, an overview of a shovel 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a side view of the shovel 100 as a work machine according to the first embodiment. An upper rotating body 3 is rotatably mounted on a lower traveling body 1 of the shovel 100 via a rotating mechanism 2. A boom 4 is attached to the upper rotating body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5 as an end attachment. The end attachment may be a slope bucket, a dredging bucket, or the like.

[0024] In the example shown in FIG. 2, the traveling direction (front-rear direction) of the shovel 100 is indicated by the X-axis, the width direction of the shovel 100 is indicated by the Y-axis, and the height direction of the shovel 100 is indicated by the Z-axis.

[0025] The boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of an attachment, and are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilt mechanism.

[0026] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor, and can detect the boom angle, which is the rotation angle of the boom 4 relative to the upper rotating body 3. For example, the boom angle is at its minimum when the boom 4 is lowered to the lowest, and increases as the boom 4 is raised.

[0027] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor, and can detect the arm angle, which is the rotation angle of the arm 5 with respect to the boom 4. For example, the arm angle is at its minimum when the arm 5 is fully closed, and increases as the arm 5 is opened.

[0028] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor, and can detect the bucket angle, which is the rotation angle of the bucket 6 with respect to the arm 5. For example, the bucket angle is at its minimum when the bucket 6 is fully closed, and increases as the bucket 6 is opened.

[0029] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, or a rotary encoder that detects the rotation angle around the connecting pin, etc. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute a posture sensor that detects the posture of the excavation attachment.

[0030] The upper rotating body 3 is equipped with a cabin 10 serving as a driver's cab, an engine 11, an aircraft tilt sensor S4, a turning angular velocity sensor S5, an imaging device S6, a positioning device S7, a microphone array A1, a communication device T1, and the like.

[0031] An excavator controller 30 is installed in the cabin 10. Also installed in the cabin 10 are a driver's seat, an operating device, and the like.

[0032] The shovel controller 30 is a calculation device that executes various calculations. The shovel controller 30 is provided, for example, in the cabin 10 and controls the drive of the shovel 100. The functions of the shovel controller 30 may be realized by any hardware, software, or a combination thereof. For example, the shovel controller 30 is mainly configured with a microcomputer including a central processing unit (CPU), a memory device such as a random access memory (RAM), a non-volatile auxiliary storage device such as a read only memory (ROM), and an interface device for various inputs and outputs. The shovel controller 30 executes various programs installed in the non-volatile auxiliary storage device on the CPU, for example, to realize various functions.

[0033] The engine 11 is a drive source of the excavator 100. In this embodiment, the engine 11 is a diesel engine. An output shaft of the engine 11 is connected to input shafts of the main pump 14 and the pilot pump 15.

[0034] The machine body inclination sensor S4 is configured to detect the inclination of the upper rotating body 3 with respect to a predetermined plane. In this embodiment, the machine body inclination sensor S4 is an acceleration sensor that detects the inclination angle about the front-rear axis and the left-right axis of the upper rotating body 3 with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper rotating body 3 are, for example, perpendicular to each other and pass through the shovel center point, which is a point on the rotation axis of the shovel 100.

[0035] The rotation angular velocity sensor S5 is configured to detect the rotation angular velocity of the upper rotating body 3. In this embodiment, the rotation angular velocity sensor S5 is a gyro sensor. The rotation angular velocity sensor S5 may be a resolver or a rotary encoder, etc. The rotation angular velocity sensor S5 may detect a rotation speed. The rotation speed may be calculated from the rotation angular velocity.

[0036] The imaging device S6 is configured to acquire images of the periphery of the shovel 100. In this embodiment, the imaging device S6 includes a front camera S6F that images the space in front of the shovel 100, a left camera S6L that images the space to the left of the shovel 100, a right camera S6R that images the space to the right of the shovel 100, and a rear camera S6B that images the space behind the shovel 100.

[0037] The imaging device S6 may be, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and may output the captured image to the display device DI.

[0038] The front camera S6F is attached, for example, to the roof of the cabin 10. The left camera S6L is attached to the left end of the upper surface of the upper rotating body 3. The right camera S6R is attached to the right end of the upper surface of the upper rotating body 3. The rear camera S6B is attached to the rear end of the upper surface of the upper rotating body 3.

[0039] In this embodiment, by providing the imaging device S6 in the above-mentioned arrangement, it is possible to capture an image of an object existing around the shovel 100. Note that the imaging device S6 may be a camera capable of recognizing the distance to the subject (for example, an RGBD camera or a stereo camera).

[0040] The positioning device S7 is configured to acquire information related to the position of the shovel 100. In this embodiment, the positioning device S7 is configured to measure the position and orientation of the shovel 100 in a reference coordinate system. Specifically, the positioning device S7 is a GNSS receiver incorporating an electronic compass, and measures the latitude, longitude, and altitude of the current position of the shovel 100, and measures the orientation of the shovel 100. The reference coordinate system according to this embodiment is, for example, the World Geodetic System. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the origin at the center of gravity of the earth, the X axis in the direction of the intersection of the Greenwich meridian and the equator, the Y axis in the direction of 90 degrees east longitude, and the Z axis in the direction of the North Pole.

[0041] The communication device T1 is configured to control communication with devices external to the shovel 100. In this embodiment, the communication device T1 is configured to control communication between the communication device T1 and devices external to the shovel 100 via a wireless communication network. The communication device T1 includes, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), a satellite communication module for connecting to a satellite communication network, and the like.

[0042] Furthermore, the communication device T1 controls wireless communication between the excavator 100 and an external GNSS (Global Navigation Satellite System) surveying system, for example.

[0043] The microphone array A1 has a plurality of microphones and is configured to collect sounds generated around the excavator 100. In this embodiment, the microphone array A1 is a plurality of microphones attached to the upper rotating body 3.

[0044] 3 is a diagram showing an example of the arrangement of microphones included in the microphone array A1 according to this embodiment. As shown in FIG. 3, the microphone array A1 is provided at the front side of the cabin 10. Specifically, a total of 15 microphones constituting the microphone array A1 are arranged in an arrangement of three microphones in the Z-axis direction and five microphones in the Y-axis direction. The microphones constituting the microphone array A1 are spaced apart by a distance 1. Note that the distance between the microphones is shown as an example and is to be determined according to the embodiment.

[0045] Each of the microphones constituting the microphone array A1 can collect sound in a predetermined direction within a range of, for example, -90 degrees to +90 degrees from the front. The range in which the microphone array A1 can collect sound is determined so that it can collect sound emitted within at least the imaging range of the front camera S6F. This allows the sound emitted within the imaging range to be displayed so as to be recognizable when displaying image information captured by the front camera S6F. The display mode of the image information will be described later. In this embodiment, the range in which the microphone array A1 can collect sound is shown as an example, and it is sufficient that it includes at least the imaging range of the front camera S6F.

[0046] In this embodiment, an example using the microphone array A1 shown in FIG. 3 has been described as an example of a plurality of sound collecting devices. This embodiment shows an example of a plurality of sound collecting devices, and is not limited to the microphone array A1 shown in FIG. 3. Specifically, the number of microphones is not limited to 15 (3×5) shown in FIG. 3. For example, the number of microphones in the width direction (Y-axis direction) may be other than five, and the number in the height direction (Z-axis direction) may be other than three. Furthermore, the microphones are not limited to a two-dimensional arrangement in the Y-axis direction and the Z-axis direction, and may be, for example, a one-dimensional arrangement in the Y-axis direction.

[0047] [Excavator drive control system] Fig. 4 is a diagram showing an example of the configuration of a drive control system of the excavator 100 in Fig. 2. In Fig. 4, a mechanical power transmission system is indicated by a double line, a hydraulic oil line by a thick solid line, a pilot line by a dashed line, and an electric drive and control system by a dotted line.

[0048] The drive system of the excavator 100 according to this embodiment includes the engine 11, the regulator 13, the main pump 14, and the control valve unit 17. Moreover, the hydraulic drive system of the excavator 100 according to this embodiment includes hydraulic actuators such as the traveling hydraulic motors 1L, 1R, the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, which hydraulically drive the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6, respectively, as described above.

[0049] The engine 11 is a main power source in the hydraulic drive system, and is mounted, for example, on the rear of the upper rotating body 3. Specifically, the engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a shovel controller 30 (described later), and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses light oil as fuel.

[0050] The regulator 13 controls the discharge amount of the main pump 14. For example, the regulator 13 adjusts the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the shovel controller 30. The regulator 13 includes, for example, regulators 13L and 13R, as described later.

[0051] The main pump 14 is mounted on the rear of the upper rotating body 3, for example, like the engine 11, and supplies hydraulic oil to the control valve unit 17 through a high-pressure hydraulic line. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the shovel controller 30, the tilt angle of the swash plate is adjusted by the regulator 13 to adjust the stroke length of the piston, thereby controlling the discharge flow rate (discharge pressure). The main pump 14 includes, for example, main pumps 14L and 14R, as described below.

[0052] The control valve unit 17 is a hydraulic control device that controls the hydraulic system in the excavator 100. In this embodiment, the control valve unit 17 includes control valves 171 to 176. The control valve 175 includes a control valve 175L and a control valve 175R, and the control valve 176 includes a control valve 176L and a control valve 176R. The control valve unit 17 is configured to selectively supply hydraulic oil discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control, for example, the flow rate of hydraulic oil flowing from the main pump 14 to the hydraulic actuators and the flow rate of hydraulic oil flowing from the hydraulic actuators to a hydraulic oil tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, traveling hydraulic motors 1L and 1R, and a swing hydraulic motor 2A. More specifically, the control valve 171 corresponds to the left traveling hydraulic motor 1L, the control valve 172 corresponds to the right traveling hydraulic motor 1R, and the control valve 173 corresponds to the swing hydraulic motor 2A. Furthermore, the control valve 174 corresponds to the bucket cylinder 9, the control valve 175 corresponds to the boom cylinder 7, and the control valve 176 corresponds to the arm cylinder 8. Furthermore, the control valve 175 includes, for example, control valves 175L and 175R as described later, and the control valve 176 includes, for example, control valves 176L and 176R as described later. The control valves 171 to 176 will be described in detail later.

[0053] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to be able to supply hydraulic oil to hydraulic control devices via a pilot line. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generating device may be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to various hydraulic control devices via a pilot line, in addition to a function of supplying hydraulic oil to the control valve unit 17 via a hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0054] The operating device 26 is a device that an operator uses to operate the actuator. The actuator includes at least one of a hydraulic actuator and an electric actuator.

[0055] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In this embodiment, the discharge pressure sensor 28 outputs a detected value to the shovel controller 30. The discharge pressure sensor 28 includes, for example, discharge pressure sensors 28L, 28R, as described later.

[0056] The operation sensor 29 is configured to detect the operation content of the operator using the operating device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operating device 26 corresponding to each actuator, and outputs the detected value to the shovel controller 30. In this embodiment, the shovel controller 30 controls the opening area of ​​the proportional valve 31 according to the output of the operation sensor 29. Then, the shovel controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the hydraulic oil (pilot pressure) supplied to each pilot port is, in principle, a pressure according to the operation direction and operation amount of the operating device 26 corresponding to each hydraulic actuator. In this way, the operating device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0057] The proportional valve 31, which functions as a control valve for machine control, is disposed in a pipe connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured so as to be able to change the flow path area of ​​the pipe. In this embodiment, the proportional valve 31 operates in response to a control command output by the shovel controller 30. Therefore, the shovel controller 30 can supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the proportional valve 31, regardless of the operation of the operating device 26 by the operator.

[0058] With this configuration, the shovel controller 30 can operate the hydraulic actuator corresponding to a specific operating device 26 even when no operation is being performed on that specific operating device 26.

[0059] For example, the shovel controller 30 sets a target rotation speed based on a work mode or the like that is set in advance by a predetermined operation by an operator or the like, and performs drive control to rotate the engine 11 at a constant speed.

[0060] Furthermore, for example, the shovel controller 30 outputs a control command to the regulator 13 as necessary to change the discharge rate of the main pump 14.

[0061] In addition, for example, the shovel controller 30 performs control related to a machine guidance function that guides (guides) the manual operation of the shovel 100 by the operator through the operation device 26. In addition, the shovel controller 30 performs control related to a machine control function that automatically assists the manual operation of the shovel 100 by the operator through the operation device 26.

[0062] Note that some of the functions of the shovel controller 30 may be realized by another controller (control device). That is, the functions of the shovel controller 30 may be realized in a distributed manner by a plurality of controllers. For example, the machine guidance function and the machine control function may be realized by a dedicated controller (control device).

[0063] [Excavator hydraulic system] Next, the hydraulic system of the shovel 100 according to this embodiment will be described with reference to FIG.

[0064] FIG. 5 is a diagram that shows an example of the configuration of a hydraulic system of the shovel 100 according to this embodiment.

[0065] In FIG. 5, the mechanical power system, hydraulic oil lines, pilot lines, and electrical control system are indicated by double lines, solid lines, dashed lines, and dotted lines, respectively.

[0066] The hydraulic system realized by the hydraulic circuit circulates hydraulic oil from each of the main pumps 14L, 14R driven by the engine 11 through center bypass oil passages C1L, C1R and parallel oil passages C2L, C2R to a hydraulic oil tank.

[0067] The center bypass oil passage C1L starts from the main pump 14L, passes through the control valves 171, 173, 175L, and 176L arranged in the control valve unit 17 in this order, and reaches the hydraulic oil tank.

[0068] The center bypass oil passage C1R starts at the main pump 14R, passes through the control valves 172, 174, 175R, and 176R arranged in the control valve unit 17 in this order, and reaches the hydraulic oil tank.

[0069] The control valve 171 is a spool valve that supplies the hydraulic oil discharged from the main pump 14L to the traveling hydraulic motor 1L, and discharges the hydraulic oil discharged from the traveling hydraulic motor 1L to a hydraulic oil tank.

[0070] The control valve 172 is a spool valve that supplies the hydraulic oil discharged from the main pump 14R to the traveling hydraulic motor 1R, and discharges the hydraulic oil discharged from the traveling hydraulic motor 1R to a hydraulic oil tank.

[0071] The control valve 173 is a spool valve that supplies the hydraulic oil discharged from the main pump 14L to the swing hydraulic motor 2A and discharges the hydraulic oil discharged from the swing hydraulic motor 2A to a hydraulic oil tank.

[0072] The control valve 174 is a spool valve that supplies the hydraulic oil discharged from the main pump 14R to the bucket cylinder 9 and also discharges the hydraulic oil in the bucket cylinder 9 to a hydraulic oil tank.

[0073] The control valves 175L and 175R are spool valves that supply the hydraulic oil discharged from the main pumps 14L and 14R to the boom cylinder 7 and discharge the hydraulic oil in the boom cylinder 7 to a hydraulic oil tank.

[0074] The control valves 176L, 176R supply the hydraulic oil discharged by the main pumps 14L, 14R to the arm cylinder 8, and discharge the hydraulic oil in the arm cylinder 8 to a hydraulic oil tank.

[0075] The control valves 171, 172, 173, 174, 175L, 175R, 176L, and 176R adjust the flow rate of hydraulic oil supplied to or discharged from the hydraulic actuator and switch the flow direction according to the pilot pressure acting on the pilot port.

[0076] The parallel oil passage C2L supplies hydraulic oil of the main pump 14L to the control valves 171, 173, 175L, and 176L in parallel with the center bypass oil passage C1L. Specifically, the parallel oil passage C2L branches off from the center bypass oil passage C1L on the upstream side of the control valve 171, and is configured to be able to supply hydraulic oil of the main pump 14L in parallel to each of the control valves 171, 173, 175L, and 176R. This allows the parallel oil passage C2L to supply hydraulic oil to a more downstream control valve when the flow of hydraulic oil through the center bypass oil passage C1L is restricted or blocked by any of the control valves 171, 173, and 175L.

[0077] The parallel oil passage C2R supplies hydraulic oil of the main pump 14R to the control valves 172, 174, 175R, and 176R in parallel with the center bypass oil passage C1R. Specifically, the parallel oil passage C2R branches off from the center bypass oil passage C1R on the upstream side of the control valve 172, and is configured to be able to supply hydraulic oil of the main pump 14R in parallel to each of the control valves 172, 174, 175R, and 176R. When the flow of hydraulic oil through the center bypass oil passage C1R is restricted or blocked by any of the control valves 172, 174, and 175R, the parallel oil passage C2R can supply hydraulic oil to a more downstream control valve.

[0078] Under the control of the shovel controller 30, the regulators 13L and 13R adjust the tilt angles of the swash plates of the main pumps 14L and 14R, thereby adjusting the discharge amounts of the main pumps 14L and 14R, respectively.

[0079] The discharge pressure sensor 28L detects the discharge pressure of the main pump 14L, and a detection signal corresponding to the detected discharge pressure is input to the shovel controller 30. The same is true for the discharge pressure sensor 28R. This allows the shovel controller 30 to control the regulators 13L, 13R in accordance with the discharge pressures of the main pumps 14L, 14R.

[0080] Negative control throttles (hereinafter, "negative control throttles") 18L, 18R are provided in the center bypass oil passages C1L, C1R between the hydraulic oil tank and the most downstream control valves 176L, 176R, respectively. As a result, the flow of hydraulic oil discharged by the main pumps 14L, 14R is restricted by the negative control throttles 18L, 18R. The negative control throttles 18L, 18R generate a control pressure (hereinafter, "negative control pressure") for controlling the regulators 13L, 13R.

[0081] The negative control pressure sensors 19L, 19R detect the negative control pressure, and a detection signal corresponding to the detected negative control pressure is input to the shovel controller 30.

[0082] The shovel controller 30 may control the regulators 13L, 13R in response to the discharge pressure of the main pumps 14L, 14R detected by the discharge pressure sensors 28L, 28R to adjust the discharge volume of the main pumps 14L, 14R. For example, the shovel controller 30 may control the regulator 13L in response to an increase in the discharge pressure of the main pump 14L to adjust the swash plate tilt angle of the main pump 14L to reduce the discharge volume. The same applies to the regulator 13R. As a result, the shovel controller 30 can perform total horsepower control of the main pumps 14L, 14R so that the absorption horsepower of the main pumps 14L, 14R, which is expressed by the product of the discharge pressure and the discharge volume, does not exceed the output horsepower of the engine 11.

[0083] Furthermore, the shovel controller 30 may adjust the discharge rate of the main pumps 14L, 14R by controlling the regulators 13L, 13R according to the negative control pressure detected by the negative control pressure sensors 19L, 19R. For example, the shovel controller 30 decreases the discharge rate of the main pumps 14L, 14R as the negative control pressure increases, and increases the discharge rate of the main pumps 14L, 14R as the negative control pressure decreases.

[0084] Specifically, in the case of a standby state (state shown in FIG. 5) in which none of the hydraulic actuators in the excavator 100 are operated, the hydraulic oil discharged from the main pumps 14L, 14R passes through the center bypass oil passages C1L, C1R and reaches the negative control throttles 18L, 18R. Then, the flow of the hydraulic oil discharged from the main pumps 14L, 14R increases the negative control pressure generated upstream of the negative control throttles 18L, 18R. As a result, the excavator controller 30 reduces the discharge amount of the main pumps 14L, 14R to the allowable minimum discharge amount, suppressing the pressure loss (pumping loss) when the discharged hydraulic oil passes through the center bypass oil passages C1L, C1R.

[0085] On the other hand, when any of the hydraulic actuators is operated through the operating device 26, the hydraulic oil discharged from the main pumps 14L, 14R flows into the hydraulic actuator to be operated through the control valve corresponding to the hydraulic actuator to be operated. Then, the flow of the hydraulic oil discharged from the main pumps 14L, 14R reduces or eliminates the amount of hydraulic oil reaching the negative control throttles 18L, 18R, and reduces the negative control pressure generated upstream of the negative control throttles 18L, 18R. As a result, the excavator controller 30 increases the discharge amount of the main pumps 14L, 14R, circulates sufficient hydraulic oil to the hydraulic actuator to be operated, and can reliably drive the hydraulic actuator to be operated.

[0086] The operating device 26 includes a left operating lever 26L, a right operating lever 26R, and a travel lever 26D. The travel lever 26D includes a left traveling lever 26DL and a right traveling lever 26DR.

[0087] The left operating lever 26L and the right operating lever 26R receive operations in at least one of the forward / rearward direction and the left / right direction.

[0088] The left operating lever 26L is used for the swing operation and the operation of the arm 5. When the left operating lever 26L is operated in the forward / backward direction, the hydraulic oil discharged by the pilot pump 15 is used to introduce a control pressure according to the amount of lever operation into the pilot ports of the control valves 176L and 176R. When the left operating lever 26L is operated in the left / right direction, the hydraulic oil discharged by the pilot pump 15 is used to introduce a control pressure according to the amount of lever operation into the pilot port of the control valve 173.

[0089] Specifically, when the left operation lever 26L is operated in the arm closing direction, the shovel controller 30 introduces hydraulic oil to the right pilot port of the control valve 176L and introduces hydraulic oil to the left pilot port of the control valve 176R. When the left operation lever 26L is operated in the arm opening direction, the shovel controller 30 introduces hydraulic oil to the left pilot port of the control valve 176L and introduces hydraulic oil to the right pilot port of the control valve 176R. When the left operation lever 26L is operated in the left turning direction, the shovel controller 30 introduces hydraulic oil to the left pilot port of the control valve 173, and when the left operation lever 26L is operated in the right turning direction, the shovel controller 30 introduces hydraulic oil to the right pilot port of the control valve 173. In order for the shovel controller 30 to perform this control, outputs from the operation sensors 29LA and 29LB are used.

[0090] The right operating lever 26R is used for operating the boom 4 and the bucket 6. When the right operating lever 26R is operated in the forward / backward direction, it uses the hydraulic oil discharged by the pilot pump 15 to introduce a control pressure corresponding to the amount of lever operation into the pilot port of the control valve 175. When the right operating lever 26R is operated in the left / right direction, it uses the hydraulic oil discharged by the pilot pump 15 to introduce a control pressure corresponding to the amount of lever operation into the pilot port of the control valve 174.

[0091] Specifically, when the right operating lever 26R is operated in the boom lowering direction, the shovel controller 30 introduces hydraulic oil to the left pilot port of the control valve 175R. When the right operating lever 26R is operated in the boom raising direction, the shovel controller 30 introduces hydraulic oil to the right pilot port of the control valve 175L and also introduces hydraulic oil to the left pilot port of the control valve 175R. When the right operating lever 26R is operated in the bucket closing direction, the shovel controller 30 introduces hydraulic oil to the right pilot port of the control valve 174, and when the right operating lever 26R is operated in the bucket opening direction, the shovel controller 30 introduces hydraulic oil to the left pilot port of the control valve 174. In order for the shovel controller 30 to perform this control, outputs from the operation sensors 29RA and 29RB are used.

[0092] The left travel lever 26DL is used to operate the left crawler 1CL. It may be configured to be interlocked with the left travel pedal. When the left travel lever 26DL is operated in the forward / backward direction, the shovel controller 30 uses the hydraulic oil discharged by the pilot pump 15 to introduce a control pressure corresponding to the lever operation amount into the pilot port of the control valve 171. The right travel lever 26DR is used to operate the right crawler 1CR. It may be configured to be interlocked with the right travel pedal. When the right travel lever 26DR is operated in the forward / backward direction, the shovel controller 30 uses the hydraulic oil discharged by the pilot pump 15 to introduce a control pressure corresponding to the lever operation amount into the pilot port of the control valve 172. In order for the shovel controller 30 to perform this control, outputs from operation sensors 29DL and 29DR, which will be described later, are used.

[0093] The operation sensor 29 is configured to detect the content of an operation of the operation device 26 by an operator. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each actuator, and outputs the detected values ​​to the shovel controller 30.

[0094] The operation sensor 29 includes operation sensors 29LA, 29LB, 29RA, 29RB, 29DL, and 29DR. The operation sensor 29LA detects the content of the operation of the left operation lever 26L by the operator in the forward / rearward direction, and outputs the detected value to the shovel controller 30. The content of the operation is, for example, the lever operation direction, the lever operation amount (lever operation angle), etc.

[0095] Similarly, the operation sensor 29LB detects the content of the operation of the left operation lever 26L by the operator in the left-right direction, and outputs the detected value to the shovel controller 30. The operation sensor 29RA detects the content of the operation of the right operation lever 26R by the operator in the forward-backward direction, and outputs the detected value to the shovel controller 30. The operation sensor 29RB detects the content of the operation of the right operation lever 26R by the operator in the left-right direction, and outputs the detected value to the shovel controller 30.

[0096] The operation sensor 29DL detects the content of the operation of the left traveling lever 26DL in the forward / backward direction by the operator, and outputs the detected value to the shovel controller 30. The operation sensor 29DR detects the content of the operation of the right traveling lever 26DR in the forward / backward direction by the operator, and outputs the detected value to the shovel controller 30.

[0097] The shovel controller 30 receives the output of the operation sensor 29, and outputs a control command to the regulator 13 as necessary, thereby changing the discharge rate of the main pump 14. The shovel controller 30 also receives the output of the control pressure sensor 19 provided upstream of the orifice 18, and outputs a control command to the regulator 13 as necessary, thereby changing the discharge rate of the main pump 14. The orifice 18 includes a left orifice 18L and a right orifice 18R, and the control pressure sensor 19 includes negative control pressure sensors 19L and 19R.

[0098] <Block configuration of remote control system> Fig. 6 is a functional block diagram showing a configuration example of the remote operation system SYS according to this embodiment. In the example shown in Fig. 6, block configurations of the remote operation room RC and the shovel 100 included in the remote operation system SYS are shown. Note that the hardware configuration of the shovel 100 is as described above, and therefore description thereof will be omitted.

[0099] <Configuration of the remote control room RC> The remote control room RC includes a remote controller 40, a communication device T2, a speaker A2, an operation sensor 43, and a display device D1. The speaker A2, the communication device T2, and the operation sensor 43 have been described above, so their description will be omitted.

[0100] Next, the remote control room RC will be described. Fig. 7 is a diagram showing an example of the layout of the remote control room RC. In the remote control room RC, a plurality of operating devices 42 are provided with respect to an operating seat DS.

[0101] In this embodiment, the display device D1 is a multi-display consisting of nine monitors arranged in three vertical rows and three horizontal columns, as shown in Fig. 7. Specifically, the display device D1 includes a center monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i.

[0102] In this embodiment, a speaker A2 is provided above the display device D1. The speaker A2 is provided to output sounds collected by the microphone array A1 of the shovel 100, for example.

[0103] <<Excavator functional blocks>> Returning to FIG. 6, each functional block in the shovel controller 30 of the shovel 100 will be described. Each functional block in the shovel controller 30 is conceptual, and does not necessarily have to be physically configured as shown in the figure. All or a part of each functional block can be functionally or physically distributed and integrated in any unit. All or any part of each processing function performed in each functional block is realized by a program executed by a CPU. Alternatively, each functional block may be realized as hardware using wired logic. By realizing a program, the shovel controller 30 includes an acquisition unit 301, an shovel state identification unit 302, a map generation unit 303, a transmission control unit 304, a reception control unit 305, and an actuator drive unit 306.

[0104] Fig. 8 is a conceptual diagram showing the surroundings of the shovel 100 according to this embodiment. In the example shown in Fig. 8, it is assumed that the shovel 100, a worker 801, and a truck 802 are present.

[0105] A worker 801 is talking to the excavator 100. Since a truck 802 is in operation, the engine noise can be heard all the way to the excavator 100.

[0106] The imaging area of ​​the front camera S6F of the excavator 100 includes a worker 801 and a truck 802.

[0107] Furthermore, a microphone array A1 collects the speech of a worker 801 and the engine sound of a truck 802.

[0108] Then, the shovel controller 30 transmits a sound signal indicating the sound collected by the microphone array A1 of the shovel 100 to the communication device T2 of the remote control room RC via the communication device T1. Furthermore, the shovel 100 generates information indicating the sound generation situation around the shovel 100 (corresponding to a sound heat map described later) based on the sound signal acquired by the microphone array A1, and transmits the generated information from the communication device T1 to the communication device T2 of the remote control room RC. A specific configuration of the shovel 100 for realizing the transmission of the information will be described.

[0109] Returning to FIG. 6, the acquisition unit 301 acquires signals from various detection devices provided on the shovel 100. For example, the acquisition unit 301 acquires detection results from a boom angle sensor S1, an arm angle sensor S2, and a bucket angle sensor S3. The acquisition unit 301 also acquires measurement results such as the position and orientation of the shovel 100 from a positioning device S7. The acquisition unit 301 also acquires image information from an imaging device S6. The acquisition unit 301 also acquires sound signals collected by each of the microphones constituting the microphone array A1 from the microphone array A1.

[0110] The shovel state identifying unit 302 is configured to identify the state of the shovel 100 based on the signal acquired by the acquiring unit 301. In this embodiment, the state of the shovel 100 includes the position and orientation of the shovel 100 and the states of the attachments of the shovel 100 (for example, the positions of the boom 4, the arm 5, and the bucket 6). The position of the shovel 100 is, for example, the position of the shovel 100 in the reference coordinate system (the latitude, longitude, and altitude of the reference point of the shovel 100). The shovel state identifying unit 302 identifies the position and orientation of the shovel 100 based on the output of the positioning device S7.

[0111] The state of the attachment (e.g., the positions of the boom 4, arm 5, and bucket 6) can be determined from the detection results of the angle sensors (boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3) and the respective sizes of the boom 4, arm 5, and bucket 6.

[0112] The map generating unit 303 generates a sound heat map that represents the intensity of sounds around the shovel 100, based on the sound signals collected by each of the microphones that make up the microphone array A1.

[0113] The sound heat map according to this embodiment is a two-dimensional map in the height direction (Z-axis direction) and width direction (Y-axis direction) of the shovel 100, and is a map that represents the strength of the collected sound for each sound collection direction within the range in which the microphone array A1 can collect sound. The strength of the sound represents the magnitude of the amplitude in the sound signal.

[0114] Fig. 9 is a conceptual diagram showing generation of a sound heat map by sound signals of the microphone array A1 in the map generating unit 303 according to this embodiment. For ease of explanation, in the example shown in Fig. 9, an example will be described in which the microphone array A1 is composed of microphones A1a to A1e.

[0115] 9, the microphones A1a to A1e of the microphone array A1 output sound signals collected from each sound collection direction within a range where sound can be collected to the shovel controller 30. The sound signals acquired by the acquisition unit 301 of the shovel controller 30 are output to the map generation unit 303.

[0116] The map generating unit 303 sets delay devices for the sound signals collected by each of the microphones A1a to A1e according to the sound collection direction.

[0117] 9, in order to recognize the intensity of a sound signal collected from a direction tilted at an angle θ in the width direction (Y-axis direction) from the front, the map generating unit 303 assigns a delay unit 303a corresponding to the microphone A1a, a delay unit 303b corresponding to the microphone A1b, a delay unit 303c corresponding to the microphone A1c, and a delay unit 303d corresponding to the microphone A1d. The delay units 303a to 303d are determined according to the angle θ indicating the sound collection direction.

[0118] For example, the delay unit 303a delays the sound signal input from the microphone A1a by -t4·s" A sound signal is generated taking into account the delay of time t4 by a calculation using the above formula. s is a Laplace operator. Time t4 indicates the delay time from when the microphone A1a collects sound to when the microphone A1e collects sound when the sound collection direction is inclined by an angle θ in the width direction. Specifically, time t4 can be calculated from the formula "t4=4·l·sinθ / v". Distance l is the distance between the microphones and is defined as the sound speed v. A specific calculation performed by the delay devices 303a to 303d can be performed using a well-known method, so a description thereof will be omitted. Note that the sound speed v may be a fixed value or may be calculated based on the detected temperature, etc. Note that this embodiment shows an example of the delay devices 303a to 303d, and any well-known circuit may be used as long as it takes into account the delay of the sound signal.

[0119] The delay unit 303b delays the sound signal input from the microphone A1b by -t3·s" A sound signal that takes into account the delay in time t3 is generated by a calculation using the above formula. Time t3 indicates the delay time from when microphone A1b collects sound until microphone A1e collects sound when the sound collection direction is tilted at an angle of θ in the width direction. Specifically, time t3 can be calculated using the formula "t3=3·l·sinθ / v".

[0120] The delay unit 303c delays the sound signal input from the microphone A1c by -t2·s" A sound signal that takes into account the delay in time t2 is generated by a calculation using the above formula. Time t2 indicates the delay time from when microphone A1c collects sound until microphone A1e collects sound when the sound collection direction is tilted at an angle of θ in the width direction. Specifically, time t2 can be calculated using the formula "t2=2·l·sinθ / v".

[0121] The delay unit 303d delays the sound signal input from the microphone A1d by -t1·s"A sound signal that takes into account the delay in time t1 is generated by a calculation using the above formula. Time t1 indicates the delay time from when microphone A1d collects sound until microphone A1e collects sound when the sound collection direction is tilted at an angle of θ in the width direction. Specifically, time t1 can be calculated from the formula "t1=l·sinθ / v".

[0122] Based on the values ​​output from delay units 303a to 303d and the sound signal output from microphone A1e, calculator 303f generates a sound signal in which the sound from a direction tilted by angle θ is emphasized among the sound signals collected by each of microphones A1a to A1e.

[0123] The map generating unit 303 generates a sound heat map 303g based on the amplitude of the sound signal generated by the calculator 303f. Specifically, in the heat map 303g, a corresponding area (e.g., pixel) is predefined for each sound collection direction. The map generating unit 303 then assigns a color to each area (e.g., pixel) of the heat map 303g according to the amplitude of the sound signal in which the sound in the sound collection direction corresponding to the area is emphasized. For example, the map generating unit 303 assigns a color such that the larger the amplitude, the redder (or black) the color, and the smaller the parameter, the whiter the color.

[0124] The map generation unit 303 generates a sound signal in which the sound from each sound collection direction is emphasized within the range in which the microphone array A1 can collect sound, and generates a sound heat map 303g by assigning a color according to the amplitude of the sound signal.

[0125] 9, for ease of explanation, an example inclined at an angle θ in the Y-axis direction will be explained, but the direction of sound collection will be a combination of the Y-axis direction and the Z-axis direction. Since the intensity of the sound can be recognized in the above-mentioned process for this combination direction as well, the explanation will be omitted.

[0126] The transmission control unit 304 performs control for transmitting various information based on the acquisition result of the acquisition unit 301 to the remote control room RC via the communication device T1. For example, the transmission control unit 304 performs control for transmitting to the remote control room RC image information captured by the imaging device S6, position information indicating the position and orientation of the shovel 100, status information indicating the status of the attachment, sound signals collected by the microphone array A1, and a sound heat map. The transmission by the transmission control unit 304 is performed at predetermined time intervals. The predetermined time may be any time, but is set to a time interval that allows the situation changed by the work of the shovel 100 to be recognized. For example, the transmission control unit 304 may transmit the above information at one-second intervals.

[0127] The reception control unit 305 performs control for receiving various information from the remote control room RC via the communication device T1. For example, the reception control unit 305 receives an operation signal for controlling the operation of the shovel 100 from the remote control room RC.

[0128] The actuator driving unit 306 is configured to drive an actuator mounted on the excavator 100. In this embodiment, the actuator driving unit 306 generates and outputs an actuation signal for each of the plurality of solenoid valves included in the proportional valve 31, based on an operation signal transmitted from the remote control room RC.

[0129] Each solenoid valve that receives the actuation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed according to the stroke amount of the control valve.

[0130] <<Remote control room function block>> Each functional block in the remote controller (one example of a control unit) 40 of the remote operation room RC will be described. Each functional block in the remote controller 40 is conceptual, and does not necessarily have to be physically configured as shown in the figure. All or part of each functional block can be functionally or physically distributed and integrated in any unit. All or any part of each processing function performed in each functional block is realized by a program executed by a CPU. Alternatively, each functional block may be realized as hardware using wired logic. By realizing a program, the remote controller 40 includes a reception control unit 401, a display screen generation unit 402, an output control unit 403, a signal generation unit 404, and a transmission control unit 405.

[0131] In this embodiment, a case will be described in which the remote controller 40 in the remote operation room RC is a control unit of a display device (one example of a display unit) D1. In other words, the configuration in the remote operation room RC functions as a display device for supporting the remote operation of the excavator 100.

[0132] The reception control unit 401 performs control for receiving various information from the shovel 100 via the communication device T2.

[0133] As another example, the reception control unit 401 receives (acquires) from the shovel 100 image information captured by the imaging device S6, position information, status information, sound signals acquired by the microphone array A1, and a sound heat map (an example of information related to sound). The image information is information captured by the imaging device S6. The position information is information indicating the position and orientation of the shovel 100 in a reference coordinate system measured by the positioning device S7. The status information is information indicating the states of the attachments (for example, the positions of the boom 4, arm 5, and bucket 6). The sound heat map is map information indicating the intensity of sound around the shovel 100 generated by the map generation unit 303.

[0134] The display screen generating unit 402 generates a display image by superimposing a sound heat map, which indicates the sound generation situation around the shovel 100, on the image information. Specifically, the display screen generating unit 402 generates the display image by superimposing the positions indicated in the image information and the sound collection directions of the heat map so as to correspond to each other.

[0135] The output control unit 403 outputs the sound signals of the microphone array A1 from the speaker A2. Any method may be used to output the sound signals from the multiple microphones from the speaker A2, regardless of whether it is a well-known method.

[0136] Moreover, the output control unit 403 outputs the display image generated by the display screen generating unit 402 to the display device D1.

[0137] The signal generation unit 404 generates an operation signal for controlling the operation of the shovel 100 in accordance with the operation received by the operation sensor 43. The transmission control unit 405 transmits the generated operation signal to the shovel 100.

[0138] Fig. 10 is a diagram showing an example of a display screen displayed on the display device D1 according to this embodiment. In the example shown in Fig. 10, a worker 801 and a truck 802 shown in image information captured by the front camera S6F are displayed. Furthermore, sound detection areas 1001 and 1002 shown in a sound heat map are displayed superimposed on the image information. The sound detection areas 1001 and 1002 are displayed semi-transparently (transmittance is arbitrary).

[0139] The sound detection area 1001 is an area displayed based on the sound heat map. The sound detection area 1001 includes a dark red area 1001a, a red area 1001b, an orange area 1001c, a yellow area 1001d, and a light yellow area 1001e. The sound detection area 1001 shows that the sound gets quieter as it changes from area 1001a to area 1001e. In other words, the area 1001a is near the sound source and shows that the worker 801 is speaking.

[0140] Similarly, the sound detection area 1002 includes a red area 1002a, an orange area 1002b, a yellow area 1002c, and a light yellow area 1002d. The sound detection area 1002 also shows that the sound decreases from area 1002a to area 1002d. In other words, area 1002a is near the sound source and shows that the engine of the truck 802 is running.

[0141] As shown in the screen in FIG. 10, in this embodiment, the output control unit 403 displays the sound generation status around the excavator 100 in a display mode that allows the strength of the sound acquired from the direction corresponding to each area displayed on the screen to be recognized. In this embodiment, an example in which the strength of the sound is represented as a color is shown, but the present invention is not limited to the example in which the strength of the sound is represented as a color, and other modes may be used. For example, the display screen may be shaded, or the brightness may be changed. In this embodiment, the display screen is displayed so that the strength of the sound can be recognized, so that the operator can grasp the source of the sound. In other words, when the operator hears a sound through the speaker A2, the operator can recognize the source of the sound, and therefore can grasp the current situation. Then, the operator can perform an operation according to the current situation. Therefore, improvement of safety can be achieved.

[0142] The sound heat map used to generate the display screen according to the present embodiment shows, as colors, the strength of the sound acquired for each direction included in the range in which the microphone array A1 can collect sound. As a result, when the microphone array A1 collects sound from multiple sources, the display image output by the output control unit 403 shows the multiple sound sources as multiple sound detection areas. In other words, when the sound signal acquired by the microphone array A1 includes multiple sound sources, the output control unit 403 can display each of the multiple sound sources in a recognizable display mode. As a result, the operator can operate the device after recognizing the multiple sound sources, thereby improving safety.

[0143] In addition, the imaging area of ​​the front camera S6F may differ from the range in which the microphone array A1 can collect sound. In this case, the microphone array A1 may acquire a sound signal from a sound source that is not shown in the imaging area of ​​the front camera S6F. In this case, the display screen generating unit 402 may generate a display image by superimposing the sound detection area on the image information, even if the sound source is not shown in the image information.

[0144] Fig. 11 is a diagram showing an example of a display screen displayed on the display device D1 according to this embodiment. In the example shown in Fig. 11, the image information captured by the front camera S6F does not show the source of the sound, but a sound detection area 1101 indicating that the source of the sound is on the right side is displayed superimposed on the image information.

[0145] The sound detection area 1101 is an area displayed based on a sound heat map. The sound detection area 1101 includes an orange area 1101a, a yellow area 1101b, and a light yellow area 1101c. When sound is output from the speaker A2, the operator can recognize the sound detection area 1101 and thereby recognize that the source of the sound is on the right side.

[0146] 11 is an example of a display mode, and is not limited to this display mode. In other words, any display mode that allows recognition that a sound signal has been acquired from a direction outside the area represented by the image information may be used.

[0147] In this way, when the output control unit 403 according to this embodiment acquires a sound signal from a direction outside the area represented by the image information captured by the front camera S6F, it displays on the display device D1 that the sound signal was acquired from outside the area represented by the image information. This allows the operator to estimate the direction in which the sound source is located even if it is not captured on the display device D1. The operator can perform work after estimating the direction in which the sound source is located, thereby improving safety.

[0148] In the present embodiment, an example has been described in which a sound heat map (an example of information related to sound) is generated by the shovel controller 30. However, the generation of a sound heat map (an example of information related to sound) is not limited to being performed on the shovel controller 30 side, and may be performed on the remote controller 40 side, for example.

[0149] (Modification 1 of the first embodiment) In the above embodiment, an example has been described in which a sound detection area showing the strength of a sound is superimposed on image information. However, the display mode is not limited to the above embodiment. For example, spoken content may be displayed as text information.

[0150] The display screen generating unit 402 of the remote controller 40 extracts the voice contained in the sound signal received by the reception control unit 401, and converts the voice into text information. A well-known voice recognition method may be used to convert the voice into text.

[0151] Then, the display screen generating unit 402 overlays the converted text information, together with the sound detection area, on the image information to generate a display screen.

[0152] Then, the output control unit 403 displays a display screen on which the character information is superimposed. Fig. 12 is a diagram showing an example of a display screen displayed on the display device D1 according to this embodiment. In the example shown in Fig. 12, character string information 1201 is superimposed on image information captured by the front camera S6F together with sound detection areas 1001 and 1002.

[0153] The voice indicated by the character string information 1201 is output from the speaker A2, but in this modification, it is also represented as character string information 1201. This allows the operator to recognize the spoken content even if he or she misses or does not hear the voice output from the speaker A2, and therefore allows the operator to operate the shovel 100 while taking into consideration the spoken content. This makes it possible to improve safety.

[0154] (Modification 2 of the first embodiment) In the above-described embodiment, an example has been described in which the sound signals acquired from a specific direction are synthesized based on the delay sum of the sound signals collected by the microphones constituting the microphone array A1, the strength of the sound in the specific direction is recognized, and the recognition result is generated as a sound heat map. However, the method of recognizing the sound generation state is not limited to the method of the above-described embodiment.

[0155] For example, the sound signals collected by the microphones constituting the microphone array A1 are a combination of sound signals from different sources. Therefore, the map generating unit 303 may separate the sound signals acquired by each microphone into sound signals different for each source, and identify the sound source from the difference in arrival time of the separated sound signals to the microphone. Then, the transmission control unit 304 transmits information about the sound, which indicates the source of the sound and the intensity of the separated sound signal, to the remote controller 40. Then, the display screen generating unit 402 generates a display image by superimposing a sound detection area indicating the source of the sound and the intensity of the sound on the image information. This makes it possible to display a display screen similar to that of the above-mentioned embodiment. As a method for extracting sound signals different for each source, for example, a well-known method combining duplication of a sound signal and filtering processing may be used.

[0156] Second Embodiment The above-described embodiment and modified examples have shown a method for displaying a sound heat map showing the generation status of sound around the shovel 100. For the purpose of this display, a case has been described in which sounds are collected from an operator 801 and a truck 802 as sounds around the shovel 100. However, the targets of sounds collected by the microphone array A1 are not limited to objects existing around the shovel 100, and also include objects existing inside the shovel 100. Therefore, in the second embodiment, a case in which sounds from objects existing inside the shovel 100 are taken into consideration will be described.

[0157] Fig. 13 is a conceptual diagram showing the situation around the shovel 100 according to this embodiment. In the example shown in Fig. 13, it is assumed that a worker 801 and a truck 802 are present around the shovel 100. Furthermore, in the example shown in Fig. 13, the engine sound of the engine 11 mounted on the shovel 100 is collected by the microphone array A1.

[0158] In the remote control room RC, if the sound output from the speaker A2 includes the engine sound of the engine 11, it may be difficult for the operator to grasp the situation around the shovel 100. Therefore, in this embodiment, control is performed to suppress the engine sound emitted from the engine 11 of the shovel 100. In this embodiment, an example of filtering the engine sound as a method of suppressing the engine sound will be described. Note that, in this embodiment, an example of filtering the engine sound will be described as a method of suppressing the engine sound, but other methods may be applied as a method of suppressing the engine sound.

[0159] Fig. 14 is a conceptual diagram showing a configuration up to outputting sound in a remote control room RC in this embodiment. In the example shown in Fig. 14, an shovel 100 is provided with a shovel controller 30A. Furthermore, a remote controller 40A is provided in the remote control room RC.

[0160] In the example shown in FIG. 14, the remote controller 40A includes a reception control unit 401, an acoustic separation unit 1401, an attenuation unit 1402, an addition unit 1403, a map generation unit 303A, an output control unit 403, and a display screen generation unit 402.

[0161] In the present embodiment, the remote controller 40A filters the engine sound. That is, in the above-described embodiment, the map generating unit 303 is provided in the shovel controller 30, but in the present embodiment, the map generating unit 303A is provided in the remote controller 40A.

[0162] In the example shown in FIG. 14, the shovel controller 30A transmits a sound signal indicating the sound collected by the microphone array A1 to the communication device T2 in the remote control room RC via the communication device T1.

[0163] The acoustic separation unit 1401 separates the sound signals acquired by each microphone, which are included in the sound signal received by the reception control unit 401, into a sound signal that differs for each source. In the example shown in Fig. 14, the sound signal is separated into a sound signal of an engine sound, a sound signal of a worker 801, and a sound signal of a truck 802.

[0164] The sound separation unit 1401 according to this embodiment holds feature information (for example, frequency band and waveform) indicating the engine sound of the engine 11. Therefore, the sound separation unit 1401 can identify a sound signal indicating the engine sound from the separated sound signals.

[0165] Then, the acoustic separation unit 1401 outputs the sound signal indicating the engine sound to the attenuation unit 1402. On the other hand, the acoustic separation unit 1401 outputs the sound signals other than the engine sound to the map generation unit 303A and also to the addition unit 1403.

[0166] The map generating unit 303A generates a sound heat map based on sound signals obtained by the microphones constituting the microphone array A1, from which engine sound has been removed. Note that the generation method is the same as that of the second modification of the first embodiment, and therefore a description thereof will be omitted.

[0167] The attenuation unit 1402 reduces the amplitude of the sound signal representing the engine sound. That is, in this embodiment, the attenuation unit 1402 reduces the amplitude of the sound signal representing the engine sound, thereby attenuating a predetermined frequency component corresponding to the engine sound. Note that the degree of reduction may be determined according to the embodiment.

[0168] Then, the adder 1403 adds the sound signal indicating the engine sound attenuated by the attenuation unit 1402 to other sound signals. In this way, the adder 1403 generates a sound signal in which the engine sound generated from the engine (drive source) of the shovel (an example of a work machine) 100 is attenuated.

[0169] Then, the output control unit 403 performs control to output the sound signal after addition from the speaker A2. At that time, the output control unit 403 may synthesize the sound signals of the microphones constituting the microphone array A1 so that the sound signals can be output from the speaker A2.

[0170] In this embodiment, the remote controller 40 performs the above-mentioned control to attenuate the engine noise, making it easier for the operator to grasp the situation around the shovel 100 by the sound output from the speaker A2.

[0171] In this embodiment, the engine sound is attenuated in the sound output from the speaker A2, so that the stress caused by the sound to the operator can be reduced.

[0172] In addition, certain frequency components such as engine noise are attenuated from the sound output from speaker A2 to make it easier to hear sounds around the shovel 100, making it easier to grasp the surrounding situation from, for example, the voices of people or abnormal sounds around the shovel 100.

[0173] (Modification 1 of the second embodiment) In the above-described embodiment, an example has been described in which the acoustic separation unit 1401 separates the engine sound into different sound signals for each source. However, the method of suppressing the engine sound is not limited to a method of extracting a sound signal indicating the engine sound and then attenuating the sound signal. In this modified example, a band-stop filter is provided instead of the acoustic separation unit 1401, the attenuation unit 1402, and the addition unit 1403. In this modified example, the map generation unit may be provided in the shovel controller or in the remote controller.

[0174] In this modification, the band-stop filter attenuates a frequency band corresponding to the engine sound from the sound signals acquired by each microphone and included in the sound signal received by the reception control unit 401. Then, the output control unit 403 outputs the sound signal in which the frequency band corresponding to the engine sound has been attenuated from the speaker A2.

[0175] In this modified example, a band-stop filter is used as a filter for attenuating the frequency band corresponding to the engine sound, but the filter for attenuating the frequency band corresponding to the engine sound is not limited to a band-stop filter, and a notch filter may be used. In other words, any filter may be used as long as it has a set frequency band for attenuating the frequency band corresponding to the engine sound and not attenuating other sounds.

[0176] In this modification, by performing the above-mentioned control, it is possible to achieve the same effects as in the second embodiment.

[0177] In the above-described embodiment and this modified example, an example of reducing the amplitude of a sound signal indicating an engine sound has been described as a filtering method. However, this embodiment shows an example of filtering, and is not limited to this filtering method. In other words, it is sufficient that the filtering attenuates sounds that are obstacles to the operator of the remote control room RC in recognizing the situation around the shovel 100 based on a predetermined condition when operating the shovel 100. Next, another example of filtering based on a predetermined condition will be described.

[0178] (Modification 2 of the second embodiment) In the above embodiment, an example has been described in which the engine sound of the engine 11 is attenuated to allow the operator to recognize the surrounding situation. However, there are other aspects of the method for allowing the operator to recognize the surrounding situation. For example, when there are multiple sound sources, it may be difficult to hear human voices. Therefore, in this modified example, a case in which human voices are easily heard will be described.

[0179] The frequency band of human voice is approximately 100 Hz to 1000 Hz, so if this frequency band is extracted and output from the speaker A2, the voice becomes easier to hear.

[0180] Therefore, in this modification, the remote controller 40 in the remote control room RC can receive an operation of whether or not to filter the frequency band. The mode of the operation device for receiving the operation may be any method, not just an arbitrary method. For example, a button for switching whether or not to filter may be provided.

[0181] When the remote controller 40 receives an operation to perform filtering, the remote controller 40 performs filtering on the received sound signal to extract a frequency band from 100 Hz to 1000 Hz.

[0182] Then, the output control unit 403 outputs the filtered sound signal from the speaker A2. In this modification, since the sound is filtered in the frequency band of the sound, it becomes easy to hear the human voice. This allows the situation around the shovel 100 to be understood, thereby improving safety.

[0183] In this modification, the remote controller 40 in the remote control room RC accepts an operation to switch whether or not to extract the frequency band of 100 Hz to 1000 Hz corresponding to human voice. However, this modification does not limit the frequency band to be extracted or not ...

[0184] In this modification, an example of switching whether or not to perform filtering depending on an operation has been described. However, the condition for switching whether or not to perform filtering is not limited to the presence or absence of an operation. For example, whether or not to perform filtering may be automatically switched depending on whether or not a human voice is detected.

[0185] In the above-described embodiment and this modified example, filtering is performed based on a predetermined condition, and one or more of generating a sound heat map and outputting from the speaker A2 are performed based on the filtered sound signal. In this way, by presenting information based on the filtered sound signal to the operator, the operator can be made to recognize necessary information. Therefore, it is possible to realize an improvement in convenience.

[0186] (Third embodiment) In the above-described embodiments, an example has been described in which the microphone array A1 is provided on the front surface of the shovel 100. However, unlike the above-described embodiments, the microphone array A1 is not limited to being provided on the front surface of the shovel 100. Therefore, in the third embodiment, an example in which the microphone array A1 is provided in each of four directions will be described.

[0187] 15 is a side view of a shovel 100A as a work machine according to the third embodiment. Other than the microphone array A1, the components are the same as those in the above-described embodiments, and therefore the description thereof will be omitted.

[0188] The microphone array A1 is configured to acquire sounds around the shovel 100. In this embodiment, the microphone array A1 includes a front microphone array A1F that collects sounds in front of the shovel 100, a left microphone array A1L that collects sounds to the left of the shovel 100, a right microphone array A1R that collects sounds to the right of the shovel 100, and a rear microphone array A1B that collects sounds behind the shovel 100.

[0189] The imaging device S6 may be, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and may output the captured image to the display device DI.

[0190] The front camera S6F is attached, for example, to the roof of the cabin 10. The left camera S6L is attached to the left end of the upper surface of the upper rotating body 3. The right camera S6R is attached to the right end of the upper surface of the upper rotating body 3. The rear camera S6B is attached to the rear end of the upper surface of the upper rotating body 3.

[0191] The left microphone array A1L is attached, for example, to a frame on the left side of the shovel 100. The right microphone array A1R is attached, for example, to a frame on the right side of the shovel 100. The rear microphone array A1B is attached, for example, to a frame on the rear side of the shovel 100.

[0192] Then, the map generation unit 303 of the shovel controller 30 generates a heat map of sounds corresponding to each of the four microphone arrays A1. Then, the transmission control unit 304 transmits the heat maps of the four sounds to the remote controller 40.

[0193] This allows the remote control room RC to display the sound detection area in an overlapping manner when displaying image information from each of the four imaging devices S6. Therefore, when an operation to switch the display from the image information captured by the four imaging devices S6 is received from the operator of the remote control room RC, the sound detection area can be displayed in an overlapping manner on the display image displayed after switching.

[0194] In this embodiment, four microphone arrays A1 are provided to accommodate the installation of the imaging device S6, so that the sound detection area can be superimposed on the image information regardless of which image information is displayed among the image information captured by the four imaging devices S6. This allows the operator in the remote control room RC to grasp the source of the sound, thereby improving safety.

[0195] Furthermore, when displaying any one of the image information captured by the four imaging devices S6 on the display device D1, the display screen generating unit 402 may generate a display image taking into consideration each of the heat maps of the sound of the microphone array A1 provided in a direction that does not correspond to the displayed image information. For example, when generating a display image based on the image information of the front camera S6F, if the heat map of the sound from the left microphone array A1L indicates that the sound is strong, the display screen generating unit 402 may superimpose information indicating that the source of the sound is on the left side on the image information of the front camera S6F. In this case, the display screen generating unit 402 may superimpose character string information converted from the sound collected from the left microphone array A1L on the image information of the front camera S6F. This allows the operator to grasp the direction in which the source of the sound is located outside the area displayed on the display screen. Therefore, the operator can recognize the direction in which the source of the sound is located, thereby improving safety.

[0196] (Fourth embodiment) In the above-described embodiment and modified example, a case where an operator performs an operation in a remote control room RC has been described. However, the mode of displaying a sound detection area superimposed on image information as in the above-described embodiment is not limited to the case where remote operation is performed. Therefore, in the fourth embodiment, a case where the sound detection area is displayed for an operator who operates the excavator 100B will be described.

[0197] 16 is a side view of a shovel 100B as a work machine according to the fourth embodiment. Components other than the image capture device S6 and the microphone array A1 are the same as those in the above-described embodiments, and therefore description thereof will be omitted.

[0198] The imaging device S6 according to this embodiment includes a left camera S6L that images the space to the left of the shovel 100, a right camera S6R that images the space to the right of the shovel 100, and a rear camera S6B that images the space behind the shovel 100. As described above, in this embodiment, the operator can visually confirm the front direction, so the front camera S6F is not provided.

[0199] Similarly, the microphone array A1 includes a left microphone array A1L that collects sound to the left of the shovel 100, a right microphone array A1R that collects sound to the right of the shovel 100, and a rear microphone array A1B that collects sound behind the shovel 100. The mounting locations are the same as in the above-described embodiment, so a description thereof will be omitted.

[0200] Then, the map generation unit 303 of the shovel controller 30 generates a sound heat map corresponding to each of the three microphone arrays A1. Then, the shovel controller 30 generates a display image by superimposing the sound heat map showing the sound generation situation around the shovel 100 on the image information, and displays the display image on the display device D1.

[0201] Next, a configuration example of the image display unit 51 and operation unit 52 of the display device DI will be described with reference to Fig. 17. Fig. 17 is a diagram showing a configuration example of the image display unit 51 and operation unit 52 of the display device DI. The example shown in Fig. 17 shows a state in which a home screen including a right image RG, a rear image BG, and an overhead image TG is displayed on the image display unit 51 as a screen displayed before the push switch 52a of the operation unit 52 is operated. The home screen is a screen displayed while the excavator 100 is in operation.

[0202] First, a description will be given of the image display unit 51. As shown in Fig. 17, the image display unit 51 includes a date and time display area 51a, a driving mode display area 51b, an attachment display area 51c, a fuel efficiency display area 51d, an engine control state display area 51e, an engine operating time display area 51f, a coolant temperature display area 51g, a remaining fuel amount display area 51h, a rotation speed level display area 51i, a urea water remaining amount display area 51j, a hydraulic oil temperature display area 51k, an air conditioner operating state display area 51m, an image display area 51n, and a switch image display area 51p.

[0203] The travel mode display area 51b, the attachment display area 51c, the engine control state display area 51e, the RPM level display area 51i, and the air conditioner operation state display area 51m are areas that display setting state information, which is information about the setting state of the excavator 100. The fuel consumption display area 51d, the engine operation time display area 51f, the coolant temperature display area 51g, the remaining fuel amount display area 51h, the remaining urea water amount display area 51j, and the hydraulic oil temperature display area 51k are areas that display operating state information, which is information about the operating state of the excavator 100.

[0204] Specifically, the date and time display area 51a is an area for displaying the current date and time. The travel mode display area 51b is an area for displaying the current travel mode. The attachment display area 51c is an area for displaying an image representing the currently attached attachment. The fuel efficiency display area 51d is an area for displaying fuel efficiency information calculated by the shovel controller 30. The fuel efficiency display area 51d includes an average fuel efficiency display area 51d1 for displaying lifetime average fuel efficiency or section average fuel efficiency, and an instantaneous fuel efficiency display area 51d2 for displaying instantaneous fuel efficiency.

[0205] The engine control state display area 51e is an area that displays the control state of the engine 11. The engine operating time display area 51f is an area that displays the accumulated operating time of the engine 11. The coolant temperature display area 51g is an area that displays the current temperature state of the engine coolant. The remaining fuel amount display area 51h is an area that displays the remaining amount of fuel stored in the fuel tank. The rotation speed level display area 51i is an area that displays, as an image, the current level set by the dial 75. FIG. 17 shows a state in which the first level is selected. The urea water remaining amount display area 51j is an area that displays, as an image, the remaining amount of urea water stored in the urea water tank. The hydraulic oil temperature display area 51k is an area that displays the temperature state of the hydraulic oil in the hydraulic oil tank.

[0206] The air conditioner operation status display area 51m includes an air outlet display area 51m1 that displays the current air outlet position, an operation mode display area 51m2 that displays the current operation mode, a temperature display area 51m3 that displays the current set temperature, and an air volume display area 51m4 that displays the current set air volume.

[0207] The image display area 51n is an area for displaying images captured by the imaging device S6. In the example shown in Fig. 17, the image display area 51n displays a right image RG, a rear image BG, and an overhead image TG.

[0208] The right image RG is an image that shows the space to the right of the shovel 100, and includes an image GC1 of the right end of the top surface of the upper rotating body 3. The right image RG is an image generated by the shovel controller 30 by superimposing a heat map of sound generated by a sound signal acquired by the right microphone array A1R on an image captured by the right camera S6R.

[0209] A worker 1501 is shown in the right image RG. A sound detection area 1511 represented by a sound heat map is further superimposed and displayed on the right image RG. The sound detection area 1001 is an area displayed based on the sound heat map. The sound detection area 1511 includes a dark red area 1511a, a red area 1511b, an orange area 1511c, a yellow area 1511d, and a light yellow area 1511e.

[0210] The rear image BG is an image that shows the space behind the shovel 100, and includes an image GC2 of the counterweight. The rear image BG is an image generated by the shovel controller 30 by superimposing a heat map of sound generated by a sound signal acquired by the rear microphone array A1B on an image captured by the rear camera S6B.

[0211] A truck 1502 is shown in the rear image BG. A sound detection area 1512, which is represented by a sound heat map, is further superimposed on the rear image BG. The sound detection area 1501 is an area displayed based on the sound heat map. The sound detection area 1521 includes a yellow area 1511a and a light yellow area 1511b.

[0212] The overhead image TG is a virtual viewpoint image generated by the shovel controller 30, and is generated based on images acquired by the rear camera S6B, the left camera S6L, and the right camera S6R. A shovel figure corresponding to the shovel 100 is arranged in the center of the overhead image. This is to allow the operator to more intuitively grasp the positional relationship between the shovel 100 and objects existing around the shovel 100.

[0213] The overhead image TG also shows the worker 1501 and the truck 1502. This allows the operator of the excavator 100 to recognize the surrounding situation.

[0214] Further, the image display area 51n has a first image display area 51n1 located above and a second image display area 51n2 located below. In the example shown in Fig. 17, the right image RG and the rear image BG are arranged in the first image display area 51n1, and the overhead image TG is arranged in the second image display area 51n2. Note that the image display area 51n may arrange the overhead image TG in the first image display area 51n1, and the rear image BG and the right image RG in the second image display area 51n2.

[0215] The image display area 51n may be configured to simultaneously display a left image. In this case, the image display area 51n may arrange the left image and the right image RG in the first image display area 51n1, and arrange the rear image BG and the overhead image TG in the second image display area 51n2. In this case, the left image may be arranged on the left side of the first image display area 51n1, and the right image RG may be arranged on the right side of the first image display area 51n1.

[0216] In the example shown in FIG. 17, the right image RG and the rear image BG are arranged adjacent to each other on the left and right, but they may be arranged with a gap therebetween. In the example shown in FIG. 17, the image display area 51n is a vertically long area, but the image display area 51n may be a horizontally long area. When the image display area 51n is a horizontally long area, the image display area 51n may arrange the overhead image TG as the first image display area 51n1 on the left side, and the rear image BG and the right image RG as the second image display area 51n2 on the right side. In this case, they may be arranged with a gap therebetween on the left and the position of the overhead image TG and the positions of the rear image BG and the right image RG may be interchanged.

[0217] 17, the switch images 51p1 to 51p7 are arranged at intervals from each other on the left and right at the bottom of the image display unit 51. The switch images 51p1 to 51p7 display icons representing the functions of the corresponding push switches 52a1 to 52a7.

[0218] Menu detail item icons for displaying menu detail items are displayed on the switch image 51p1. When the operator operates the push switch 52a1 corresponding to the switch image 51p1, the icons displayed on the switch images 51p2 to 51p7 are switched to icons associated with the menu detail items.

[0219] In the example shown in FIG. 17, an icon for displaying information related to the digital level is displayed on the switch image 51p4. When the operator operates the push switch 52a4 corresponding to the switch image 51p4, the overhead image TG displayed in the second image display area 51n2 is switched to an image (sub-screen) showing information related to the digital level. Meanwhile, the rear image BG and right image RG displayed in the first image display area 51n1 are continuously displayed in the same size. However, the image (sub-screen) showing information related to the digital level may be displayed in the first image display area 51n1. In this case, the rear image BG and right image RG displayed in the first image display area 51n1 may be displayed in the second image display area 51n2 instead of the overhead image TG.

[0220] 17, no icons are displayed on switch images 51p2, 51p3, 51p5, 51p6, and 51p7. Therefore, even if the operator operates push switches 52a2, 52a3, 52a5, 52a6, and 52a7 corresponding to switch images 51p2, 51p3, 51p5, 51p6, and 51p7, no change occurs in the image displayed on image display unit 51.

[0221] It should be noted that the icons displayed on the switch images 51p1 to 51p7 are not limited to the above examples, and icons for displaying other information may be displayed.

[0222] Next, the operation unit 52 will be described. As shown in FIG. 17, the operation unit 52 is composed of push switches 52a, which are button-type switches corresponding to the switch images 51p1 to 51p7. In the example shown in FIG. 17, the operation unit 52 includes seven push switches 52a1 to 52a7 arranged in the upper row and seven push switches 52a8 to 52a14 arranged in the lower row. The push switches 52a8 to 52a14 are arranged below the push switches 52a1 to 52a7, respectively. However, the number, form, and arrangement of the push switches 52a of the operation unit 52 are not limited to the above example, and may be a form in which the functions of multiple button-type switches are integrated into one, such as a jog wheel or a jog switch. The operation unit 52 may be configured as a member separate from the display device DI. Also, a method in which the switch images 51p1 to 51p7 are directly touched and operated on a touch panel in which the image display unit 51 and the operation unit 52 are integrated may be used.

[0223] Push switches 52a1-52a7 are arranged below switch images 51p1-51p7 in correspondence with switch images 51p1-51p7, respectively, and function as push switches for selecting switch images 51p1-51p7. Because push switches 52a1-52a7 are arranged below switch images 51p1-51p7 in correspondence with switch images 51p1-51p7, respectively, the operator can intuitively select switch image 51p1-51p7.

[0224] The push switch 52a8 is a switch for switching the captured image displayed in the image display area 51n. Each time the push switch 52a8 is operated, the captured image displayed in the first image display area 51n1 of the image display area 51n is switched between, for example, a rear image, a left image, a right image, and an overhead image. Also, each time the push switch 52a8 is operated, the captured image displayed in the second image display area 51n2 of the image display area 51n may be switched between, for example, a rear image, a left image, a right image, and an overhead image. Also, each time the push switch 52a8 is operated, the captured image displayed in the first image display area 51n1 of the image display area 51n and the captured image displayed in the second image display area 51n2 may be replaced with each other. In this way, the push switch 52a8 as the operation unit 52 may be used to switch the captured image displayed in the first image display area 51n1 or the second image display area 51n2, or to switch between the captured image displayed in the first image display area 51n1 and the captured image displayed in the second image display area 51n2. Also, a separate switch may be provided to switch the screen displayed in the second image display area 51n2.

[0225] Push switches 52a9 and 52a10 are switches for adjusting the volume of air from the air conditioner. In the example shown in Fig. 17, the volume of air from the air conditioner is decreased when push switch 52a9 is operated, and increased when push switch 52a10 is operated.

[0226] The push switch 52a11 is a switch for switching the cooling / heating function ON / OFF. In the example shown in Fig. 17, the push switch 52a11 is configured to switch the cooling / heating function ON / OFF every time it is operated.

[0227] Push switches 52a12 and 52a13 are switches for adjusting the set temperature of the air conditioner. In the example shown in Fig. 17, the set temperature is lowered when push switch 52a12 is operated, and is raised when push switch 52a13 is operated.

[0228] The push switch 52a14 is a switch capable of switching the display of the engine operating time display area 51f.

[0229] The push switches 52a2 to 52a6 and 52a9 to 52a13 are configured to allow input of numbers displayed on or near the respective switches. The push switches 52a3, 52a4, 52a5 and 52a11 are configured to allow the cursor to be moved left, up, right and down, respectively, when the cursor is displayed in the image display area 51n.

[0230] The functions assigned to the push switches 52a1 to 52a7 and 52a8 to 52a14 described above are merely examples, and other functions may be assigned to the switches.

[0231] As described above, when the push switch 52a1 corresponding to the switch image 51p1 is operated in a state in which the right image RG, the rear image BG, and the overhead image TG are displayed in the image display area 51n, new icons (icons representing functions newly assigned to the push switches 52a1 to 52a7, respectively) are displayed on the switch images 51p2 to 51p7 in a state in which the right image RG and the rear image BG are displayed. Therefore, the operator can check the new icons while checking the right image RG and the rear image BG.

[0232] In the above example, when one of the push switches 52a1-52a7 corresponding to the switch images 51p2-51p7 is operated while the right image RG, rear image BG, and overhead image TG are displayed in the image display area 51n, the overhead image TG is switched to a menu image showing information corresponding to the selected switch image. In this way, since the menu image is displayed while the right image RG and rear image BG are displayed, the operator can continue to monitor the surroundings (the space behind and to the right) even while the menu image is displayed. Therefore, the operator can operate the excavator 100 while the menu image is displayed.

[0233] In this embodiment, the sound detection area is displayed so as to be superimposed on the right image RG and the rear image BG included in the image display area 51n of the display device D1.

[0234] That is, in this embodiment, the operator riding on the shovel 100 can visually confirm the situation in front of the shovel 100, and can therefore recognize the source of the sound as well as the sound coming from in front of the shovel 100.

[0235] However, it may be difficult for an operator riding on the shovel 100 to recognize the source of a sound in an area that cannot be seen. Therefore, the shovel controller 30 according to the present embodiment displays the above-mentioned screen on the display device DI.

[0236] This allows the operator of the shovel 100 to recognize the source of sound around the shovel 100 even in areas that cannot be seen. For example, by referring to the right image RG and the rear image BG, the source of the sound can be recognized, and the operator can grasp the source of the sound that he or she is currently hearing, and can perform an operation corresponding to the sound that he or she has heard. This can therefore improve safety.

[0237] <effect> In the above-described embodiment and modified example, the operator can recognize the source of the sound by visually checking the display image, and therefore it becomes easy to check the situation at the work site. Therefore, the operator can perform operations according to the situation at the work site, and therefore it is possible to improve safety. Furthermore, the operator can perform operations according to the situation at the work site, and therefore it is possible to improve work efficiency.

[0238] In the above-described embodiment and modified examples, a case has been described in which a shovel is used as an example of a work machine. However, the configurations shown in the embodiment and modified examples are not limited to being applied to a shovel as a work machine, and may be applied to, for example, a crane, a forklift, etc.

[0239] The above describes an embodiment showing an example of a work machine display device, a work machine, and a work machine remote control system, but the present invention is not limited to the above embodiment. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the claims. Naturally, these also fall within the technical scope of the present invention. [Explanation of symbols]

[0240] 100 Shovel 1 Undercarriage 2 Swivel mechanism 3. Upper rotating body 4. Boom 5 Arm 6 Bucket S6 imaging device T1 Communications Equipment DI display device A1 Microphone Array 30 Excavator Controller 301 Acquisition Department 302 Excavator condition identification section 303 Map Generation Unit 304 Transmission control section 305 Reception control section 306 Actuator drive unit 31 Proportional valve RC Remote Control Room T2 Communication Equipment D1 display device A2 Speaker 40 Remote Controller 401 Reception control section 402 Display screen generation section 403 Output control section 404 Signal Generator 405 Transmission control section

Claims

1. Acquiring image information captured by an imaging device provided on the work machine, and acquiring information related to sound signals acquired by each of a plurality of sound collecting devices provided on the work machine; a control unit configured to display, on a display unit, a sound generation situation around the work machine based on information about the sound signals acquired from each of the plurality of sound collecting devices, superimposed on the image information; Display device for work machine.

2. the control unit displays, as the sound generation status, the acquired sound intensity from a direction corresponding to each area represented in the image information in a recognizable display mode. The display device of a work machine according to claim 1.

3. When the sound signal acquired from each of the plurality of sound collection devices includes a plurality of sound sources, the control unit displays the plurality of sound sources in a recognizable display manner. The display device of a work machine according to claim 2.

4. When the control unit acquires the sound signal from a direction outside the area represented by the image information, the control unit displays a message indicating that the sound signal was acquired from outside the area represented by the image information. The display device of a work machine according to claim 1.

5. The control unit converts the voice included in the sound signal into characters, and displays the converted characters on the display unit. The display device of a work machine according to claim 1.

6. The control unit filters the sound signals acquired from each of the plurality of sound collection devices based on a predetermined condition, and outputs the filtered sound signals. The display device of a work machine according to claim 1.

7. The control unit performs filtering in which it is determined whether or not to attenuate a predetermined frequency component, and outputs the filtered sound signal. The display device of a work machine according to claim 6.

8. The control unit performs filtering to attenuate the sound signal generated from the drive source of the work machine. The display device of a work machine according to claim 7.

9. The display device for a work machine according to claim 6 , wherein the control unit switches between performing filtering and not performing filtering depending on an operation received from an operating device as the predetermined condition.

10. The control unit outputs the filtered sound signal from a speaker. The display device for a work machine according to any one of claims 6 to 9.

11. An imaging device; A plurality of sound collecting devices; a control device that acquires image information from the imaging device and acquires sound signals from each of the plurality of sound collection devices, and outputs to a display device a sound generation situation around the work machine based on the sound signals acquired from each of the plurality of sound collection devices, superimposed on the image information; A work machine comprising:

12. The plurality of sound collecting devices are provided so as to be capable of acquiring sounds emitted from a range including an imaging area captured by the imaging device.

12. A work machine according to claim 11.

13. A remote operation system including a work machine and an information processing device that operates the work machine, a work machine that acquires image information from an imaging device provided on the work machine, acquires sound signals from each of a plurality of sound collecting devices provided on the work machine, and transmits the image information and information relating to the sound signals; an information processing device that displays, on a display device, a sound generation situation around the work machine based on information about the sound signal, superimposed on the received image information; A remote control system for a work machine comprising:

Images