Display device for construction machine, construction machine, and remote operation system for construction machine

Abstract

The present invention addresses the problem of improving safety. A display device for a construction machine according to one embodiment of the present invention is provided with: a control unit configured to acquire image information captured by an imaging device provided in the construction machine, and to acquire information relating to sound signals acquired by each of a plurality of sound collection devices provided in the construction machine; and superimposing, on the image information, the generation status of sound around the construction machine on the basis of the information relating to the sound signals acquired from each of the plurality of sound collection devices, and displaying the same on a display unit.

Description

Technical Field

[0001] This application claims priority based on Japanese Patent Application No. 2023-108227 filed on June 30, 2023. The entire contents of this Japanese application are incorporated herein by reference.

[0002] The invention relates to a display device of a construction machine, the construction machine and a remote operation system of the construction machine. Background Art

[0003] In the past, it was required to accurately provide the operator of a construction machine with information about the surrounding environment where the construction machine is working. For example, a technology has been proposed that calculates the degree of danger for each position outside the machine and extracts and outputs sounds from positions with high degrees of danger (see Patent Document 1). This allows the operator to avoid danger immediately.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2012-58314

[0005] However, the system described in Patent Document 1 does not consider displaying the source of the sound. That is, it is believed that when displaying image information showing the surrounding conditions of the construction machine, if the position of the sound source existing in the surrounding conditions can be visually recognized, safety can be further improved. Summary of the invention

[0006] In view of the above circumstances, safety is improved by superimposing the sound generation status around the construction machine on the image information captured by the camera device.

[0007] A display device of a construction machine involved in one embodiment of the present invention includes a control unit, which is configured to: obtain image information captured by a camera device installed in the construction machine, and obtain information related to sound signals respectively obtained by multiple sound collection devices installed in the construction machine; and superimpose the generation status of sounds around the construction machine based on the information related to the sound signals respectively obtained from the multiple sound collection devices on the image information and display it on the display unit.

[0008] Effects of the Invention

[0009] According to one embodiment of the present invention, the generation status of sounds around a construction machine can be recognized, thereby improving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram showing an example of the remote operation system according to the first embodiment.

[0011] Figure 2 It is a side view showing the shovel (excavator) according to the first embodiment.

[0012] Figure 3 It is a diagram showing an example of arrangement of microphones included in the microphone array provided in the shovel according to the first embodiment.

[0013] Figure 4 It is a diagram showing a configuration example of a drive control system of the shovel according to the first embodiment.

[0014] Figure 5 It is a diagram schematically showing an example of the configuration of a hydraulic system of the shovel according to the first embodiment.

[0015] Figure 6 This is a functional block diagram showing a configuration example of the remote operation system according to the first embodiment.

[0016] Figure 7 It is a diagram showing an example of arrangement of the remote operation room RC according to the first embodiment.

[0017] Figure 8 This is a conceptual diagram showing the situation around the shovel according to the first embodiment.

[0018] Fig. 9 This is a conceptual diagram showing generation of a sound heat map based on a sound signal from a microphone array in a map generation unit according to the first embodiment.

[0019] Fig.10 This is a diagram showing an example of a display screen displayed on the display device according to the first embodiment.

[0020] Fig.11 This is a diagram showing an example of a display screen displayed on the display device according to the first embodiment.

[0021] Fig.12 It is a diagram showing an example of a screen displayed on a display device of a remote operation room according to a modification.

[0022] Fig.13 This is a conceptual diagram showing the situation around the shovel according to the second embodiment.

[0023] Fig.14 This is a conceptual diagram showing the configuration in the remote operation room until the sound is output in the second embodiment.

[0024] Fig.15 It is a side view of a shovel as a construction machine according to the third embodiment.

[0025] Fig.16 It is a side view of a shovel as a construction machine according to a fourth embodiment.

[0026] Fig.17 It is a diagram showing a configuration example of an image display unit and an operation unit of a display device for a shovel according to a fourth embodiment.

[0027] In the figure: 100-excavator, 1-lower walking body, 2-slewing mechanism, 3-upper slewing body, 4-boom, 5-arm, 6-bucket, S6-camera, T1-communication device, DI-display device, A1-microphone array, 30-excavator controller, 301-acquisition unit, 302-excavator state determination unit, 303-image generation unit, 304-transmission control unit, 305-reception control unit, 306-actuator drive unit, 31-proportional valve, RC-remote operation room, T2-communication device. D1-display device, A2-speaker, 40-remote controller, 401-reception control unit, 402-display screen generation unit, 403-output control unit, 404-signal generation unit, 405-transmission control unit. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, the embodiments described below are examples and do not limit the invention, and all features or combinations thereof described in the embodiments are not necessarily essential features or combinations thereof of the invention. In addition, in each of the drawings, the same or corresponding structures are sometimes marked with the same or corresponding symbols, and the description is omitted.

[0029] (First embodiment)

[0030] First, refer to Figure 1 An overview of the remote operation system SYS according to the first embodiment will be described. Figure 1 This is a schematic diagram showing an example of the remote operation system SYS according to the first embodiment.

[0031] <Equipment that constitutes the remote operation system>

[0032] like Figure 1 As shown in FIG. 1 , the remote operation system SYS according to the first embodiment includes a shovel 100 and a remote operation room RC.

[0033] The shovel 100 and the remote control room RC are connected via a communication line NW so as to be able to transmit and receive data.

[0034] The shovel 100 is capable of wireless communication and can transmit and receive data with a device (for example, a remote control room RC) connected to the communication line NW.

[0035] Furthermore, the shovel 100 can transmit information related to the construction site to the remote operation room RC. Thus, the remote operation room RC can confirm the construction site based on the information from the shovel 100. In addition, the present embodiment does not limit the device for measuring the construction site to the shovel 100, but may be other devices such as a drone flying over the construction site, a fixed-point camera, or a camera device that can be carried by a user.

[0036] For example, the shovel 100 is provided with an imaging device S6. The shovel 100 transmits imaging information indicating the imaging result of the construction site by the imaging device S6 to the remote operation room RC.

[0037] The remote operation system SYS may include one or more shovels 100 , so that the remote operation system SYS can provide the remote operation room RC with information related to the construction site through the plural shovels 100 .

[0038] <Example of remote operation room configuration>

[0039] The remote operation room RC is provided with a communication device T2, a remote controller 40, an operation device 42, an operation sensor 43, a display device D1, and a speaker A2. The remote operation room RC is also provided with an operation seat DS where an operator OP who remotely operates the shovel 100 sits.

[0040] The communication device T2 is configured to control the communication device T1 (see Figure 2 ) between them.

[0041] The remote controller 40 is an information processing device that performs various operations. In the present embodiment, the remote controller 40 is composed of a microcomputer including a CPU and a memory. In addition, various functions of the remote controller 40 are realized by the CPU executing a program stored in the memory.

[0042] The display device D1 displays a screen based on information transmitted from the shovel 100 so that the operator OP in the remote operation room RC can visually recognize the surroundings of the shovel 100. Although the operator is in the remote operation room RC, the display device D1 can confirm the status of the construction site including the surroundings of the shovel 100.

[0043] The operating device 42 (an example of an operating unit) is provided with an operating sensor 43 for detecting the operation content of the operating device 42. The operating sensor 43 is, for example, a tilt sensor for detecting the tilt angle of the operating lever or an angle sensor for detecting the swing angle of the operating lever around the swing axis. The operating sensor 43 may also be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor or a distance sensor. The operating sensor 43 outputs information related to the detected operation content of the operating device 42 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and sends the generated operation signal to the excavator 100. The operating sensor 43 may also be configured to generate an operation signal. At this time, the operating sensor 43 may output the operation signal to the communication device T2 without passing through the remote controller 40. Thereby, remote operation of the excavator 100 can be achieved from the remote operation room RC.

[0044] The speaker A2 outputs sound information transmitted from the shovel 100 so that the operator OP in the remote operator's cabin RC can recognize sounds generated around the shovel 100 .

[0045] <Example of the structure of an excavator>

[0046] Next, refer to Figure 2 The outline of the shovel 100 according to the present embodiment will be described. Figure 2 This is a side view of an excavator 100 as a construction machine according to the first embodiment. An upper revolving body 3 is rotatably mounted on a lower traveling body 1 of the excavator 100 via a revolving mechanism 2. A boom 4 is mounted on the upper revolving body 3. A dipper arm 5 is mounted on the front end of the boom 4, and a bucket 6 as an end attachment is mounted on the front end of the dipper arm 5. The end attachment may be a bucket for slopes or a bucket for dredging, etc.

[0047] exist Figure 2 In the example shown, the X-axis represents the traveling direction (front-rear direction) of the shovel 100 , the Y-axis represents the width direction of the shovel 100 , and the Z-axis represents the height direction of the shovel 100 .

[0048] The boom 4, the arm 5 and the bucket 6 constitute an excavation attachment as an example of an attachment, and are respectively hydraulically driven by a boom cylinder 7, an arm cylinder 8 and a bucket cylinder 9. A boom angle sensor S1 is mounted on the boom 4, an arm angle sensor S2 is mounted on the arm 5, and a bucket angle sensor S3 is mounted on the bucket 6. The excavation attachment may also be provided with a bucket tilting mechanism.

[0049] The boom angle sensor S1 detects the rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is an acceleration sensor capable of detecting the boom angle as the rotation angle of the boom 4 relative to the upper swing body 3. The boom angle becomes the minimum angle when the boom 4 is maximally lowered, for example, and increases as the boom 4 is raised.

[0050] The arm angle sensor S2 detects the rotation angle of the arm 5. In the present embodiment, the arm angle sensor S2 is an acceleration sensor that can detect the arm angle as the rotation angle of the arm 5 relative to the boom 4. The arm angle becomes the minimum angle when the arm 5 is retracted to the maximum, for example, and increases as the arm 5 is opened.

[0051] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is an acceleration sensor that can detect the bucket angle as the rotation angle of the bucket 6 relative to the boom 5. The bucket angle is, for example, the smallest angle when the bucket 6 is retracted to the maximum, and increases as the bucket 6 is opened.

[0052] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may also be a potentiometer using a variable resistor, a stroke sensor for detecting the stroke amount of the corresponding hydraulic cylinder, or a rotary encoder for detecting the rotation angle around a connecting pin, etc. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute a posture sensor for detecting the posture of the excavating attachment.

[0053] The upper rotating body 3 is equipped with a cab 10 as a cockpit, an engine 11, a fuselage tilt sensor S4, a rotation angular velocity sensor S5, a camera device S6, a positioning device S7, a microphone array A1, a communication device T1, and the like.

[0054] A shovel controller 30 is provided in the cab 10. Also, a driver's seat, an operating device, and the like are provided in the cab 10.

[0055] The excavator controller 30 is a computing device that performs various operations. The excavator controller 30 is, for example, disposed in the cab 10 to perform drive control of the excavator 100. The functions of the excavator controller 30 can be implemented by any hardware, software, or a combination thereof. For example, the excavator controller 30 is constructed with a microcomputer including a CPU (Central Processing Unit), a RAM (Random Access Memory) and other memory devices, a ROM (Read Only Memory) and other non-volatile auxiliary storage devices, and various input and output interface devices as the center. The excavator controller 30 implements various functions by, for example, executing various programs installed in the non-volatile auxiliary storage device on the CPU.

[0056] The engine 11 is a driving source of the shovel 100. In the present embodiment, the engine 11 is a diesel engine. The output shaft of the engine 11 is connected to the input shafts of the main pump 14 and the pilot pump 15, respectively.

[0057] The body tilt sensor S4 is configured to detect the tilt of the upper rotating body 3 relative to a predetermined plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects the tilt angle of the upper rotating body 3 relative to the horizontal plane about the front-rear axis and the tilt angle about the left-right axis. The front-rear axis and the left-right axis of the upper rotating body 3 are, for example, orthogonal to each other and pass through the center point of the shovel, which is a point on the rotation axis of the shovel 100.

[0058] The rotation angular velocity sensor S5 is configured to detect the rotation angular velocity of the upper rotating body 3. In the present embodiment, the rotation angular velocity sensor S5 is a gyro sensor. The rotation angular velocity sensor S5 may also be a rotary transformer or a rotary encoder. The rotation angular velocity sensor S5 may also detect the rotation speed. The rotation speed may be calculated from the rotation angular velocity.

[0059] The camera device S6 is configured to acquire images of the periphery of the shovel 100. In the present embodiment, the camera device S6 includes a front camera S6F for photographing the space in front of the shovel 100, a left camera S6L for photographing the space on the left side of the shovel 100, a right camera S6R for photographing the space on the right side of the shovel 100, and a rear camera S6B for photographing the space on the rear side of the shovel 100.

[0060] The camera device S6 may be, for example, a single-lens camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the display device DI.

[0061] The front camera S6F is installed, for example, on the top of the cab 10. The left camera S6L is installed on the left end of the upper surface of the upper rotating body 3. The right camera S6R is installed on the right end of the upper surface of the upper rotating body 3. The rear camera S6B is installed on the rear end of the upper surface of the upper rotating body 3.

[0062] In this embodiment, by disposing the imaging device S6 as described above, it is possible to capture an object around the shovel 100. Alternatively, the imaging device S6 may be a camera (eg, an RGBD camera or a stereo camera) that can recognize the distance to the imaging object.

[0063] The positioning device S7 is configured to obtain information related to the position of the excavator 100. In the present embodiment, the positioning device S7 is configured to measure the position and orientation of the excavator 100 in the reference coordinate system. Specifically, the positioning device S7 is a GNSS receiver assembled with an electronic compass, which measures the latitude, longitude and altitude of the current position of the excavator 100, and measures the orientation of the excavator 100. The reference coordinate system involved in the present embodiment is, for example, the world geodetic system. The world geodetic system is a three-dimensional orthogonal XYZ coordinate system with the center of gravity of the earth as the origin, the direction of the intersection of the Greenwich meridian and the equator as the X-axis, the direction of 90 degrees east longitude as the Y-axis, and the direction of the North Pole as the Z-axis.

[0064] The communication device T1 is configured to control communication with a device located outside the shovel 100. In the present embodiment, the communication device T1 is configured to control communication between the communication device T1 and the device located outside the shovel 100 via a wireless communication network. The communication device T1 includes, for example, a mobile communication module corresponding to mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), or a satellite communication module for connecting to a satellite communication network.

[0065] Furthermore, the communication device T1 controls wireless communication between, for example, an external GNSS (Global Navigation Satellite System) measurement system and the shovel 100 .

[0066] The microphone array A1 includes a plurality of microphones and is configured to collect sounds generated around the shovel 100. In the present embodiment, the microphone array A1 is a plurality of microphones attached to the upper swing body 3.

[0067] Figure 3 1 is a diagram showing an example of the arrangement of microphones included in the microphone array A1 according to the present embodiment. Figure 3As shown, a microphone array A1 is provided at the front side of the cab 10. Specifically, the microphones constituting the microphone array A1 are arranged in a form of 3 microphones arranged in the Z-axis direction and 5 microphones arranged in the Y-axis direction, and a total of 15 microphones are arranged. The distance between the microphones constituting the microphone array A1 is set to 1. In addition, the distance between the microphones is only shown as an example and is determined according to the embodiment.

[0068] The microphones constituting the microphone array A1 are respectively capable of collecting sounds in a specified direction within a range of -90 degrees to +90 degrees, for example, centered on the front. The range in which the microphone array A1 can acquire sounds is set to at least be capable of acquiring sounds emitted within the imaging range of the front camera S6F. Thus, when the image information captured by the front camera S6F is displayed, it can be displayed so that the sounds emitted within the imaging range can be recognized. In addition, the display method of the image information will be described later. In the present embodiment, the range in which the microphone array A1 can collect sounds is merely shown as an example, and at least includes the imaging range of the front camera S6F.

[0069] In this embodiment, as an example of a plurality of sound collecting devices, Figure 3 The present embodiment merely shows an example of a plurality of sound collecting devices and is not limited to Figure 3 Specifically, the number of microphones is not limited to Figure 3 15 (3×5) are shown. For example, the number of microphones in the width direction (Y-axis direction) may be other than 5, and the number in the height direction (Z-axis direction) may be other than 3. Furthermore, the microphones are not limited to a two-dimensional arrangement in the Y-axis direction and the Z-axis direction, and may be a one-dimensional arrangement in the Y-axis direction, for example.

[0070] [Drive control system for excavators]

[0071] Figure 4 Yes means Figure 2 FIG. 1 is a diagram showing a configuration example of a drive control system of an excavator 100. Figure 4 In the figure, the mechanical power transmission system is represented by a double line, the working oil pipeline route is represented by a thick solid line, the pilot pipeline route is represented by a dotted line, and the electric drive / control system is represented by a dotted line.

[0072] The drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. As described above, the hydraulic drive system of the excavator 100 according to the present embodiment includes hydraulic actuators such as the travel hydraulic motors 1L and 1R, the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, which respectively drive the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

[0073] The engine 11 is a main power source in the hydraulic drive system, and is mounted, for example, at the rear of the upper swing body 3. Specifically, the engine 11 rotates constantly at a preset target speed under direct or indirect control of an excavator controller 30 described later, and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses diesel as fuel.

[0074] The regulator 13 controls the discharge amount of the main pump 14. For example, the regulator 13 adjusts the angle (deflection angle) of the swash plate of the main pump 14 according to a control command from the shovel controller 30. As described later, the regulator 13 includes, for example, regulators 13L and 13R.

[0075] The main pump 14 is mounted, for example, at the rear of the upper swing body 3, similarly to the engine 11, and supplies hydraulic oil to the control valve unit 17 through a high-pressure hydraulic line. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the excavator controller 30, the stroke length of the piston is adjusted by adjusting the deflection angle of the swash plate by the regulator 13, thereby controlling the discharge flow rate (discharge pressure). As described later, the main pump 14 includes, for example, main pumps 14L and 14R.

[0076] The control valve unit 17 is a hydraulic control device for controlling the hydraulic system in the excavator 100. In the present embodiment, the control valve unit 17 includes control valves 171 to 176. The control valve 175 includes a control valve 175L and a control valve 175R, and the control valve 176 includes a control valve 176L and a control valve 176R. The control valve unit 17 is configured to selectively supply the hydraulic oil discharged by the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control, for example, the flow rate of the hydraulic oil flowing from the main pump 14 to the hydraulic actuator and the flow rate of the hydraulic oil flowing from the hydraulic actuator to the hydraulic oil tank. The hydraulic actuator includes a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, travel hydraulic motors 1L, 1R, and a swing hydraulic motor 2A. More specifically, the control valve 171 corresponds to the left travel hydraulic motor 1L, the control valve 172 corresponds to the right travel hydraulic motor 1R, and the control valve 173 corresponds to the swing hydraulic motor 2A. Furthermore, the control valve 174 corresponds to the bucket cylinder 9, the control valve 175 corresponds to the boom cylinder 7, and the control valve 176 corresponds to the arm cylinder 8. Furthermore, as described later, the control valve 175 includes, for example, control valves 175L and 175R, and as described later, the control valve 176 includes, for example, control valves 176L and 176R. The details of the control valves 171 to 176 will be described later.

[0077] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to supply working oil to the hydraulic control device via a pilot line. In the present embodiment, the pilot pump 15 is a fixed capacity hydraulic pump. However, the pilot pressure generating device can also be implemented by the main pump 14. That is, in addition to the function of supplying working oil to the control valve unit 17 via the working oil pipeline, the main pump 14 can also have the function of supplying working oil to various hydraulic control devices via the pilot line. In this case, the pilot pump 15 can be omitted.

[0078] The operating device 26 is a device for an operator to operate the actuator. The actuator includes at least one of a hydraulic actuator and an electric actuator.

[0079] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the present embodiment, the discharge pressure sensor 28 outputs the detected value to the shovel controller 30. As described later, the discharge pressure sensor 28 includes, for example, discharge pressure sensors 28L and 28R.

[0080] The operation sensor 29 is configured to detect the operation content of the operator who uses the operating device 26. In the present embodiment, the operation sensor 29 detects the operation direction and operation amount of the operating device 26 corresponding to each actuator, and outputs the detected value to the excavator controller 30. In the present embodiment, the excavator controller 30 controls the opening area of ​​the proportional valve 31 according to the output of the operation sensor 29. In addition, the excavator controller 30 supplies the working oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the working oil supplied to each pilot port is, in principle, a pressure corresponding to the operation direction and operation amount of the operating device 26 corresponding to each hydraulic actuator. In this way, the operating device 26 is configured to be able to supply the working oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0081] The proportional valve 31, which functions as a control valve for controlling the equipment, is configured to be arranged in a pipeline connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and the flow path area of ​​the pipeline can be changed. In the present embodiment, the proportional valve 31 is operated according to the control command output by the shovel controller 30. Therefore, the shovel controller 30 can supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the proportional valve 31 regardless of the operation of the operating device 26 by the operator.

[0082] With this configuration, even when no operation is performed on a specific operating device 26 , the shovel controller 30 can operate the hydraulic actuator corresponding to the specific operating device 26 .

[0083] For example, the shovel controller 30 sets a target rotation speed according to an operation mode set in advance by a predetermined operation of an operator or the like, and performs drive control to rotate the engine 11 at a constant speed.

[0084] Furthermore, for example, the shovel controller 30 outputs a control command to the regulator 13 as necessary to change the discharge rate of the main pump 14 .

[0085] Furthermore, for example, the shovel controller 30 performs control related to a device guidance function that guides the operator's manual operation of the shovel 100 through the operating device 26. Furthermore, the shovel controller 30 performs control related to a device control function that automatically supports the operator's manual operation of the shovel 100 through the operating device 26.

[0086] In addition, part of the functions of the shovel controller 30 may be implemented by other controllers (control devices). That is, the functions of the shovel controller 30 may be implemented in a distributed manner by a plurality of controllers. For example, the equipment guidance function and the equipment control function may be implemented by a dedicated controller (control device).

[0087] [Hydraulic system of excavator]

[0088] Next, refer to Figure 5 A hydraulic system of the shovel 100 according to the present embodiment will be described.

[0089] Figure 5 It is a diagram schematically showing an example of the configuration of the hydraulic system of the shovel 100 according to the present embodiment.

[0090] In addition, Figure 5 In the figure, the mechanical power system, the working oil pipeline, the pilot pipeline and the electrical control system are represented by double lines, solid lines, dashed lines and dotted lines respectively.

[0091] The hydraulic system implemented by the hydraulic circuit circulates hydraulic oil from each main pump 14L, 14R driven by the engine 11 to the hydraulic oil tank through the center bypass oil passages C1L, C1R and the parallel oil passages C2L, C2R.

[0092] The center bypass oil passage C1L starts from the main pump 14L, passes through the control valves 171 , 173 , 175L, and 176L arranged in the control valve unit 17 in sequence, and then reaches the hydraulic oil tank.

[0093] The center bypass oil passage C1R starts from the main pump 14R, passes through the control valves 172 , 174 , 175R, and 176R arranged in the control valve unit 17 in sequence, and then reaches the hydraulic oil tank.

[0094] The control valve 171 is a spool valve that supplies hydraulic oil discharged from the main pump 14L to the traveling hydraulic motor 1L and discharges the hydraulic oil discharged from the traveling hydraulic motor 1L to the hydraulic oil tank.

[0095] The control valve 172 is a spool valve that supplies hydraulic oil discharged from the main pump 14R to the traveling hydraulic motor 1R and discharges the hydraulic oil discharged from the traveling hydraulic motor 1R to the hydraulic oil tank.

[0096] The control valve 173 is a spool valve that supplies hydraulic oil discharged from the main pump 14L to the swing hydraulic motor 2A and discharges the hydraulic oil discharged from the swing hydraulic motor 2A to the hydraulic oil tank.

[0097] The control valve 174 is a spool valve that supplies hydraulic oil discharged from the main pump 14R to the bucket cylinder 9 and discharges the hydraulic oil in the bucket cylinder 9 to the hydraulic oil tank.

[0098] The control valves 175L and 175R are spool valves for supplying hydraulic oil discharged from the main pumps 14L and 14R to the boom cylinder 7 and for discharging the hydraulic oil in the boom cylinder 7 to the hydraulic oil tank, respectively.

[0099] The control valves 176L and 176R supply the hydraulic oil discharged from the main pumps 14L and 14R to the arm cylinder 8 and discharge the hydraulic oil in the arm cylinder 8 to the hydraulic oil tank.

[0100] The control valves 171 , 172 , 173 , 174 , 175L, 175R, 176L, and 176R adjust the flow rate of hydraulic oil supplied to or discharged from the hydraulic actuator or switch the flow direction of the hydraulic oil according to the pilot pressure acting on the pilot port.

[0101] The parallel oil passage C2L supplies the working oil of the main pump 14L to the control valves 171, 173, 175L, and 176L in parallel with the center bypass oil passage C1L. Specifically, the parallel oil passage C2L is configured to branch from the center bypass oil passage C1L on the upstream side of the control valve 171, and can supply the working oil of the main pump 14L in parallel with each of the control valves 171, 173, 175L, and 176R. Thus, when the flow of the working oil through the center bypass oil passage C1L is restricted or cut off by any one of the control valves 171, 173, and 175L, the parallel oil passage C2L can supply the working oil to the control valve further downstream.

[0102] The parallel oil passage C2R supplies the working oil of the main pump 14R to the control valves 172, 174, 175R, and 176R in parallel with the central bypass oil passage C1R. Specifically, the parallel oil passage C2R is configured to branch from the central bypass oil passage C1R on the upstream side of the control valve 172, and can supply the working oil of the main pump 14R in parallel with each of the control valves 172, 174, 175R, and 176R. When the flow of the working oil through the central bypass oil passage C1R is restricted or cut off by any one of the control valves 172, 174, and 175R, the parallel oil passage C2R can supply the working oil to the control valve further downstream.

[0103] The regulators 13L and 13R adjust the discharge amounts of the main pumps 14L and 14R by adjusting the tilt angles of the swash plates of the main pumps 14L and 14R, respectively, under the control of the shovel controller 30 .

[0104] The discharge pressure sensor 28L detects the discharge pressure of the main pump 14L, and a detection signal corresponding to the detected discharge pressure is input to the shovel controller 30. The same is true for the discharge pressure sensor 28R. Thus, the shovel controller 30 can control the regulators 13L and 13R according to the discharge pressure of the main pumps 14L and 14R.

[0105] In the center bypass oil passages C1L and C1R, negative control throttles (hereinafter referred to as "negative control throttles") 18L and 18R are provided between the control valves 176L and 176R at the most downstream and the hydraulic oil tank. As a result, the flow of the hydraulic oil discharged by the main pumps 14L and 14R is restricted by the negative control throttles 18L and 18R. In addition, the negative control throttles 18L and 18R generate control pressures (hereinafter referred to as "negative control pressures") for controlling the regulators 13L and 13R.

[0106] The negative control pressure sensors 19L and 19R detect the negative control pressure, and a detection signal corresponding to the detected negative control pressure is input to the shovel controller 30 .

[0107] The excavator controller 30 can control the regulators 13L and 13R according to the discharge pressure of the main pumps 14L and 14R detected by the discharge pressure sensors 28L and 28R, thereby adjusting the discharge volume of the main pumps 14L and 14R. For example, the excavator controller 30 can control the regulator 13L according to the increase in the discharge pressure of the main pump 14L, adjust the swash plate deflection angle of the main pump 14L, and thereby reduce the discharge volume. The same is true for the regulator 13R. Thus, the excavator controller 30 can perform total horsepower control of the main pumps 14L and 14R so that the absorption horsepower of the main pumps 14L and 14R represented by the product of the discharge pressure and the discharge volume does not exceed the output horsepower of the engine 11.

[0108] Furthermore, the excavator controller 30 can control the regulators 13L and 13R according to the negative control pressure detected by the negative control pressure sensors 19L and 19R, thereby adjusting the discharge amount of the main pumps 14L and 14R. For example, the excavator controller 30 reduces the discharge amount of the main pumps 14L and 14R as the negative control pressure increases, and increases the discharge amount of the main pumps 14L and 14R as the negative control pressure decreases.

[0109] Specifically, in the standby state ( Figure 5 In the case of the state shown in FIG. 1 , the hydraulic oil discharged from the main pumps 14L and 14R reaches the negative control throttles 18L and 18R through the central bypass oil passages C1L and C1R. Furthermore, the flow of the hydraulic oil discharged from the main pumps 14L and 14R increases the negative control pressure generated upstream of the negative control throttles 18L and 18R. As a result, the excavator controller 30 reduces the discharge amount of the main pumps 14L and 14R to the minimum allowable discharge amount, thereby suppressing the pressure loss (pumping loss) of the discharged hydraulic oil when passing through the central bypass oil passages C1L and C1R.

[0110] On the other hand, when one of the hydraulic actuators is operated by the operating device 26, the hydraulic oil discharged from the main pumps 14L and 14R flows into the hydraulic actuator to be operated via the control valve corresponding to the hydraulic actuator to be operated. In addition, the flow of the hydraulic oil discharged from the main pumps 14L and 14R reduces or eliminates the amount reaching the negative control throttles 18L and 18R, thereby reducing the negative control pressure generated upstream of the negative control throttles 18L and 18R. As a result, the excavator controller 30 can increase the discharge amount of the main pumps 14L and 14R, so that sufficient hydraulic oil circulates in the hydraulic actuator to be operated, thereby reliably driving the hydraulic actuator to be operated.

[0111] The operating device 26 includes a left operating lever 26L, a right operating lever 26R, and a travel lever 26D. The travel lever 26D includes a left travel lever 26DL and a right travel lever 26DR.

[0112] The left operation lever 26L and the right operation lever 26R receive operations in one or more directions of the front-rear direction and the left-right direction.

[0113] The left operating lever 26L is used for the swing operation and the operation of the boom 5. When the left operating lever 26L is operated in the forward and backward direction, the control pressure corresponding to the lever operation amount is introduced into the pilot ports of the control valves 176L and 176R by the hydraulic oil discharged from the pilot pump 15. And, when the left operating lever 26L is operated in the left and right direction, the control pressure corresponding to the lever operation amount is introduced into the pilot port of the control valve 173 by the hydraulic oil discharged from the pilot pump 15.

[0114] Specifically, when the left operating lever 26L is operated in the boom retracting direction, the shovel controller 30 introduces the hydraulic oil into the right pilot port of the control valve 176L and the left pilot port of the control valve 176R. Furthermore, when the left operating lever 26L is operated in the boom opening direction, the shovel controller 30 introduces the hydraulic oil into the left pilot port of the control valve 176L and the right pilot port of the control valve 176R. Furthermore, when the left operating lever 26L is operated in the left rotation direction, the shovel controller 30 introduces the hydraulic oil into the left pilot port of the control valve 173, and when the left operating lever 26L is operated in the right rotation direction, the shovel controller 30 introduces the hydraulic oil into the right pilot port of the control valve 173. In order to perform this control, the shovel controller 30 uses the outputs from the operation sensors 29LA and 29LB.

[0115] The right operating lever 26R is used to operate the boom 4 and the bucket 6. When the right operating lever 26R is operated in the forward and backward directions, the hydraulic oil discharged from the pilot pump 15 introduces a control pressure corresponding to the lever operation amount into the pilot port of the control valve 175. And when it is operated in the left and right directions, the hydraulic oil discharged from the pilot pump 15 introduces a control pressure corresponding to the lever operation amount into the pilot port of the control valve 174.

[0116] Specifically, if the right operating lever 26R is operated in the boom lowering direction, the shovel controller 30 introduces the hydraulic oil into the left pilot port of the control valve 175R. Furthermore, if the right operating lever 26R is operated in the boom raising direction, the shovel controller 30 introduces the hydraulic oil into the right pilot port of the control valve 175L and introduces the hydraulic oil into the left pilot port of the control valve 175R. Furthermore, if the right operating lever 26R is operated in the bucket retracting direction, the shovel controller 30 introduces the hydraulic oil into the right pilot port of the control valve 174, and if the right operating lever 26R is operated in the bucket opening direction, the shovel controller 30 introduces the hydraulic oil into the left pilot port of the control valve 174. In order to perform this control, the shovel controller 30 uses the outputs from the operation sensors 29RA and 29RB.

[0117] The left travel rod 26DL is used for operating the left crawler 1CL. It can also be configured to be linked with the left travel pedal. If the left travel rod 26DL is operated in the forward and backward directions, the excavator controller 30 uses the working oil discharged from the pilot pump 15 to introduce the control pressure corresponding to the rod operation amount into the pilot port of the control valve 171. The right travel rod 26DR is used for operating the right crawler 1CR. It can also be configured to be linked with the right travel pedal. If the right travel rod 26DR is operated in the forward and backward directions, the excavator controller 30 uses the working oil discharged from the pilot pump 15 to introduce the control pressure corresponding to the rod operation amount into the pilot port of the control valve 172. In order to perform this control, the excavator controller 30 uses the outputs from the operation sensors 29DL and 29DR described later.

[0118] The operation sensor 29 is configured to detect the contents of the operation performed by the operator on the operation device 26. In the present embodiment, the operation sensor 29 detects the operation direction and the operation amount of the operation device 26 corresponding to each actuator, and outputs the detected values ​​to the shovel controller 30.

[0119] The operation sensor 29 includes operation sensors 29LA, 29LB, 29RA, 29RB, 29DL, and 29DR. The operation sensor 29LA detects the content of the operation performed by the operator on the left operation lever 26L in the front-rear direction, and outputs the detected value to the shovel controller 30. The operation content is, for example, the lever operation direction, the lever operation amount (lever operation angle), etc.

[0120] Similarly, the operation sensor 29LB detects the contents of the operation performed by the operator on the left operating lever 26L in the left-right direction, and outputs the detected value to the shovel controller 30. The operation sensor 29RA detects the contents of the operation performed by the operator on the right operating lever 26R in the front-back direction, and outputs the detected value to the shovel controller 30. The operation sensor 29RB detects the contents of the operation performed by the operator on the right operating lever 26R in the left-right direction, and outputs the detected value to the shovel controller 30.

[0121] The operation sensor 29DL detects the operator's operation on the left travel rod 26DL in the front-rear direction and outputs the detected value to the shovel controller 30. The operation sensor 29DR detects the operator's operation on the right travel rod 26DR in the front-rear direction and outputs the detected value to the shovel controller 30.

[0122] The shovel controller 30 receives the output of the operation sensor 29, and outputs a control command to the regulator 13 as needed to change the discharge amount of the main pump 14. In addition, the shovel controller 30 receives the output of the control pressure sensor 19 provided upstream of the throttle 18, and outputs a control command to the regulator 13 as needed to change the discharge amount of the main pump 14. The throttle 18 includes a left throttle 18L and a right throttle 18R, and the control pressure sensor 19 includes negative control pressure sensors 19L and 19R.

[0123] <Framework structure of remote operation system>

[0124] Figure 6 1 is a functional block diagram showing a configuration example of a remote operating system SYS according to the present embodiment. Figure 6 In the illustrated example, the block configurations of the remote operation room RC included in the remote operation system SYS and the shovel 100 are shown. In addition, since the hardware configuration of the shovel 100 is as described above, the description thereof is omitted.

[0125] <Structure of Remote Control Room RC>

[0126] The remote control room RC includes a remote controller 40, a communication device T2, a speaker A2, an operation sensor 43, and a display device D1. The speaker A2, the communication device T2, and the operation sensor 43 have been described above, and thus their description is omitted.

[0127] Next, the remote control room RC will be described. Figure 7 2 is a diagram showing an example of arrangement of a remote operation room RC. A plurality of operation devices 42 are installed in the remote operation room RC with the operation seat DS as a reference.

[0128] In this embodiment, if Figure 7As shown, the display device D1 is a multi-screen display consisting of nine monitors in three vertical rows and three horizontal columns. Specifically, the display device D1 includes a central monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i.

[0129] In the present embodiment, a speaker A2 is provided above the display device D1. The speaker A2 is provided, for example, to output sounds collected by the microphone array A1 of the shovel 100.

[0130] <<Functional frame of excavator>>

[0131] Back to Figure 6 , the functional blocks within the excavator controller 30 of the excavator 100 are explained. The functional blocks within the excavator controller 30 are conceptual and do not necessarily need to be physically constructed as shown in the figure. All or part of each functional block can be functionally or physically dispersed / integrated in arbitrary units. All or any part of each processing function performed in each functional block is implemented by a program executed by the CPU. Alternatively, each functional block can also be implemented in the form of hardware based on wired logic. The excavator controller 30 includes an acquisition unit 301, an excavator state determination unit 302, a map generation unit 303, a sending control unit 304, a receiving control unit 305 and an actuator drive unit 306 by implementing the program.

[0132] Figure 8 2 is a conceptual diagram showing the surrounding conditions of the shovel 100 according to the present embodiment. Figure 8 In the example shown, it is assumed that the shovel 100 , the worker 801 , and the truck 802 exist.

[0133] The worker 801 is talking to the shovel 100. Since the truck 802 is in operation, the engine sound is transmitted to the shovel 100.

[0134] The imaging area of ​​the front camera S6F of the shovel 100 includes the worker 801 and the truck 802 .

[0135] Furthermore, the microphone array A1 collects the voice of the worker 801 and the engine sound of the truck 802 .

[0136] Then, the shovel controller 30 transmits a sound signal representing the sound collected by the microphone array A1 of the shovel 100 to the communication device T2 of the remote operation room RC via the communication device T1. Furthermore, the shovel 100 generates information representing the generation status of the sound around the shovel 100 (equivalent to the sound heat map described later) based on the sound signal acquired by the microphone array A1, and transmits the generated information from the communication device T1 to the communication device T2 of the remote operation room RC. The specific structure of the shovel 100 for realizing the transmission of this information is described.

[0137] Back to Figure 6 , the acquisition unit 301 acquires signals from various detection devices installed in the excavator 100. For example, the acquisition unit 301 acquires the detection results of the boom angle sensor S1, the dipper angle sensor S2, and the bucket angle sensor S3. In addition, the acquisition unit 301 acquires the measurement results of the position and direction of the excavator 100 from the positioning device S7. In addition, the acquisition unit 301 acquires image information from the camera device S6. In addition, the acquisition unit 301 acquires the sound signals collected by each microphone constituting the microphone array A1 from the microphone array A1.

[0138] The excavator state determination unit 302 is configured to determine the state of the excavator 100 based on the signal acquired by the acquisition unit 301. In the present embodiment, the state of the excavator 100 includes the position and orientation of the excavator 100 and the state of the attachments of the excavator 100 (for example, the positions of the boom 4, the arm 5, and the bucket 6). The position of the excavator 100 is, for example, the position in the reference coordinate system of the excavator 100 (the latitude, longitude, and altitude of the reference point of the excavator 100). The excavator state determination unit 302 determines the position and orientation of the excavator 100 based on the output of the positioning device S7.

[0139] The state of the attachment (for example, the position of the boom 4, the arm 5 and the bucket 6) can be determined based on the detection results of the angle sensors (the boom angle sensor S1, the arm angle sensor S2 and the bucket angle sensor S3) and the respective sizes of the boom 4, the arm 5 and the bucket 6.

[0140] The map generation unit 303 generates a sound heat map indicating the strength of sounds around the shovel 100 based on sound signals collected by the microphones constituting the microphone array A1 .

[0141] The heat map of the sound involved in this embodiment is set as a two-dimensional map in the height direction (Z-axis direction) of the excavator 100 and the width direction (Y-axis direction) of the excavator 100, and is set as a map showing the strength of the collected sound in each sound collection direction within the range that the microphone array A1 can collect sound. The strength of the sound indicates the amplitude of the sound signal.

[0142] Fig. 9 : is a conceptual diagram showing that a sound heat map is generated based on a sound signal of the microphone array A1 in the map generation unit 303 according to the present embodiment. Fig. 9 In the illustrated example, for convenience of explanation, an example in which the microphone array A1 is composed of microphones A1a to A1e is described.

[0143] like Fig. 9 As shown, microphones A1a to A1e of microphone array A1 output sound signals collected in each sound collecting direction within the sound collecting range to shovel controller 30. Then, the sound signals acquired by acquisition unit 301 of shovel controller 30 are output to map generation unit 303.

[0144] The map generation unit 303 sets delay devices corresponding to the sound collection directions for the sound signals collected by the microphones A1a to A1e, respectively.

[0145] exist Fig. 9 In the example shown, in order to identify the strength of the sound signal collected from the direction inclined at an angle θ from the front along the width direction (Y-axis direction), the map generation unit 303 assigns a corresponding delay 303a to the microphone A1a, a corresponding delay 303b to the microphone A1b, a corresponding delay 303c to the microphone A1c, and a corresponding delay 303d to the microphone A1d. The delays 303a to 303d are determined according to the angle θ indicating the sound collection direction.

[0146] For example, the delay device 303a uses “e -t4·s " generates a sound signal taking into account the delay of time t4. s is set to be a Laplace operator. Time t4 represents the delay time from the time when microphone A1a collects the sound to the time when microphone A1e collects the sound when the sound collection direction is inclined at an angle θ along the width direction. Specifically, time t4 can be calculated according to the formula "t4=4·l·sinθ / v". Distance l is the distance between the microphones, and the speed of sound is defined as v. The specific operations performed by delay devices 303a to 303d may be performed using known methods, so their description is omitted. In addition, the speed of sound v may be a fixed value or may be calculated based on the detected temperature, etc. In addition, the present embodiment only shows an example of delay devices 303a to 303d, and any known circuit may be used as long as the circuit takes into account the delay of the sound signal.

[0147] The delay device 303b uses the “e -t3·s" generates a sound signal taking into account the delay of time t3. Time t3 represents the delay time from when microphone A1b collects sound to when microphone A1e collects sound when the sound collection direction is inclined at an angle θ along the width direction. Specifically, time t3 can be calculated according to the formula "t3=3·l·sinθ / v".

[0148] The delay device 303c uses the “e -t2·s " generates a sound signal taking into account the delay of time t2. Time t2 represents the delay time from when microphone A1c collects sound to when microphone A1e collects sound when the sound collection direction is inclined at an angle θ along the width direction. Specifically, time t2 can be calculated according to the formula "t2=2·l·sinθ / v".

[0149] The delay device 303d uses the “e -t1·s " generates a sound signal taking into account the delay of time t1. Time t1 represents the delay time from when microphone A1d collects sound to when microphone A1e collects sound when the sound collection direction is inclined at an angle θ along the width direction. Specifically, time t1 can be calculated according to the formula "t1=l·sinθ / v".

[0150] The operator 303f generates a sound signal that emphasizes the sound from the direction of the tilt angle θ among the sound signals collected by the microphones A1a to A1e, based on the values ​​output from the delay devices 303a to 303d and the sound signal output from the microphone A1e.

[0151] The map generation unit 303 generates a heat map 303g of the sound based on the amplitude of the sound signal generated by the operator 303f. Specifically, a corresponding area (e.g., pixel) is pre-set for each sound collection direction in the heat map 303g. Then, the map generation unit 303 assigns a color corresponding to the amplitude of the sound signal of the sound in the sound collection direction corresponding to the area to the area (e.g., pixel) of the heat map 303g. For example, the map generation unit 303 assigns colors as follows: the larger the amplitude, the redder (or, the darker), and the smaller the parameter, the whiter.

[0152] The map generation unit 303 generates a sound signal emphasizing the sound in each sound collection direction within the range in which the microphone array A1 can collect sound, and assigns a color corresponding to the amplitude of the sound signal, thereby generating a sound heat map 303g.

[0153] In addition, Fig. 9In the example shown, for the sake of convenience, the example of tilting the angle θ along the Y axis is described, but the direction of collecting sound is a combination of the Y axis direction and the Z axis direction. The strength of the sound can also be identified by the above processing for this combined direction, so the description is omitted.

[0154] The sending control unit 304 controls the sending of various information based on the acquisition result of the acquisition unit 301 to the remote operation room RC via the communication device T1. For example, the sending control unit 304 controls the sending of image information captured by the camera S6, position information indicating the position and orientation of the excavator 100, status information indicating the status of the accessories, sound signals collected by the microphone array A1, and a heat map of the sound to the remote operation room RC. The sending by the sending control unit 304 is performed at predetermined intervals. The predetermined time may be any time, but is set to a time interval that can identify a condition that has changed due to the operation of the excavator 100. For example, the sending control unit 304 may send the above information at intervals of 1 second.

[0155] The reception control unit 305 performs control for receiving various information from the remote operation room RC via the communication device T1. For example, the reception control unit 305 receives an operation signal for controlling the operation of the shovel 100 from the remote operation room RC.

[0156] The actuator driving unit 306 is configured to drive an actuator mounted on the shovel 100. In the present embodiment, the actuator driving unit 306 generates and outputs operation signals for each of the plurality of electromagnetic valves included in the proportional valve 31 based on an operation signal transmitted from the remote operation room RC.

[0157] Each solenoid valve receiving the actuation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0158] <<Functional frame of remote operation room>>

[0159] The functional blocks in the remote controller (an example of a control unit) 40 of the remote operation room RC are described. The functional blocks in the remote controller 40 are conceptual and do not necessarily need to be physically configured as shown in the figure. All or part of each functional block can be functionally or physically dispersed / integrated in any unit. All or any part of each processing function performed in each functional block is implemented by a program executed by the CPU. Alternatively, each functional block can also be implemented in the form of hardware based on wired logic. The remote controller 40 has a receiving control unit 401, a display screen generation unit 402, an output control unit 403, a signal generation unit 404 and a sending control unit 405 by implementing the program.

[0160] In this embodiment, the remote controller 40 in the remote operation room RC is described as a control unit of the display device (an example of a display unit) D1. That is, the structure in the remote operation room RC functions as a display device for supporting remote operation of the shovel 100.

[0161] The reception control unit 401 performs control for receiving various information from the shovel 100 via the communication device T2.

[0162] As another example, the receiving control unit 401 receives (acquires) image information captured by the camera S6, position information, state information, sound signals acquired by the microphone array A1, and a sound heat map (an example of information related to sound) from the excavator 100. The image information is set to information captured by the camera S6. The position information is set to information indicating the position and orientation of the excavator 100 in the reference coordinate system measured by the positioning device S7. The state information is set to information indicating the state of the attachment (for example, the position of the boom 4, the arm 5, and the bucket 6). The sound heat map is set to be graph information generated by the graph generation unit 303 indicating the strength of the sound around the excavator 100.

[0163] The display screen generator 402 generates a display image by superimposing a sound heat map indicating the sound generation status around the shovel 100 on the image information. Specifically, the display screen generator 402 generates a display image by superimposing the sound heat map so that the position indicated by the image information corresponds to the sound collection direction of the heat map.

[0164] The output control unit 403 outputs the sound signal of the microphone array A1 from the speaker A2. The method of outputting the sound signals from the plurality of microphones from the speaker A2 may use any method, whether or not it is a publicly known method.

[0165] Furthermore, the output control unit 403 outputs the display image generated by the display screen generation unit 402 to the display device D1 .

[0166] The signal generation unit 404 generates an operation signal for controlling the operation of the shovel 100 according to the operation received by the operation sensor 43. The transmission control unit 405 transmits the generated operation signal to the shovel 100.

[0167] Fig.10 FIG. 1 is a diagram showing an example of a display screen displayed on the display device D1 according to the present embodiment. Fig.10In the example shown, the worker 801 and the truck 802 represented by the image information captured by the front camera S6F are displayed. Furthermore, the sound detection areas 1001 and 1002 represented by the sound heat map are superimposed on the image information. The sound detection areas 1001 and 1002 are displayed semi-transparently (the transmittance is arbitrary).

[0168] The sound detection area 1001 is an area displayed based on the heat map of the sound. The sound detection area 1001 includes a dark red area 1001a, a red area 1001b, an orange area 1001c, a yellow area 1001d, and a light yellow area 1001e. In the sound detection area 1001, it is indicated that the sound decreases as it changes from area 1001a to area 1001e. In other words, it is indicated that area 1001a is near the sound source and the operator 801 is making a sound.

[0169] Similarly, the sound detection area 1002 includes a red area 1002a, an orange area 1002b, a yellow area 1002c, and a light yellow area 1002d. In the sound detection area 1002, it is also shown that the sound decreases as it changes from area 1002a to area 1002d. In other words, it means that area 1002a is near the sound source and the engine of the truck 802 is running.

[0170] like Fig.10 As shown in the screen, in the present embodiment, the output control unit 403 displays in a display manner that can identify the strength of the sound obtained from the direction corresponding to each area displayed on the screen as the sound generation status around the excavator 100. In addition, in the present embodiment, an example of indicating the strength of the sound in the form of color is shown, but it is not limited to the example of indicating the strength of the sound in the form of color, and other methods can also be used. For example, it can also be a shadow, and the brightness can be changed. In the present embodiment, by displaying the display screen in a manner that can identify the strength of the sound, the operator can grasp the source of the sound. That is, when the sound is heard through the speaker A2, the operator can recognize the source of the sound, and thus can grasp the current situation. Then, the operator can perform operations corresponding to the current situation. Therefore, it is possible to achieve improved safety.

[0171] In the heat map of sound used to generate the display screen involved in this embodiment, the strength of the sound acquired for each direction included in the range where the microphone array A1 can collect sound is shown in the form of color. Thus, in the case where the microphone array A1 collects sound from multiple sources, in the display image output by the output control unit 403, the sources of multiple sounds are represented as multiple sound detection areas. That is, in the case where the sound signal acquired by the microphone array A1 includes multiple sources of sound, the output control unit 403 can display it in a display mode that can separately identify the sources of the multiple sounds. Thus, the operator can operate after identifying the sources of multiple sounds, thereby achieving improved safety.

[0172] In addition, the camera area captured by the front camera S6F and the range in which the microphone array A1 can collect sound may be different. In this case, the microphone array A1 may also obtain a sound signal from the source of the sound that is not indicated in the camera area captured by the front camera S6F. In this case, even if the source is not indicated in the image information, the display screen generation unit 402 may generate a display image by superimposing the sound detection area on the image information.

[0173] Fig.11 FIG. 1 is a diagram showing an example of a display screen displayed on the display device D1 according to the present embodiment. Fig.11 In the example shown, the image information captured by the front camera S6F does not capture the source of the sound, but the sound detection area 1101 indicating that the source of the sound exists on the right side is superimposed on the image information and displayed.

[0174] The sound detection area 1101 is an area displayed according to the heat map of the sound. The sound detection area 1101 includes an orange area 1101a, a yellow area 1101b, and a light yellow area 1101c. And, when the sound is output from the speaker A2, the operator can recognize that the source of the sound is on the right side by recognizing the sound detection area 1101.

[0175] Fig.11 The sound detection area 1101 shown is merely an example of a display method, and the display method is not limited to this display method. That is, any display method may be used as long as it can recognize that a sound signal is acquired from a direction outside the area indicated by the image information.

[0176] In this way, when a sound signal is acquired from a direction outside the area represented by the image information captured by the front camera S6F, the output control unit 403 according to the present embodiment displays the fact that the sound signal is acquired from outside the area represented by the image information on the display device D1. Thus, even if it is not displayed on the display device D1, the operator can infer the direction from which the sound is coming. The operator can perform the operation after inferring the direction from which the sound is coming, thereby improving safety.

[0177] In the present embodiment, an example of generating a sound heat map (an example of information related to sound) by the excavator controller 30 is described. However, the heat map of sound (an example of information related to sound) is not limited to being generated on the excavator controller 30 side, and for example, it can also be generated on the remote controller 40 side.

[0178] (Variation 1 of the First Embodiment)

[0179] In the above embodiment, an example of superimposing a sound detection area indicating the strength of a sound on image information for display is described. However, the display method is not limited to the above embodiment. For example, the content of the speech may also be displayed in character information.

[0180] The display screen generator 402 of the remote controller 40 extracts the sound included in the sound signal received by the reception controller 401, and converts the sound into character information. A known method can be used as a sound recognition method to convert the sound into characters.

[0181] Then, the display screen generation unit 402 generates a display screen by superimposing the converted character information together with the sound detection area on the image information.

[0182] Then, the output control unit 403 displays the display screen on which the character information is superimposed. Fig.12 FIG. 1 is a diagram showing an example of a display screen displayed on the display device D1 according to the present embodiment. Fig.12 In the example shown, the character string information 1201 is superimposed on the image information captured by the front camera S6F and displayed together with the sound detection areas 1001 and 1002 .

[0183] The sound represented by the character string information 1201 is output from the speaker A2, but in this modification, it is also displayed in the form of the character string information 1201. Thus, even if the sound output from the speaker A2 is missed or not heard, the operator can recognize the content of the speech, and can therefore operate the shovel 100 in consideration of the content of the speech. Thus, safety can be improved.

[0184] (Variation 2 of the first embodiment)

[0185] In the above embodiment, the following example is described: the sound signal acquired from a specific direction is synthesized based on the delay sum of the sound signals collected by the microphones constituting the microphone array A1, thereby identifying the strength of the sound in the specific direction and generating the recognition result as a heat map of the sound. However, the method of identifying the generation condition of the sound is not limited to the method in the above embodiment.

[0186] For example, the sound signals collected by the microphones constituting the microphone array A1 are a combination of sound signals from different sources. Therefore, the image generation unit 303 can also separate the sound signals obtained from each microphone into different sound signals according to the source, and determine the source of the sound based on the time difference between the separated sound signals reaching the microphone. Then, the sending control unit 304 sends the sound-related information indicating the source of the sound and the intensity of the separated sound signal to the remote controller 40. Then, the display screen generation unit 402 superimposes the sound detection area indicating the source of the sound and the intensity of the sound on the image information to generate a display image. Thereby, the same display screen as the above-mentioned embodiment can be displayed. As a method for extracting different sound signals by source, for example, a known method of combining replication and filtering processing of sound signals can be used.

[0187] (Second embodiment)

[0188] In the above-mentioned embodiment and modification, a method for displaying a sound heat map indicating the generation status of sounds around the excavator 100 is shown. In order to perform this display, a case where sounds are collected from the worker 801 and the truck 802 as sounds around the excavator 100 is described. However, the objects from which the microphone array A1 collects sounds are not limited to objects existing around the excavator 100, but also include objects existing inside the excavator 100. Therefore, in the second embodiment, a case where sounds from objects existing inside the excavator 100 are considered is described.

[0189] Fig.13 2 is a conceptual diagram showing the surrounding conditions of the shovel 100 according to the present embodiment. Fig.13 In the example shown, it is assumed that there are a worker 801 and a truck 802 around the shovel 100. Fig.13 In the illustrated example, the microphone array A1 collects the engine sound of the engine 11 mounted on the shovel 100 .

[0190] When the sound output from the speaker A2 includes the engine sound of the engine 11, it may be difficult for the operator in the remote control room RC to understand the situation around the shovel 100. Therefore, in the present embodiment, control is performed to suppress the engine sound emitted from the engine 11 of the shovel 100. In the present embodiment, an example of filtering the engine sound is described as a method of suppressing the engine sound. In addition, in the present embodiment, an example of filtering the engine sound is described as a method of suppressing the engine sound, but other methods may also be applied as a method of suppressing the engine sound.

[0191] Fig.14 This is a conceptual diagram showing the structure of the remote control room RC until the sound is output in this embodiment. Fig.14 In the example shown, the shovel 100 is provided with a shovel controller 30A. Furthermore, the remote controller 40A is installed in the remote control room RC.

[0192] exist Fig.14 In the example shown, the remote controller 40A is assumed to include a reception control unit 401 , an acoustic separation unit 1401 , an attenuation unit 1402 , an adder 1403 , a map generator 303A, an output control unit 403 , and a display screen generator 402 .

[0193] In the present embodiment, the remote controller 40A filters the engine sound. That is, in the above embodiment, the map generation unit 303 is provided in the shovel controller 30. However, in the present embodiment, the map generation unit 303A is provided in the remote controller 40A.

[0194] exist Fig.14 In the example shown, the shovel controller 30A transmits a sound signal representing the sound collected from the microphone array A1 to the communication device T2 of the remote operation room RC via the communication device T1.

[0195] The acoustic separation unit 1401 separates the sound signals acquired by each microphone included in the sound signal received by the reception control unit 401 into different sound signals according to their sources. Fig.14 In the example shown, the sound signal is separated into the sound signal of the engine sound, the sound signal of the worker 801 , and the sound signal of the truck 802 .

[0196] The acoustic separation unit 1401 according to the present embodiment stores characteristic information (for example, frequency band and waveform) representing the engine sound of the engine 11. Therefore, the acoustic separation unit 1401 can identify the sound signal representing the engine sound from the separated sound signal.

[0197] Then, the acoustic separation unit 1401 outputs the sound signal indicating the engine sound to the attenuation unit 1402. On the other hand, the acoustic separation unit 1401 outputs the sound signal other than the engine sound to the map generation unit 303A and the addition unit 1403.

[0198] The map generation unit 303A generates a sound heat map based on the sound signal obtained by each microphone constituting the microphone array A1 without the engine sound. The generation method is the same as that of the second modification of the first embodiment, and the description thereof is omitted.

[0199] The attenuation unit 1402 reduces the amplitude of the sound signal representing the engine sound. That is, in this embodiment, the attenuation unit 1402 reduces the amplitude of the sound signal representing the engine sound, thereby attenuating a predetermined frequency component corresponding to the engine sound. The degree of reduction may be determined according to the embodiment.

[0200] Then, the adding unit 1403 adds the sound signal representing the engine sound attenuated by the attenuating unit 1402 and the other sound signal. Thus, the adding unit 1403 generates a sound signal in which the engine sound generated by the engine (drive source) of the shovel (an example of a construction machine) 100 is attenuated.

[0201] Then, the output control unit 403 performs control for outputting the added audio signal from the speaker A2. At this time, the output control unit 403 can synthesize the audio signals of the microphones constituting the microphone array A1 so that the audio signals can be output from the speaker A2.

[0202] In the present embodiment, the remote controller 40 performs the above-described control to attenuate the engine sound, and the operator can easily grasp the situation around the shovel 100 through the sound output from the speaker A2.

[0203] In the present embodiment, the engine sound is attenuated in the sound output from the speaker A2, so that the stress caused by the sound to the operator can be reduced.

[0204] Furthermore, by attenuating a predetermined frequency component such as the engine sound from the sound output from the speaker A2, the sound around the shovel 100 becomes easier to hear, so that, for example, the surrounding conditions can be easily understood from the human voice or abnormal sound around the shovel 100.

[0205] (Variation 1 of the Second Embodiment)

[0206] In the above embodiment, an example in which the acoustic separation unit 1401 separates different sound signals according to the source is described. However, as a method of suppressing the engine sound, it is not limited to the method of extracting the sound signal representing the engine sound and then attenuating the sound signal. In this modification, an example is set to provide a band stop filter instead of the acoustic separation unit 1401, the attenuation unit 1402 and the addition unit 1403. In addition, in this modification, the graph generation unit can be provided in the excavator controller or in the remote controller.

[0207] In this modification, the band rejection filter attenuates the frequency band corresponding to the engine sound from the sound signal acquired by each microphone included in the sound signal received by the reception control unit 401. Then, the output control unit 403 outputs the sound signal with the attenuated frequency band corresponding to the engine sound from the speaker A2.

[0208] In addition, in this modification, an example of using a band-stop filter as a filter for attenuating the frequency band corresponding to the engine sound is described, but the filter for attenuating the frequency band corresponding to the engine sound is not limited to the band-stop filter, and a notch filter may also be used. In other words, any filter may be used as long as the attenuation frequency band is set to attenuate the frequency band corresponding to the engine sound without attenuating other sounds.

[0209] In this modification, by performing the above-mentioned control, it is possible to produce the same effects as those of the second embodiment.

[0210] In the above-mentioned embodiment and this modification, an example of reducing the amplitude of the sound signal representing the engine sound as a filtering method is described. However, this embodiment only shows an example of filtering and is not limited to this filtering method. That is, as long as the operator in the remote control room RC attenuates the sound that hinders the recognition of the conditions around the excavator 100 according to the prescribed conditions when operating the excavator 100, it will be sufficient. Next, another example of filtering according to the prescribed conditions will be described.

[0211] (Variation 2 of the Second Embodiment)

[0212] In the above embodiment, an example of attenuating the engine sound of the engine 11 in order to make the operator recognize the surrounding conditions is described. However, there are other ways as a method for making the operator recognize the surrounding conditions. For example, when there are multiple sources of sound, it is sometimes difficult to hear the voice of a person. Therefore, in this modified example, a case where it is easy to hear the voice of a person is described.

[0213] The frequency band of human voice is approximately 100 Hz to 1000 Hz. Therefore, if this frequency band is extracted and output from the speaker A2, the voice becomes easier to hear.

[0214] Therefore, in this modification, the remote controller 40 set as the remote operation room RC can receive an operation of whether to filter in the frequency band. The operating device for receiving the operation can use any method, whether it is a well-known method or not. For example, a button for switching whether to filter can be set.

[0215] Then, when receiving an operation to perform filtering, the remote controller 40 performs filtering to extract a frequency band of 100 Hz to 1000 Hz from the received sound signal.

[0216] Then, the output control unit 403 outputs the filtered sound signal from the speaker A2. In this modification, since the sound is filtered in the frequency band, it becomes easier to hear the human voice. This allows the situation around the shovel 100 to be understood, thereby improving safety.

[0217] In addition, in this modification, an example is described in which the remote controller 40 of the remote control room RC receives an operation to switch whether to extract the frequency band of 100 Hz to 1000 Hz corresponding to the human voice. However, in this modification, the frequency band to be extracted is not limited to the frequency band corresponding to the human voice. For example, it is also possible to switch whether to attenuate the frequency band corresponding to the engine sound as shown in the above embodiment according to the operation.

[0218] In this variation, an example of switching whether to filter based on an operation is described. However, the condition for switching whether to filter is not limited to whether an operation is performed. For example, it is also possible to automatically switch whether to filter based on whether a human voice is detected.

[0219] In the above-mentioned embodiment and this variation, filtering is performed according to a predetermined condition, and one or more examples of generating a sound heat map and outputting from the speaker A2 are described based on the filtered sound signal. In this way, by prompting the operator with information based on the filtered sound signal, the operator can recognize necessary information. Therefore, it is possible to achieve improved convenience.

[0220] (Third embodiment)

[0221] In the above embodiment, an example in which the microphone array A1 is provided on the front surface of the shovel 100 is described. However, the present invention is not limited to the example in which the microphone array A1 is provided on the front surface of the shovel 100 as in the above embodiment. Therefore, in the third embodiment, an example in which the microphone array A1 is provided in four directions is described.

[0222] Fig.15It is a side view of a shovel 100A as a construction machine according to the third embodiment. Since the parts other than the microphone array A1 are the same as those in the above-mentioned embodiment, the description thereof will be omitted.

[0223] The microphone array A1 is configured to acquire sounds around the shovel 100. In the present embodiment, the microphone array A1 includes a front microphone array A1F for collecting sounds in front of the shovel 100, a left microphone array A1L for collecting sounds on the left side of the shovel 100, a right microphone array A1R for collecting sounds on the right side of the shovel 100, and a rear microphone array A1B for collecting sounds behind the shovel 100.

[0224] The camera device S6 may be, for example, a single-lens camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the display device DI.

[0225] The front camera S6F is installed, for example, on the top of the cab 10. The left camera S6L is installed on the left end of the upper surface of the upper rotating body 3. The right camera S6R is installed on the right end of the upper surface of the upper rotating body 3. The rear camera S6B is installed on the rear end of the upper surface of the upper rotating body 3.

[0226] The left microphone array A1L is attached to, for example, the left frame of the shovel 100. The right microphone array A1R is attached to, for example, the right frame of the shovel 100. The rear microphone array A1B is attached to, for example, the rear frame of the shovel 100.

[0227] Then, the map generation unit 303 of the shovel controller 30 generates heat maps of the sounds corresponding to the four microphone arrays A1 , respectively. Then, the transmission control unit 304 transmits the heat maps of the four sounds to the remote controller 40 .

[0228] Thus, when the image information of each of the four camera devices S6 is displayed in the remote operation room RC, the sound detection area can be superimposed and displayed. Therefore, when an operation of switching the display from the image information captured by the four camera devices S6 is received from the operator of the remote operation room RC, the sound detection area can be superimposed and displayed on the display image after the switching.

[0229] In this embodiment, four microphone arrays A1 are arranged in a manner corresponding to the arrangement of the camera S6, so that the sound detection area can be superimposed on the image information for display regardless of which image information among the image information captured by the four cameras S6 is displayed. Thus, the operator of the remote operation room RC can grasp the source of the sound, thereby achieving improved safety.

[0230] Furthermore, when any one of the image information captured by the four cameras S6 is displayed on the display device D1, the display screen generation unit 402 may also generate a display image that takes into account the heat map of each sound of the microphone array A1 set in a direction that does not correspond to the displayed image information. For example, when the heat map of the sound from the left microphone array A1L indicates a strong sound when generating a display image based on the image information of the front camera S6F, the display screen generation unit 402 may superimpose information indicating that the source of the sound exists on the left side on the image information of the front camera S6F. At this time, the display screen generation unit 402 may superimpose the character string information converted from the sound collected from the left microphone array A1L on the image information of the front camera S6F for display. Thus, the operator can grasp the direction in which the source of the sound exists in an area other than the area displayed on the display screen. Therefore, the operator can identify the direction in which the source of the sound exists, thereby improving safety.

[0231] (Fourth embodiment)

[0232] In the above-mentioned embodiment and modification, the case where the operator performs the operation in the remote operation room RC is described. However, the method of superimposing the sound detection area on the image information and displaying it as in the above-mentioned embodiment is not limited to the case of remote operation. Therefore, in the fourth embodiment, the case of displaying to the operator who performs the operation in the shovel 100B is described.

[0233] Fig.16 It is a side view of a shovel 100B as a construction machine according to the fourth embodiment. The parts other than the imaging device S6 and the microphone array A1 are the same as those in the above-mentioned embodiment, and the description thereof will be omitted.

[0234] The camera device S6 according to the present embodiment includes a left camera S6L for photographing the space on the left side of the shovel 100, a right camera S6R for photographing the space on the right side of the shovel 100, and a rear camera S6B for photographing the space behind the shovel 100. As described above, in the present embodiment, the front can be confirmed by the operator with naked eyes, so the front camera S6F is not provided.

[0235] Similarly, the microphone array A1 includes a left microphone array A1L for collecting sounds on the left side of the shovel 100, a right microphone array A1R for collecting sounds on the right side of the shovel 100, and a rear microphone array A1B for collecting sounds behind the shovel 100. The installation positions are the same as those in the above embodiment and the description thereof is omitted.

[0236] Then, the map generation unit 303 of the shovel controller 30 generates sound heat maps corresponding to the three microphone arrays A1 , respectively. Then, the shovel controller 30 superimposes the sound heat map indicating the sound generation status around the shovel 100 on the image information to generate a display image, and displays it on the display device D1 .

[0237] Next, refer to Fig.17 A configuration example of the image display unit 51 and the operation unit 52 of the display device DI will be described. Fig.17 DI is a diagram showing an example of the configuration of an image display unit 51 and an operation unit 52 of a display device DI. Fig.17 In the example shown, a main screen including a right image RG, a rear image BG, and a bird's-eye view image TG is displayed as a screen displayed on the image display unit 51 before the button switch 52a of the operation unit 52 is operated. The main screen is a screen displayed when the shovel 100 is operating.

[0238] First, the image display unit 51 will be described. Fig.17 As shown, the image display unit 51 includes a date and time display area 51a, a travel mode display area 51b, an accessory display area 51c, a fuel consumption display area 51d, an engine control status display area 51e, an engine operating time display area 51f, a cooling water temperature display area 51g, a fuel remaining amount display area 51h, a speed level display area 51i, a urea water remaining amount display area 51j, a working oil temperature display area 51k, an air conditioning operating status display area 51m, an image display area 51n and a switch image display area 51p.

[0239] The travel mode display area 51b, the accessory display area 51c, the engine control state display area 51e, the rotation speed level display area 51i, and the air conditioning operation state display area 51m are areas for displaying setting state information, which is information related to the setting state of the shovel 100. The fuel consumption display area 51d, the engine operation time display area 51f, the cooling water temperature display area 51g, the fuel remaining amount display area 51h, the urea water remaining amount display area 51j, and the hydraulic oil temperature display area 51k are areas for displaying operation state information, which is information related to the operation state of the shovel 100.

[0240] Specifically, the date and time display area 51a is an area for displaying the current date and time. The travel mode display area 51b is an area for displaying the current travel mode. The attachment display area 51c is an area for displaying an image representing the attachment currently installed. The fuel consumption display area 51d is an area for displaying fuel consumption information calculated by the excavator controller 30. The fuel consumption display area 51d includes an average fuel consumption display area 51d1 for displaying the lifetime average fuel consumption or the interval average fuel consumption and an instantaneous fuel consumption display area 51d2 for displaying the instantaneous fuel consumption.

[0241] The engine control state display area 51e is an area for displaying the control state of the engine 11. The engine operation time display area 51f is an area for displaying the accumulated operation time of the engine 11. The cooling water temperature display area 51g is an area for displaying the current temperature state of the engine cooling water. The remaining fuel amount display area 51h is an area for displaying the remaining amount of fuel stored in the fuel tank. The rotation speed level display area 51i is an area for displaying the current level set by the control panel 75 in an image. Fig.17 , the state where the first level is selected is shown. The urea water remaining amount display area 51j is an area for displaying the remaining amount of urea water stored in the urea water tank in an image. The hydraulic oil temperature display area 51k is an area for displaying the temperature state of the hydraulic oil in the hydraulic oil tank.

[0242] The air conditioning operation status display area 51m includes an outlet display area 51m1 displaying the current outlet position, an operation mode display area 51m2 displaying the current operation mode, a temperature display area 51m3 displaying the current set temperature, and an air volume display area 51m4 displaying the current set air volume.

[0243] The image display area 51n is an area for displaying images captured by the camera S6. Fig.17 In the example shown, the right image RG, the rear image BG, and the bird's-eye view image TG are displayed in the image display area 51 n.

[0244] The right image RG is an image showing the space on the right side of the shovel 100, including an image GC1 of the right end of the upper surface of the upper rotating body 3. The right image RG is an image generated by the shovel controller 30 by superimposing a sound heat map generated based on the sound signal acquired by the right microphone array A1R on the image captured by the right camera S6R.

[0245] The right image RG shows a worker 1501. The right image RG also shows a sound detection area 1511 represented by a sound heat map. The sound detection area 1511 is an area displayed based on the sound heat map. The sound detection area 1511 includes a dark red area 1511a, a red area 1511b, an orange area 1511c, a yellow area 1511d, and a light yellow area 1511e.

[0246] The rear image BG is an image showing the space behind the shovel 100, including the image GC2 of the counterweight. The rear image BG is an image generated by the shovel controller 30 by superimposing a sound heat map generated based on the sound signal acquired by the rear microphone array A1B on the image captured by the rear camera S6B.

[0247] A truck 1502 is displayed on the rear image BG. A sound detection area 1512 represented by a sound heat map is also superimposed on the rear image BG. The sound detection area 1001 is an area displayed based on the sound heat map. The sound detection area 1521 includes a yellow area 1511a and a light yellow area 1511b.

[0248] The bird's-eye view image TG is a virtual viewpoint image generated by the shovel controller 30, and is generated based on the images acquired by the rear camera S6B, the left camera S6L, and the right camera S6R. In addition, a shovel graphic corresponding to the shovel 100 is arranged in the central part of the bird's-eye view image. This is to enable the operator to more intuitively understand the positional relationship between the shovel 100 and the objects existing around the shovel 100.

[0249] The bird's-eye view image TG also displays a worker 1501 and a truck 1502. This allows the operator of the shovel 100 to recognize the surrounding situation.

[0250] Furthermore, the image display area 51n includes a first image display area 51n1 located at the top and a second image display area 51n2 located at the bottom. Fig.17 In the example shown, the right image RG and the rear image BG are arranged in the first image display area 51n1, and the bird's-eye view image TG is arranged in the second image display area 51n2. In addition, in the image display area 51n, the bird's-eye view image TG may be arranged in the first image display area 51n1, and the rear image BG and the right image RG may be arranged in the second image display area 51n2.

[0251] Furthermore, the image display area 51n may be configured to simultaneously display the left image. In this case, in the image display area 51n, the left image and the right image RG may be arranged in the first image display area 51n1, and the rear image BG and the bird's-eye view image TG may be arranged in the second image display area 51n2. In this case, the left image may be arranged on the left side of the first image display area 51n1, and the right image RG may be arranged on the right side of the first image display area 51n1.

[0252] And, in Fig.17 In the example shown, the right image RG and the back image BG are arranged adjacent to each other on the left and right, but they may be arranged at a distance from each other. Fig.17 In the example shown, the image display area 51n is a vertically long area, but the image display area 51n may be a horizontally long area. When the image display area 51n is a horizontally long area, in the image display area 51n, the bird's-eye view image TG may be arranged on the left side as the first image display area 51n1, and the rear image BG and the right image RG may be arranged on the right side as the second image display area 51n2. In this case, the left and right images may be arranged at intervals, or the position of the bird's-eye view image TG and the positions of the rear image BG and the right image RG may be replaced.

[0253] The switch image display area 51p has switch images 51p1 to 51p7. Fig.17 In the example shown, switch images 51p1 to 51p7 are arranged at intervals on the left and right sides of the lowermost portion of the image display unit 51. Icons representing the functions of the corresponding push button switches 52a1 to 52a7 are displayed on the switch images 51p1 to 51p7.

[0254] A menu detail item icon for displaying a menu detail item is displayed on the switch image 51p1. When the operator operates the push switch 52a1 corresponding to the switch image 51p1, the icons displayed on the switch images 51p2 to 51p7 are switched to icons associated with the menu detail items.

[0255] exist Fig.17In the example shown, an icon for displaying information related to the digital level is displayed on the switch image 51p4. If the operator operates the push button switch 52a4 corresponding to the switch image 51p4, the bird's-eye view image TG originally displayed in the second image display area 51n2 is switched to an image (sub-screen) representing information related to the digital level. On the other hand, the rear image BG and the right image RG originally displayed in the first image display area 51n1 continue to be displayed in the same size. However, the image (sub-screen) representing information related to the digital level can also be displayed in the first image display area 51n1. At this time, the rear image BG and the right image RG originally displayed in the first image display area 51n1 can also be displayed in the second image display area 51n2 instead of the bird's-eye view image TG.

[0256] exist Fig.17 In the example shown, no icon is displayed on the switch images 51p2, 51p3, 51p5, 51p6, 51p7. Therefore, even if the operator operates the push button switches 52a2, 52a3, 52a5, 52a6, 52a7 corresponding to the switch images 51p2, 51p3, 51p5, 51p6, 51p7, the image displayed on the image display unit 51 does not change.

[0257] In addition, the icons displayed on the switch images 51p1 to 51p7 are not limited to the above-mentioned examples, and icons for displaying other information may be displayed.

[0258] Next, the operation unit 52 will be described. Fig.17 As shown in FIG. 1 , the operation unit 52 is composed of a push button switch 52a which is a push button switch corresponding to each of the switch images 51p1 to 51p7. Fig.17 In the example shown, the operating unit 52 includes seven button switches 52a1 to 52a7 arranged in the upper row and seven button switches 52a8 to 52a14 arranged in the lower row. The button switches 52a8 to 52a14 are arranged below each of the button switches 52a1 to 52a7. However, the number, form and arrangement of the button switches 52a of the operating unit 52 are not limited to the above example. For example, it can also be a form that integrates the functions of multiple button switches like a micro roller or a micro switch. In addition, the operating unit 52 can also be composed of a component different from the display device DI. In addition, it can also be a method of directly touching the switch images 51p1 to 51p7 on the touch panel formed as a whole with the image display unit 51 and the operating unit 52.

[0259] The button switches 52a1 to 52a7 are arranged below the switch images 51p1 to 51p7 in correspondence with the switch images 51p1 to 51p7, respectively, and function as button switches for selecting the switch images 51p1 to 51p7. The button switches 52a1 to 52a7 are arranged below the switch images 51p1 to 51p7 in correspondence with the switch images 51p1 to 51p7, respectively, so that the operator can intuitively select the switch images 51p1 to 51p7.

[0260] The button switch 52a8 is a switch for switching the camera image displayed in the image display area 51n. The camera image displayed in the first image display area 51n1 of the image display area 51n is switched between, for example, a rear image, a left image, a right image, and a bird's-eye view image, every time the button switch 52a8 is operated. Also, the camera image displayed in the second image display area 51n2 of the image display area 51n is switched between, for example, a rear image, a left image, a right image, and a bird's-eye view image, every time the button switch 52a8 is operated. Also, the camera image displayed in the first image display area 51n1 of the image display area 51n and the camera image displayed in the second image display area 51n2 are swapped each time the button switch 52a8 is operated. In this way, the button switch 52a8 as the operation unit 52 can be used to switch the captured image displayed in the first image display area 51n1 or the second image display area 51n2, and can also be used to switch the captured image displayed in the first image display area 51n1 and the captured image displayed in the second image display area 51n2. In addition, a switch for switching the screen displayed in the second image display area 51n2 may be separately provided.

[0261] The button switches 52a9 and 52a10 are switches for adjusting the air volume of the air conditioner. Fig.17 In the example shown, it is configured that when the push button switch 52a9 is operated, the air volume of the air conditioner is reduced, and when the push button switch 52a10 is operated, the air volume of the air conditioner is increased.

[0262] The button switch 52a11 is a switch for switching the cooling / heating function on / off. Fig.17 In the example shown, each time the push button switch 52a11 is operated, the cooling / heating function is switched on / off.

[0263] The button switches 52a12 and 52a13 are switches for adjusting the set temperature of the air conditioner. Fig.17 In the example shown, it is configured that when the push button switch 52a12 is operated, the set temperature becomes lower, and when the push button switch 52a13 is operated, the set temperature becomes higher.

[0264] The push button switch 52a14 is a switch capable of switching the display of the engine operation time display area 51f.

[0265] Furthermore, the button switches 52a2 to 52a6 and 52a9 to 52a13 are configured to input numbers displayed on the switches or near the switches. Furthermore, the button switches 52a3, 52a4, 52a5, and 52a11 are configured to move the cursor leftward, upward, rightward, and downward respectively when the cursor is displayed in the image display area 51n.

[0266] The functions assigned to the push button switches 52a1 to 52a7 and 52a8 to 52a14 are merely examples, and other functions may be assigned.

[0267] As described above, when the push button switch 52a1 corresponding to the switch image 51p1 is operated while the right image RG, the rear image BG, and the bird's-eye view image TG are displayed in the image display area 51n, new icons (icons indicating functions newly assigned to the push button switches 52a1 to 52a7) are displayed on the switch images 51p2 to 51p7 while the right image RG and the rear image BG are displayed. Therefore, the operator can check the new icons while checking the right image RG and the rear image BG.

[0268] Furthermore, in the above example, if one of the button switches 52a1 to 52a7 corresponding to the switch images 51p2 to 51p7 is operated in a state where the right image RG, the rear image BG, and the bird's-eye view image TG are displayed in the image display area 51n, the bird's-eye view image TG is switched to a menu image showing information corresponding to the selected switch image. In this way, the menu image is displayed in a state where the right image RG and the rear image BG are displayed, so the operator can continue to monitor the surroundings (the space behind and on the right) even in a state where the menu image is displayed. Therefore, the operator can operate the excavator 100 in a state where the menu image is displayed.

[0269] In the present embodiment, the sound detection area is displayed superimposed on the right image RG and the rear image BG included in the image display area 51n of the display device D1.

[0270] That is, in the present embodiment, the operator riding on the shovel 100 can visually check the situation in front of the shovel 100 , and can therefore recognize the source of the sound along with the sound heard from the front of the shovel 100 .

[0271] However, it may be difficult for the operator riding on the shovel 100 to recognize the source of the sound in the area that cannot be visually recognized. Therefore, the shovel controller 30 according to the present embodiment displays the above-mentioned screen on the display device DI.

[0272] Thus, the operator of the shovel 100 can recognize the source of the sound around the shovel 100 even in the area that cannot be visually recognized. For example, by referring to the right image RG and the rear image BG, the source of the sound can be recognized, so the operator can understand the source of the sound currently heard and perform operations corresponding to the sound heard. Therefore, safety can be improved.

[0273] <Function>

[0274] In the above-mentioned embodiment and modification, the operator can identify the source of the sound by visually recognizing the displayed image, so it is easy to confirm the situation at the construction site. Therefore, the operator can perform operations corresponding to the situation at the construction site, so that safety can be improved. Furthermore, the operator can perform operations corresponding to the situation at the construction site, so that work efficiency can be improved.

[0275] In the above-mentioned embodiment and modification, the case where a shovel is used as an example of a construction machine is described. However, the structure shown in the embodiment and modification is not limited to the example of applying to the shovel as a construction machine, for example, it can also be applied to a crane, a forklift, etc.

[0276] The above describes an embodiment of a display device of a construction machine, a construction machine, and a remote control system of a construction machine, but the present invention is not limited to the above embodiment, etc. Various changes, corrections, replacements, additions, deletions, and combinations can be made within the scope described in the technical solution. Of course, these contents also belong to the technical scope of the present invention.

Claims

1. A display device for a construction machine, comprising a control unit, wherein the control unit is configured as follows: Acquiring image information captured by a camera installed in the construction machine, and acquiring information related to sound signals respectively acquired by a plurality of sound collecting devices installed in the construction machine; and A state of generation of sound around the construction machine based on information related to the sound signals acquired from the plurality of sound collecting devices is superimposed on the image information and displayed on a display unit.

2. The display device for construction machinery according to claim 1, wherein: The control unit displays the intensity of the sound acquired from the directions corresponding to the respective areas indicated by the image information in a display mode that allows identification as the generation status of the sound.

3. The display device for construction machinery according to claim 2, wherein: When the sound signal acquired from each of the plurality of sound collecting devices includes a plurality of sound sources, the control unit displays the plurality of sound sources in a display mode that allows identification of the plurality of sound sources.

4. The display device for construction machinery according to claim 1, wherein: When the sound signal is acquired from a direction outside the area indicated by the image information, the control unit displays a content that the sound signal is acquired from outside the area indicated by the image information.

5. The display device for construction machinery according to claim 1, wherein: The control unit converts the sound included in the sound signal into characters, and displays the converted characters on the display unit.

6. The display device for construction machinery according to claim 1, wherein: The control unit filters the sound signals acquired from the plurality of sound collecting devices based on a predetermined condition, and outputs the filtered sound signals.

7. The display device for construction machinery according to claim 6, wherein: The control unit performs filtering to determine whether to attenuate a predetermined frequency component, and outputs the audio signal after filtering.

8. The display device for construction machinery according to claim 7, wherein: The control unit performs filtering to attenuate the sound signal generated from a driving source of the construction machine.

9. The display device for a construction machine according to claim 6, wherein: The control unit switches whether to perform filtering as the predetermined condition according to an operation received from an operating device.

10. The display device for a construction machine according to any one of claims 6 to 9, wherein: The control unit outputs the filtered sound signal from a speaker.

11. A construction machine comprising: Camera device; a plurality of sound collecting devices; and The control device obtains image information from the camera device and sound signals from the multiple sound collecting devices respectively, superimposes the sound generation conditions around the construction machine based on the sound signals respectively obtained from the multiple sound collecting devices on the image information and outputs it to the display device.

12. The construction machine according to claim 11, wherein: The plurality of sound collecting devices are arranged so as to be able to acquire sounds emitted from a range including an imaging area captured by the imaging device.

13. A remote operating system for a construction machine, comprising a construction machine and an information processing device for operating the construction machine, The construction machine acquires image information from a camera device installed in the construction machine, acquires sound signals from a plurality of sound collecting devices installed in the construction machine, and transmits the image information and information related to the sound signals. The information processing device superimposes the generation status of the sound around the construction machine based on the information related to the sound signal on the received image information and displays the result on the display device.

Images