Working machine periphery monitoring system

Abstract

Provided is a working machine periphery monitoring system that is capable of maintaining an intuitive feeling obtained with a touch panel. Provided is a working machine periphery monitoring system that employs an intuitive and easy operation method, by displaying an operation guide for a touch operation relating to a setting of a camera video display (an operation guide indicating an operation method for the touch operation on a monitor 12 to change the setting of a displayed video), together with the displayed video in a form including an icon that resembles an operation mode composed of a hand, a finger or the like of a person.

Description

Surroundings monitoring system for work machines

[0001] The present invention relates to a surroundings monitoring system for a work machine equipped with a work implement, such as a hydraulic excavator.

[0002] In recent years, work machines such as hydraulic excavators have become increasingly computerized and information-based, and as the number of electronic devices installed in them has increased, the number of functions that support work has also increased.

[0003] For example, there is a function that notifies the operator of the situation around the work machine by displaying images on a monitor installed in the cab based on information obtained from multiple cameras mounted on the work machine. The images displayed on the monitor can be selected from several patterns, such as an image obtained from multiple cameras combined into an image seen from directly above the vehicle body as if looking down (looking down), or an image obtained from multiple cameras displayed side by side (without viewpoint conversion, etc.), and the operator can switch the image displayed on the monitor according to his or her preference.

[0004] In addition to the camera image, the monitor may also display icons indicating the status of various vehicle settings, meters showing the remaining fuel level and engine coolant temperature, and the status of various devices such as the radio and air conditioner.

[0005] In recent years, there has also been a function called machine guidance (MG) that assists work by calculating the position and attitude of the work machine and work equipment from information acquired by various sensors attached to the work machine and work equipment of the work machine, and notifying the operator of the positional relationship between the work machine and work equipment and the target surface during construction.

[0006] Furthermore, there is also an area restriction function that assists in working within the work area by calculating the degree to which the work machine and work equipment are approaching the boundary of the work area from information on the position and posture of the work machine and work equipment and information on the pre-set work area, and notifying the operator and imposing operational restrictions on the work machine and work equipment according to the degree of proximity.

[0007] While the number of functions to assist with work is increasing, the number of items that can be set on the monitor is also increasing, which raises concerns that the setting operations on the monitor may become cumbersome and reduce usability.

[0008] In the work machine of Patent Document 1, at least one of the display position of operational information such as meters on the monitor and the camera image to be displayed can be switched by operating a single selector switch. Therefore, when an operator wants to switch the display, the effort of searching for the switch button is reduced, and the display can be switched with a simple operation, improving operational efficiency.

[0009] Patent No. 6692744

[0010] In the work machine of Patent Document 1, switching of at least one of the display position of operational information such as meters on the monitor and the camera image to be displayed is achieved by operating a single selector switch. Meanwhile, in recent years, work machine monitors have increasingly been equipped with touch panels, making it possible to perform various settings and switches by directly operating (touching) the information displayed on the monitor. Being able to directly operate the information displayed on the monitor allows for more intuitive operation.

[0011] Furthermore, it is possible to change the content to be switched and the assigned functions depending on the type of touch operation (tap, swipe, drag, etc.). Therefore, even when touching the same area on the monitor, various types of switching and changes can be performed depending on the operation method, which is expected to improve convenience.

[0012] On the other hand, even if the touch area is the same, if the type of switching differs depending on the operation, it can be difficult to know which operation will result in which switching. In recent years, with the spread of smartphones, touch panel operations have become commonplace, and there is an increasing number of devices where functions can be associated with the operation method, such as pinching in and out to zoom in and out. However, there are also cases where the information displayed on the monitor of a work machine and the functions assigned to each piece of information cannot be used to associate a function with the operation method. For example, in the case of operations unique to work machines, such as switching the display pattern of camera images, it is difficult to associate a function with the operation method, making it difficult to understand the acceptable operations and how to operate the operation corresponding to the function, which reduces the usability of touch operations.

[0013] Therefore, an object of the present invention is to provide a surroundings monitoring system for a work machine that can maintain the intuitiveness achieved by using a touch panel.

[0014] In order to achieve the above object, the present invention provides a surroundings monitoring system for a work machine, which includes a camera attached to a vehicle body, a monitor, and a control device that generates a display image including camera images acquired from the camera and displays the generated display image on the monitor, wherein the monitor is capable of accepting input via touch operation, and the control device simultaneously displays on the monitor an operation guide that imitates the operating style of an operator, showing how to touch the monitor to change the settings of the display image, and the display image.

[0015] According to the present invention, by displaying an operation guide showing how to operate the displayed image of the camera image, including an icon that resembles the operating form of a human hand or finger, it is possible to intuitively understand the function of the operation method. Furthermore, by displaying the operation guide at the appropriate time and in the appropriate place, it is possible to achieve both ease of checking the surroundings through the displayed image and ease of understanding the operation through the operation guide, thereby improving work efficiency.

[0016] 1 is a right side view of a hydraulic excavator 1 which is an example of a work machine according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing the interior of the operator's cab 1f. FIG. 3 is a system configuration diagram of the hydraulic excavator 1. FIG. 4 is a functional block diagram of a control device 100 according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a pattern of a display image. FIG. 6 is a flowchart of a display image generation unit 101. FIG. 7 is a diagram showing an example of a pattern of correspondence between setting contents and operation contents. FIG. 8 is a diagram showing an example of switching of a display image by a touch operation. FIG. 9 is a flowchart of a display image setting unit 102. FIG. 10 is a flowchart of a manipulation guide setting unit 105. FIG. 11 is a flowchart of a manipulation guide display determination unit 107 for determining whether to perform normal display. FIG. 12 is a flowchart of a manipulation guide display determination unit 107 for determining whether to perform reduced display. FIG. 13 is a flowchart of a manipulation guide generation unit 104. FIG. 14 is a diagram showing an example of a manipulation guide display (when the manipulation guide is not displayed). FIG. 15 is a diagram showing an example of a manipulation guide display (when the manipulation guide is displayed normally). FIG. 16 is a diagram showing an example of a manipulation guide display (when the manipulation guide is displayed reduced). FIG. 17 is a diagram showing an example of a manipulation guide display (when the manipulation guide is displayed normally). FIG. 18 is a diagram showing an example of a manipulation guide display (when the manipulation guide is displayed superimposed on a vehicle body icon). FIG. 19 is a diagram showing an example of a manipulation guide display (when the manipulation guide is not displayed). Example of operation guide display (when the operation guide is superimposed on the image of the vehicle body). Example of operation guide display (when the operation guide is superimposed on an empty area).

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] FIG. 1 shows an overview of a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention. The hydraulic excavator 1 comprises an articulated front working implement 1A consisting of a boom 1a, an arm 1b, and a bucket 1c, each of which rotates vertically, and a vehicle body 1B consisting of an upper rotating body 1d and a lower traveling body 1e. The upper rotating body 1d is attached to the top of the lower traveling body 1e so as to be able to rotate left and right. The upper rotating body 1d is also provided with a driver's compartment (also referred to as a cab) 1f. The base end of the boom 1a of the front working implement 1A is supported at the front of the upper rotating body 1d. The boom 1a, arm 1b, bucket 1c, upper rotating body 1d, and lower traveling body 1e are driven by actuators for a boom cylinder 2a, an arm cylinder 2b, a bucket cylinder 2c, a swing motor 3d, and left and right traveling motors 3e, 3f, respectively. The upper rotating body 1d is equipped with an engine 4, a pump 5, and a valve 6 for operating the hydraulic excavator 1. The pump 5 is driven by the rotation of the engine 4 and discharges hydraulic oil to each actuator. The valve 6 controls the flow rate and direction of the hydraulic oil in response to inputs from operation levers 7a, 7b, 7c, and 7d (FIG. 2) provided in the operator's cab 1f, thereby operating each actuator. Furthermore, the boom 1a, the arm 1b, and the bucket 1c are equipped with angle sensors 8a, 8b, and 8c that detect the rotation angles of each part, and the upper rotating body 1d is equipped with GNSS antennas 9a and 9b for acquiring the position of the hydraulic excavator 1, and cameras 10a, 10b, and 10c that capture images of the surroundings of the work machine are attached to the left, rear, and right sides of the upper rotating body 1d. In addition, the upper rotating body 1d is equipped with a control device 100 that generates an image (display image) to be displayed on the monitor 12 (Figure 2) based on the images (camera images) acquired from the cameras 10a, 10b, and 10c, and performs processing to display the generated display image on the monitor 12.

[0019] 2 shows the interior of the operator's cab 1f. The operator's cab 1f is equipped with control levers 7a, 7b, 7c, and 7d, a shutoff lever 17 that locks the operation input from the control levers to the actuators, a speaker 11 that audibly conveys information to the operator, and a monitor 12 that displays information. The monitor 12 of this embodiment is also equipped with a touch panel 15 that accepts input by touch operation. Such a monitor 12 is also referred to as a touch panel monitor.

[0020] FIG. 3 shows the overall configuration of an information input system for a hydraulic excavator 1, an example of a work machine according to an embodiment of the present invention. The control device 100 for the hydraulic excavator 1 according to this embodiment includes a main C / U (control unit) 50, a GNSS C / U 40 that processes information from the GNSS antennas 9a and 9b, and a notification C / U 60 that controls various notifications to the operator. The main C / U 50 controls basic systems of the work machine, such as the engine 4 and pump 5, and is connected to a key cylinder, which controls the electrical activation of the work machine and the start of the engine 4, which is the prime mover. The main C / U 50 is also connected to a communication terminal and communicates with various terminals and servers outside the hydraulic excavator 1. Based on the processing status of the main C / U 50, the notification C / U 60 calculates display information for the monitor 12 and sound information for the speaker 11 and sends signals. Furthermore, the notification C / U 60 sends various setting signals to the main C / U 50 in response to inputs made by touch operations on the touch panel 15.

[0021] Figure 4 is a functional block diagram of a control device 100 according to an embodiment of the present invention. The control device 100 includes a display image generation unit 101, a display image setting unit 102, a touch operation acquisition unit 103, an operation guide generation unit 104, an operation guide setting unit 105, a display control unit 106, an operation guide display determination unit 107, and a vehicle body state acquisition unit 108. Each block will be described below. The devices shown in Figure 4 constitute a surroundings monitoring system that monitors the situation around the hydraulic excavator 1.

[0022] The display image generation unit 101 synthesizes the images (camera images) acquired from the cameras 10a, 10b, and 10c according to the settings, and generates a display image to be displayed on the monitor 12. Five patterns of display images generated in this embodiment are shown in FIG. 5. Display images 201 and 202 are display images of the same size. Display images 203, 204, and 205 are display images of the same size, but are different sizes from the display images 201 and 202.

[0023] Display image 201 is an image (bird's-eye view image) obtained by projecting images acquired from cameras 10a, 10b, and 10c onto an image from a viewpoint looking down from above (directly above the hydraulic excavator). Of display image 201, image 201a is an area onto which image from left camera 10a is projected, image 201b is an area onto which image from rear camera 10b is projected, and image 201c is an area onto which image from right camera 10c is projected. Vehicle body icon 201d is an icon imitating the shape of a hydraulic excavator, and is displayed to assist in understanding the positional relationship between the projected surrounding image and the vehicle body.

[0024] Display image 202 is an image in which images from cameras 10b and 10c are displayed side by side without projection using viewpoint conversion, etc. Of display image 202, image 202b displays an image from camera 10b on the left side, and image 202c displays an image from camera 10c on the right side.

[0025] Display image 203 is an image in which images acquired from cameras 10a, 10b, and 10c are projected onto an image from a viewpoint looking down from above (seen directly above the hydraulic excavator) (bird's-eye image), and the image from camera 10b is displayed side by side. Of display image 203, image 203a is an area onto which image from left camera 10a is projected, image 203b is an area onto which image from rear camera 10b is projected, and image 203c is an area onto which image from right camera 10c is projected. Image 203e is an image in which image from rear camera 10b is displayed without being projected. Vehicle body icon 203d is an icon imitating the shape of a hydraulic excavator, and like vehicle body icon 201d, it is displayed to assist in understanding the positional relationship between the projected surrounding image and the vehicle body.

[0026] Display image 204 is an image in which images from cameras 10a, 10b, and 10c are displayed side by side without projection using viewpoint conversion, etc. Of display image 204, image 204a displays an image from left camera 10a, image 204b displays an image from rear camera 10b, and image 204c displays an image from right camera 10c.

[0027] Display image 205 is an image (bird's-eye view image) in which images acquired from cameras 10a, 10b, and 10c are projected onto an image of a viewpoint looking down from the sky (as seen from directly above the hydraulic excavator). It differs in display size and display area from display image 201 and display image 203, and displays up to the front of the vehicle body. Of display image 205, image 205a is an area onto which image from left camera 10a is projected, image 205b is an area onto which image from rear camera 10b is projected, and image 205c is an area onto which image from right camera 10c is projected. Vehicle body icon 205d is an icon modeled after the shape of a hydraulic excavator, and, like vehicle body icon 201d, is displayed to assist in understanding the positional relationship between the projected surrounding image and the vehicle body itself.

[0028] FIG. 6 shows a flowchart of the display image generation unit 101. First, in step S300, the display image generation unit 101 checks whether camera images have been acquired from the cameras 10a, 10b, and 10c. If camera images have not been acquired, the process ends. If camera images have been acquired, the process proceeds to step S301. In step S301, the display image settings are checked. In this embodiment, which display pattern should be displayed (which display pattern should be selected) from the display patterns shown in FIG. 5 is checked in this step. After checking the settings, the process proceeds to step S302, where the images acquired from the cameras 10a, 10b, and 10c are actually processed according to the settings to generate a display image. Once the generation of the display image is complete, the process proceeds to step S303, where the generated display image is output, and the process ends.

[0029] The display image setting unit 102 changes the settings of the display image generated by the display image generation unit 101 by a touch operation on the display image. In this embodiment, the setting is changed by switching the pattern of the display image shown in FIG. 5 by a touch operation on the display image. The change in the setting varies depending on the type of touch operation. FIG. 7 shows a table 400 showing the correspondence between the type of touch operation (operation content 401) and the change in the setting (setting content 410).

[0030] In this embodiment, four types of touch operations are assumed: operation details 402, 403, 404, and 405. Operation detail 402 is a swipe operation in which one finger touches the touch panel 15 and moves the finger. Operation detail 403 is a pinch-in operation in which two fingers touch the touch panel 15 and move the fingers closer together. Operation detail 404 is a pinch-out operation in which two fingers touch the touch panel 15 and move the fingers farther apart. Operation detail 405 is a tap operation in which one finger touches the touch panel 15 and then releases it.

[0031] In this embodiment, three types of setting changes are assumed: setting contents 411, 412, and 413. As the setting corresponding to operation content 402, switching the pattern of the display image within the same display size is assigned as setting content 411. As the setting corresponding to operation content 403, reducing the size of the displayed image is assigned as setting content 412. As the setting corresponding to operation content 404, enlarging the size of the displayed image is assigned as setting content 413. As the setting corresponding to operation content 405, no setting content is assigned, and therefore setting content 414 has no setting content.

[0032] 8 shows the relationship between touch operations and display images generated by display image generator 101. Monitor displays 211, 212, 213, 214, and 215 are the display images described in FIG. 5 displayed on monitor 12 equipped with touch panel 15. Touch panel 15 has touch area 15a that responds to touch operations, and monitor 12 has liquid crystal display area 12a where the image is actually displayed. Monitor display 211 displays display image 201, monitor display 212 displays display image 202, monitor display 213 displays display image 203, monitor display 214 displays display image 204, and monitor display 215 displays display image 205. Group 220 includes monitor displays 211 and 212, which have the same display image size. Similarly, group 221 includes monitor displays 213, 214, and 215, which have the same display image size. In the table 400 of FIG. 7, the operation content 402 and the setting content 411 indicate that a swipe operation is assigned to switching patterns within a group having the same size of display images.

[0033] To switch between group 220 and group 221, which have different sizes of display images, pinch-in and pinch-out operations are assigned according to operation contents 403 and 404 and setting contents 412 and 413. When switching from group 220 to group 221, the size of the display image increases, so switching is performed when a pinch-out operation is input. When switching from group 221 to group 220, the size of the display image decreases, so switching is performed when a pinch-in operation is input.

[0034] FIG. 9 shows a flowchart of the display image setting unit 102. In step S310, it is determined whether a touch operation has been performed on the currently displayed display image. If a touch operation has not been performed on the display image, the process ends. If a touch operation has been performed on the display image, the process proceeds to step S311. In step S311, it is determined whether the type of touch operation is an operation to which a setting has been assigned. Since settings are assigned to swipes, pinches in, and pinches out listed in table 400 of FIG. 7, if any of the touch operations of swipes, pinches in, and pinches out has been input, the process proceeds to step S312. If a touch operation other than swipes, pinches in, or pinches out has been input, no setting has been assigned, and the process ends. In step S312, the setting content assigned to the content of the input touch operation is confirmed based on table 400 of FIG. 7. After confirmation of the setting content has been completed, the process proceeds to step S313. In step S313, the setting content of the display image generation unit 101 is changed in accordance with the setting content. When the change of the setting contents of the display video generating unit 101 is completed, the process ends.

[0035] The touch operation acquisition unit 103 acquires and outputs information on inputs made by the operator through touch operations on the touch panel 15 .

[0036] The operation guide setting unit 105 changes the setting of the operation guide generated by the operation guide generating unit 104 by touching the monitor.

[0037] FIG. 10 shows a flowchart of the operation guide setting unit 105. In this embodiment, the flow shows a case where switching between displaying and hiding an operation guide and changing the display position of the operation guide can be performed. In step S320, it is confirmed whether a touch operation has been input to the monitor 12 (the touch panel 15 thereof). If a touch operation has not been input, the process ends. If a touch operation has been input, the process proceeds to step S321. In step S321, it is determined whether the input touch operation is an operation to change the operation guide display. An example of an operation to change the operation guide display is an operation to select and confirm the change content of the operation guide display on a menu screen. If an operation to change the operation guide display has not been input, the process ends. If an operation to change the operation guide display has been input, the process proceeds to step S322. In step S322, it is determined whether the content of the operation to change the operation guide display is an operation to switch between displaying and hiding the operation guide. If it is an operation to switch between displaying and hiding the operation guide, the process proceeds to step S323. In step S323, the display / hide of the operation guide is changed according to the content of the operation to change the operation guide display, and the process ends. If it is determined in step S322 that the operation to change the operation guide display is not an operation to switch between displaying and hiding the operation guide, the process proceeds to step S324. In step S324, it is determined whether the operation to change the operation guide display is a change in the display position of the operation guide. If it is not an operation to change the display position of the operation guide, the process ends. If it is an operation to change the display position of the operation guide, the process proceeds to step S325. In step S325, the display position of the operation guide is changed according to the operation to change the operation guide display, and the process ends. In this embodiment, the display position of the operation guide can be changed to the top or left side of the displayed image. Note that a case where the top of the displayed image is selected as the display position of the operation guide is shown in FIGS. 14B and 14C, and a case where the left side of the displayed image is selected is shown in FIG. 15.

[0038] The operation guide display determination unit 107 determines whether to display the operation guide in a normal display or a reduced display. FIG. 11 shows a flowchart for the operation guide display determination unit 107 to determine whether to display the operation guide in a normal display. In step S340, it is determined whether a touch operation has been performed on the display image on the monitor 12. If a touch operation has been performed, the process proceeds to step S342. In step S342, the operation guide is set to be displayed in a normal display, and the process ends. If it is determined in step S340 that a touch operation has not been performed, the process proceeds to step S341. In step S341, it is determined whether the display image is being displayed on the monitor 12 for the first time after system startup. If the display image is being displayed on the monitor 12 for the first time after system startup, the process proceeds to step S342. As described above, in step S342, the operation guide is set to be displayed in a normal display, and the process ends. If the display image is not being displayed on the monitor 12 for the first time after system startup, the process ends.

[0039] FIG. 12 shows a flowchart for determining whether to perform a reduced display in the operation guide display determination unit 107. In step S350, it is determined whether a touch operation has not been input to the display image on the monitor 12 for a certain period of time or longer. If a touch operation has not been input to the display image for a certain period of time or longer, the process proceeds to step S355. In step S355, a setting is made to display the operation guide in a reduced display (in other words, to switch the operation guide to a reduced display), and the process ends (similarly below). In step S350, if a touch operation has been input to the display image within the certain period of time, the process proceeds to step S351. In step S351, it is determined whether a touch operation has been input on a part other than the display image on the monitor 12 (in other words, a touch operation other than a touch operation on the display image). If a touch operation has been input on a part other than the display image, the process proceeds to step S355. If a touch operation has not been input on a part other than the display image, the process proceeds to step S352. In step S352, it is determined whether a predetermined operation has been input. The predetermined operation here refers to an operation performed to reduce the display of the operation guide (in other words, to switch the display of the operation guide to a reduced size). In this embodiment, a reduced display switching icon 256 is provided on the operation guide, and when the reduced display switching icon 256 is touched, it is determined that the predetermined operation has been performed. The reduced display switching icon 256 will be described later with reference to FIG. 15 . If it is determined that the predetermined operation has been input, the process proceeds to step S355. If it is determined that the predetermined operation has not been input, the process proceeds to step S353. In step S353, it is determined whether the display on the monitor 12 has switched to a screen on which no display image is displayed (in other words, whether the display image has become hidden). If the display has switched to a screen on which no display image is displayed (in other words, whether the display image has become hidden), the process proceeds to step S355. An example of a screen on which no display image is displayed is when the display image disappears while a menu screen for configuring various settings is displayed. If the screen on which a display image is displayed is a screen on which no display image is displayed, the process proceeds to step S354.In step S354, it is determined whether the operating state of the work machine has changed. In this embodiment, whether the operating state of the work machine has changed is determined based on whether the lock state of the shut-off lever 17 (acquired by the vehicle body state acquisition unit 108, described below) has changed. If it is determined that the operating state of the work machine has changed, the process proceeds to step S355. If it is determined that the operating state of the work machine has not changed, the process ends (in other words, the operation guide is not switched to a reduced display).

[0040] The operation guide generation unit 104 generates an operation guide showing an operation method for touching the monitor 12 to change the settings of the display image (in other words, an operation guide for touch operations related to the settings of the camera image display on the monitor) based on information from the display image setting unit 102, the touch operation acquisition unit 103, the operation guide setting unit 105, the operation guide display determination unit 107, etc. In this embodiment, the operation guide includes icons imitating the operation form of an operator, such as an icon imitating a human hand or finger or the movement of a human hand or finger, and an icon indicating the setting content 410 of the display image corresponding to the operation content 401 of the touch operation (see also table 400 in FIG. 7 ). In addition, in this embodiment, the operation guide generation unit 104 generates an operation guide for normal display or reduced display based on a predetermined condition (in other words, switches between the normal display and reduced display of the operation guide), and the operation guide includes information on the display position, etc.

[0041] FIG. 13 shows a flowchart of the manipulation guide generation unit 104. In step S330, it is confirmed whether the display setting of the manipulation guide is hidden. If it is set to hidden, the process proceeds to step S336. In step S336, the manipulation guide is hidden, and the process ends without generating the manipulation guide, etc. In step S330, if the display setting of the manipulation guide is not set to hidden (it is set to display), the process proceeds to step S331. In step S331, information on the setting contents 410 and operation contents 401 that can be set by the display image setting unit 102 is acquired. Once acquisition of the information from the display image setting unit 102 is completed, the process proceeds to step S332. In step S332, the display setting of the manipulation guide is confirmed. The display setting of the manipulation guide includes information on the display position of the manipulation guide, etc. Once confirmation of the display setting of the manipulation guide is completed, the process proceeds to step S333. In step S333, it is determined whether the display setting of the manipulation guide is reduced display or normal display. If it is determined that the display setting of the manipulation guide is reduced display, the process proceeds to step S334. In step S334, an operation guide for reduced display is generated, and the process ends. If it is determined in step S333 that the display setting of the operation guide is normal display, the process proceeds to step S335. In step S335, an operation guide for normal display is generated, and the process ends.

[0042] The display control unit 106 performs processing to display the display image generated by the display image generation unit 101 and the operation guide generated by the operation guide generation unit 104 on the monitor 12. In this embodiment, only the display image and operation guide are displayed, but information indicating the setting status of the work machine and the setting status of various devices in the cab, such as the air conditioner and radio, may also be displayed together.

[0043] The vehicle body state acquisition unit 108 acquires the operating state of the work machine to be used in the determination by the operation guide display determination unit 107. In this embodiment, the locked state of the shut-off lever 17 is acquired and output to the operation guide display determination unit 107.

[0044] 14A, 14B, 14C, and 15 show display examples of operation guides. Each display example shows a case where a display image 205 is displayed. Fig. 14A shows a monitor display 215 when the operation guide is not displayed, Fig. 14B shows a monitor display 230 when the operation guide is displayed in the normal mode, and Fig. 14C shows a monitor display 240 when the operation guide is displayed in a reduced size.

[0045] On the monitor display 215, the display setting for the operation guide is set to non-display (step S336 in FIG. 13), so the display image 205 is displayed but the operation guide is not displayed.

[0046] The monitor display 230 is a display example when the operation guide is normally displayed (step S335 in FIG. 13 ). As the operation guide, setting content guides 233 and 234 are displayed together with touch operation guides 231 and 232. The touch operation guides 231 and 232 indicate the operation method (operation form) of the touch operation, and the setting content guides 233 and 234 indicate the display image settings that are made by the touch operation (i.e., the display image settings corresponding to the operation content of the touch operation). The touch operation guide 231 is an icon indicating a swipe operation, and the setting content guide 233 corresponding to the touch operation guide 231 indicates a change in the pattern of the display image. These two icons indicate that the pattern of the display image is changed by performing a swipe operation. The touch operation guide 232 is an icon indicating a pinch-in operation, and the setting content guide 234 corresponding to the touch operation guide 232 indicates the reduction of the display image. These two icons indicate that the display image is reduced by performing a pinch-in operation. The background image 235 is displayed around the operation guides (231 to 234) and is a background image that is superimposed on the displayed image together with the operation guides (231 to 234), and is displayed to make the operation guides easy to see regardless of the state of the displayed image. The operation guides (231 to 234) and the background image 235 are displayed in a semi-transparent color. The background image 235 is generated by the operation guide generating unit 104 in accordance with the operation guides.

[0047] The monitor display 240 is a display example when the operation guides are displayed in a reduced size (step S334 in FIG. 13 ). Touch operation guides 241 and 242 are displayed as the operation guides. The touch operation guide 241 is an icon indicating a swipe operation, and the touch operation guide 242 is an icon indicating a pinch-in operation. The touch operation guides 241 and 242 are displayed in a smaller size than the touch operation guides 231 and 232 in normal display. As in normal display, a background image 243 is also displayed around the operation guides (241 and 242), and the operation guides (241 and 242) and the background image 243 are displayed in a semi-transparent color. Note that the background image 243 is generated by the operation guide generation unit 104 in accordance with the operation guide. In this embodiment, the size of the displayed icons is reduced for reduced display, but the display color and transparency may be changed, or the operation guides may not be displayed at all.

[0048] FIG. 15 shows a monitor display 250 as an example of another display format when operation guides are normally displayed. The operation guides include touch operation guides 251 and 252, setting content guides 253 and 254, and an operation area guide 257. The touch operation guide 251 is an icon indicating a swipe operation, and the setting content guide 253 corresponding to the touch operation guide 251 indicates a change in the pattern of the displayed image. These two icons indicate that the pattern of the displayed image changes when a swipe operation is performed. The touch operation guide 252 is an icon indicating that a pinch-in operation is not possible, and the setting content guide 254 corresponding to the touch operation guide 252 indicates that the displayed image cannot be reduced. These two icons indicate that the displayed image cannot be reduced by a pinch-in operation. The operation area guide 257 is a guide indicating the area in which the operation indicated in the operation guide is accepted. In this embodiment, the operation area guide 257 is displayed to surround the periphery of the displayed image, since the operation area guide 257 targets touch operations on the displayed image. The reduced display switching icon 256 is an icon that accepts an input to switch the display from normal display to reduced display, and when this icon is tapped, the display of the operation guide switches to reduced display. The background image 255 is displayed around each icon (251 to 254) of the operation guide and the reduced display switching icon 256. The icons (251 to 254) of the operation guide, the reduced display switching icon 256, and the background image 255 that are superimposed on the displayed image use semi-transparent colors. The background image 255 is generated by the operation guide generating unit 104 to match each icon (251 to 254) of the operation guide and the reduced display switching icon 256. The operation area guide 257 is also generated by the operation guide generating unit 104 to match the displayed image and emphasize its surroundings.

[0049] Furthermore, in monitor display 250 (FIG. 15), the position at which the operation guide is displayed is changed from that of monitor display 230 (FIG. 14B), which also normally displays the operation guide. Monitor display 230 (FIG. 14B) is a display example in which the top of the displayed image is selected as the display position for the operation guide (and the background image around it), while monitor display 250 (FIG. 15) is a display example in which the left side of the displayed image is selected.

[0050] 16 shows a display example in which an operation guide is superimposed on an icon resembling a car body. Monitor display 260 shows an example in which display image 205 is displayed. Touch operation guides 261 and 262 are displayed as operation guides. Monitor display 260 displays touch operation guides 261 and 262 and background image 263 superimposed on car body icon 205d. By displaying touch operation guides 261 and 262 superimposed on car body icon 205d, touch operation guides 261 and 262 do not superimpose on the camera image in the display image, and it is possible to display touch operation guides 261 and 262 without obstructing the view of the surroundings.

[0051] 17A, 17B, and 17C show display examples in which an operation guide is superimposed on the display image 204. In FIG. 17A, the monitor display 270 shows an example in which an image of the surroundings is displayed on the display image 204. Vehicle body images 271a, 271b, and 271c are displayed on the camera image, and the shape and size of the images vary depending on the shape of the vehicle body, the camera mounting position, lens distortion, and the like. Furthermore, on the vehicle body side of the camera image are ground areas 273a, 273b, and 273c, which show the ground surface, etc. Conversely, on the side of the camera image farther from the vehicle body are sky areas 274a, 274b, and 274c, which show the sky and tall three-dimensional objects. In this embodiment, for the sake of explanation, imaginary horizon lines 272a, 272b, and 272c are displayed as boundaries between ground areas 273a, 273b, and 273c and sky areas 274a, 274b, and 274c.

[0052] 17B shows an example in which an operation guide is displayed superimposed on the vehicle body shown in the camera image on the monitor display 280. Touch operation guides 281 and 282 are displayed as the operation guides. The touch operation guides 281 and 282 and a background image 283 are displayed superimposed on the vehicle body image 271b. By displaying the operation guides superimposed on the vehicle body shown in the camera image, it is possible to display the touch operation guides 281 and 282 without obstructing the view of the surroundings.

[0053] 17C shows an example in which an operation guide is displayed in an area of ​​the camera image that shows the sky on monitor display 290. Touch operation guides 291 and 292 are displayed as operation guides. Touch operation guides 291 and 292 and background image 293 are displayed superimposed on sky area 274a. The sky area is an area that shows the sky or the area far from the vehicle body, and is unlikely to get in the way when checking the area near the vehicle body or to obstruct the surrounding field of view.

[0054] As described above, according to this embodiment, by displaying an icon imitating the operator's operating form, such as the shape of a human hand or fingers, on the monitor 12, as in the touch operation guides 231 and 232 on the monitor display 230 in Figure 14B, it is possible to easily understand that some operation or setting can be performed on the display of the camera image by inputting a touch operation.

[0055] Furthermore, by displaying operation guides that represent the movements of a human hand and fingers, such as the touch operation guides 231 and 232, it becomes possible to easily grasp the types of touch operations that can be input.

[0056] Furthermore, by displaying setting content guides 233 and 234 on the monitor 12 together with the touch operation guides 231 and 232, it becomes possible to easily grasp what setting operations are assigned to the types of touch operations.

[0057] Furthermore, by displaying the background image 235 around the touch operation guides 231 and 232, it is possible to reduce the likelihood that the touch operation guides 231 and 232 will become difficult to see due to the display content of the displayed image.

[0058] Furthermore, by making the touch operation guides 231 and 232 and the background image 235 semi-transparent, it becomes possible to see through the superimposed display image, and it becomes possible to operate while checking what is shown in the display image.

[0059] Furthermore, as shown in the monitor display 290 of Figure 17C, by superimposing the operation guide on the upper side of the monitor of the display image 204, the operation guide is superimposed and displayed in an area far from the work machine or in an area showing the sky, making it possible to display the operation guide while ensuring visibility near the work machine.

[0060] Furthermore, as shown in the touch operation guide 252 on the monitor display 250 in FIG. 15, if it is not possible to perform settings on the displayed image by touch operation for some reason (in other words, if some or all of the setting operations for changing the settings of the displayed image cannot be performed due to certain conditions), by indicating this state (generating an operation guide including such an icon), it is possible to easily grasp the state in which touch operation cannot be performed, and work efficiency is also improved.

[0061] Furthermore, by displaying the operation area guide 257 of the monitor display 250 in Figure 15 together with the touch operation guide 251, it becomes possible to easily grasp the area (in this embodiment, the displayed image) that accepts touch operations shown in the operation guide.

[0062] Furthermore, as shown in FIG. 8, the pattern and size of the displayed image can be changed by touch operation, which makes it easy to change the displayed image to suit the work content and the operator's preferences, thereby improving work efficiency.

[0063] 9, by assigning setting operations to touch operations on the displayed image, intuitive operation can be realized, improving work efficiency. Furthermore, since there is no need to place button icons for setting in an area separate from the displayed image, the display area of ​​the monitor 12 can be saved and can be used to display other functions.

[0064] 10, the operator can change the display position of the operation guide, which can be changed in accordance with the operator's preference or the display image pattern to be used, improving usability. Furthermore, the operator can set whether to display or hide the operation guide, so that operators who understand the operation method even without the operation guide or operators who find the display of the operation guide bothersome can hide the operation guide, and it becomes possible to set a display that is easy to use according to the operator's preference.

[0065] Furthermore, as shown in the flowchart of Figure 11, by normally displaying an operation guide when a touch operation is input to the display image, the operation guide is displayed when some operation is about to be performed on the display image, reducing confusion about how to operate it and improving work efficiency. Furthermore, by displaying the operation guide when the display image is displayed on the screen after system startup, it becomes possible to easily understand how to set the display image when starting to use the work machine, improving work efficiency.

[0066] Furthermore, as shown in the flowchart of FIG. 12 , by reducing the display of the operation guide depending on various conditions, the display of the operation guide is reduced when no operation is being performed on the displayed image, making it easier to check the displayed image and improving work efficiency. As for various conditions, when no touch operation is performed on the displayed image for a certain period of time, the display is switched to a reduced display, assuming that the setting operation on the displayed image has ended. When a touch operation other than a touch operation on the displayed image is input or when the operating state of the work machine changes, the setting operation on the displayed image is also considered to have ended, and the display of the operation guide is reduced. Furthermore, when the display on the monitor 12 is switched to a screen that does not display the displayed image itself (when the displayed image is no longer displayed), the display of the operation guide is also reduced. In this way, the work machine determines that the operation on the displayed image has ended and switches to a reduced display, improving work efficiency. The display is also switched to a reduced display when the reduced display switch icon 256 is operated. Switching to a reduced display of the operation guide is also possible when the operator intentionally wants to do so.

[0067] In this embodiment, an example is given in which the operation guide is displayed in a reduced size compared to the normal display in order to make the displayed image easier to check (to increase visibility); however, instead of changing (reducing) the display size of the operation guide, or at the same time as changing (reducing) the display size of the operation guide, it is also possible to make the displayed image easier to check by changing the display color or changing (increasing) the display transmittance to create a specific display in which the visibility of the operation guide is lower than the normal display, for example.

[0068] [Summary] As described above, the surroundings monitoring system for a work machine (hydraulic excavator 1) of this embodiment is a system for monitoring the surroundings of a work machine (hydraulic excavator 1) that includes cameras 10a, 10b, 10c mounted on the hydraulic excavator 1, a monitor 12, and a control device 100 that generates a display image including camera images acquired from the cameras and displays the generated display image on the monitor, in which the monitor 12 is capable of receiving input via a touch operation (touch panel 15), and the control device 100 acquires the input via the touch operation on the monitor (touch operation acquisition unit 103), changes the settings of the display image according to the acquired touch operation (display image setting unit 102), processes the camera image acquired from the camera in accordance with the changed display image settings, generates the display image to be displayed on the monitor (display image generation unit 101), generates an operation guide imitating the operation form of an operator showing how to operate the touch operation on the monitor to change the settings of the display image (operation guide generation unit 104), and displays the generated operation guide and the display image simultaneously on the monitor (display control unit 106).

[0069] The operation guide includes a (human) hand or finger, or a representation (icon) showing the movement of a (human) hand or finger.

[0070] When a touch operation is performed on the display image, the control device 100 (the display image setting unit 102) changes the setting of the display image in response to the touch operation.

[0071] The control device 100 (the operation guide generating unit 104) generates the operation guide including an expression (icon) indicating the setting content of the display image according to the operation content of the touch operation.

[0072] When (some or all) setting operations for changing the settings of the displayed image cannot be performed due to specified conditions, the control device 100 (operation guide generation unit 104) generates the operation guide including an expression (icon) indicating that (some or all) setting operations cannot be performed.

[0073] The control device 100 (the operation guide generating unit 104 and the display control unit 106) displays the operation guide on the monitor 12 by superimposing it on the display image. Also, the control device 100 (the operation guide generating unit 104) generates an operation guide including an icon in a semi-transparent color.

[0074] The control device 100 (the operation guide generating unit 104) also generates a semi-transparent background image around the operation guide.

[0075] The control device 100 (operation guide generating unit 104) emphasizes the periphery of the displayed image together with the operation guide (operation area guide 257).

[0076] The control device 100 (the operation guide setting unit 105 and the operation guide generating unit 104) can change at least one of display, non-display and display position of the operation guide as the display setting of the operation guide.

[0077] The control device 100 (the operation guide display determination unit 107) can switch between a normal display of the operation guide and a special display in which the visibility of the operation guide is lower than the normal display in order to increase the visibility of the displayed image (make it easier to see) based on specified conditions, and the special display includes at least one of changing (reducing) the display size of the operation guide or changing (increasing) the transparency at which it is displayed.

[0078] The control device 100 (operation guide display determination unit 107) normally displays the operation guide when a touch operation is performed on the display image. The control device 100 (display image setting unit 102) does not change the setting of the display image when a predetermined touch operation is performed on the display image, and the control device 100 (operation guide display determination unit 107) normally displays the operation guide even when the setting of the display image is not changed.

[0079] The control device 100 (operation guide display determination unit 107) normally displays the operation guide when the surroundings monitoring system is started and the display image is displayed on the monitor.

[0080] The control device 100 (operation guide display determination unit 107) switches the operation guide from normal display to specific display when a predetermined condition is satisfied.

[0081] The control device 100 (the display image setting unit 102) can change at least one of the size or pattern of the display image in response to the touch operation, and the pattern of the display image includes at least one of an overhead image in which the camera image is converted (synthesized) into an image from a viewpoint seen from directly above the vehicle body (looking down from above), or an image in which the camera image is displayed without converting the viewpoint.

[0082] The control device 100 (operation guide generation unit 104, display control unit 106) displays the operation guide by superimposing it on the area in the displayed image in which the sky or the vehicle body is shown, or on a representation (icon) that resembles the vehicle body.

[0083] The control device 100 is capable of switching between a normal display of the operation guide and a specific display in which the visibility of the operation guide is lower than the normal display in order to increase the visibility (make it easier to see) of the displayed image based on predetermined conditions, and switches the operation guide from the normal display to the specific display when at least one of the following occurs: no touch operation is made on the displayed image for a certain period of time; a touch operation other than a touch operation is made on the displayed image; a predetermined operation to switch to the specific display is input by the operator; the displayed image is no longer displayed; or the operating state of the vehicle body changes.

[0084] According to this embodiment, the operation guide showing how to operate the displayed image of the camera image is displayed with an icon that resembles the operation form of a human hand or finger, so that the function of the operation method can be intuitively understood. Furthermore, by displaying the operation guide at the appropriate time and in the appropriate place, it is possible to achieve both ease of checking the surroundings through the displayed image and ease of understanding the operation through the operation guide, thereby improving work efficiency.

[0085] Note that the components of the control device 100, as well as their functions and execution processes, may be partially or entirely implemented by hardware (e.g., by designing logic for executing each function using an integrated circuit). The components of the control device 100 may also be implemented as a program (software) that is read and executed by an arithmetic processing device (e.g., a CPU) to implement each function of the control device 100. Information related to the program may be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), or a recording medium (magnetic disk, optical disk, etc.).

[0086] In addition, in the above description of the embodiment, the control lines and information lines are those that are considered necessary for the description of the embodiment, but they do not necessarily represent all the control lines and information lines related to the product. In reality, it can be considered that almost all components are interconnected.

[0087] DESCRIPTION OF SYMBOLS 1 Hydraulic excavator (work machine) 1A Front work machine (work device) 1B Vehicle body (work machine main body) 1a Boom 1b Arm 1c Bucket 1d Upper rotating body 1e Lower traveling body 1f Operator's room (cab) 7a, 7b, 7c, 7d Operation lever 8a, 8b, 8c Angle sensor 9a, 9b GNSS antenna 10a, 10b, 10c Camera 11 Speaker 12 Monitor 15 Touch panel 17 Shut-off lever 100 Control device 101 Display image generation unit 102 Display image setting unit 103 Touch operation acquisition unit 104 Operation guide generation unit 105 Operation guide setting unit 106 Display control unit 107 Operation guide display determination unit 108 Vehicle body state acquisition unit

Claims

1. A surroundings monitoring system for a work machine comprising: a camera attached to a vehicle body; a monitor; and a control device which generates a display image including camera footage acquired from said camera and displays said generated display image on said monitor, wherein said monitor is capable of receiving input via a touch operation, and said control device simultaneously displays on said monitor an operation guide which imitates an operator's operating style, which shows how to touch the monitor to change settings of said display image, and the display image.

2. A surroundings monitoring system for a work machine as described in claim 1, wherein the operation guide includes an expression showing a hand or finger, or a hand or finger movement.

3. A work machine surroundings monitoring system as described in claim 1, characterized in that, when a touch operation is performed on the displayed image, the control device changes the settings of the displayed image in response to the touch operation.

4. A surrounding monitoring system for a work machine as described in claim 1, characterized in that the control device generates the operation guide including an expression indicating the setting content of the displayed image according to the operation content of the touch operation.

5. A surrounding monitoring system for a work machine as described in claim 1, characterized in that the control device generates the operation guide including a representation indicating that the setting operation cannot be performed when a setting operation for changing the settings of the displayed image cannot be performed due to specified conditions.

6. A surroundings monitoring system for a work machine as described in claim 1, characterized in that the control device displays the operation guide on the monitor by superimposing it on the displayed image.

7. A surroundings monitoring system for a work machine as described in claim 1, wherein said control device also generates a semi-transparent background image around said operation guide.

8. A surroundings monitoring system for a work machine as described in claim 1, wherein said control device emphasizes the surroundings of said displayed image together with said operation guide.

9. A surroundings monitoring system for a work machine as described in claim 1, characterized in that the control device is capable of changing at least one of the display settings of the operation guide, namely, display, non-display or display position of the operation guide.

10. A surroundings monitoring system for a work machine as described in claim 1, wherein the control device is capable of switching between a normal display of the operation guide and a special display in which the visibility of the operation guide is lower than that of the normal display in order to improve the visibility of the displayed image based on predetermined conditions, and the special display includes at least one of changing the display size of the operation guide or changing the transparency at which it is displayed.

11. A work machine surroundings monitoring system as described in claim 1, wherein the control device normally displays the operation guide when a touch operation is performed on the displayed image.

12. A surroundings monitoring system for a work machine as described in claim 1, characterized in that the control device normally displays the operation guide when the surroundings monitoring system is started and the display image is displayed on the monitor.

13. A surroundings monitoring system for a work machine as described in claim 10, wherein the control device switches the operation guide from normal display to specific display when a predetermined condition is satisfied.

14. A surroundings monitoring system for a work machine as described in claim 1, wherein the control device is capable of changing at least one of the size or pattern of the displayed image in response to the touch operation, and the pattern of the displayed image includes at least one of an overhead image in which the camera image is converted into an image from a viewpoint seen from directly above the vehicle body, or an image in which the camera image is displayed without performing viewpoint conversion.

15. A surrounding monitoring system for a work machine as described in claim 1, characterized in that the control device displays the operation guide by superimposing it on the sky or the area in which the vehicle body is shown in the displayed image, or on a representation that resembles the vehicle body.

16. A surroundings monitoring system for a work machine as described in claim 1, wherein the control device is capable of switching, based on predetermined conditions, between a normal display of the operation guide and a specific display in which the visibility of the operation guide is lower than the normal display in order to improve the visibility of the displayed image, and wherein the operation guide is switched from the normal display to the specific display in response to at least one of the following: no input via touch operation is made to the displayed image for a certain period of time; a touch operation other than a touch operation is input to the displayed image; a predetermined operation for switching to the specific display is input by the operator; the displayed image is made invisible; or the operating state of the vehicle body has changed.

Images