Periphery monitoring system, and work machine comprising the same

Abstract

To provide a periphery monitoring system that can maintain intuitiveness achieved by using a touch panel, and a work machine comprising the same.SOLUTION: A periphery monitoring system that makes it easy to understand the operation method in an intuitive manner is provided in which an operation guide for touch operations relating to settings of a camera image display (an operation guide showing an operation method of touch operation to a monitor 12 to change the settings of a displayed image) is displayed together with the displayed image in a form including icons that mimic an operating form of a human hand or finger. Also provided is a work machine comprising the periphery monitoring system.SELECTED DRAWING: Figure 14B

Description

[Technical field]

[0001] The present invention relates to a surroundings monitoring system for a work machine equipped with a work implement, such as a hydraulic excavator, and to a work machine equipped with the same. [Background technology]

[0002] In recent years, work machines such as hydraulic excavators have become increasingly computerized and informatized, and as the number of electronic devices mounted on the machines increases, the number of functions that support work has also increased.

[0003] For example, there is a function that notifies the operator of the situation around the work machine by displaying images on a monitor installed in the cab based on information obtained from multiple cameras mounted on the work machine. The images displayed on the monitor can be selected from several patterns, such as an image obtained from multiple cameras that is synthesized into an image seen from directly above the vehicle body as if looking down, or an image obtained from multiple cameras that is displayed side by side (without viewpoint conversion, etc.), and the operator can switch between the images displayed on the monitor according to his / her preference.

[0004] In addition to the camera image, the monitor may also display icons indicating the various settings of the vehicle, meters showing the remaining fuel and engine coolant temperature, and the status of various devices such as the radio and air conditioner.

[0005] In recent years, a function called machine guidance (MG) has been developed that calculates the position and attitude of the work machine and the work equipment from information acquired by various sensors attached to the work machine and the work equipment of the work machine, and assists the work by notifying the operator of the positional relationship between the work machine and the work equipment and the target surface during construction.

[0006] In addition, there is also an area restriction function that calculates the degree to which the work machine and work equipment are approaching the boundary of the work area based on information on the position and attitude of the work machine and work equipment and information on a preset work area, and notifies the operator and imposes operational restrictions on the work machine and work equipment according to the degree of proximity, thereby assisting in working within the work area.

[0007] Thus, while the number of functions to assist with work is increasing, the number of items that can be set on the monitor is also increasing along with the increase in functions, which raises concerns that setting operations on the monitor may become cumbersome and usability may decrease.

[0008] In the work machine of Patent Document 1, at least one of the display position of the driving information such as meters on the monitor and the camera image to be displayed can be switched by operating a single changeover switch. Therefore, when the operator wants to switch the display, the time and effort required to search for the button to switch is reduced, and the display can be switched with a simple operation, improving the efficiency of operation. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 6692744 Summary of the Invention [Problem to be solved by the invention]

[0010] In the work machine of Patent Document 1, at least one of the display position of the operating information such as meters on the monitor and the camera image to be displayed is changed by operating a single changeover switch. Meanwhile, in recent years, the monitors of work machines have become increasingly equipped with touch panels, and it has become possible to change various settings and switches by directly operating (touching) the information displayed on the monitor. Being able to directly operate the information displayed on the monitor allows for more intuitive operation.

[0011] Furthermore, it is possible to change the content to be switched and the assigned function depending on the type of touch operation (tap, swipe, drag, etc.). Therefore, even if the same area on the monitor is touched, various types of switching and changes can be performed depending on the operation method, which is expected to improve convenience.

[0012] On the other hand, even if the touch area is the same, if the type of switching differs depending on the operation, it becomes difficult to know which operation can be used to perform which switching. In recent years, with the spread of smartphones, operations using touch panels have become common, and there are an increasing number of operations in which functions can be associated with the operation method, such as zooming in and out by pinching in and out. However, there are also cases in which the information displayed on the monitor of a work machine and the functions assigned to each piece of information cannot be associated with the function of the operation method. For example, in the case of an operation specific to a work machine, such as an operation to switch the display pattern of a camera image, it is difficult to associate the function with the operation method, and the operation method for the acceptable operations or the operation corresponding to the function becomes unclear, resulting in a decrease in usability of touch operations.

[0013] Therefore, an object of the present invention is to provide a surroundings monitoring system that can maintain the intuitiveness provided by a touch panel, and a work machine equipped with the same. [Means for solving the problem]

[0014] In order to achieve the above-mentioned object, the present invention provides an ambient monitoring system comprising a camera, a monitor, and a control device that generates a display image including a camera image acquired from the camera and displays the generated display image on the monitor, wherein the monitor is capable of accepting input via a touch operation, and the control device simultaneously displays on the monitor an operation guide that mimics an operator's operating form, which shows how to touch the monitor to change the settings of the display image, and the display image. Effect of the Invention

[0015] According to the present invention, the operation guide showing the operation method of the displayed image of the camera image is displayed including an icon that resembles the operation form of a human hand or finger, so that the function of the operation method can be intuitively understood. Also, by displaying the operation guide at the appropriate time and in the appropriate place, it is possible to achieve both ease of checking the surroundings through the displayed image and ease of understanding the operation through the operation guide, thereby improving work efficiency. [Brief description of the drawings]

[0016] [Figure 1] 1 is a right side view of a hydraulic excavator 1 which is an example of a work machine according to an embodiment of the present invention. [Diagram 2] Schematic diagram showing the interior of the driver's cab 1F. [Diagram 3] System configuration diagram of hydraulic excavator 1. [Figure 4] FIG. 1 is a functional block diagram of a control device 100 according to an embodiment of the present invention. [Diagram 5] 11A to 11C are diagrams showing examples of display image patterns. [Figure 6] 4 is a flowchart of the display video generating unit 101. [Figure 7] 11A and 11B are diagrams showing examples of patterns of settings corresponding to operation contents. [Figure 8] 11A and 11B are diagrams showing examples of switching of display images by touch operations. [Figure 9] 4 is a flowchart of the display video setting unit 102. [Figure 10] 4 is a flowchart of the operation guide setting unit 105. [Figure 11] 6 is a flowchart showing the operation guide display determination unit 107 determining whether to perform normal display. [Figure 12] 6 is a flowchart showing the operation guide display determining unit 107 determining whether to perform reduced display. [Figure 13] 4 is a flowchart of the operation guide generating unit 104. [Figure 14A] An example of how the operation guide is displayed (when the operation guide is not displayed). [Figure 14B] An example of how the operation guide is displayed (when the operation guide is displayed normally). [Figure 14C] An example of how the operation guide is displayed (when the operation guide is displayed in a reduced size). [Figure 15] An example of how the operation guide is displayed (when the operation guide is displayed normally). [Figure 16] Example of operation guide display (when the operation guide is superimposed on the vehicle icon). [Figure 17A] An example of how the operation guide is displayed (when the operation guide is not displayed). [Figure 17B] An example of the operation guide display (when the operation guide is superimposed on the image of the vehicle). [Figure 17C] Example of operation guide display (when operation guide is superimposed on empty area). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] FIG. 1 shows an overview of a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention. The hydraulic excavator 1 is composed of a multi-joint front working machine 1A consisting of a boom 1a, an arm 1b, and a bucket 1c, each of which rotates vertically, and a vehicle body 1B consisting of an upper rotating body 1d and a lower traveling body 1e. The upper rotating body 1d is attached to the upper part of the lower traveling body 1e so as to be able to rotate left and right. The upper rotating body 1d is also equipped with a driver's room (also called a cab) 1f. The base end of the boom 1a of the front working machine 1A is supported at the front part of the upper rotating body 1d. The boom 1a, the arm 1b, the bucket 1c, the upper rotating body 1d, and the lower traveling body 1e are driven by actuators of a boom cylinder 2a, an arm cylinder 2b, a bucket cylinder 2c, a swing motor 3d, and left and right traveling motors 3e, 3f, respectively. The upper rotating body 1d is equipped with an engine 4, a pump 5, and a valve 6 for operating the hydraulic excavator 1. The pump 5 is driven by the rotation of the engine 4 and discharges hydraulic oil to each actuator. The valve 6 controls the flow rate and direction of the hydraulic oil by input from operation levers 7a, 7b, 7c, and 7d (FIG. 2) provided in the driver's cab 1f, and operates each actuator. Furthermore, the boom 1a, the arm 1b, and the bucket 1c are equipped with angle sensors 8a, 8b, and 8c for detecting the rotation angles of each part, and the upper rotating body 1d is equipped with GNSS antennas 9a and 9b for acquiring the position of the hydraulic excavator 1, and cameras 10a, 10b, and 10c for photographing the surroundings of the work machine are attached to the left, rear, and right sides of the upper rotating body 1d. In addition, the upper rotating body 1d is equipped with a control device 100 that generates an image (display image) to be displayed on the monitor 12 (Figure 2) based on the images (camera images) obtained from the cameras 10a, 10b, and 10c, and performs processing such as displaying the generated display image on the monitor 12.

[0019] Fig. 2 shows the interior of the cab 1f. In the cab 1f, there are installed operation levers 7a, 7b, 7c, and 7d, a shutoff lever 17 that locks the operation input from the operation levers to the actuators, a speaker 11 that transmits information to the operator by sound, and a monitor 12 as a display device for displaying information. The monitor 12 of this embodiment is also equipped with a touch panel 15 that accepts input by touch operation. Such a monitor 12 is also called a touch panel monitor.

[0020] FIG. 3 shows the overall configuration of an information input system for a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention. The control device 100 for the hydraulic excavator 1 according to this embodiment includes a main C / U (control unit) 50, a GNSS C / U 40 that processes information from the GNSS antennas 9a and 9b, and a notification C / U 60 that controls various notifications to the operator. The main C / U 50 controls basic systems for the work machine, such as the engine 4 and the pump 5, and is connected to a key cylinder, and also controls the electrical start of the work machine and the start of the engine 4, which is the prime mover. The main C / U 50 is also connected to a communication terminal, and communicates with various terminals and servers outside the hydraulic excavator 1. The notification C / U 60 calculates display information for the monitor 12 and sound information from the speaker 11 based on the processing state of the main C / U 50, and sends a signal. Furthermore, the notification C / U 60 sends various setting signals to the main C / U 50 in accordance with input by touch operation from the touch panel 15.

[0021] Fig. 4 is a functional block diagram of a control device 100 in an embodiment of the present invention. The control device 100 includes a display image generating unit 101, a display image setting unit 102, a touch operation acquiring unit 103, an operation guide generating unit 104, an operation guide setting unit 105, a display control unit 106, an operation guide display determining unit 107, and a vehicle body state acquiring unit 108. Each block will be described below. The devices shown in Fig. 4 constitute a surroundings monitoring system that monitors the surroundings of the hydraulic excavator 1.

[0022] Display image generating unit 101 synthesizes images (camera images) acquired from cameras 10a, 10b, and 10c in accordance with settings, and generates a display image to be displayed on monitor 12. Five patterns of display images generated in this embodiment are shown in FIG. 5. Display images 201 and 202 are display images of the same size. Display images 203, 204, and 205 are display images of the same size, but are different sizes from display images 201 and 202.

[0023] Displayed image 201 is an image (overhead image) obtained by projecting images acquired from cameras 10a, 10b, and 10c onto an image of a viewpoint looking down from the sky (seen directly above the hydraulic excavator). Of displayed image 201, image 201a is an area onto which image from left camera 10a is projected, image 201b is an image from rear camera 10b, and image 201c is an area onto which image from right camera 10c is projected. Vehicle body icon 201d is an icon imitating the shape of a hydraulic excavator, and is displayed to assist in understanding the positional relationship between the projected image of the surroundings and the vehicle body itself.

[0024] Display image 202 is an image in which images from cameras 10b and 10c are displayed side by side without projection using viewpoint conversion, etc. Of display image 202, image 202b displays an image from camera 10b on the left side, and image 202c displays an image from camera 10c on the right side.

[0025] Display image 203 is an image in which images acquired from cameras 10a, 10b, and 10c are projected onto an image of a viewpoint looking down from the sky (seen directly above the hydraulic excavator) (overhead image), and the image from camera 10b are displayed side by side. Of display image 203, image 203a is an area onto which the image from left camera 10a is projected, image 203b is an area onto which the image from rear camera 10b is projected, and image 203c is an area onto which the image from right camera 10c is projected. Image 203e is an image in which the image from rear camera 10b is displayed without being projected. Vehicle body icon 203d is an icon imitating the shape of a hydraulic excavator, and like vehicle body icon 201d, is displayed to assist in understanding the positional relationship between the projected image of the surroundings and the vehicle body itself.

[0026] Displayed image 204 is an image in which images from cameras 10a, 10b, and 10c are displayed side by side without projection using viewpoint conversion, etc. Of displayed image 204, image 204a displays an image from left camera 10a, image 204b displays an image from rear camera 10b, and image 204c displays an image from right camera 10c.

[0027] Display image 205 is an image (bird's-eye view image) obtained by projecting images acquired from cameras 10a, 10b, and 10c onto an image of a viewpoint looking down from the sky (seen directly above the hydraulic excavator). Display image 205 differs in display size and display area from display image 201 and display image 203, and displays up to the front side of the vehicle body. Of display image 205, image 205a is an area onto which image from left camera 10a is projected, image 205b is image from rear camera 10b, and image 205c is image from right camera 10c. Vehicle body icon 205d is an icon imitating the shape of a hydraulic excavator, and like vehicle body icon 201d, is displayed to assist in understanding the positional relationship between the projected surrounding image and the vehicle body itself.

[0028] FIG. 6 shows a flowchart of the display image generating unit 101. First, in step S300, the display image generating unit 101 checks whether camera images have been acquired from the cameras 10a, 10b, and 10c. If camera images have not been acquired, the process ends. If camera images have been acquired, the process proceeds to step S301. In step S301, the display image settings are checked. In this embodiment, which display pattern should be displayed (which display pattern should be selected) among the display patterns shown in FIG. 5 is checked in this step. After checking the settings, the process proceeds to step S302, where the images acquired from the cameras 10a, 10b, and 10c are actually processed according to the settings to generate a display image. When the generation of the display image is completed, the process proceeds to step S303, where the generated display image is output, and the process ends.

[0029] The display image setting unit 102 changes the settings of the display image generated by the display image generating unit 101 by a touch operation on the display image. In this embodiment, the setting is changed by switching the pattern of the display image shown in Fig. 5 by a touch operation on the display image. The change in the setting varies depending on the type of touch operation. Fig. 7 shows a table 400 showing the correspondence between the type of touch operation (operation content 401) and the change in the setting (setting content 410).

[0030] In this embodiment, four types of touch operations are assumed: operation contents 402, 403, 404, and 405. Operation content 402 is a swipe operation in which one finger is touched on the touch panel 15 and the finger is moved. Operation content 403 is a pinch-in operation in which two fingers are touched on the touch panel 15 and the fingers are moved to narrow the distance between them. Operation content 404 is a pinch-out operation in which two fingers are touched on the touch panel 15 and the fingers are moved to widen the distance between them. Operation content 405 is a tap operation in which one finger is touched on the touch panel 15 and then released.

[0031] In this embodiment, three types of setting changes are assumed: setting contents 411, 412, and 413. As the setting contents corresponding to operation content 402, switching the pattern of the display image within the same display size is assigned as setting content 411. As the setting contents corresponding to operation content 403, reducing the size of the displayed image is assigned as setting content 412. As the setting contents corresponding to operation content 404, enlarging the size of the displayed image is assigned as setting content 413. As the setting contents corresponding to operation content 405, no setting contents are assigned, so setting content 414 has no setting contents.

[0032] FIG. 8 shows the relationship between touch operations and display images generated by display image generating unit 101. Monitor displays 211, 212, 213, 214, and 215 are the display images described in FIG. 5 displayed on monitor 12 equipped with touch panel 15. Touch panel 15 has touch area 15a that responds to touch operations, and monitor 12 has liquid crystal display area 12a where images are actually displayed. Monitor display 211 displays display image 201, monitor display 212 displays display image 202, monitor display 213 displays display image 203, monitor display 214 displays display image 204, and monitor display 215 displays display image 205. Group 220 includes monitor display 211 and monitor display 212, which have the same size of display images. Group 221 also includes monitor displays 213, 214, and 215, which have the same size of display images. In table 400 of FIG. 7, from operation content 402 and setting content 411, a swipe operation is assigned to pattern switching within a group having the same size of display video.

[0033] For switching between groups 220 and 221, which have different sizes of display images, pinch-in and pinch-out operations are assigned according to operation contents 403 and 404 and setting contents 412 and 413. When switching from group 220 to group 221, the size of the display image increases, so switching is performed when a pinch-out operation is input. When switching from group 221 to group 220, the size of the display image decreases, so switching is performed when a pinch-in operation is input.

[0034] FIG. 9 shows a flowchart of the display image setting unit 102. In step S310, it is determined whether a touch operation has been performed on the currently displayed display image. If a touch operation has not been performed on the display image, the process ends. If a touch operation has been performed on the display image, the process proceeds to step S311. In step S311, it is determined whether the type of touch operation is an operation to which a setting has been assigned. Since settings are assigned to the swipe, pinch in, and pinch out in the table 400 in FIG. 7, if any of the touch operations of swipe, pinch in, and pinch out has been input, the process proceeds to step S312. If a touch operation other than swipe, pinch in, and pinch out has been input, since no setting has been assigned, the process ends. In step S312, the setting contents assigned to the contents of the input touch operation are confirmed based on the table 400 in FIG. 7. After confirmation of the setting contents has been completed, the process proceeds to step S313. In step S313, the setting contents of the display image generating unit 101 are changed according to the setting contents. When the change in the settings of display video generating unit 101 is complete, the process ends.

[0035] The touch operation acquisition unit 103 acquires and outputs information on input by an operator's touch operation on the touch panel 15 .

[0036] The operation guide setting unit 105 changes the settings of the operation guide generated by the operation guide generating unit 104 by a touch operation on the monitor.

[0037] FIG. 10 shows a flowchart of the operation guide setting unit 105. In this embodiment, the flow shows a case where the display / non-display switching of the operation guide and the display position of the operation guide can be changed. In step S320, it is confirmed whether a touch operation is input to the monitor 12 (the touch panel 15). If a touch operation is not input, the process is terminated. If a touch operation is input, the process proceeds to step S321. In step S321, it is determined whether the input touch operation is an operation to change the operation guide display. An example of an operation to change the operation guide display is an operation to select and confirm the change content of the operation guide display on a menu screen. If an operation to change the operation guide display is not input, the process is terminated. If an operation to change the operation guide display is input, the process proceeds to step S322. In step S322, it is determined whether the content of the change operation of the operation guide display is an operation to switch between displaying and hiding the operation guide. If it is an operation to switch between displaying and hiding the operation guide, the process proceeds to step S323. In step S323, the display / non-display of the operation guide is changed according to the content of the change operation of the operation guide display, and the process is terminated. If it is determined in step S322 that the operation to change the operation guide display is not a switching operation to display or hide the operation guide, the process proceeds to step S324. In step S324, it is determined whether the operation to change the operation guide display is a change in the display position of the operation guide. If it is not an operation to change the display position of the operation guide, the process ends. If it is an operation to change the display position of the operation guide, the process proceeds to step S325. In step S325, the display position of the operation guide is changed according to the content of the operation to change the operation guide display, and the process ends. In this embodiment, the display position of the operation guide can be changed to the top or left side of the displayed image. Note that the case where the top of the displayed image is selected as the display position of the operation guide is shown in Figs. 14B and 14C, and the case where the left side of the displayed image is selected is shown in Fig. 15.

[0038] The operation guide display determination unit 107 determines whether the operation guide is to be displayed in a normal display or in a reduced display. FIG. 11 shows a flowchart for determining whether the operation guide is to be displayed in a normal display in the operation guide display determination unit 107. In step S340, it is determined whether a touch operation has been performed on the display image on the monitor 12. If a touch operation has been performed, the process proceeds to step S342. In step S342, the operation guide is set to be displayed in a normal display, and the process ends. If it is determined in step S340 that a touch operation has not been performed, the process proceeds to step S341. In step S341, it is determined whether the display image is displayed for the first time on the monitor 12 after the system is started. If the display image is displayed for the first time on the monitor 12 after the system is started, the process proceeds to step S342. As above, in step S342, the operation guide is set to be displayed in a normal display, and the process ends. If the display image is not displayed for the first time on the monitor 12 after the system is started, the process ends.

[0039] FIG. 12 shows a flowchart for determining whether or not to perform a reduced display in the operation guide display determination unit 107. In step S350, it is determined whether or not a touch operation has not been input to the display image on the monitor 12 for a certain period of time or more. If a touch operation has not been input to the display image for a certain period of time or more, the process proceeds to step S355. In step S355, a setting is made to display the operation guide in a reduced display (in other words, the operation guide is switched to a reduced display), and the process ends (similar below). In step S350, if a touch operation has been input to the display image within a certain period of time, the process proceeds to step S351. In step S351, it is determined whether or not a touch operation has been input to a part other than the display image on the monitor 12 (in other words, a touch operation other than a touch operation to the display image). If a touch operation has been input to a part other than the display image, the process proceeds to step S355. If a touch operation has not been input to a part other than the display image, the process proceeds to step S352. In step S352, it is determined whether or not a predetermined operation has been input. The predetermined operation here refers to an operation performed to reduce the display of the operation guide (in other words, to switch the display of the operation guide to a reduced display). In this embodiment, a reduced display switching icon 256 is provided on the operation guide, and when the reduced display switching icon 256 is touched, it is determined that the predetermined operation has been performed. The reduced display switching icon 256 will be described later with reference to FIG. 15. If it is determined that the predetermined operation has been input, the process proceeds to step S355. If it is determined that the predetermined operation has not been input, the process proceeds to step S353. In step S353, it is determined whether the display on the monitor 12 has been switched to a screen on which the display image is not displayed (in other words, whether the display image has become hidden). If it has been switched to a screen on which the display image is not displayed (in other words, the display image has become hidden), the process proceeds to step S355. An example of a screen on which the display image is not displayed is a case where the display image is not displayed when a menu screen for performing various settings is displayed. If the display image is displayed on the screen, the process proceeds to step S354.In step S354, it is determined whether the operating state of the work machine has changed. In this embodiment, whether the operating state of the work machine has changed is determined based on whether the lock state of the shutoff lever 17 (acquired by the vehicle body state acquisition unit 108, described below) has changed. If it is determined that the operating state of the work machine has changed, the process proceeds to step S355. If it is determined that the operating state of the work machine has not changed, the process ends (i.e., the operation guide is not switched to a reduced display).

[0040] The operation guide generating unit 104 generates an operation guide showing an operation method of touch operation on the monitor 12 for changing the setting of the display image (in other words, an operation guide of a touch operation related to the setting of the camera image display on the monitor) based on information from the display image setting unit 102, the touch operation acquisition unit 103, the operation guide setting unit 105, the operation guide display determination unit 107, etc. In this embodiment, the operation guide includes an icon imitating the operation form of the operator, such as an icon imitating a human hand or finger, or an icon imitating the movement of a human hand or finger, an icon showing the setting content 410 of the display image corresponding to the operation content 401 of the touch operation (also see table 400 in FIG. 7). In addition, in this embodiment, the operation guide generating unit 104 generates an operation guide for normal display or reduced display based on a predetermined condition (in other words, it switches between normal display and reduced display of the operation guide), and the operation guide includes information on the display position, etc.

[0041] FIG. 13 shows a flowchart of the operation guide generating unit 104. In step S330, it is confirmed whether the display setting of the operation guide is hidden. If it is set to hidden, the process proceeds to step S336. In step S336, the operation guide is hidden, and the process is terminated without generating the operation guide or the like. In step S330, if the display setting of the operation guide is not set to hidden (set to display), the process proceeds to step S331. In step S331, information on the setting contents 410 and the operation contents 401 that can be set by the display image setting unit 102 is acquired. When the acquisition of information from the display image setting unit 102 is completed, the process proceeds to step S332. In step S332, the display setting of the operation guide is confirmed. The display setting of the operation guide includes information on the display position of the operation guide, etc. When the confirmation of the display setting of the operation guide is completed, the process proceeds to step S333. In step S333, it is determined whether the display setting of the operation guide is reduced display or normal display. When it is determined that the display setting of the operation guide is reduced display, the process proceeds to step S334. In step S334, a manipulation guide for reduced display is generated, and the process ends. If it is determined in step S333 that the display setting of the manipulation guide is normal display, the process proceeds to step S335. In step S335, a manipulation guide for normal display is generated, and the process ends.

[0042] The display control unit 106 performs processing to display the display image generated by the display image generation unit 101 and the operation guide generated by the operation guide generation unit 104 on the monitor 12. In this embodiment, only the display image and the operation guide are displayed, but information indicating the setting state of the work machine and the setting states of various devices in the cab such as the air conditioner and radio may also be displayed together.

[0043] The vehicle body state acquisition unit 108 acquires the operating state of the work machine to be used in the determination by the operation guide display determination unit 107. In this embodiment, the locked state of the shut-off lever 17 is acquired and output to the operation guide display determination unit 107.

[0044] Figures 14A, 14B, 14C, and 15 show display examples of the operation guide. Each display example shows a case where display image 205 is displayed. Figure 14A shows monitor display 215 when the operation guide is not displayed, Figure 14B shows monitor display 230 when the operation guide is normally displayed, and Figure 14C shows monitor display 240 when the operation guide is displayed in a reduced size.

[0045] On the monitor display 215, the display setting for the operation guide is set to non-display (step S336 in FIG. 13), so the display image 205 is displayed but the operation guide is not displayed.

[0046] The monitor display 230 is a display example in the case where the operation guide is normally displayed (step S335 in FIG. 13). As the operation guide, setting content guides 233 and 234 are displayed together with touch operation guides 231 and 232. The touch operation guides 231 and 232 indicate the operation method (operation form) of the touch operation, and the setting content guides 233 and 234 indicate what kind of display image is set by the touch operation (i.e., the setting content of the display image according to the operation content of the touch operation). The touch operation guide 231 is an icon indicating a swipe operation, and the setting content guide 233 corresponding to the touch operation guide 231 indicates a pattern switch of the display image, and these two icons indicate that the pattern of the display image is switched by performing a swipe operation. The touch operation guide 232 is an icon indicating a pinch-in operation, and the setting content guide 234 corresponding to the touch operation guide 232 indicates the reduction of the display image, and these two icons indicate that the display image is reduced by performing a pinch-in operation. The background image 235 is displayed around the operation guides (231 to 234) and is a background image that is superimposed on the display image together with the operation guides (231 to 234) to make the operation guides easy to see regardless of the state of the display image. The operation guides (231 to 234) and the background image 235 are displayed in a semi-transparent color. The background image 235 is generated by the operation guide generating unit 104 in accordance with the operation guides.

[0047] The monitor display 240 is a display example in the case where the operation guide is displayed in a reduced size (step S334 in FIG. 13). As the operation guide, touch operation guides 241 and 242 are displayed. The touch operation guide 241 is an icon indicating a swipe operation, and the touch operation guide 242 is an icon indicating a pinch-in operation. The touch operation guides 241 and 242 are displayed in a smaller size than the touch operation guides 231 and 232 in the normal display. As in the normal display, the background image 243 is also displayed around the operation guides (241 and 242), and the operation guides (241 and 242) and the background image 243 are displayed in a semi-transparent color. The background image 243 is generated by the operation guide generating unit 104 in accordance with the operation guide. In this embodiment, the size of the icon to be displayed is reduced as the reduced display, but the display color and transparency may be changed, or the operation guide may not be displayed.

[0048] FIG. 15 shows a monitor display 250 as a display example of another form when the operation guide is normally displayed. As the operation guide, touch operation guides 251 and 252, setting content guides 253 and 254, and an operation area guide 257 are displayed. The touch operation guide 251 is an icon indicating a swipe operation, and the setting content guide 253 corresponding to the touch operation guide 251 indicates a pattern change of the display image, and these two icons indicate that the pattern of the display image is changed by performing a swipe operation. The touch operation guide 252 is an icon indicating that a pinch-in operation is not possible, and the setting content guide 254 corresponding to the touch operation guide 252 indicates that the display image cannot be reduced, and these two icons indicate that the display image cannot be reduced by the pinch-in operation. The operation area guide 257 is a guide indicating an area in which the operation shown in the operation guide is accepted, and in this embodiment, since the touch operation on the display image is the target, it is displayed together with the operation guide so as to surround the periphery of the display image. The reduced display switching icon 256 is an icon that accepts an input to switch the display from normal display to reduced display, and when this icon is tapped, the display of the operation guide is switched to reduced display. The background image 255 is displayed around each icon (251 to 254) of the operation guide and the reduced display switching icon 256. Each icon (251 to 254) of the operation guide, the reduced display switching icon 256, and the background image 255 that are superimposed on the display image use semi-transparent colors. The background image 255 is generated by the operation guide generating unit 104 in accordance with each icon (251 to 254) of the operation guide and the reduced display switching icon 256. In addition, the operation area guide 257 is also generated by the operation guide generating unit 104 so as to emphasize its surroundings in accordance with the display image.

[0049] Furthermore, in monitor display 250 (FIG. 15), the position at which the operation guide is displayed has been changed from that in monitor display 230 (FIG. 14B), which also normally displays the operation guide. Monitor display 230 (FIG. 14B) is a display example in which the top of the displayed image is selected as the display position for the operation guide (and the background image surrounding it), while monitor display 250 (FIG. 15) shows a display example in which the left side of the displayed image is selected.

[0050] 16 shows a display example in which an operation guide is displayed superimposed on an icon resembling a car body. Monitor display 260 shows an example in which display image 205 is displayed. Touch operation guides 261 and 262 are displayed as operation guides. Monitor display 260 displays touch operation guides 261 and 262 and background image 263 in a form superimposed on car body icon 205d. By displaying touch operation guides 261 and 262 in a form superimposed on car body icon 205d, touch operation guides 261 and 262 do not superimpose on the camera image in the display image, and it is possible to display touch operation guides 261 and 262 without blocking the surrounding field of view.

[0051] 17A, 17B, and 17C show display examples in which an operation guide is superimposed on the display image 204. In FIG. 17A, the monitor display 270 shows an example in which the surrounding image is displayed on the display image 204. On the camera image, the subject vehicle body images 271a, 271b, and 271c are displayed, and the shape and size of the images vary depending on the shape of the subject vehicle body, the mounting position of the camera, lens distortion, and the like. In addition, the subject vehicle body side of the camera image is the ground area 273a, 273b, and 273c, which shows the ground surface, etc. Conversely, the farther side of the camera image from the subject vehicle body are the sky areas 274a, 274b, and 274c in which the sky and tall three-dimensional objects are displayed. In this embodiment, for the sake of explanation, virtual horizons 272a, 272b, and 272c that are boundaries between ground areas 273a, 273b, and 273c and sky areas 274a, 274b, and 274c are displayed.

[0052] 17B, ​​an example is shown in which an operation guide is displayed superimposed on the vehicle body captured in the camera image on monitor display 280. Touch operation guides 281 and 282 are displayed as operation guides. Touch operation guides 281 and 282 and a background image 283 are displayed superimposed on vehicle body image 271b. By displaying the operation guide superimposed on the vehicle body in the camera image, it is possible to display touch operation guides 281 and 282 without blocking the surrounding field of view.

[0053] 17C, monitor display 290 shows an example in which an operation guide is displayed in an area showing the sky on the camera image. Touch operation guides 291 and 292 are displayed as the operation guide. Touch operation guides 291 and 292 and background image 293 are displayed superimposed on sky area 274a. The sky area is an area showing the sky or the distance from the vehicle body, and is unlikely to get in the way when checking the vicinity of the vehicle body, and is unlikely to obstruct the surrounding field of view.

[0054] As described above, according to this embodiment, by displaying an icon imitating the operating form of an operator, such as a human hand or finger shape, on the monitor 12, as in the touch operation guides 231, 232 of the monitor display 230 in FIG. 14B, it is possible to easily grasp that some operation or setting can be performed on the display of the camera image by inputting a touch operation.

[0055] Also, by displaying operation guides that represent the movements of human hands and fingers, such as the touch operation guides 231 and 232, it becomes possible to easily grasp the types of touch operations that can be input.

[0056] Furthermore, by displaying setting content guides 233 and 234 on the monitor 12 together with the touch operation guides 231 and 232, it becomes possible to easily grasp what setting operations are assigned to the types of touch operations.

[0057] Furthermore, by displaying the background image 235 around the touch operation guides 231 and 232, the touch operation guides 231 and 232 are less likely to become difficult to view due to the display content of the display image.

[0058] Furthermore, by making the touch operation guides 231 and 232 and the background image 235 semi-transparent, it becomes possible to see through the superimposed display image, and it becomes possible to perform operations while checking what is shown in the display image.

[0059] Furthermore, as shown in monitor display 290 in FIG. 17C, by superimposing the operation guide on the upper side of the monitor of the display image 204, the operation guide is superimposed and displayed in an area far from the work machine or in an area showing the sky, so that it is possible to display the operation guide while ensuring visibility in the vicinity of the work machine.

[0060] Furthermore, as in the touch operation guide 252 of the monitor display 250 in FIG. 15, if settings on the displayed image cannot be made by touch operations for some reason (in other words, if some or all of the setting operations for changing the settings of the displayed image cannot be made due to specified conditions), by indicating this state (generating an operation guide including such an icon), the state in which touch operations cannot be made can be easily understood, and work efficiency is also improved.

[0061] Furthermore, by displaying the operation area guide 257 of the monitor display 250 in FIG. 15 together with the touch operation guide 251, it becomes possible to easily grasp the area (display image in this embodiment) that accepts touch operations shown in the operation guide.

[0062] In addition, as shown in Figure 8, the pattern and size of the displayed image can be changed by touch operation, making it easy to change the displayed image to suit the work content and the operator's preferences, thereby improving work efficiency.

[0063] In addition, as shown in the flowchart of Fig. 9, by assigning a setting operation to a touch operation on a displayed image, an intuitive operation can be realized, improving work efficiency. Furthermore, since it is no longer necessary to provide button icons for settings in an area separate from the displayed image, the display area of ​​the monitor 12 can be saved and can be used for displaying other functions.

[0064] Also, as shown in the flowchart in Fig. 10, the operator can change the display position of the operation guide, which can be changed according to the operator's preference or the display image pattern to be used, improving usability. Furthermore, the operator can also set whether to display or hide the operation guide, so that operators who understand the operation method even without the operation guide or operators who find the display of the operation guide bothersome can hide the operation guide, making it possible to set a display that is easy to use according to the operator's preference.

[0065] Also, as shown in the flowchart of Fig. 11, by normally displaying the operation guide when a touch operation is input to the displayed image, the operation guide is displayed when some operation is about to be performed on the displayed image, reducing confusion about how to operate and improving work efficiency. Also, by displaying the operation guide when the displayed image is displayed on the screen after the system is started, it becomes possible to easily understand how to set the displayed image when starting to use the work machine, improving work efficiency.

[0066] Also, as shown in the flowchart of FIG. 12, by reducing the display of the operation guide according to various conditions, when no operation is being performed on the display image, the display of the operation guide is reduced, making it easier to check the display image, and improving work efficiency. As for various conditions, when no touch operation is performed on the display image for a certain period of time, the display is switched to a reduced display, assuming that the setting operation on the display image has ended. When a touch operation other than a touch operation on the display image is input or when the operating state of the work machine changes, the display is also reduced, assuming that the setting operation on the display image has ended. Furthermore, when the display on the monitor 12 is switched to a screen where the display image itself is not displayed (when the display image is hidden), the display of the operation guide is reduced. In this way, the work machine determines that the operation on the display image has ended and switches to a reduced display, thereby improving work efficiency. Also, when the reduced display switching icon 256 is operated, the display is switched to a reduced display. When the operator intentionally wants to switch the operation guide to a reduced display, it can be switched.

[0067] In this embodiment, an example is shown in which the operation guide is displayed in a reduced display size compared to the normal display in order to make the displayed image easier to check (to increase visibility). However, instead of changing (reducing) the display size of the operation guide, or at the same time as changing (reducing) the display size of the operation guide, it is also possible to make the displayed image easier to check by changing the display color or changing (increasing) the display transmittance to create a specific display in which the visibility of the operation guide is lower than the normal display, thereby making it easier to check the displayed image.

[0068] [summary] As described above, the surroundings monitoring system of this embodiment includes cameras 10a, 10b, and 10c mounted on a hydraulic excavator 1, a monitor 12, and a control device 100 that generates a display image including camera images acquired from the cameras and displays the generated display image on the monitor, in which the monitor 12 is capable of accepting input via a touch operation (touch panel 15), and the control device 100 acquires the input via the touch operation on the monitor (touch operation acquisition unit 103), changes a setting of the display image in accordance with the acquired touch operation (display image setting unit 102), processes the camera image acquired from the camera in accordance with the changed setting of the display image, generates the display image to be displayed on the monitor (display image generation unit 101), generates an operation guide imitating an operator's operation form showing a method of operating the touch operation on the monitor to change the setting of the display image (operation guide generation unit 104), and displays the generated operation guide and the display image simultaneously on the monitor (display control unit 106).

[0069] The operation guide includes a representation (icon) of a (human) hand or finger, or a movement of a (human) hand or finger.

[0070] When a touch operation is performed on the display image, the control device 100 (the display image setting unit 102) changes the setting of the display image in response to the touch operation.

[0071] The control device 100 (the operation guide generating unit 104) generates the operation guide including an expression (icon) indicating the setting content of the display image corresponding to the operation content of the touch operation.

[0072] When (some or all) setting operations for changing the settings of the displayed image cannot be performed due to specified conditions, the control device 100 (operation guide generation unit 104) generates the operation guide including an expression (icon) indicating that (some or all) setting operations cannot be performed.

[0073] The control device 100 (operation guide generating unit 104, display control unit 106) displays the operation guide on the monitor 12 by superimposing it on the display image. In addition, the control device 100 (operation guide generating unit 104) generates an operation guide including an icon in a semi-transparent color.

[0074] The control device 100 (the operation guide generating unit 104) also generates a semi-transparent background image around the operation guide.

[0075] The control device 100 (the operation guide generating unit 104) emphasizes the surroundings of the displayed image together with the operation guide (operation area guide 257).

[0076] The control device 100 (the operation guide setting unit 105 and the operation guide generating unit 104) can change at least one of display, non-display and the display position of the operation guide as the display setting of the operation guide.

[0077] The control device 100 (the operation guide display determination unit 107) is capable of switching between a normal display of the operation guide and a special display in which the visibility of the operation guide is lower than that of the normal display in order to increase the visibility of the displayed image (make it easier to see) based on predetermined conditions, and the special display includes at least one of changing (reducing) the display size of the operation guide or changing (increasing) the transparency at which it is displayed.

[0078] The control device 100 (operation guide display determination unit 107) normally displays the operation guide when a touch operation is performed on the display image. The control device 100 (display image setting unit 102) does not change the setting of the display image when a predetermined touch operation is performed on the display image, and the control device 100 (operation guide display determination unit 107) normally displays the operation guide even when the setting of the display image is not changed.

[0079] The control device 100 (the operation guide display determination unit 107) normally displays the operation guide when the surroundings monitoring system is started and the display image is displayed on the monitor.

[0080] The control device 100 (the operation guide display determination unit 107) switches the operation guide from normal display to specific display when a predetermined condition is satisfied.

[0081] Furthermore, the work machine (hydraulic excavator 1) of this embodiment comprises a vehicle body 1B, a work implement (front work implement 1A) attached to the vehicle body 1B, at least one camera 10a, 10b, 10c attached to the vehicle body, a monitor 12, and a control device 100 that generates a display image including camera images acquired from the cameras and displays the generated display image on the monitor, the monitor 12 being capable of accepting input via touch operation (touch panel 15), the control device 100 generating an operation guide imitating the operation form of an operator showing the operation method of the touch operation on the monitor to change the settings of the display image (operation guide generation unit 104), and displaying the generated operation guide and the display image simultaneously on the monitor (display control unit 106).

[0082] The control device 100 (display image setting unit 102) can change at least one of the size or pattern of the display image in response to the touch operation, and the pattern of the display image includes at least one of an overhead image in which the camera image is converted (synthesized) into an image from a viewpoint seen from directly above the work machine (looking down from the sky), or an image in which the camera image is displayed without performing viewpoint conversion.

[0083] The control device 100 (the operation guide generation unit 104 and the display control unit 106) displays the operation guide by superimposing it on an area in the displayed image showing the sky or the vehicle body, or on a representation (icon) that resembles the vehicle body.

[0084] The control device 100 is capable of switching between a normal display of the operation guide and a specific display in which the visibility of the operation guide is lower than the normal display in order to increase the visibility (make it easier to see) of the displayed image based on predetermined conditions, and switches the operation guide from the normal display to the specific display in response to at least one of the following: no touch operation is performed on the displayed image for a certain period of time; a touch operation other than a touch operation is input on the displayed image; a predetermined operation for switching to the specific display is input by the operator; the displayed image is made invisible; or the operating state of the work machine has changed.

[0085] According to this embodiment, the operation guide showing the operation method of the displayed image of the camera image is displayed including an icon that resembles the operation form of a human hand or finger, so that the function of the operation method can be intuitively understood. Also, by displaying the operation guide at the appropriate time and in the appropriate place, it is possible to achieve both ease of checking the surroundings through the displayed image and ease of understanding the operation through the operation guide, thereby improving work efficiency.

[0086] The present invention is not limited to the above-described embodiment, and includes various modified forms within the scope of the gist of the present invention. For example, the present invention is not limited to those having all of the configurations described in the above-described embodiment, and includes those in which some of the configurations are deleted.

[0087] Further, each configuration of the control device 100 described above and the functions and execution processes of each configuration may be realized in part or in whole by hardware (for example, by designing logic for executing each function in an integrated circuit). Further, the configuration of the control device 100 described above may be a program (software) that is read and executed by an arithmetic processing device (for example, a CPU) to realize each function of the configuration of the control device 100. Information related to the program may be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), and a recording medium (magnetic disk, optical disk, etc.).

[0088] In addition, in the above description of the embodiment, the control lines and information lines are those that are considered to be necessary for the description of the embodiment, but they do not necessarily show all the control lines and information lines related to the product. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0089] 1. Hydraulic excavator (working machine) 1A Front work equipment (working device) 1B Body (main body of work machine) 1a Boom 1b Arm 1c Bucket 1d Upper rotating body 1e Undercarriage 1f Driver's cab 7a,7b,7c,7d Operation lever 8a, 8b, 8c Angle sensors 9a,9b GNSS antenna 10a, 10b, 10c Camera 11 Speaker 12 Monitors 15 Touch Panel 17 Shut-off lever 100 Control device 101 Display image generation unit 102 Display image setting section 103 Touch operation acquisition unit 104 Operation guide generation unit 105 Operation guide setting section 106 Display control unit 107 Operation guide display determination unit 108 Vehicle status acquisition unit

Claims

1. A camera and A monitor and a control device that generates a display image including a camera image acquired from the camera and displays the generated display image on the monitor, the monitor is capable of accepting input by touch operation; The control device is characterized in that it simultaneously displays on the monitor an operation guide that imitates an operator's operating style, which shows how to operate the touch operation on the monitor to change the settings of the displayed image, and the displayed image.

2. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that the operation guide includes an expression showing a hand or finger, or a hand or finger movement.

3. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that, when a touch operation is performed on the displayed image, the control device changes a setting of the displayed image in response to the touch operation.

4. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that the control device generates the operation guide including an expression indicating a setting content of the displayed image corresponding to the operation content of the touch operation.

5. The surroundings monitoring system according to claim 1, The control device is characterized in that, when a setting operation to change the settings of the displayed image cannot be performed due to specified conditions, it generates the operation guide including an expression indicating that the setting operation cannot be performed.

6. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that the control device displays the operation guide on the monitor by superimposing it on the displayed image.

7. The surroundings monitoring system according to claim 1, The control device also generates a semi-transparent background image around the operation guide.

8. The surroundings monitoring system according to claim 1, The control device, in conjunction with the operation guide, emphasizes the surroundings of the displayed image.

9. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that the control device is capable of changing at least one of display settings, whether to display or not display the operation guide, and a display position of the operation guide, as a display setting of the operation guide.

10. The surroundings monitoring system according to claim 1, the control device is capable of switching between a normal display of the operation guide and a specific display in which visibility of the operation guide is lowered than that of the normal display in order to enhance visibility of the displayed image, based on a predetermined condition; The surroundings monitoring system, wherein the specific display includes at least one of a change in the display size of the operation guide and a change in the transparency at which the operation guide is displayed.

11. The surroundings monitoring system according to claim 1, The surroundings monitoring system is characterized in that the control device normally displays the operation guide when a touch operation is performed on the displayed image.

12. The surroundings monitoring system according to claim 1, The control device normally displays the operation guide when the surroundings monitoring system is started and the display image is displayed on the monitor.

13. The surroundings monitoring system according to claim 10, The surroundings monitoring system is characterized in that the control device switches the operation guide from a normal display to a specific display when a predetermined condition is satisfied.

14. The car body and A working device attached to the vehicle body; At least one camera mounted on the vehicle body; A monitor and a control device that generates a display image including a camera image acquired from the camera and displays the generated display image on the monitor, the monitor is capable of accepting input by touch operation; The control device is characterized in that the control device simultaneously displays, on the monitor, an operation guide that imitates an operator's operation form, which shows how to operate the touch operation on the monitor to change settings of the displayed image, and the displayed image.

15. A work machine according to claim 14, the control device is capable of changing at least one of a size or a pattern of the display image in response to the touch operation; A work machine characterized in that the display image pattern includes at least one of an overhead image in which the camera image is converted into an image from a viewpoint seen from directly above the work machine, and an image in which the camera image is displayed without performing viewpoint conversion.

16. A work machine according to claim 14, A working machine characterized in that the control device displays the operation guide by superimposing it on an area in the displayed image showing the sky or the vehicle body, or on a representation that resembles the vehicle body.

17. A work machine according to claim 14, the control device is capable of switching between a normal display of the operation guide and a specific display in which visibility of the operation guide is lowered than that of the normal display in order to enhance visibility of the displayed image, based on a predetermined condition; A work machine characterized in that the operation guide is switched from a normal display to a specific display in response to at least one of the following: no touch operation is made on the displayed image for a certain period of time; a touch operation other than a touch operation is made on the displayed image; a specified operation is input by the operator to switch to the specific display; the displayed image is no longer displayed; or the operating state of the work machine has changed.

Images