Periphery monitoring system, and work machine equipped with the same

Abstract

To provide a periphery monitoring system, and a work machine equipped with the same which can maintain intuitiveness obtained by a touch panel.SOLUTION: Provided are: a periphery monitoring system by which an operation method can be easily understood intuitively by displaying an operation guide for a touch operation relating to settings of a camera video display (an operation guide indicating an operation method of the touch operation on a monitor 12 for changing settings of a displayed video) together with the displayed video in a manner in which the guide includes an icon simulating an operation mode formed of a hand or a finger of a person; and a work machine equipped with the same.SELECTED DRAWING: Figure 14B

Description

[Technical Field]

[0001] The present invention relates to a surroundings monitoring system for a work machine equipped with a work implement such as a hydraulic excavator, and a work machine equipped with the same. [Background technology]

[0002] In recent years, the computerization and informationization of hydraulic excavators and other work machines has progressed, and the electronic devices installed As the number of users increases, the number of functions that support work is also increasing.

[0003] For example, based on information acquired from multiple cameras installed in the work machine, By displaying images on the monitor, the operator is informed of the situation around the work machine. The images displayed on the monitor are taken from multiple cameras and displayed from directly above the vehicle. It is possible to combine images captured from multiple cameras into a bird's-eye view (top-down view) Select from multiple patterns, such as images displayed side by side (without viewpoint conversion, etc.) This allows operators to switch the images displayed on the monitor according to their preference. It is possible.

[0004] In addition to the camera images, the monitor also displays icons showing the various settings of the vehicle and the remaining fuel. The gauges also show the amount of fuel and engine coolant temperature, as well as the status of various devices such as the radio and air conditioner. They may be displayed together.

[0005] In recent years, various sensors attached to work machines and their work equipment have been used to acquire data. The position and posture of the work machine and the work equipment are calculated from the information obtained, and the work machine and the work equipment are Machine guidance assists the operator by notifying them of the positional relationship between the equipment and the target surface. There is a function called MG.

[0006] Furthermore, the information on the position and posture of the work machine and the work device and the information on the preset work area are used to The degree to which the work machine and the work device are approaching the boundary of the work area is calculated, and the degree of approach is By notifying the operator and restricting the operation of the work machine or work equipment according to the situation, the work can be There is also a feature called area restrictions that helps you work within your business area.

[0007] In this way, the number of functions to support work has increased, but the number of settings on the monitor has also increased. As the number of possible settings increases, the setup operations on the monitor can become complicated and difficult to use. There are also concerns that it may be prone to decline.

[0008] In the work machine of Patent Document 1, the display position of the operation information such as the meter on the monitor, At least one of the camera images is switched in response to the operation of a single switch. Therefore, when an operator wants to switch the display, the This reduces the time and effort required to search for the icon, and the display can be switched with simple operations, making operation easier. Improved efficiency. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 6692744 Summary of the Invention [Problem to be solved by the invention]

[0010] In the work machine of Patent Document 1, the display position of the operation information such as the meter on the monitor, At least one of the camera images can be switched by operating a single switch. On the other hand, in recent years, the use of touch panels for monitors on work machines has been increasing. Various settings and switching can be done by directly operating (touching) the information displayed on the monitor. It is becoming more intuitive because you can directly manipulate the information displayed on the monitor. This makes it possible to perform simple operations.

[0011] Furthermore, you can switch between different touch operations (tap, swipe, drag, etc.). It is also possible to change the content and functions to be assigned. Even if you touch the This will increase convenience and is expected to improve.

[0012] On the other hand, even if the touch area is the same, if the type of switching differs depending on the operation, In recent years, the popularity of smartphones has made it difficult to understand which switches can be made by operating the With the spread of smartphones, touch panel operations have become commonplace, for example, pinch in / out. There are also an increasing number of things that can be associated with functions depending on the operation method, such as zooming in and out. The information displayed on the monitor of the industrial machine and the functions assigned to each piece of information can be displayed in a similar manner depending on the operation method. There are also cases where functions cannot be associated with the device. For example, there are cases where functions such as switching the display pattern of camera images cannot be associated with the device. In the case of operations specific to any work machine, it is difficult to associate functions with the operation method, and it is difficult to accept I can no longer understand how to operate the controls that correspond to the various functions, and the usability of the touch controls has deteriorated. will decrease.

[0013] Therefore, in the present invention, a surroundings monitoring system that can maintain the intuitiveness of a touch panel is provided. The present invention aims to provide a stem and a work machine equipped with the stem. [Means for solving the problem]

[0014] In order to achieve the above object, the present invention provides a camera, a monitor, and a A display image including a camera image is generated, and the generated display image is displayed on the monitor. In the surroundings monitoring system, the monitor receives input by touch operation. The control device can be connected to the monitor for changing the settings of the display image. An operation guide imitating the operation form of an operator showing how to operate the touch operation on the monitor and and a display image on the monitor simultaneously. do. [Effects of the Invention]

[0015] According to the present invention, the operation guide showing the operation method of the displayed image of the camera image can be displayed by hand or by a person. By displaying icons that mimic the operation of fingers, etc., the operation method can be intuitively It also allows users to understand the functions of the device by displaying operation guides at the right time and in the right place. By displaying the image, it is easy to check the surroundings and the operation guide makes it easy to understand the operation. This allows for both high speed and low cost, improving work efficiency. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a right side view of a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention. [Figure 2] Schematic diagram showing the interior of the driver's cab on the 1st floor. [Figure 3] System configuration diagram of hydraulic excavator 1. [Figure 4] FIG. 1 is a functional block diagram of a control device 100 according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams showing examples of display image patterns. [Figure 6] 4 is a flowchart of the display video generation unit 101. [Figure 7] FIG. 10 is a diagram showing an example of a pattern of correspondence between operation contents and setting contents. [Figure 8] 10A and 10B are diagrams showing examples of switching of display images by touch operations. [Figure 9] 10 is a flowchart of the display video setting unit 102. [Figure 10] 10 is a flowchart of the operation guide setting unit 105. [Figure 11] 10 is a flowchart showing the operation guide display determining unit 107 determining whether to perform normal display. [Figure 12] 10 is a flowchart showing the operation guide display determining unit 107 determining whether to perform reduced display. [Figure 13] 10 is a flowchart of the operation guide generation unit 104. [Figure 14A] Example of operation guide display (when operation guide is not displayed). [Figure 14B] Example of operation guide display (when operation guide is displayed normally). [Figure 14C] Example of operation guide display (when operation guide is displayed in reduced size). [Figure 15] Example of operation guide display (when operation guide is displayed normally). [Figure 16] Example of operation guide display (when the operation guide is superimposed on the vehicle icon). [Figure 17A] Example of operation guide display (when operation guide is not displayed). [Figure 17B] Example of operation guide display (when operation guide is superimposed on the image of the vehicle). [Figure 17C] Example of operation guide display (when operation guide is superimposed on empty area). DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] An overview of a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention, is shown in FIG. The pressure shovel 1 has a boom 1a, an arm 1b, and a bucket 1c that rotate vertically. and a front working machine 1A of a multi-joint type consisting of an upper rotating body 1d and a lower traveling body 1e. The upper rotating body 1d is mounted on the lower traveling body 1e so as to be able to rotate in the left and right directions. The upper rotating body 1d is also equipped with an operator's room (also called a cab) 1f. The base end of the boom 1a of the front working implement 1A is supported at the front of the upper rotating body 1d. The boom 1a, arm 1b, bucket 1c, upper rotating body 1d and lower traveling body 1e are the boom cylinder 2a, arm cylinder 2b, bucket cylinder 2c, and swing The left and right travel motors 3e and 3f are driven by respective actuators. The upper rotating body 1d is provided with an engine 4, a pump 5, a valve The pump 5 is driven by the rotation of the engine 4, and each actuator The valve 6 is operated by operating levers 7a, 7b, The flow rate and direction of hydraulic oil are controlled by inputs from 7c and 7d (Fig. 2), and each actuator Furthermore, the boom 1a, arm 1b, and bucket 1c are equipped with a detection system for detecting the rotation angle of each part. The upper rotating body 1d is provided with angle sensors 8a, 8b, and 8c for detecting the position of the hydraulic excavator 1. GNSS antennas 9a and 9b are attached to acquire the surroundings of the work machine. Cameras 10a, 10b, and 10c for photographing the surrounding area are mounted on the left, rear, and right sides of the upper rotating body 1d. The upper rotating body 1d is also provided with cameras 10a, 10b, and 10c. Generates the image (display image) to be displayed on the monitor 12 (Figure 2) based on the acquired image (camera image). The control device 100 is installed to perform processing such as displaying the generated display image on the monitor 12. are.

[0019] Figure 2 shows the interior of the driver's cab 1f. The driver's cab 1f has control levers 7a and 7b. b, 7c, 7d, and the shutoff function that locks the operation input from the operating lever to the actuator. Flavor 17, a speaker 11 for transmitting information to the operator by sound, and a display for displaying information. A monitor 12 is provided as a display device for the image data. The touch panel 15 is equipped to receive inputs by touch operation. The monitor 12 is also called a touch panel monitor.

[0020] FIG. 3 is a block diagram showing an information input system for a hydraulic excavator 1, which is an example of a work machine according to an embodiment of the present invention. The control device 100 of the hydraulic excavator 1 according to this embodiment includes a memory. In addition to the in-C / U (control unit) 50, information from the GNSS antennas 9a and 9b is The GNSS C / U40 processes the signal and the notification C / U60 controls various notifications to the operator. The main C / U 50 controls the basic systems for the work machine, such as the engine 4 and pump 5. It also controls the system and is connected to a key cylinder, allowing electrical start-up of the work machine. It also controls the starting of engine 4, which is the prime mover. Main C / U 50 is also connected to a communication terminal. It communicates with various terminals and servers outside the hydraulic excavator 1. Based on the processing status of the main C / U 50, the display information of the monitor 12 is calculated and the output from the speaker 11 is The notification C / U60 calculates the sound information from the touch panel 15 and sends a signal. Various setting signals are sent to the main C / U 50 in accordance with inputs made by touch operation from the

[0021] 4 is a functional block diagram of the control device 100 according to the embodiment of the present invention. Reference numeral 100 denotes a display image generating unit 101, a display image setting unit 102, a touch operation acquiring unit 103, and an operation An operation guide generating unit 104, an operation guide setting unit 105, a display control unit 106, and an operation guide display determination unit 107 The system includes a determination unit 107 and a vehicle body state acquisition unit 108. Each block will be described below. A surroundings monitoring system that monitors the situation around the hydraulic excavator 1 using the devices shown in FIG. is configured.

[0022] The display image generating unit 101 generates images acquired from the cameras 10a, 10b, and 10c (camera images). The images are then combined according to the settings to generate a display image to be displayed on the monitor 12. Five patterns of display images generated in this format are shown in FIG. 5. Display images 201 and 202 are The display images 203, 204, and 205 are all of the same size. The displayed image is a different size from the displayed images 201 and 202.

[0023] The display image 201 is a picture of the images captured by the cameras 10a, 10b, and 10c viewed from above. The image is a bird's-eye view projected onto the image from a point of view (seen directly above the hydraulic excavator). Of the displayed images 201, image 201a is an image from the left camera 10a, and image 201b is an image from the rear. The image 201c is projected from the camera 10b on the left side, and the image 201b is projected from the camera 10c on the right side. The vehicle icon 201d is an icon that resembles the shape of a hydraulic excavator. , which is displayed to help understand the positional relationship between the projected image of the surroundings and the vehicle itself. .

[0024] The display image 202 is the image from the cameras 10b and 10c projected without any viewpoint conversion or the like. Among the displayed images 202, image 202b shows the image captured by camera 1 on the left side. Image 202c displays an image from camera 10c on the right side.

[0025] The display image 203 is a picture of the images acquired by the cameras 10a, 10b, and 10c viewed from above. The image projected onto the image from the viewpoint (seen directly above the hydraulic excavator) (bird's-eye view image) and the camera The image 203a is a display image of the images from the cameras 10a and 10b. Image 203b is an image from the left camera 10a, image 203b is an image from the rear camera 10b, and image 203c is the area where the image from the right camera 10c is projected. The image from the camera 10b on the side is displayed without projection. d is an icon that resembles the shape of a hydraulic excavator, and like the vehicle icon 201d, It is displayed to help understand the positional relationship between the projected image of the surroundings and the vehicle itself.

[0026] The display image 204 is a projection image of the images from the cameras 10a, 10b, and 10c, which is obtained by performing viewpoint conversion, etc. Among the displayed images 204, image 204a has the left The image from the camera 10a is shown in image 204b, the image from the rear camera 10b is shown in image 20. 4c displays the image from the right camera 10c.

[0027] The display image 205 is a picture obtained from the cameras 10a, 10b, and 10c, looking down from above. The image is a bird's-eye view projected onto the image from a point of view (seen directly above the hydraulic excavator). The display size and display area of ​​the displayed image 201 and the displayed image 203 are different. The image 205a of the displayed image 205 is captured by the left camera 10. a, image 205b is an image from the rear camera 10b, and image 205c is an image from the right side This is the area where the image from the camera 10c is projected. The icon is shaped like a car body, and like the car body icon 201d, it is It is displayed to help you understand the positional relationship between the image and your vehicle.

[0028] FIG. 6 shows a flowchart of the display image generation unit 101. The display image generation unit 101: First, in step S300, camera images are acquired from the cameras 10a, 10b, and 10c. If the camera image is not available, the process will be terminated. If the camera image has been acquired, the process proceeds to step S301. Now, check the display image settings. In this embodiment, among the display patterns shown in FIG. In this step, you will confirm which display should be used (which display pattern should be selected). After confirming the above, the process proceeds to step S302, and the cameras 10a, 10b, and 10c are actually set according to the settings. The image acquired from c is processed and the display image is generated. When the display image generation is completed, The process proceeds to step S303, where the generated display image is output, and the process ends.

[0029] The display image setting unit 102 sets the display image generated by the display image generating unit 101 as the display image. In this embodiment, the setting can be changed by touching the up arrow. The displayed image pattern can be changed by touching the image. The change of the setting differs depending on the type of touch operation (operation content 401). A table 400 showing the correspondence of the changes (settings 410) is shown.

[0030] In this embodiment, the types of touch operations are operation contents 402, 403, 404, and 405. The operation content 402 is a type in which the touch panel 15 is touched with one finger. The operation 403 is a swipe operation in which two fingers are moved on the touch panel 15. This is a pinch-in operation where you touch the screen and move your fingers closer together. 404 is a diagram showing how to move two fingers apart while touching the touch panel 15 with two fingers. The operation content 405 is a pinch-out operation in which the user touches the touch panel 15 with one finger. This is a tap operation where you tap the button and then release it.

[0031] In this embodiment, there are three types of setting changes: setting contents 411, 412, and 413. The setting contents corresponding to the operation contents 402 are the display contents within the same display size. The pattern switching of the displayed image is assigned as the setting content 411. The corresponding setting is to reduce the size of the displayed image as setting 412. The setting contents corresponding to the operation contents 404 are the display image being displayed. Enlarging the image size is assigned as setting 413. This corresponds to operation 405. As for the setting contents, since no setting contents are assigned, setting contents 414 is set to "No setting contents". are.

[0032] FIG. 8 shows the relationship between the touch operation and the display image generated by the display image generating unit 101. The monitor displays 211, 212, 213, 214, and 215 show the images shown in FIG. The image is displayed on the monitor 12 equipped with the touch panel 15. There is a touch area 15a that responds to touch operations, and the monitor 12 actually displays an image. The monitor display 211 has a liquid crystal display area 12a. The monitor display 212 displays the display image 202, the monitor display 213 displays the display image 203, and the monitor display 21 Display image 204 is displayed on monitor display 215, and display image 205 is displayed on monitor display 215. Group 220 includes monitor display 211 and monitor display 212, which have the same display image size. Also, the group 221 includes a monitor display 213, which has the same display image size. 7, the operation content 402 and the setting content 41 are included. From 1, swipe operation is assigned to switch patterns within a group with the same display image size. It's hitting.

[0033] To switch between group 220 and group 221, which have different display image sizes, The pinch-in and pinch-out operations are performed according to the contents 403 and 404 and the setting contents 412 and 413. When switching from group 220 to group 221, the displayed image The size of the screen will increase, so a pinch out gesture will cause the switch to occur. When switching from group 221 to group 220, the size of the displayed image is reduced. The switchover occurs when a pinch-in operation is input.

[0034] FIG. 9 shows a flowchart of the display image setting unit 102. In step S310, It is determined whether a touch operation has been performed on the displayed image. If no touch operation is performed on the displayed image, the process ends. If a touch operation is performed, the process proceeds to step S311. It is determined whether the type of operation is an operation to which a setting is assigned. Settings are assigned to the swipe, pinch in, and pinch out described in When any of the touch operations, pinch in or pinch out, is input, the screen Proceed to step S312. If touch operations other than swiping, pinching in, or pinching out are input, If so, the setting has not been assigned, and the process is terminated. The settings assigned to the input touch operation are displayed in the table 400 of FIG. After the setting contents have been confirmed, the process proceeds to step S313. In step 13, the setting contents of the display image generating unit 101 are changed in accordance with the setting contents. When the change of the setting contents of the generation unit 101 is completed, the process ends.

[0035] The touch operation acquisition unit 103 receives input from the touch panel 15 by the operator through touch operations. Obtain and output information.

[0036] The operation guide setting unit 105 monitors the settings of the operation guide generated by the operation guide generating unit 104. Change it by touching the screen.

[0037] FIG. 10 shows a flowchart of the operation guide setting unit 105. This shows the flow when you can show or hide the operation guide and change the display position of the operation guide. In step S320, a touch operation is input to the monitor 12 (touch panel 15). If no touch operation has been input, the process ends. If a touch operation has been input, the process proceeds to step S321. It is determined whether the input touch operation is an operation to change the operation guide display. For example, on the menu screen, you can select the change content for the operation guide display, If no operation to change the operation guide display has been entered, If an operation to change the operation guide display is input, the process is terminated. In step S322, the content of the operation to change the operation guide display is determined to be the display of the operation guide, Determine whether the operation is to hide or not. If there is, the process proceeds to step S323. In step S323, the change of the operation guide display is The display or non-display of the operation guide is changed according to the content of the change operation, and the process is completed. In step S322, the operation guide display change operation content is displayed or hidden. If it is determined that the operation is not a key operation, the process proceeds to step S324. It is determined whether the operation to change the display of the operation guide is a change in the display position of the operation guide. If the operation is not a change operation of the display position of the operation guide, the process ends. If the operation is to change the display position, the process proceeds to step S325. , Change the display position of the operation guide according to the operation content, and perform the process. In this embodiment, the display position of the operation guide can be changed to either the top or the left side of the displayed image. It is possible. In addition, if you select the top of the displayed image as the display position of the operation guide, 4B and 14C, and FIG. 15 shows a case where the left side of the displayed image is selected.

[0038] The operation guide display determination unit 107 determines whether the operation guide is displayed in a normal display or in a reduced display. In FIG. 11, the operation guide display determining unit 107 determines whether to perform normal display. In step S340, the monitor 12 displays a flowchart for determining whether the It is determined whether a touch operation has been performed on the displayed image. If so, the process proceeds to step S342. In step S342, the operation guide is displayed in the normal display. In step S340, if a touch operation is not performed, the setting is changed to display. If it is determined that the system is in a normal state, the process proceeds to step S341. Then, it is determined whether the image is displayed on the monitor 12 for the first time. If the display image is displayed on the monitor 12 for the first time after the system is started, step S In the same manner as above, in step S342, the operation guide is displayed in the normal display mode. The setting is completed and the process is terminated. The display image is displayed on the monitor 12 for the first time after the system is started. If it is not the right timing, the process ends.

[0039] As shown in FIG. 12, the operation guide display determining unit 107 determines whether to perform a reduced display. In step S350, the display image on the monitor 12 is It determines whether or not touch operation has been performed for a certain period of time. If no touch operation has been input to the displayed image, the process proceeds to step S355. In step S355, the operation guide is displayed in a reduced size (in other words, the operation guide is displayed in a reduced size). In step S350, the setting is changed to "switch to display" and the process is terminated (similarly below). If a touch operation is input to the displayed image within the time limit, step S351 In step S351, the process proceeds to step S352. If the touch panel is set to "ON", it determines whether a touch operation other than a touch operation on the displayed image has been input. If a touch operation is input to a part other than the displayed image, the process proceeds to step S35. Proceed to step 5. If no touch operations are being input to areas other than the displayed image, In step S352, it is determined whether a predetermined operation has been input. The predetermined operation here is to reduce the display of the operation guide (in other words, to reduce the display of the operation guide). In this embodiment, the operation guide is displayed with the The display switching icon 256 is installed, and the reduced display switching icon 256 is touched. When the reduced display switching icon 256 is pressed, it is determined that a predetermined operation has been performed. This will be described later with reference to FIG. 15. If it is determined that a predetermined operation has been input, If it is determined that the predetermined operation has not been input, the process proceeds to step S355. In step S353, the display on the monitor 12 is changed to the display image. Whether the screen has switched to one without the image displayed (in other words, whether the displayed image has been hidden) The display image is switched to a screen where no image is displayed (in other words, the display image is If the display image is not displayed, the process proceeds to step S355. For example, when the menu screen for various settings is displayed, the display image is If the display image is not displayed on the screen, In step S354, it is determined whether the operating state of the work machine has changed. In this embodiment, the lock state of the shut-off lever 17 (the state of the vehicle body to be described later) is determined. The operating state of the work machine is determined based on whether the operating state of the work machine has changed or not (acquired by the state acquisition unit 108). If it is determined that the operating state of the work machine has changed, step S The process proceeds to 355. If it is determined that the operating state of the work machine has not changed, the process ends. (i.e., the operation guide will not be switched to a reduced display).

[0040] The operation guide generation unit 104 is a unit that includes the display image setting unit 102, the touch operation acquisition unit 103, and the operation guide. Based on information from the guide setting unit 105, the operation guide display determination unit 107, etc., An operation guide (in other words, a touch operation guide) showing how to operate the monitor 12 to change the settings For example, it generates a touch operation guide for setting up the camera image display on the monitor. In the embodiment, the operation guide is an icon that imitates the operation form of the operator, Hands or fingers between the screen, or icons that mimic the movements of a human hand or finger, or touch operation Icons indicating the setting contents 410 of the display image corresponding to the operation contents 401 (also shown in the table 400 of FIG. 7) In this embodiment, the manipulation guide generating unit 104 generates a manipulation guide under a predetermined condition. Based on the conditions, an operation guide for normal display or reduced display is generated (in other words, The operation guide also has a display position switch (switching between normal display and reduced display of the guide). It contains information etc.

[0041] FIG. 13 shows a flowchart of the operation guide generation unit 104. In step S330, Check whether the display setting for the operation guide is hidden. If it is set to hidden, The process proceeds to step S336. In step S336, the operation guide is hidden, and the operation guide In step S330, the display setting of the operation guide is set to hidden. If the setting is not set to display (if the setting is set to display), the process proceeds to step S331. In step S331, the setting contents 410 and operation contents 40 that can be set in the display image setting unit 102 are displayed. After the information acquisition from the display image setting unit 102 is completed, the process advances to step S In step S332, the display setting of the operation guide is confirmed. The display settings include information on the display position of the operation guide. Check the display settings of the operation guide. When the above is completed, the process proceeds to step S333. In step S333, the display setting of the operation guide is Determine whether the display setting of the operation guide is reduced or normal. In step S334, an operation guide for a reduced display is generated. In step S333, if it is determined that the display setting of the operation guide is normal display, If so, the process proceeds to step S335. In step S335, the operation guide for normal display is The ID is generated and the process ends.

[0042] The display control unit 106 controls the display image generated by the display image generation unit 101 and the operation guide generation unit 103. The operation guide generated by the generating unit 104 is displayed on the monitor 12. Although the display only shows the image and operation guide, it also shows the setting status of the work machine and air The setting status of various devices in the driver's cab, such as the computer and radio, may also be displayed.

[0043] The vehicle body state acquisition unit 108 acquires the behavior of the work machine used in the judgment of the operation guide display judgment unit 107. In this embodiment, the lock state of the shut-off lever 17 is acquired, and the operation state is acquired. The result is output to the production guide display determination unit 107.

[0044] 14A, 14B, 14C, and 15 show examples of operation guide displays. 14A shows an example in which the display image 205 is displayed. 14B shows a monitor display 215 when the operation guide is normally displayed. FIG. 14C shows the monitor display 240 when the operation guide is displayed in a reduced size.

[0045] The monitor display 215 is not displayed because the display setting of the operation guide is set to non-display (see FIG. 13). In step S336, the display image 205 is displayed, but the operation guide is not displayed. do not have.

[0046] The monitor display 230 is displayed when the operation guide is normally displayed (step S33 in FIG. 13). 5) is a display example. As an operation guide, it is set together with touch operation guides 231 and 232. The touch operation guides 231 and 232 are displayed. The setting guides 233 and 234 indicate the operation method (operation form) of the touch panel. What display image settings are made by the operation (i.e., the operation content of the touch operation) The touch operation guide 231 shows the setting contents of the display image according to the swipe operation. The setting content guide 233 corresponding to the touch operation guide 231 is an icon indicating the operation. These two icons indicate the pattern change of the displayed image. This indicates that the displayed image pattern will change when you perform the touch operation. Touch Operation Guide 2 32 is an icon indicating a pinch-in operation, and the setting corresponding to the touch operation guide 232 is The fixed content guide 234 indicates the reduction of the displayed image, and these two icons The background image 235 is shown as , are displayed around the operation guides (231 to 234) and together with the operation guides (231 to 234) It is a background image that is superimposed on the display image and is displayed as an operation guide regardless of the state of the display image. are displayed for easy visibility. Operation guide (231-234) and background image 2 The background image 235 is displayed in a semi-transparent color. It is generated to better suit the operation guide.

[0047] The monitor display 240 is displayed when the operation guide is displayed in a reduced size (step S33 in FIG. 13). 4) is a display example. As an operation guide, touch operation guides 241 and 242 are displayed. The touch operation guide 241 is an icon that indicates a swipe operation. The touch operation guide 241, 242 is an icon that indicates a pinch-in operation. 2 is displayed in a smaller size than the touch operation guides 231 and 232 in normal display. As in normal display, the background image 243 is also displayed around the operation guides (241, 242). The operation guides (241, 242) and background image 243 are displayed in semi-transparent colors. The background image 243 is generated by the manipulation guide generating unit 104 in accordance with the manipulation guide. In this embodiment, the reduced display is achieved by reducing the size of the displayed icon. However, the display color and transparency may be changed, or the operation guide may not be displayed.

[0048] FIG. 15 shows monitor display 2 as an example of a different display format when the operation guide is normally displayed. 50. As operation guides, touch operation guides 251 and 252 and setting content guide 25 3, 254, and operation area guide 257 are displayed. The icon indicates the type of operation, and corresponds to the touch operation guide 251. 53 indicates the pattern change of the displayed image, and these two icons allow you to This shows that the displayed image pattern changes when you perform a touch operation. The guide 252 is an icon indicating that the pinch-in operation is not possible, and is used as a touch operation guide. The setting guide 254 corresponding to the guide 252 indicates that the displayed image cannot be reduced. These two icons prevent you from pinching in to reduce the size of the displayed image. The operation area guide 257 indicates that the operation indicated in the operation guide is accepted. In this embodiment, the guide is for touch operations on the displayed image. Therefore, it is displayed around the displayed image together with the operation guide. Icon 256 is an icon that accepts input to switch the display from normal to reduced view. When this icon is tapped, the operation guide will be displayed in a reduced size. The scenery image 255 includes the icons (251 to 254) of the operation guide and the reduced display switching icon. The icons (256) of the operation guide superimposed on the displayed image are displayed around the 1 to 254), thumbnail display switching icon 256, and background image 255 use semi-transparent colors. The background image 255 is generated by the operation guide generating unit 104 to display each icon of the operation guide. The icons (251 to 254) and the reduced display switching icon 256 are generated in accordance with the The operation area guide 257 is also generated by the operation guide generating unit 104 in accordance with the displayed image. It is generated to highlight the surrounding area.

[0049] Also, in the monitor display 250 (FIG. 15), the operation guide is normally displayed on the monitor. The position where the operation guide is displayed has been changed for the display 230 (FIG. 14B). In the data display 230 (FIG. 14B), the display position of the operation guide (and the background image around it) This is a display example when the upper part of the display image is selected, and the monitor display 250 (FIG. 1 5) shows an example of a display when the left side of the displayed image is selected.

[0050] Figure 16 shows an example of a display in which an operation guide is superimposed on an icon resembling a vehicle body. The monitor display 260 shows an example of the display image 205. The monitor display 260 displays touch operation guides 261 and 262 as touch guides. Touch operation guides 261 and 262 and a background image 263 are superimposed on the vehicle body icon 205d. By displaying it in a form that is superimposed on the vehicle body icon 205d, it is possible to provide a touch operation guide. 261 and 262 do not overlap with the camera image in the displayed image and obstruct the view of the surroundings. Therefore, the touch operation guides 261 and 262 can be displayed.

[0051] 17A, 17B, and 17C show a display in which an operation guide is superimposed on the display image 204. In FIG. 17A, the monitor display 270 shows the surrounding image in the display image 204. The camera image shows the vehicle body images 271a, 271b, and 271c. The shape and size of the image may vary depending on the shape of the vehicle, the camera installation position, and the lens position. The camera image on the vehicle side is divided into ground areas 273a, 273b, and 273c. 73c, and the ground surface is visible. On the other hand, the sky and the height are visible on the far side of the camera image. In this embodiment, there are sky areas 274a, 274b, and 274c in which a solid object is reflected. Therefore, the ground regions 273a, 273b, and 273c and the sky regions 274a, 274b, and 274c The imaginary horizontal lines 272a, 272b, and 272c that are the boundaries between the two are shown.

[0052] In FIG. 17B, the monitor display 280 displays operation guidance superimposed on the vehicle body shown in the camera image. An example of displaying the touch operation guide 281 and 282 is shown. Touch operation guides 281 and 282 and a background image 283 are displayed on the vehicle body image 271b. By superimposing the operation guide on the vehicle body in the camera image, , it is possible to display touch operation guides 281 and 282 without obstructing the surrounding view. This becomes:

[0053] In FIG. 17C, the monitor display 290 displays an operation guide in the area where the sky is displayed on the camera image. The following shows an example of the touch operation guide 291 and 292. The touch operation guides 291 and 292 and the background image 293 are superimposed on the sky area 274a. The sky area is the area where the sky or the distant area from the vehicle is displayed, and the area near the vehicle is clearly visible. It is less likely to get in the way when checking the surroundings and less likely to obstruct your view of the surroundings.

[0054] As described above, according to this embodiment, the touch operation guide 23 on the monitor display 230 in FIG. 14B 1, 232, the operator's operation is performed by a human hand or finger shape or the like on the monitor 12. By displaying icons that resemble the shape of a device, the camera image can be displayed by touch operation. It is easy to understand that some operation or setting is possible for the display.

[0055] In addition, touch operation guides 231 and 232 are also available, which represent the movements of human hands and fingers. By displaying the guide, it becomes easy to understand the types of touch operations that can be input.

[0056] In addition, monitor the setting guides 233 and 234 along with the touch operation guides 231 and 232. By displaying the settings on the touch panel 12, you can easily see what settings are assigned to each type of touch operation. This makes it easier to understand what is happening.

[0057] In addition, by displaying a background image 235 around the touch operation guides 231 and 232, the display The touch operation guides 231 and 232 are less likely to be difficult to see depending on the content of the displayed image. It will be reduced.

[0058] In addition, by making the touch operation guides 231 and 232 and the background image 235 semi-transparent, This allows you to see through the displayed image, so you can see what is being displayed without having to check. Operation is possible.

[0059] Also, as shown in the monitor display 290 of FIG. 17C, the operation guide is displayed on the monitor of the display image 204. By superimposing the image on the upper side of the monitor, it is possible to operate the machine in a distant area or in an area where the sky is visible. The operation guide is superimposed on the display, ensuring visibility of the area around the work machine while providing operation guidance. It becomes possible to display.

[0060] Also, as shown in the touch operation guide 252 of the monitor display 250 in FIG. If the settings for the displayed image cannot be made for some reason (in other words, if the image is not displayed due to certain conditions) If you are unable to perform some or all of the setting operations to change the display settings, By showing (generating an operation guide including an icon like this), it is possible to prevent touch operations from being performed. This makes it easier to understand the situation and improves work efficiency.

[0061] 15. Also, the operation area guide 257 of the monitor display 250 of FIG. 15 can be changed to the touch operation guide 251 By displaying the area shown in the operation guide together with In this case, it is possible to easily grasp the displayed image.

[0062] In addition, as shown in Figure 8, the pattern and size of the displayed image can be changed by touch operation. This makes it easy to change the displayed image to suit the work content and operator preferences. This makes it possible to improve work efficiency.

[0063] As shown in the flowchart in Figure 9, the touch operation on the displayed image can be set. By assigning operations, intuitive operation can be realized, improving work efficiency. This eliminates the need to place setting button icons in a separate area from the displayed image. The display area of ​​the monitor 12 can be saved and can be used to display other functions. do.

[0064] As shown in the flowchart in Figure 10, the display position of the operation guide can be adjusted by the operator. By being able to change the position, the operation guide can be tailored to suit the operator's preferences and the display image pattern used. The display position of the operation guide can be changed, improving usability. The display can also be configured by the operator, so the operation method can be understood even without an operation guide. Operators who are not familiar with the operation guide or find it bothersome will choose to ignore the operation guide. This allows the operator to set the display to suit their preferences. It becomes possible.

[0065] Also, as shown in the flowchart in Figure 11, when a touch operation is input to the displayed image, By displaying the operation guide normally, it is possible to perform some operation on the displayed image. Operation guides are sometimes displayed, reducing confusion about how to operate the device and improving work efficiency. Also, when the display image is displayed on the screen after the system starts up, an operation guide is displayed. This makes it easy to understand how to set up the display image when starting to use the work machine. This improves work efficiency.

[0066] As shown in the flowchart in Figure 12, the operation guide is displayed in a reduced size depending on various conditions. By showing the operation guide, the display will be reduced when no operation is being performed on the displayed image. This makes it easier to check the displayed image, improving work efficiency. If no touch operation is performed on the displayed image, the setting operation on the displayed image is considered to be completed. If a touch operation other than a touch operation on the displayed image is input, Even if the operating status of the work machine changes, the setting operation for the displayed image is considered to be completed. The operation guide is displayed in a reduced size. Furthermore, the display on the monitor 12 is switched, and the displayed image is automatically Operation guide will be displayed even if the screen changes to one where the body is not displayed (when the displayed image is hidden). In this way, the completion of the operation on the displayed image is notified to the work machine side. By determining whether the display is reduced or not, work efficiency can be improved. When icon 256 is operated, the display is switched to a reduced size. You can also switch to a reduced display of the operation guide.

[0067] In this embodiment, in order to make it easier to check the displayed image (to increase visibility), This is an example of how to display the work guide in a reduced size compared to the normal display. Instead of changing (reducing) the display size of the operation guide, or At the same time as changing the size (reducing), you can change the display color and change the display transparency (increasing). By making the visibility of the operation guide lower than normal display, the display It is also possible to make it easier to check the video.

[0068] [summary] As described above, the surroundings monitoring system of this embodiment is mounted on the hydraulic excavator 1. Cameras 10a, 10b, and 10c, a monitor 12, and camera images acquired from the cameras a control device 100 that generates a display image including the image data and displays the generated display image on the monitor; In the surroundings monitoring system, the monitor 12 receives input by touch operation. The control device 100 can be attached to the monitor (touch panel 15). The touch operation input is acquired (touch operation acquisition unit 103), and the acquired touch operation In response to the request, the display image setting unit 102 changes the setting of the display image, and the changed display image The camera image acquired from the camera is processed according to the image settings and displayed on the monitor. The display image is generated (display image generating unit 101), and the setting of the display image is an operator's operation mode indicating an operation method of the touch operation on the monitor for changing the A manipulation guide that resembles the above is generated (the manipulation guide generating unit 104), and the generated manipulation guide and the above The display image is simultaneously displayed on the monitor (display control unit 106).

[0069] The operation guide may be a (human) hand or finger, or a (human) hand or finger movement. Includes representations (icons) to show the

[0070] The control device 100 (the display image setting unit 102) is configured to perform a touch operation on the display image. When the touch operation is performed, the setting of the display image is changed in response to the touch operation.

[0071] The control device 100 (the operation guide generating unit 104) generates a The operation guide includes an expression (icon) indicating the setting content of the display image.

[0072] The control device 100 (operation guide generating unit 104) adjusts the display image according to a predetermined condition. If you are unable to perform any of the setting operations to change the image settings, The operation guide includes an expression (icon) indicating that the setting operation cannot be performed (or all of the setting operation cannot be performed). Generate.

[0073] The control device 100 (operation guide generating unit 104, display control unit 106) The operation guide is displayed on the monitor 12 by being superimposed on the image. The operation guide generating unit 104 generates an operation guide including semi-transparent color icons.

[0074] The control device 100 (operation guide generating unit 104) displays a semi-transparent image around the operation guide. A background image is also generated.

[0075] The control device 100 (operation guide generating unit 104) generates the operation guide together with the The periphery of the displayed image is emphasized (operation area guide 257).

[0076] The control device 100 (the operation guide setting unit 105 and the operation guide generating unit 104) As the display setting of the operation guide, at least one of the following may be selected: display, non-display, or display position of the operation guide. One of them can also be changed.

[0077] The control device 100 (operation guide display determination unit 107) determines whether or not the previous operation is performed based on a predetermined condition. The normal display of the operation guide and the display image are displayed in the above-mentioned manner to improve visibility (to make it easier to see). A specific display in which the visibility of the operation guide is lower than that of the normal display can be switched, and the specific display The display can be changed by changing the display size (reduction) or the transparency (increase) of the operation guide. ) includes at least one of the following.

[0078] The control device 100 (operation guide display determination unit 107) When the operation guide is set to normal display, the display image setting of the control device 100 is When a predetermined touch operation is performed on the display image, the display image is The setting is not changed, and the control device 100 (operation guide display determination unit 107) The operation guide will be displayed normally even if the settings are not changed.

[0079] The control device 100 (operation guide display determination unit 107) When the display image is displayed on the monitor, the operation guide is displayed normally.

[0080] When a predetermined condition is satisfied, the control device 100 (operation guide display determination unit 107) Then, the operation guide is switched from the normal display to the specific display.

[0081] The working machine (hydraulic excavator 1) of this embodiment includes a vehicle body 1B and a and at least one work implement (front work implement 1A) attached to the vehicle body. Three cameras 10a, 10b, and 10c, a monitor 12, and camera images acquired from the cameras. a control device 10 that generates a display image including the image and displays the generated display image on the monitor; 0, and the monitor 12 is capable of receiving input by touch operation ( The control device 100 is a touch panel 15 for changing the settings of the display image. An operation guide imitating the operation form of an operator is provided to show the operation method of the touch operation on the monitor. The operation guide generating unit 104 generates the operation guide and the display image. The image is simultaneously displayed on the monitor (display control unit 106).

[0082] The control device 100 (the display image setting unit 102) sets the display image in response to the touch operation. At least one of the size and pattern of the displayed image can be changed, In the pattern (2), the camera image is viewed from directly above the work machine (looking down from above). A bird's-eye view image converted (combined) into a viewpoint image, or the camera image without viewpoint conversion At least one of the displayed images is included.

[0083] The control device 100 (operation guide generating unit 104, display control unit 106) The area in the image where the sky or the vehicle itself is reflected, or an icon that resembles the vehicle itself, The operation guide is displayed superimposed.

[0084] The control device 100 controls the display of the operation guide to be a normal display or a In order to improve the visibility of the displayed image (to make it easier to see), the visibility of the operation guide is set to be higher than normal display. It is possible to switch a specific display with the lowest brightness, and the display image can be touched for a certain period of time. No input was made by the touch panel, and no touch operation other than the touch operation on the displayed image was made. a predetermined operation for switching to the specific display is input by the operator; The display image has been hidden, or the operating state of the work machine has changed, At least one of the operations switches the operation guide from a normal display to a specific display.

[0085] According to this embodiment, an operation guide showing how to operate the displayed image of the camera image can be displayed without the need for human hands. By displaying icons that mimic the operation of a mouse or finger, the device can be operated intuitively. It also allows users to understand the functions by the operation method. By displaying the image, it is easy to check the surroundings and the operation guide makes it easy to understand the operation. This allows for both ease of use and improved work efficiency.

[0086] The present invention is not limited to the above-described embodiment, and may be modified within the scope of the present invention without departing from the gist thereof. For example, the present invention includes all of the above-described embodiments. It is not limited to those having the configuration, but also includes those in which part of the configuration is removed.

[0087] In addition, the components of the control device 100 and the functions and execution processes of the components are Some or all of these functions may be designed as hardware (for example, the logic that executes each function may be designed as an integrated circuit). The configuration of the control device 100 may be realized by a calculation processing device (e.g., The control device 100 is configured by a CPU (for example) that reads and executes the data. The information relating to the program may be a program (software) that realizes the function. For example, semiconductor memory (flash memory, SSD, etc.), magnetic storage device (hard disk It can be stored on a hard drive, etc., and on a recording medium (magnetic disk, optical disk, etc.). .

[0088] In addition, in the above description of the embodiment, the control lines and information lines are not necessary for the description of the embodiment. However, this does not necessarily mean that all control lines and information lines related to the product are shown. In reality, it is safe to assume that almost all components are interconnected. [Explanation of symbols]

[0089] 1. Hydraulic excavator (work machine) 1A Front work equipment (working equipment) 1B Body (main body of work machine) 1a Boom 1b Arm 1c Bucket 1d Upper rotating body 1e Undercarriage 1F Driver's cab 7a,7b,7c,7d Operation lever 8a, 8b, 8c Angle sensors 9a, 9b GNSS antenna 10a, 10b, 10c Camera 11 Speaker 12 monitors 15 Touch Panel 17 Shut-off lever 100 control device 101 Display image generation unit 102 Display image setting section 103 Touch operation acquisition unit 104 Operation guide generation unit 105 Operation guide setting section 106 Display control unit 107 Operation guide display determination unit 108 Vehicle body status acquisition unit

Claims

1. A camera and The monitor and A display image including a camera image acquired from the camera is generated, and the generated display image is displayed. a control device for displaying on the monitor, The monitor is capable of accepting input by touch operation, The control device receives the touch operation on the monitor for changing the setting of the display image. An operation guide imitating the operation form of an operator showing the operation method of the display image is displayed on the monitor. A surroundings monitoring system characterized by simultaneously displaying the surroundings on a monitor.

2. The surroundings monitoring system according to claim 1, The operation guide includes an expression showing a hand or a finger, or a movement of a hand or a finger. A perimeter monitoring system.

3. The surroundings monitoring system according to claim 1, When a touch operation is performed on the display image, the control device and a surroundings monitoring system characterized by changing the settings of the displayed image.

4. The surroundings monitoring system according to claim 1, The control device displays a table showing settings of the display image according to the operation content of the touch operation. A surroundings monitoring system that generates the operation guide including the current state.

5. The surroundings monitoring system according to claim 1, The control device is configured to perform a setting operation for changing the settings of the display image according to a predetermined condition. If the setting operation cannot be performed, the operation guide is generated, which includes a message indicating that the setting operation cannot be performed. A surroundings monitoring system characterized by:

6. The surroundings monitoring system according to claim 1, The control device displays the operation guide on the monitor by superimposing it on the display image. A perimeter monitoring system characterized by:

7. The surroundings monitoring system according to claim 1, The control device also generates a semi-transparent background image around the operation guide. A surrounding surveillance system.

8. The surroundings monitoring system according to claim 1, The control device is characterized in that it emphasizes the surroundings of the displayed image together with the operation guide. A perimeter monitoring system.

9. The surroundings monitoring system according to claim 1, The control device may set the display setting of the operation guide to display, hide, or A surroundings monitoring system characterized in that at least one of the display position or the display position can be changed.

10. The surroundings monitoring system according to claim 1, The control device controls the normal display of the operation guide and the display image based on a predetermined condition. In order to increase the visibility of the operation guide, a specific display is switched in which the visibility of the operation guide is lower than that of the normal display. It is interchangeable, The specific display may be at least one of changing the display size of the operation guide or changing the display transparency. A perimeter monitoring system comprising at least one

11. The surroundings monitoring system according to claim 1, The control device displays the operation guide in a normal state when a touch operation is performed on the display image. A surroundings monitoring system characterized by displaying.

12. The surroundings monitoring system according to claim 1, The control device starts the surroundings monitoring system and displays the display image on the monitor. When the user presses the button, the operation guide is displayed normally.

13. The surroundings monitoring system according to claim 10, When a predetermined condition is satisfied, the control device changes the operation guide from a normal display to a specific display. A surroundings monitoring system characterized by switching to a display.

14. The car body and a working device attached to the vehicle body; at least one camera mounted on the vehicle body; The monitor and A display image including a camera image acquired from the camera is generated, and the generated display image is displayed. a control device for displaying on the monitor, The monitor is capable of accepting input by touch operation, The control device receives the touch operation on the monitor for changing the setting of the display image. An operation guide imitating the operation form of an operator showing the operation method of the display image is displayed on the monitor. A work machine characterized by simultaneously displaying the information on the monitor.

15. 15. The work machine according to claim 14, The control device changes the size or pattern of the displayed image in response to the touch operation. At least one of these can be changed, The display image pattern includes an image of the camera image viewed from directly above the work machine. A bird's-eye view image converted into a camera image, or an image displayed without converting the camera image. A work machine characterized in that it includes at least one image of the work machine.

16. 15. The work machine according to claim 14, The control device may select an area in the displayed image where the sky or the vehicle itself is displayed, or A working machine characterized in that the operation guide is displayed superimposed on a representation that resembles a body.

17. 15. The work machine according to claim 14, The control device controls the normal display of the operation guide and the display image based on a predetermined condition. In order to increase the visibility of the operation guide, a specific display is switched in which the visibility of the operation guide is lower than that of the normal display. It is interchangeable, No input by touch operation has been made to the displayed image for a certain period of time, The operator inputs a touch operation other than a touch operation on the image. When a predetermined operation to switch to the display is input, the displayed image is hidden, a change in the operating state of the work machine, A work machine characterized by switching from a normal display to a specific display.

Images