Peripheral image display device and peripheral image display method for construction machinery

Abstract

The objective is to easily provide images that allow the driver to properly perceive the ground conditions around the crawler. [Solution] The peripheral image display device for construction machinery comprises a camera mounted on the construction machinery and a monitor that displays the camera's image. The camera includes a side camera mounted on the side of the upper slewing body such that its shooting range faces forward. The side camera is positioned below the floor of the cab of the upper slewing body, and the distance from the pivot axis of the upper slewing body to the center of the side camera's lens is longer than the distance from the pivot axis to the outer side of the crawler. The side camera has a field of view that allows it to photograph the outer side of the front of the crawler and the ground to which the crawler is in contact when the upper slewing body is facing forward, and a field of view that allows it to photograph the outer side of the rear of the crawler and the ground when the upper slewing body is facing backward.

Description

Technical Field

[0001] The present invention relates to a peripheral image display device and a peripheral image display method for construction machinery.

Background Art

[0002] A remote control system capable of indicating the forward or backward direction of a working machine such as a construction machine is described in Patent Document 1. This Patent Document 1 states that "in a first captured image acquired through a first imaging device provided on the upper slewing body and provided so as to include at least one of the crawlers of the lower traveling body in an imaging region, in each of the front-rear directions of the pair of crawlers, they are unevenly arranged, have different forms from each other, and based on the respective reflection modes of a first marker provided in front of the pair of crawlers and a second marker provided behind the pair of crawlers, a first support processing element for recognizing the forward or backward direction of the lower traveling body, and a second support processing element for transmitting a composite image in which an index image representing the forward or backward direction of the lower traveling body recognized by the first support processing element is at least partially superimposed on the working machine or its surroundings in the first captured image to the remote control device."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When operating large construction machinery, even when the operator is seated in the driver's seat, it is often difficult to directly see the crawler from the driver's seat, making it difficult to determine which direction the crawler is facing and which direction it is moving. If there are obstacles or cliffs around the construction machinery, there is a risk of the machinery tipping over or falling if it is driven in such conditions. Patent Document 1 refers to an image captured with the crawler included in the imaging area, but it is insufficient for recognizing the ground conditions around the crawler.

[0005] The present invention has been made in view of the above, and aims to easily provide an image that allows the driver to appropriately recognize the condition of the ground around the crawler. [Means for solving the problem]

[0006] To solve the above problems, the present invention provides a peripheral image display device for construction machinery, comprising: a camera attached to a construction machine having a lower traveling body including a crawler and an upper rotating body rotatably mounted relative to the lower traveling body; and a monitor for displaying images captured by the camera, wherein the camera includes a side camera attached to the side of the upper rotating body such that its shooting range faces forward of the upper rotating body, the side camera is positioned below the floor of the driver's cab of the upper rotating body, and the distance from the pivot axis of the upper rotating body to the center of the lens of the side camera is longer than the distance from the pivot axis to the outer side of the crawler, and the angle of view is such that when the upper rotating body is facing forward of the lower traveling body, it can capture the outer side of the front of the crawler and the ground to which the crawler is in contact, and when the upper rotating body is facing backward of the lower traveling body, it can capture the outer side of the rear of the crawler and the ground. [Effects of the Invention]

[0007] According to the present invention, it is possible to easily provide an image that allows the driver to appropriately recognize the ground conditions around the crawler. [Brief explanation of the drawing]

[0008] [Figure 1] A side view of the shovel according to the first embodiment. [Figure 2] Figure 1 shows a top view of the shovel. [Figure 3] Front view of the shovel shown in Figure 1. [Figure 4] Figure 1 shows a monitor screen displaying an image captured by the side camera. [Figure 5] Figure 1 shows a monitor screen displaying an image captured by the front-facing camera. [Figure 6] Configuration diagram of the peripheral image display device of the first embodiment. [Figure 7] This figure illustrates a composite image created by superimposing guide lines onto the camera image shown in Figure 4. [Figure 8] This figure illustrates a composite image created by superimposing guide lines onto the camera image shown in Figure 5. [Figure 9] A diagram illustrating the configuration of a peripheral image display device that displays a composite image with guide lines superimposed. [Figure 10] A flowchart showing the process performed by the processing device shown in Figure 9. [Figure 11] This figure illustrates a composite image in which guide lines and additional guide lines are superimposed on the camera image shown in Figure 7. [Figure 12] This figure illustrates a composite image created by superimposing guide lines and additional guide lines onto the camera image shown in Figure 8. [Figure 13] This figure illustrates a composite image in which the guide lines shown in Figure 7 are composed of arrow-shaped figures. [Figure 14] A diagram showing the range of rotation angles for displaying guide lines. [Figure 15] This diagram illustrates a composite image created by making the shapes that make up the guide lines transparent so that the ground is visible. [Figure 16] This diagram illustrates a composite image in which the length of the shapes constituting the guide lines changes according to the speed of the lower vehicle. [Figure 17] A diagram showing the configuration of a peripheral image display device that displays the composite image shown in Figure 16. [Figure 18] A flowchart showing the processing performed by the processing device of the peripheral image display device that displays the composite image shown in FIG. 16. [Figure 19] A diagram explaining the composite image in which the guide line is restricted so as not to cross the step. [Figure 20] A configuration diagram of the peripheral image display device that displays the composite image shown in FIG. 19. [Figure 21] A flowchart showing the processing performed by the processing device of the peripheral image display device that displays the composite image shown in FIG. 19. [Figure 22] A diagram explaining the composite image in which the guide line is bent along the inclined surface. [Figure 23] A flowchart showing the processing performed by the processing device of the peripheral image display device that displays the composite image shown in FIG. 22. [Figure 24] A configuration diagram of the peripheral image display device according to the second embodiment. [Embodiments for Carrying Out the Invention]

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each embodiment, components denoted by the same reference numerals have the same components in each embodiment and the description thereof will be omitted unless otherwise specified.

[0010] [First Embodiment] The first embodiment of the present invention will be described with reference to FIGS. 1 to 23. The peripheral image display device 10 according to the first embodiment is a device that acquires and displays an image around a construction machine. In the present embodiment, a large hydraulic excavator 100 (hereinafter referred to as “excavator 100”), which is a type of construction machine, will be described as an example. However, the construction machine to which the present invention is applied may be a small or medium-sized hydraulic excavator, or may be a construction machine other than a hydraulic excavator.

[0011] FIG. 1 is a side view of the excavator 100 according to the first embodiment. FIG. 2 is a top view of the excavator 100 shown in FIG. 1. FIG. 3 is a front view of the excavator 100 shown in FIG. 1.

[0012] The shovel 100 is a construction machine comprising a lower traveling body 102 and an upper rotating body 101 that is rotatably mounted relative to the lower traveling body 102. The shovel 100 has a structure in which the upper rotating body 101 and the lower traveling body 102 are connected by a pivot axis. The lower traveling body 102 is equipped with crawler tracks 103, which are continuous tracks for traveling on the ground 403. The upper rotating body 101 is equipped with a driver's cab 104 in which a driver who is seated to operate the shovel 100 is provided. The upper rotating body 101 is also equipped with a bucket 107 for excavating soil and sand.

[0013] The lower vehicle 102 has a defined front and rear due to its structure. In other words, the front and rear of the lower vehicle 102 are not structurally identical. This is also related to the driver's operation method. When the driver operates the vehicle forward, the direction in which the lower vehicle 102 moves is the front for the lower vehicle 102. When the driver operates the vehicle backward, the direction in which the lower vehicle 102 moves is the rear for the lower vehicle 102. This remains true regardless of the rotation of the upper rotating vehicle 101. For example, consider a state where the upper rotating vehicle 101 rotates 180 degrees, and the front for the lower vehicle 102 becomes the rear when viewed from the driver's cab 104 of the upper rotating vehicle 101. In this state, when the driver operates the vehicle forward, the lower vehicle 102 moves forward for the lower vehicle 102, and backward for the driver's cab 104 and the upper rotating vehicle 101. In this embodiment, the position in which the upper rotating body 101 faces forward of the lower traveling body 102 is also referred to as the "forward-facing position," and the position in which the upper rotating body 101 faces backward of the lower traveling body 102 is also referred to as the "rearward-facing position."

[0014] The upper rotating body 101 of the shovel 100 is equipped with cameras 105 and 106 for photographing the area around the crawler 103 (including the ground 403). Cameras 105 and 106 include a side camera 105 mounted on the side of the upper rotating body 101 such that its shooting range or lens (light-receiving surface) faces forward of the upper rotating body 101, and a front camera 106 mounted on the front of the upper rotating body 101 such that its shooting range faces forward of the upper rotating body 101.

[0015] As shown in Figures 1 and 3, the side camera 105 is positioned below the floor surface 104a of the driver's cab 104 of the upper rotating body 101. As shown in Figure 2, the side camera 105 is positioned to protrude outward from the crawler 103. That is, as shown in Figure 3, the side camera 105 is positioned such that the distance 109 from the pivot axis 108 of the upper rotating body 101 to the center of the lens of the side camera 105 is longer than the distance 110 from the pivot axis 108 to the outer side surface 402 of the crawler 103 (outer side surface of the lower traveling body 102).

[0016] Furthermore, the side camera 105 has a field of view that allows it to capture the outer side surface 402 of the front 405 of the crawler 103 and the ground 403 to which the front 405 of the crawler 103 makes contact when the upper rotating body 101 is in a forward-facing position. The side camera 105 also has a field of view that allows it to capture the outer side surface 402 of the rear 406 of the crawler 103 and the ground 403 to which the rear 406 of the crawler 103 makes contact when the upper rotating body 101 is in a rear-facing position. As a result, the side camera 105 can capture the outer side surface 402 of the front 405 or rear 406 of the crawler 103 and the line in which the outer side surface 402 of the crawler 103 makes contact with the ground 403. Therefore, the surrounding image display device 10 can easily provide images that allow the driver to appropriately recognize the condition of the ground 403 around the crawler 103 and the condition of the ground 403 in front of the crawler 103.

[0017] In this embodiment, "outside" means that the horizontal distance from the pivot axis 108 of the upper rotating body 101 is far, and "inside" means that the horizontal distance from the pivot axis 108 of the upper rotating body 101 is close.

[0018] As shown in Figures 1 and 3, the front camera 106 is positioned below the floor surface 104a of the driver's cab 104 of the upper rotating body 101. When the upper rotating body 101 is in a forward-facing position, the front camera 106 has a field of view that allows it to capture the inner side surface 501 of the front part 405 of the crawler 103 and the ground 403 to which the front part 405 of the crawler 103 makes contact. When the upper rotating body 101 is in a rearward-facing position, the front camera 106 has a field of view that allows it to capture the inner side surface 501 of the rear part 406 of the crawler 103 and the ground 403 to which the rear part 406 of the crawler 103 makes contact. As a result, the front camera 106 can capture the inner side surface 501 of the front part 405 or rear part 406 of the crawler 103 and the line in which the inner side surface 501 of the crawler 103 makes contact with the ground 403. Therefore, the surrounding image display device 10 can easily provide images captured by the front camera 106 that allow the driver to appropriately recognize the condition of the ground 403 around the crawler 103 and the condition of the ground 403 in front of the crawler 103.

[0019] Furthermore, the line in which the outer side surface 402 of the crawler 103 makes contact with the ground 403 is the intersection of the plane containing the outer side surface 402 of the crawler 103 and the ground 403 on which the crawler 103 makes contact. Similarly, the line in which the inner side surface 501 of the crawler 103 makes contact with the ground 403 is the intersection of the plane containing the inner side surface 501 of the crawler 103 and the ground 403 on which the crawler 103 makes contact. In this embodiment, these intersection lines, i.e., the line in which the outer side surface 402 of the crawler 103 makes contact with the ground 403 and the line in which the inner side surface 501 of the crawler 103 makes contact with the ground 403, are collectively referred to as the "crawler ground contact line".

[0020] Furthermore, in this embodiment, the tip of the crawler 103 is the region (or space) extending from the front part 405 of the crawler 103 in front of the lower running body 102 when the upper rotating body 101 is in a forward-facing position. The tip of the crawler 103 is the region (or space) extending from the rear part 406 of the crawler 103 in rearward-facing position of the lower running body 102. The tip of the crawler 103 roughly coincides with the region (or space) extending in front of the upper rotating body 101.

[0021] Figure 4 shows the screen 401 of the monitor 601 displaying an image captured by the side camera 105 shown in Figure 1.

[0022] In this embodiment, screen 401 refers to the display screen of the monitor 601, which will be described later. In the screen 401 shown in Figure 4 and subsequent figures, the images displayed on screen 401 will be explained using as an example images taken by cameras 105 and 106 when the upper rotating body 101 is in a forward-facing position (hereinafter also referred to as "camera images"). When the upper rotating body 101 is in a rearward-facing position, the same camera images as when the upper rotating body 101 is in a forward-facing position are displayed on screen 401 of the monitor 601. However, when the upper rotating body 101 is in a rearward-facing position, the camera image including the rear part 406 of the crawler 103 is displayed on the monitor 601 instead of the front part 405.

[0023] As shown in Figure 4, the camera image from the side camera 105 includes the line where the outer side 402 of the crawler 103 touches the ground 403, and the ground 403 surrounding the front part 405 of the crawler 103. This allows the driver to notice if there is a cliff or obstacle near the crawler 103. The camera image from the side camera 105 also includes the ground 403 in front of the upper rotating body 101, that is, the ground 403 ahead of the crawler 103. This allows the driver to recognize where the crawler 103 will reach on the ground 403 if it continues to move forward. By checking this information on screen 401, the driver can avoid moving towards cliffs or obstacles around the crawler 103 and move the shovel 100 to the target location. The reference numeral 404 in Figure 4 indicates the edge of the ground 403, and beyond the edge 404 of the ground 403 there is a large step, such as a cliff or a steep slope, that the shovel 100 cannot travel over. The operator can recognize how far away the edge 404 of the ground 403 is from the crawler 103 and avoid the risk of tipping over or falling by approaching the edge 404 of the ground 403.

[0024] Figure 5 shows the screen 401 of the monitor 601 displaying an image captured by the front camera 106 shown in Figure 1.

[0025] The camera image from the front camera 106 includes the inner side 501 of the front 405 of the crawler 103, and the ground 403 located in front of the crawler 103. This allows the driver to recognize where on the ground 403 the crawler 103 will reach if it continues to move forward. The driver can then move the shovel 100 to the target location while avoiding cliffs or obstacles in front of the crawler 103.

[0026] Figure 6 is a configuration diagram of the peripheral image display device 10 according to the first embodiment.

[0027] The peripheral image display device 10 includes cameras 105 and 106 attached to the shovel 100, and a monitor 601 that displays images (camera images) captured by cameras 105 and 106. Cameras 105 and 106 are each connected to the monitor 601. The monitor 601 is installed in the operator's cab 104, and the operator can operate the shovel 100 while viewing the images displayed on the monitor 601. Although Figure 6 illustrates the left and right side cameras 105 and the front camera 106, configurations with some of these cameras 105 and 106 omitted are also possible. For example, a configuration with only the left and right side cameras 105 and no front camera 106 is also possible. Furthermore, although in Figure 6 cameras 105 and 106 are each connected to their respective monitors 601, it is also possible to install a relay device between cameras 105 and 106 and the monitor 601, allowing switching between cameras 105 and 106 and the connected monitor 601. Furthermore, the number of cameras 105 and 106 does not necessarily have to be the same as the number of monitors 601. For example, it is possible to have a configuration where there is only one monitor 601, and the cameras 105 and 106 connected to it are switched by a relay device.

[0028] Figure 7 illustrates a composite image obtained by superimposing guide lines 702 onto the camera image shown in Figure 4.

[0029] The peripheral image display device 10 can display a composite image on the screen 401 of the monitor 601 by superimposing a guide line 702 onto the camera image shown in Figure 4. The guide line 702 is a line extending forward from the line 701 (hereinafter also referred to as the "crawler ground contact line 701") where the outer side surface 402 of the crawler 103 touches the ground 403. In this case, "forward" refers to the front for the upper rotating body 101, and is in the depth direction of the camera image. As mentioned above, the line where the outer side surface 402 of the crawler 103 touches the ground 403 is the intersection line of the plane containing the outer side surface 402 of the crawler 103 and the ground 403 on which the crawler 103 touches. That is, the guide line 702 is a line extending along the intersection line of the plane containing the outer side surface 402 of the crawler 103 and the ground 403.

[0030] The length of the guide line 702 is predetermined and stored in the memory device 903. For example, the length of the guide line 702 is set to extend from the tip of the crawler ground line 701 (the front end for the upper rotating body 101) in the camera image by the length of the crawler 103 in the front-rear direction. Alternatively, the length of the guide line 702 is set to extend to the furthest point that the bucket 107 can reach. This guide line 702 allows the operator to recognize where the crawler 103 will pass on the ground 403 if it continues to move forward. If the operator sees a cliff or obstacle on the ground 403 that the crawler 103 will pass through, they can avoid these and move the shovel 100 to the target point.

[0031] Figure 8 illustrates a composite image obtained by superimposing guide lines 801 onto the camera image shown in Figure 5.

[0032] The peripheral image display device 10 can display a composite image on the screen 401 of the monitor 601 by superimposing a guide line 801 onto the camera image shown in Figure 5. The guide line 801 is a line extending forward from the crawler ground contact line, which is the line where the inner side surface 501 of the crawler 103 touches the ground 403, relative to the upper rotating body 101. As mentioned above, the line where the inner side surface 501 of the crawler 103 touches the ground 403 is the intersection line of the plane containing the inner side surface 501 of the crawler 103 and the ground 403 on which the crawler 103 makes contact. In other words, the guide line 801 is a line extending along the intersection line of the plane containing the inner side surface 501 of the crawler 103 and the ground 403.

[0033] The length of the guide line 801 is the same as the guide line 702 shown in Figure 7, and is set to a length extending from the tip of the crawler grounding line in the camera image by the length of the crawler 103 in the front-rear direction, or to the furthest point that the bucket 107 can reach.

[0034] In addition, in the composite images shown in Figures 7 and 8, when the upper rotating body 101 rotates, the position and direction of the crawler 103 in the camera image change, and the position and direction of the guide lines 702 and 801 also change in accordance with the change in the position and direction of the crawler 103.

[0035] Figure 9 is a diagram showing the configuration of the peripheral image display device 10, which displays a composite image with guide lines 702 and 801 superimposed.

[0036] In addition to the components of the peripheral image display device 10 shown in Figure 6, the peripheral image display device 10 includes a rotation angle measuring device 901 for measuring the rotation angle of the upper rotating body 101 relative to the lower traveling body 102, a processing device 902 for processing camera images and displaying them on the monitor 601, a storage device 903 for storing parameters necessary for processing by the processing device 902, an input device 904 which is an input interface to the processing device 902, and an output device 905 which is an output interface to the monitor 601.

[0037] In the rotation angle measuring device 901, for example, the rotation angle when the upper rotation body 101 is at a reference position where the forward direction for the lower traveling body 102 and the forward direction for the upper rotation body 101 coincide is defined as 0 degrees. The rotation angle measuring device 901 measures the rotation angle of the upper rotation body 101 by acquiring information such as, for example, that the upper rotation body 101 has rotated 60 degrees to the right from the reference position. The rotation angle measured by the rotation angle measuring device 901, and the camera images from cameras 105 and 106, are sent to the processing device 902 via the input device 904.

[0038] The mounting positions and shooting directions of cameras 105 and 106 are fixed relative to the upper rotating body 101. The field of view of cameras 105 and 106 is also constant. The installation position of the crawler 103 on the lower traveling body 102 is also constant. Therefore, any change in the positional relationship between cameras 105 and 106 and the crawler 103 is solely due to the rotation of the upper rotating body 101. Once the rotation angle is determined, the positional relationship between cameras 105 and 106 and the crawler 103 is determined, and the position and direction of the crawler 103 within the camera image are also determined. The storage device 903 stores parameters that associate the coordinates of the crawler ground contact line within the camera image with the rotation angle of the upper rotating body 101. These parameters represent the positional relationship between the rotation angle of the upper rotating body 101 and the crawler ground contact line, and indicate where the crawler ground contact line moves in response to changes in the rotation angle.

[0039] The processing unit 902 calculates the coordinates of the crawler ground line in the camera image based on the rotation angle measured by the rotation angle measuring device 901 and the aforementioned parameters stored in the storage device 903. Then, the processing unit 902 calculates the coordinates of the guide lines 702 and 801 in the camera image based on the calculated coordinates of the crawler ground line in the camera image. In this process, the processing unit 902 calculates the coordinates of the guide lines 702 and 801 in the camera image so that they are along the crawler ground line and extend in front of the upper rotating body 101 in the camera image. The processing unit 902 generates guide lines 702 and 801 that match the calculated coordinates and superimposes them onto the coordinates. As a result, the processing unit 902 can generate a composite image by superimposing the guide lines 702 and 801 onto the camera image. That is, the processing unit 902 can generate a composite image by superimposing the generated guide lines 702 onto the camera image of the side camera 105. The processing unit 902 can generate a composite image by superimposing the generated guide lines 801 onto the camera image of the front camera 106. The processing unit 902 then outputs the generated composite image to the output device 905, which sends it to the monitor 601 in the driver's cab 104. The monitor 601 displays the received composite image on the screen 401. In this way, the processing unit 902 can display the generated composite image on the monitor 601.

[0040] Figure 10 is a flowchart showing the process performed by the processing unit 902 shown in Figure 9.

[0041] In step S1001, the processing unit 902 acquires camera images, which are images captured by cameras 105 and 106, from the input device 904.

[0042] In step S1002, the processing unit 902 obtains the rotation angle measured by the rotation angle measuring device 901 from the input device 904.

[0043] In step S1003, the processing unit 902 obtains parameters from the storage device 903 that associate the coordinates of the crawler grounding line in the camera image with the turning angle.

[0044] In step S1004, the processing unit 902 calculates the coordinates of the crawler ground contact line within the camera image based on the turning angle acquired in step S1002 and the parameters acquired in step S1003.

[0045] In step S1005, the processing unit 902 generates guide lines 702 and 801 that run along the crawler ground line and extend forward of the upper rotating body 101, based on the coordinates of the crawler ground line in the camera image calculated in step S1004.

[0046] In step S1006, the processing unit 902 superimposes the guide lines 702 and 801 generated in step S1005 onto the camera image acquired in step S1001 to generate a composite image.

[0047] In step S1007, the processing unit 902 outputs the composite image generated in step S1006 to the monitor 601 for display. After that, the processing unit 902 terminates the process shown in Figure 10.

[0048] Using Figure 11, we will explain another example of a composite image with the guide line 702 superimposed. Figure 11 is a diagram illustrating a composite image in which the guide line 702 and additional guide line 1101 are superimposed on the camera image shown in Figure 7.

[0049] The processing unit 902 may generate additional guide lines 1101 along the crawler ground line, which is the intersection line between the plane including the outer side surface 402 of the crawler 103 and the ground 403, in addition to the guide lines 702 that extend along the crawler ground line, which is the intersection line between the plane including the inner side surface 501 of the crawler 103 and the ground 403. The processing unit 902 may then superimpose the generated guide lines 702 and additional guide lines 1101 onto the camera image of the side camera 105 to generate a composite image.

[0050] In the camera image from the side camera 105, the inner side 501 of the crawler 103 is not visible because it is hidden by the crawler 103 itself, but the width of the crawler 103 is known in advance and is constant. The processing unit 902 can calculate the coordinates of the crawler grounding line relating to the inner side 501 in the camera image based on the coordinates of the crawler grounding line relating to the outer side 402 in the camera image. Based on the calculated coordinates of the crawler grounding line relating to the inner side 501 in the camera image, the processing unit 902 calculates the coordinates of the additional guide line 1101 in the camera image. As a result, the processing unit 902 can generate the additional guide line 1101 and superimpose it on the camera image from the side camera 105 to generate a composite image. The crawler 103 will pass between the outer guide line 702 and the inner additional guide line 1101. If an obstacle or other object is detected between guide line 702 and additional guide line 1101, the operator can recognize that if crawler 103 continues moving forward, crawler 103 will collide with the obstacle or other object. If an obstacle or other object is detected between guide line 702 and additional guide line 1101, the operator can avoid it and move shovel 100 to the target point.

[0051] Using Figure 12, we will explain another example of a composite image with the guide line 801 superimposed. Figure 12 is a diagram illustrating a composite image in which the guide line 801 and additional guide line 1201 are superimposed on the camera image shown in Figure 8.

[0052] The processing unit 902 may generate additional guide lines 1201 along the crawler ground contact lines, which are the intersection lines between the plane including the inner sides 501 of the left and right crawlers 103 and the ground 403, in addition to the guide lines 801 extending along the crawler ground contact lines 701, which are the intersection lines between the plane including the outer sides 402 of the left and right crawlers 103 and the ground 403. The processing unit 902 may then superimpose the generated guide lines 801 and additional guide lines 1201 onto the camera image of the front camera 106 to generate a composite image.

[0053] In the camera image from the front camera 106, the outer side 402 of the crawler 103 is not visible because it is hidden by the crawler 103 itself, but the width of the crawler 103 is known in advance and is constant. The processing unit 902 can calculate the coordinates of the crawler grounding line 701 relating to the outer side 402 in the camera image based on the coordinates of the crawler grounding line relating to the inner side 501 in the camera image. Based on the calculated coordinates of the crawler grounding line 701 relating to the outer side 402 in the camera image, the processing unit 902 calculates the coordinates of the additional guide line 1201 in the camera image. As a result, the processing unit 902 can generate the additional guide line 1201 and superimpose it on the camera image from the front camera 106 to generate a composite image. The crawler 103 will pass between the inner guide line 801 and the outer additional guide line 1201. If an obstacle or other object is detected between guide line 801 and additional guide line 1201, the operator can recognize that if crawler 103 continues moving forward, crawler 103 will collide with the obstacle or other object. If an obstacle or other object is detected between guide line 801 and additional guide line 1201, the operator can avoid it and move shovel 100 to the target point.

[0054] Using Figure 13, another example of the display method of the guide lines 702 and 801 will be explained. Figure 13 is a diagram illustrating a composite image in which the guide line 702 shown in Figure 7 is composed of arrow-shaped figures 1302.

[0055] The processing unit 902 may generate a guide line 1301 by arranging multiple arrow-shaped figures 1302, which indicate the front of the lower traveling body 102, along the crawler ground contact line. The shape of the figure 1302 is not particularly limited as long as it has a pointed tip (i.e., arrow-shaped), and may be a pentagon, arrow, feather, or triangle as shown in Figure 13. The processing unit 902 is positioned so that the pointed direction of the figure 1302 faces the front of the lower traveling body 102. In Figure 13, the pointed direction of the figure 1302 is oriented in the depth direction of the camera image. When the operator performs a forward operation, the shovel 100 moves in the direction of the pointed figure 1302, that is, forward from the upper rotating body 101. When the upper rotating body 101 is in a rearward position, the rear 406 of the crawler 103 is captured in the camera image, and the pointed direction of the figure 1302 faces towards the camera image. When the operator moves forward, the shovel 100 moves in the direction of the pointed part of figure 1302, that is, backward relative to the upper slewing body 101. In this way, the operator can recognize which direction the upper slewing body 101 is facing relative to the lower traveling body 102 by checking the direction of the pointed part of figure 1302.

[0056] Furthermore, the processing unit 902 may generate not only the guide line 702 but also the guide line 801 using the figure 1302. The processing unit 902 may also generate not only the guide lines 702 and 801 but also the additional guide lines 1101 and 1201 using the figure 1302.

[0057] Using Figure 14, another example of the display method for guide lines 702 and 801 will be explained. Figure 14 is a diagram showing the range of rotation angles for displaying guide lines 702 and 801.

[0058] The processing unit 902 may generate guide lines 702 and 801 only if the rotation angle of the upper rotating body 101, as measured by the rotation angle measuring device 901, is within a predetermined range 1401. If the rotation angle is outside the predetermined range 1401, that is, if the upper rotating body 101 is in a position close to lateral relative to the lower traveling body 102, the crawler ground contact line may fall outside the field of view of the cameras 105 and 106. In this case, it is difficult to generate guide lines 702 and 801 along the crawler ground contact line. Generating and displaying guide lines 702 and 801 that do not follow the crawler ground contact line may confuse the driver. Therefore, the processing unit 902 generates guide lines 702 and 801 only if the rotation angle of the upper rotating body 101 is within the predetermined range 1401. This allows the driver to recognize which direction the upper rotating body 101 is facing relative to the lower traveling body 102 without confusion. The predetermined range 1401 is set in advance according to the field of view of cameras 105 and 106.

[0059] Furthermore, the processing device 902 may also generate the guide line 1301 and the additional guide lines 1101 and 1201 only when the rotation angle of the upper rotating body 101 is within a predetermined range 1401.

[0060] Using Figure 15, another example of how the guide line 1301 is displayed will be explained. Figure 15 is a diagram illustrating a composite image in which the ground 403 is visible through the shape 1302 that constitutes the guide line 1301.

[0061] The processing unit 902 generates the guide line 1301 by arranging multiple partially transparent or semi-transparent shapes 1302 along the crawler ground line so that the ground 403 present at the location where the guide line 1301 is superimposed is visible in the camera image. Partially transparent or semi-transparent shapes 1302 are, for example, shapes 1302 that consist only of an outer frame and are transparent without being filled in, or shapes 1302 that are filled in with a semi-transparent color. If the shapes 1302 that constitute the guide line 1301 have a certain area, the driver may not be able to recognize the terrain at the location where the guide line 1301 is superimposed. For example, if there is an obstacle 1501 such as a stone at the location where the guide line 1301 is superimposed, the driver may not be able to recognize the obstacle 1501. Therefore, the processing unit 902 generates the guide line 1301 using partially transparent or semi-transparent shapes 1302. As a result, even when the guide line 1301 is displayed, the driver can recognize the condition of the ground 403 at the location where the guide line 1301 is superimposed, and can move the shovel 100 to the target point while avoiding the obstacle 1501 located at the location where the guide line 1301 is superimposed.

[0062] Furthermore, the processing device 902 may generate not only the guide line 1301 but also the additional guide lines 1101 and 1201 using a partially transparent or semi-transparent figure 1302.

[0063] Figures 16 to 18 illustrate another example of the display mode of the guide line 1301. Figure 16 is a diagram illustrating a composite image in which the length of the shapes 1302 constituting the guide line 1301 changes according to the travel speed of the lower traveling body 102. Figure 17 is a configuration diagram of the peripheral image display device 10 that displays the composite image shown in Figure 16.

[0064] The processing unit 902 changes the length of the shapes 1302 that make up the guide line 1301 (the length along the crawler ground contact line) according to the travel speed of the lower traveling body 102 (the travel speed of the shovel 100). For example, the faster the travel speed of the lower traveling body 102, the longer the length of the shapes 1302. However, the processing unit 902 does not change the overall length of the guide line 1301. Therefore, if the length of one shape 1302 is increased, the number of shapes 1302 to be arranged decreases. To achieve this, the peripheral image display device 10 has the configuration shown in Figure 17.

[0065] The surrounding image display device 10 shown in Figure 17 has a speed measuring device 1701 added to the surrounding image display device 10 shown in Figure 9, which measures the travel speed of the lower traveling body 102. The speed measuring device 1701 is constructed using existing technology. For example, the speed measuring device 1701 may count the number of rotations of the crawler 103 axle per unit time and calculate the travel speed from the number of rotations and the length of the crawler 103 per rotation. Alternatively, the speed measuring device 1701 may calculate the travel speed from the time change in the position of the shovel 100 identified by a GNSS (Global Navigation Satellite System) receiver. Alternatively, the speed measuring device 1701 may measure the distance of the shovel 100 to surrounding objects using LiDAR (Light Detection and Ranging) or ultrasound, etc., and calculate the travel speed from the time change in the measured distance. The speed measuring device 1701 sends the measured travel speed to the processing device 902 via the input device 904.

[0066] Furthermore, the storage device 903 of the peripheral image display device 10 shown in Figure 17 stores parameters that associate the length of the figures 1302 that constitute the guide line 1301 with the travel speed of the lower traveling body 102. The processing device 902 calculates the length and number of figures 1302 based on the travel speed measured by the speed measuring device 1701 and the parameters stored in the storage device 903. The processing device 902 then generates the guide line 1301 by arranging the calculated number of figures 1302 with the calculated length along the crawler ground contact line. Details of the processing of the processing device 902 will be described later with reference to Figure 18.

[0067] Figure 18 is a flowchart showing the processing performed by the processing unit 902 of the peripheral image display device 10 that displays the composite image shown in Figure 16.

[0068] Steps S1001 to S1004 are the same as the flowchart shown in Figure 10.

[0069] In step S1801, the processing unit 902 obtains the travel speed of the lower traveling body 102, which has been measured by the speed measuring device 1701, from the input device 904.

[0070] In step S1802, the processing unit 902 obtains parameters from the storage device 903 that associate the length of the figure 1302 constituting the guide line 1301 with the travel speed of the lower traveling body 102. These parameters include the minimum and maximum values ​​of the length of the figure 1302, and the upper limit of the travel speed used to calculate the length of the figure 1302. The minimum value of the length of the figure 1302 corresponds to the length of the figure 1302 when the travel speed of the lower traveling body 102 is zero (when stopped). The maximum value of the length of the figure 1302 corresponds to the length of the figure 1302 when the travel speed of the lower traveling body 102 is at the upper limit. The values ​​of these parameters are fixed values ​​that have been stored in the storage device 903 in advance.

[0071] In step S1803, the processing unit 902 calculates the length and number of arrangements of the figure 1302 based on the travel speed acquired in step S1801 and the parameters acquired in step S1802. For example, the processing unit 902 can calculate the length and number of arrangements of the figure 1302 using the following equations (1) and (2).

[0072] Here, among the parameters obtained from the storage device 903, the minimum value of the length of figure 1302 is n, the maximum value is x, and the upper limit of the travel speed used to calculate the length of figure 1302 is U. However, it is assumed that both the minimum value n and the maximum value x are set to values ​​greater than 0 and less than or equal to 1. The minimum value n is less than the maximum value x. The travel speed measured by the speed measuring device 1701 is v, the provisional value of the length of figure 1302 is L, and the number of figures 1302 arranged is C.

[0073] In this case, the provisional value L of the length of figure 1302 is calculated by equation (1). Note that "·" in the equation is the operator indicating multiplication. min[ ,] means that the smaller of the two numbers in [ ,] is output. Even if the driving speed v exceeds the upper limit value U, the provisional value L of the length of figure 1302 is at most x. Furthermore, as shown in equation (2), the number of figures 1302 to be placed C is calculated based on this provisional value L. The maximum value of the provisional value L is x, and since the maximum value x is set to a value of 1 or less, the number of placements C will be an integer of 1 or more. L = min[n + v·(xn) / U ,x] …(1) The integer part of C = (1 / L) ... (2)

[0074] In step S1005, the processing unit 902 generates a guide line 1301 using the length and number of shapes 1302 calculated in step S1803, similar to the flowchart shown in Figure 10. In the generated guide line 1301, C arrow-shaped shapes 1302 with pointed ends are arranged in a sequence along the crawler ground contact line.

[0075] Steps S1006 and S1007 are the same as the flowchart shown in Figure 10. After that, the processing unit 902 completes the process shown in Figure 18.

[0076] As described above, the peripheral image display device 10 shown in Figure 17 includes a speed measuring device 1701 for measuring the travel speed of the lower traveling body 102, and a storage device 903 for storing parameters that associate the length of the figure 1302 constituting the guide line 1301 along the crawler ground contact line with the travel speed. The processing device 902 calculates the length of the figure 1302 based on the travel speed measured by the speed measuring device 1701 and the parameters stored in the storage device 903, and generates the guide line 1301 by arranging multiple figures 1302 having the calculated length along the crawler ground contact line.

[0077] This allows the operator to recognize from the image how far the crawler 103 will have advanced after the shovel 100 has traveled for a certain period of time. Therefore, if there is an obstacle ahead of the crawler 103, the operator can recognize how long it will take for the crawler 103 to reach the obstacle. The operator can then safely and efficiently avoid the obstacle and move the shovel 100 to the target location.

[0078] Furthermore, the processing unit 902 may generate not only the guide line 1301, but also the additional guide lines 1101 and 1201 using a figure 1302 whose length changes according to the travel speed of the lower traveling body 102.

[0079] Figures 19 to 21 illustrate another example of the display mode of the guide line 1301. Figure 19 illustrates a composite image in which the guide line 1301 is restricted so as not to exceed the step 1903. Figure 20 is a configuration diagram of the peripheral image display device 10 that displays the composite image shown in Figure 19.

[0080] In some cases, there may be a large step 1903, such as a cliff, that the shovel 100 cannot traverse ahead of the crawler 103. In such cases, the processing unit 902 limits the length of the guide line 1301 (the length along the crawler ground contact line) so that the tip position of the guide line 1301 does not exceed the boundary 1904 between the step 1903 and the ground 403 on which the crawler 103 makes contact.

[0081] In Figure 19, the ground 403 on which the crawler 103 makes contact is shown as the first ground 1901. In Figure 19, there is a second ground 1902 that is lower than the first ground 1901, located in front of the crawler 103. In Figure 19, a cliff is shown as an example of a step 1903, located between the first ground 1901 and the second ground 1902. In Figure 19, the size of the step 1903 corresponds to the height difference Δh between the first ground 1901 and the second ground 1902, and is large enough that the shovel 100 cannot travel over it. The boundary 1904 of the first ground 1901 is the boundary with the step 1903 of the first ground 1901. In Figure 19, the edge of a cliff is shown as an example of the boundary 1904 of the first ground 1901.

[0082] If a step that the shovel 100 can travel over is closer than the length of a preset guide line 1301, the processing unit 902 generates a guide line 1301 that extends beyond the boundary 1904 of the first ground 1901. On the other hand, if a step 1903 that the shovel 100 cannot travel over is closer than the length of a preset guide line 1301, the processing unit 902 shortens the length of the guide line 1301 so that the tip of the guide line 1301 does not cross the boundary 1904. This allows the operator to recognize the limit of where the crawler 103 can travel. To achieve this, the surrounding image display device 10 has the configuration shown in Figure 20.

[0083] The surrounding image display device 10 shown in Figure 20 is an additional terrain measurement device 2001 compared to the surrounding image display device 10 shown in Figure 9, which measures terrain data around the shovel 100. The terrain measurement device 2001 is configured using existing technologies such as LiDAR. The terrain measurement device 2001 measures at least the slope angle of the ground and the height of the ground as terrain data. The terrain measurement device 2001 sends the measured terrain data to the processing device 902 via the input device 904.

[0084] The processing unit 902 determines, based on the terrain data measured by the terrain measurement device 2001, whether or not there is a step 1903 ahead of the crawler 103 that prevents the shovel 100 from passing. If it determines that such a step 1903 exists, it limits the length of the guide line 1301. Details of the processing unit 902 will be described later with reference to Figure 21.

[0085] Figure 21 is a flowchart showing the processing performed by the processing unit 902 of the peripheral image display device 10, which displays the composite image shown in Figure 19.

[0086] Steps S1001 to S1004 are the same as the flowchart shown in Figure 10.

[0087] In step S2101, the processing unit 902 acquires terrain data from the input device 904, which is measured by the terrain measurement device 2001 at the end of the crawler 103.

[0088] In step S2102, the processing unit 902 determines, based on the terrain data acquired in step S2101, whether or not there is a step 1903 ahead of the crawler 103 that the shovel 100 cannot travel over. Specifically, the processing unit 902 uses the shovel 100 itself as a horizontal reference and calculates the slope angle and slope length of the terrain ahead of the crawler 103 relative to the horizontal based on the terrain data. Then, the processing unit 902 determines, based on the slope angle and slope length of the terrain ahead of the crawler 103, whether or not there is a step 1903 that the shovel 100 cannot travel over. If the slope angle ahead of the crawler 103 exceeds a threshold angle and the slope length exceeds a threshold length, it is dangerous for the shovel 100 to proceed. The thresholds for determining this slope angle and slope length are set in advance based on the specifications and structure of the shovel 100. For example, if the terrain ahead of the crawler 103 is a steep cliff, the slope angle will be large. If the slope of the terrain continues for a long distance, there is a risk that the shovel 100 may tip over or fall. On the other hand, if the slope angle is large but the length of the slope is short, the height difference Δh will be small, and the size of the step 1903 will be small. If the step 1903 is small, the shovel 100 can travel over it.

[0089] If the processing unit 902 has a step 1903 ahead of the crawler 103 that prevents the shovel 100 from passing over, it proceeds to step S2103. If the processing unit 902 does not have a step 1903 ahead of the crawler 103, it proceeds to step S1005.

[0090] In step S2103, the processing unit 902 identifies the coordinates in the camera image of the step 1903 that the shovel 100 cannot traverse. In particular, the processing unit 902 identifies the coordinates in the camera image of the boundary 1904 of the first ground 1901, which is the starting point of the step 1903. As mentioned above, the mounting positions, shooting directions, and field of view of the cameras 105 and 106 are predetermined. In addition, the direction of the step 1903 as seen from the mounting positions of the cameras 105 and 106, and the distance to the step 1903, can be calculated based on terrain data. Based on this information, the processing unit 902 can identify where the step 1903 is located in the camera image.

[0091] The processing unit 902 then calculates the length of the guide line 1301 based on the coordinates of the step 1903 in the camera image, so that the tip position of the guide line 1301 does not exceed the boundary 1904 of the first ground surface 1901. The coordinates of the boundary 1904 of the first ground surface 1901 in the camera image are determined from the coordinates of the step 1903 in the camera image. The coordinates of the guide line 1301 in the camera image are determined from the coordinates of the crawler ground line in the camera image calculated in step S1004. The processing unit 902 compares the coordinates of the boundary 1904 of the first ground surface 1901 in the camera image with the coordinates of the guide line 1301 in the camera image, and determines whether the tip position of the guide line 1301 exceeds the boundary 1904 of the first ground surface 1901 in the camera image. If the tip position of the guide line 1301 exceeds the boundary 1904 of the first ground surface 1901 in the camera image, the length of the guide line 1301 is shortened so that the tip position of the guide line 1301 is located in front of the boundary 1904 in the camera image. In this way, the processing unit 902 calculates the length of the guide line 1301 so that the tip position of the guide line 1301 does not exceed the boundary 1904 of the first ground surface 1901.

[0092] In step S1005, if the length of the guide line 1301 was calculated in step S2103, the processing unit 902 generates the guide line 1301 using the calculated length of the guide line 1301, in the same manner as the flowchart shown in Figure 10.

[0093] Steps S1006 and S1007 are the same as the flowchart shown in Figure 10. After that, the processing unit 902 completes the process shown in Figure 21.

[0094] As shown above, the surrounding image display device 10 in Figure 20 includes a terrain measuring device 2001 that measures terrain data around the shovel 100. Based on the terrain data measured by the terrain measuring device 2001, the processing device 902 determines whether or not there is a step 1903 ahead of the crawler 103 that the shovel 100 cannot travel over. If it is determined that such a step 1903 exists, the processing device 902 limits the length of the guide line 1301 along the crawler ground line so that the tip position of the guide line 1301 does not exceed the boundary 1904 between the step 1903 and the first ground 1901 on which the crawler 103 makes contact.

[0095] This allows the driver to easily recognize the limit of how far the crawler 103 can go. The driver can avoid the risk of tipping over or falling by approaching the limit of how far the crawler 103 can go.

[0096] Furthermore, the processing device 902 may also limit the length of the additional guide lines 1101 and 1201 along the crawler ground contact line, so that the tip positions of the additional guide lines 1101 and 1201 do not exceed the boundary 1904 with the step 1903 of the first ground surface 1901.

[0097] Figures 22 and 23 illustrate another example of how the guide line 1301 is displayed. Figure 22 is a diagram illustrating a composite image in which the guide line 1301 is bent along the inclined surface 2201.

[0098] There may be an uneven, inclined surface 2201 at the end of the crawler 103. This inclined surface 2201 is assumed to have a gentle slope angle that allows the shovel 100 to travel on it. In such cases, the processing device 902 generates the guide line 1301 by bending it along the inclined surface 2201 so that the guide line 1301 extends along the inclined surface 2201.

[0099] In Figure 22, the ground 403 on which the crawler 103 makes contact is shown as the first ground surface 1901. In Figure 22, there is a second ground surface 1902 that is lower than the first ground surface 1901, located in front of the crawler 103. In Figure 22, the height difference Δh between the first ground surface 1901 and the second ground surface 1902 is small relative to the length of the inclined surface 2201, indicating a gentle inclined surface 2201 on which the shovel 100 can travel. The boundary 1904 of the first ground surface 1901 is the boundary between the first ground surface 1901 and the inclined surface 2201.

[0100] Since the shovel 100 can travel on the inclined surface 2201, the processing unit 902 can also generate a guide line 1301 that extends in a straight line beyond the boundary 1904 of the first ground 1901 if the inclined surface 2201 ahead of the crawler 103 is closer than the length of the preset guide line 1301. However, if the processing unit 902 is closer than the length of the preset guide line 1301, it generates the guide line 1301 by bending it along the inclined surface 2201 so that the guide line 1301 extends along the inclined surface 2201. This allows the operator to recognize in advance that the terrain ahead is inclined and to proceed with caution. Details of the processing unit 902 will be described later with reference to Figure 23.

[0101] Figure 23 is a flowchart showing the processing performed by the processing unit 902 of the peripheral image display device 10, which displays the composite image shown in Figure 22.

[0102] Steps S1001 to S1004 are the same as the flowchart shown in Figure 10. Step S2101 is the same as the flowchart shown in Figure 21.

[0103] In step S2301, the processing unit 902 determines, based on the terrain data acquired in step S2101, whether or not there is an inclined surface 2201 in front of the crawler 103 that the shovel 100 can travel on. Specifically, the processing unit 902 uses the shovel 100 itself as a horizontal reference and calculates the inclination angle and inclination length of the terrain in front of the crawler 103 relative to the horizontal based on the terrain data. Then, the processing unit 902 determines, based on the inclination angle and inclination length of the terrain in front of the crawler 103, whether or not there is an inclined surface 2201 that the shovel 100 can travel on. However, minute changes in terrain are not considered as inclination, and an inclination angle in front of the crawler 103 exceeds a predetermined angle and inclination length exceeds a predetermined length, and is considered an inclined surface 2201. The predetermined angle and predetermined length for being considered an inclined surface 2201 are set in advance based on the specifications and structure of the shovel 100.

[0104] If there is an inclined surface 2201 in front of the crawler 103 on which the shovel 100 can travel, the processing unit 902 proceeds to step S2302. If there is no inclined surface 2201 in front of the crawler 103, the processing unit 902 proceeds to step S1005.

[0105] In step S2302, the processing unit 902 identifies the coordinates in the camera image of the inclined surface 2201 on which the shovel 100 can travel. In particular, the processing unit 902 identifies the coordinates in the camera image of the boundary 1904 of the first ground surface 1901, which is the starting point of the inclined surface 2201. This identification method is the same as in step S2103 in Figure 21.

[0106] The processing unit 902 then calculates the position and direction in which to bend the guide line 1301 based on the coordinates of the inclined surface 2201 in the camera image and the inclination angle of the inclined surface 2201. Specifically, the processing unit 902 compares the coordinates of the boundary 1904 of the first ground 1901, which is the starting position of the inclined surface 2201, in the camera image with the coordinates of the guide line 1301 in the camera image, and determines whether the tip position of the guide line 1301 crosses the boundary 1904 of the first ground 1901 in the camera image. If the tip position of the guide line 1301 crosses the boundary 1904 of the first ground 1901 in the camera image, the coordinates of the boundary 1904 of the first ground 1901, which is the starting position of the inclined surface 2201, in the camera image are set as the position in which to bend the guide line 1301 in the camera image. Furthermore, the processing unit 902 calculates from the terrain data which direction the inclination direction of the part of the inclined surface 2201 with the maximum inclination angle appears to be from the mounting positions of the cameras 105 and 106. Then, the processing unit 902 uses the calculated direction as the direction in which the guide line 1301 is bent in the camera image. In this way, the processing unit 902 calculates the position and direction in which the guide line 1301 is bent.

[0107] In step S1005, if the position and direction for bending the guide line 1301 have been calculated in step S2302, the processing unit 902 generates the guide line 1301 using the calculated position and direction for bending the guide line 1301, in the same manner as the flowchart shown in Figure 10.

[0108] Steps S1006 and S1007 are the same as the flowchart shown in Figure 10. After that, the processing unit 902 completes the process shown in Figure 23.

[0109] As described above, the surrounding image display device 10 that displays the composite image shown in Figure 22 is equipped with a terrain measuring device 2001 that measures terrain data around the shovel 100. Based on the terrain data measured by the terrain measuring device 2001, the processing device 902 determines whether or not there is an inclined surface 2201 on which the shovel 100 can travel ahead of the crawler 103. If it is determined that such an inclined surface 2201 exists, the processing device 902 generates the guide line 1301 by bending it along the inclined surface 2201 so that the guide line 1301 extends along the inclined surface 2201.

[0110] This allows the driver to recognize in advance that the terrain ahead is sloped and to proceed with caution. The driver can avoid dangers such as tipping over when moving along the sloped surface 2201.

[0111] Furthermore, the processing device 902 may also generate additional guide lines 1101 and 1201 by bending them along the inclined surface 2201, so that they extend along the inclined surface 2201, in addition to the guide line 1301.

[0112] [Second Embodiment] A second embodiment of the present invention will be described using Figure 24. In the second embodiment, the same components as in the first embodiment will not be described.

[0113] Figure 24 is a configuration diagram of the peripheral image display device 10 according to the second embodiment.

[0114] In the peripheral image display device 10 of the second embodiment, the mounting configuration of cameras 105 and 106 to the shovel 100, and the display configuration of camera images and guide lines 702, 801, 1301, etc. on the monitor 601 are the same as in the first embodiment. The device configuration of the peripheral image display device 10 of the second embodiment is divided into a device group 2401 on the shovel 100 side and a device group 2402 on the remote control room side. In the peripheral image display device 10 of the second embodiment, the output device 905 and monitor 601 are installed in a remote control room located away from the work site of the shovel 100, rather than in the operator's cab 104 of the shovel 100. The shovel 100 of the second embodiment is operated by remote control from the remote control room.

[0115] A composite image, created by superimposing guide lines 702, 801, and 1301 onto the camera images from cameras 105 and 106, is generated by the processing unit 902 and sent to the communication device 2403 included in the device group 2401 on the shovel 100 side. The communication device 2403 on the shovel 100 side transmits the composite image to the communication device 2404 included in the device group 2402 on the remote control room side. Communication between the communication device 2403 on the shovel 100 side and the communication device 2404 on the remote control room side may be wireless or wired. The communication device 2404 on the remote control room side sends the received composite image to the monitor 601 via the output device 905.

[0116] As a result, the surrounding image display device 10 can easily provide the operator with images that allow the operator to appropriately recognize the condition of the ground 403 around the crawler 103 and the condition of the ground 403 in front of the crawler 103, even when the operator is operating the shovel 100 remotely.

[0117] The peripheral image display device 10, which is one embodiment of the present invention, has been described above. The present invention can also be implemented as a method for displaying peripheral images of construction machinery. The peripheral image display method of this embodiment may be realized by the processing device 902 executing a program that has been stored in advance.

[0118] The surrounding image display method of this embodiment is a method for displaying surrounding images of a construction machine, which displays images captured by cameras 105 and 106 attached to a construction machine having a lower traveling body 102 including a crawler 103 and an upper rotating body 101 that can rotate relative to the lower traveling body 102 on a monitor 601. In this surrounding image display method, the cameras 105 and 106 include a side camera 105 attached to the side of the upper rotating body 101 such that its shooting range faces forward of the upper rotating body 101. The surrounding image display method includes a guide line generation step (for example, step S1005 in Figure 10) which generates a guide line 702 extending along the intersection line between a plane including the outer side surface 402 of the crawler 103 and the ground 403 on which the crawler 103 makes contact; a composite image generation step (for example, step S1006 in Figure 10) which superimposes the generated guide line 702 onto an image captured by the side camera 105 to generate a composite image; and a display step (for example, step S1007 in Figure 10) which outputs the generated composite image to the monitor 601 and displays it on the monitor 601.

[0119] The peripheral image display method of this embodiment further includes an additional guideline generation step that generates an additional guideline 1101 extending along the intersection line between the plane including the inner side surface 501 of the crawler 103 and the ground 403, and a composite image generation step that generates a composite image by superimposing the generated guideline 702 and the additional guideline 1101 onto an image captured by the side camera 105.

[0120] In the peripheral image display method of this embodiment, cameras 105 and 106 further include a front camera 106 mounted on the front of the upper rotating body 101 such that its shooting range faces forward of the upper rotating body 101. The guide line generation step generates a guide line 801 that extends along the intersection line between the plane including the inner side surface 501 of the crawler 103 and the ground 403. The composite image generation step generates a composite image by superimposing the generated guide line 801 onto the image captured by the front camera 106.

[0121] In the surrounding image display method of this embodiment, the guide line generation step generates a figure 1302 by arranging multiple arrow-shaped figures 1302 that indicate the front of the lower traveling body 102 along the aforementioned intersection line.

[0122] In the peripheral image display method of this embodiment, the guide line generation step generates guide lines 702 and 801 only when the rotation angle of the upper rotating body 101 is within a predetermined range 1401.

[0123] In the peripheral image display method of this embodiment, the guide line generation step generates the guide line 1301 by arranging multiple figures 1302 that are at least partially transparent or semi-transparent along the aforementioned intersection lines so that the ground 403 located at the overlapping position of the guide line 1301 is captured.

[0124] In the surrounding image display method of this embodiment, the construction machine includes a speed measuring device 1701 for measuring the travel speed of the lower traveling body 102, and a storage device 903 for storing parameters that associate the length of the figure 1302 constituting the guide line 1301 along the aforementioned intersection line with the travel speed. The guide line generation process calculates the length of the figure 1302 based on the travel speed measured by the speed measuring device 1701 and the parameters stored in the storage device 903, and generates the guide line 1301 by arranging a plurality of figures 1302 having the calculated length along the intersection line.

[0125] In the surrounding image display method of this embodiment, the construction machine is equipped with a terrain measuring device 2001 that measures terrain data around the construction machine. The guide line generation process determines, based on the terrain data measured by the terrain measuring device 2001, whether or not there is a step 1903 ahead of the crawler 103 that the construction machine cannot travel over. If it is determined that there is a step 1903, the length of the guide line 1301 along the aforementioned crossing line is limited so that the tip position of the guide line 1301 does not exceed the boundary 1904 between the step 1903 and the first ground 1901 on which the crawler 103 makes contact.

[0126] In the surrounding image display method of this embodiment, the construction machine is equipped with a terrain measuring device 2001 that measures terrain data around the construction machine. The guide line generation process determines, based on the terrain data measured by the terrain measuring device 2001, whether or not there is an inclined surface 2201 that the construction machine can travel on in front of the crawler 103. If it is determined that there is an inclined surface 2201, the guide line 1301 is generated by bending it along the inclined surface 2201 so that the guide line 1301 extends along the inclined surface 2201.

[0127] In the peripheral image display method of this embodiment, the monitor 601 is installed in a remote control room located away from the construction machine's work site, and the construction machine is operated remotely from the remote control room.

[0128] The peripheral image display method of this embodiment can achieve the same effects as the peripheral image display device 10 described above.

[0129] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those comprising all the components described. Furthermore, it is possible to replace some of the components of one embodiment with components of another embodiment, and it is also possible to add components of another embodiment to the components of one embodiment. In addition, it is possible to add, delete, or replace some of the components of each embodiment with components of other embodiments.

[0130] Furthermore, each of the above-mentioned components, functions, processing units, or processing means may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above-mentioned components or functions may be implemented in software by a processor interpreting and executing programs that implement each function. Information such as programs, tables, or files that implement each function can be stored in a recording device such as memory, a hard disk, or an SSD (solid state drive), or in a recording medium such as an IC card, SD card, or DVD.

[0131] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0132] 10...Surrounding image display device, 100...Shovel, 101...Upper rotating body, 102...Lower traveling body, 103...Crawler, 104...Driver's cab, 104a...Floor, 105...Side camera, 106...Front camera, 108...Rotating axis, 402...Outer side, 403...Ground, 405...Front, 406...Rear, 501...Inner side, 601...Monitor, 701...Crawler ground contact line (Intersecting lines), 702...Guide lines, 801...Guide lines, 901...Swivel angle measuring device, 902...Processing device, 903...Memory device, 1101...Additional guide lines, 1201...Additional guide lines, 1301...Guide lines, 1302...Figure, 1401...Determined range, 1701...Speed ​​measuring device, 1903...Step, 1904...Boundary, 2001...Terrain measuring device, 2201...Inclined surface

Claims

1. A camera attached to a construction machine comprising a lower traveling body including a crawler and an upper slewing body that is rotatably mounted relative to the lower traveling body, The system includes a monitor that displays images captured by the aforementioned camera, The camera includes a side camera mounted on the side of the upper rotating body such that its shooting range is directed forward of the upper rotating body. The aforementioned side camera is The upper rotating body is positioned below the floor of the driver's cab, such that the distance from the pivot axis of the upper rotating body to the center of the lens of the side camera is longer than the distance from the pivot axis to the outer side of the crawler. The upper rotating body has a field of view that allows it to photograph the outer side of the front of the crawler and the ground on which the crawler makes contact when the upper rotating body is facing forward and the lower traveling body is facing forward, and the upper rotating body has a field of view that allows it to photograph the outer side of the rear of the crawler and the ground when the upper rotating body is facing backward and the lower traveling body is facing rearward. A peripheral image display device for construction machinery, characterized by the following features.

2. The system further includes a processing device that processes the image captured by the camera and displays it on the monitor, The aforementioned processing apparatus is A guideline is generated that extends along the intersection line between the plane including the outer side surface of the crawler and the ground. The generated guide lines are superimposed on the image captured by the side camera to generate a composite image. The generated composite image is displayed on the monitor. A peripheral image display device for construction machinery as described in claim 1.

3. The processing device further generates additional guide lines extending along the intersection line between the plane including the inner side surface of the crawler and the ground, The generated guide lines and additional guide lines are superimposed on the image captured by the side camera to generate the composite image. A peripheral image display device for construction machinery as described in feature 2.

4. The camera further includes a front camera mounted on the front of the upper rotating body such that its shooting range is directed forward of the upper rotating body. The aforementioned front camera is It is positioned below the floor of the driver's cab, In the forward-facing position, the field of view is capable of photographing the inner side of the front of the crawler and the ground, and in the rear-facing position, the field of view is capable of photographing the inner side of the rear of the crawler and the ground. The processing device generates a guide line extending along the intersection line between the plane including the inner side surface of the crawler and the ground, The generated guide lines are superimposed on the image captured by the front camera to generate the composite image. A peripheral image display device for construction machinery as described in feature 2.

5. The processing apparatus generates the guide lines by arranging multiple arrow-shaped figures indicating the front of the lower traveling body along the intersecting lines. A peripheral image display device for construction machinery as described in feature 2.

6. The system further includes a pivot angle measuring device for measuring the pivot angle of the upper pivot body relative to the lower traveling body, The processing device generates the guide line only when the rotation angle measured by the rotation angle measuring device is within a predetermined range. A peripheral image display device for construction machinery as described in feature 2.

7. The processing device generates the guide lines by arranging multiple partially transparent or semi-transparent figures along the intersection lines so that the ground located at the position where the guide lines are superimposed is captured in the image. A peripheral image display device for construction machinery as described in feature 2.

8. A speed measuring device for measuring the travel speed of the lower traveling body, The system further includes a storage device that stores parameters relating the length of the figure constituting the guide line along the intersecting line to the travel speed, The aforementioned processing apparatus is Based on the speed measured by the speed measuring device and the parameters stored in the memory device, the length of the figure is calculated. The guide lines are generated by arranging multiple figures having the calculated lengths along the intersecting lines. A peripheral image display device for construction machinery as described in feature 2.

9. The construction machine is further equipped with a topographic measuring device for measuring topographic data around it. The aforementioned processing apparatus is Based on the terrain data measured by the terrain measuring device, it is determined whether or not there is a step ahead of the crawler that prevents the construction machine from traveling. If the presence of the aforementioned step is determined, the length of the guide line along the intersection line is limited so that the tip position of the guide line does not exceed the boundary between the step and the ground on which the crawler makes contact. A peripheral image display device for construction machinery as described in feature 2.

10. The construction machine is further equipped with a topographic measuring device for measuring topographic data around it. The aforementioned processing apparatus is Based on the terrain data measured by the terrain measuring device, it is determined whether or not there is an inclined surface in front of the crawler that the construction machine can travel on. If it is determined that the aforementioned inclined surface exists, the guide line is generated by bending it along the inclined surface so that the guide line extends along the inclined surface. A peripheral image display device for construction machinery as described in feature 2.

11. The monitor is installed in a remote control room located away from the construction site of the construction machine. The construction machinery is operated remotely from the remote control room. A peripheral image display device for construction machinery as described in claim 1.

12. A method for displaying images of the surroundings of a construction machine, comprising a lower traveling body including a crawler and an upper slewing body that can rotate relative to the lower traveling body, wherein images captured by a camera attached to the construction machine are displayed on a monitor, The camera includes a side camera mounted on the side of the upper rotating body such that its shooting range is directed forward of the upper rotating body. The aforementioned method for displaying surrounding images is: A guide line generation step generates a guide line that extends along the intersection line between a plane including the outer side surface of the crawler and the ground on which the crawler makes contact. A composite image generation step involves superimposing the generated guide lines onto the image captured by the side camera to generate a composite image, The process includes a display step of outputting the generated composite image to the monitor and displaying it on the monitor. A method for displaying images of the surroundings of construction machinery, characterized by the features described herein.

13. The system further includes an additional guideline generation step, which generates an additional guideline extending along the intersection line between a plane including the inner side surface of the crawler and the ground. The composite image generation step generates the composite image by superimposing the generated guide lines and additional guide lines onto the image captured by the side camera. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

14. The camera further includes a front camera mounted on the front of the upper rotating body such that its shooting range is directed forward of the upper rotating body. The guide line generation step generates a guide line that extends along the intersection line between the plane including the inner side surface of the crawler and the ground, The composite image generation step generates the composite image by superimposing the generated guide lines onto the image captured by the front camera. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

15. The guide line generation step generates the guide line by arranging multiple arrow-shaped figures indicating the front of the lower traveling body along the intersecting line. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

16. The guide line generation step generates the guide line only when the rotation angle of the upper rotating body is within a predetermined range. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

17. The guide line generation step generates the guide lines by arranging multiple figures, at least partially transparent or semi-transparent, along the intersection lines so that the ground present at the overlapping position of the guide lines is reflected. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

18. The construction machine comprises a speed measuring device for measuring the travel speed of the lower traveling body, and a storage device for storing parameters that associate the length of the figure constituting the guide line along the intersecting line with the travel speed. The guide line generation step is, Based on the speed measured by the speed measuring device and the parameters stored in the memory device, the length of the figure is calculated. The guide lines are generated by arranging multiple figures having the calculated lengths along the intersecting lines. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

19. The construction machine is equipped with a topographic measuring device that measures topographic data in the area surrounding the construction machine. The guide line generation step is, Based on the terrain data measured by the terrain measuring device, it is determined whether or not there is a step ahead of the crawler that prevents the construction machine from traveling. If the presence of the aforementioned step is determined, the length of the guide line along the intersection line is limited so that the tip position of the guide line does not exceed the boundary between the step and the ground on which the crawler makes contact. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

20. The construction machine is equipped with a topographic measuring device that measures topographic data in the area surrounding the construction machine. The guide line generation step is, Based on the terrain data measured by the terrain measuring device, it is determined whether or not there is an inclined surface in front of the crawler that the construction machine can travel on. If it is determined that the aforementioned inclined surface exists, the guide line is generated by bending it along the inclined surface so that the guide line extends along the inclined surface. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

21. The monitor is installed in a remote control room located away from the construction site of the construction machine. The construction machinery is operated remotely from the remote control room. A method for displaying images of the surrounding area of ​​a construction machine as described in 12.

Images