Periphery monitoring system for work machine, work machine, and method for monitoring peripheral areas of work machine

Abstract

To suppress deterioration of visibility when a plurality of types of images are simultaneously displayed on a display.SOLUTION: A monitoring system for work machine periphery comprises: a first display image generation unit that generates a first display image indicating the periphery of a work machine on the basis of a camera image of a camera that images the periphery of the work machine; a second display image generation unit that generates a second display image having a display form different from that of the first display image on the basis of the camera images of a plurality of cameras that image the periphery of the work machine; and a display control unit that generates a signal for displaying the first display image on a part of a display screen, displaying the second display image on a part of the display screen, and displaying a boundary image on the boundary between the first display image and the second display image on the display screen.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a periphery monitoring system for a work machine, a work machine, and a periphery monitoring method for a work machine. [Background technology]

[0002] 2. Description of the Related Art In the technical field relating to work machines, a surroundings monitoring device such as that disclosed in Patent Document 1 is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-074929 A Summary of the Invention [Problem to be solved by the invention]

[0004] The periphery monitoring device includes a display unit that displays an image of the periphery of the work machine. The driver of the work machine can recognize the situation around the work machine by checking the image displayed on the display unit. There are cases where it is desired to display multiple types of images on the display unit to allow the driver to highly recognize the situation around the work machine. However, when multiple types of images are displayed on the display unit at the same time, visibility may decrease.

[0005] According to the present disclosure, it is an object to suppress a decrease in visibility when multiple types of images are simultaneously displayed on a display unit. [Means for solving the problem]

[0006] According to the present disclosure, there is provided a surrounding monitoring system for a work machine, comprising: a first display image generating unit that generates a first display image showing the surroundings of the work machine based on a camera image from a camera that images the surroundings of the work machine; a second display image generating unit that generates a second display image having a display form different from the first display image based on camera images from a plurality of cameras that image the surroundings of the work machine; and a display control unit that generates a signal for displaying the first display image on a portion of the display screen, displaying the second display image on a portion of the display screen, and displaying a boundary image at the boundary between the first display image and the second display image on the display screen. Effect of the Invention

[0007] According to the present disclosure, when multiple types of images are simultaneously displayed on a display unit, a decrease in visibility is suppressed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view showing a work machine according to an embodiment. [Diagram 2] FIG. 2 is a side view showing the work machine according to the embodiment. [Diagram 3] FIG. 3 is a top view illustrating a camera system according to an embodiment. [Figure 4] FIG. 4 is a top view diagrammatically illustrating the radar system according to the embodiment. [Diagram 5] FIG. 5 is a functional block diagram showing a motor grader according to an embodiment. [Figure 6] FIG. 6 is a diagram showing a periphery monitoring monitor according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a composite image according to the embodiment. [Figure 8] FIG. 8 is a diagram showing a composite image according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing a synthetic image generating method according to the embodiment. [Figure 10] FIG. 10 is a block diagram illustrating a computer system according to an embodiment. [Figure 11] FIG. 11 is a diagram showing a composite image according to the embodiment. [Figure 12] FIG. 12 is a diagram showing a composite image according to the embodiment. [Figure 13] FIG. 13 is a diagram showing a composite image according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Although an embodiment according to the present disclosure will be described with reference to the drawings, the present disclosure is not limited to the embodiment. The components of the embodiments described below can be appropriately combined. In addition, some components may not be used.

[0010] In the embodiment, a local coordinate system is set in the work machine 1, and the positional relationship of each part will be described with reference to the local coordinate system. In the local coordinate system, the first axis extending in the left-right direction (vehicle width direction) of the work machine 1 is the X-axis, the second axis extending in the front-rear direction of the work machine 1 is the Y-axis, and the third axis extending in the up-down direction of the work machine 1 is the Z-axis. The X-axis and Y-axis are perpendicular to each other. The Y-axis and Z-axis are perpendicular to each other. The Z-axis and X-axis are perpendicular to each other. The +X direction is the rightward direction, and the -X direction is the leftward direction. The +Y direction is the forward direction, and the -Y direction is the backward direction. The +Z direction is the upward direction, and the -Z direction is the downward direction.

[0011] [First embodiment] <Work machinery> Fig. 1 is a perspective view showing a work machine 1 according to an embodiment. Fig. 2 is a side view showing the work machine 1 according to an embodiment. In the embodiment, the work machine 1 is a motor grader. In the following description, the work machine 1 will be appropriately referred to as a motor grader 1.

[0012] As shown in Figures 1 and 2, the motor grader 1 comprises a vehicle body 2, a cab 3, a work machine 4, wheels 5, a work machine operating device 6, a traveling operating device 7, a work machine control device 8, a traveling control device 9, and a surrounding monitoring system 10.

[0013] The motor grader 1 travels on a work site using wheels 5. The motor grader 1 performs work using a work implement 4 at the work site. The work implement 4 performs work at the work site. Examples of work performed by the motor grader 1 include ground leveling, road cutting, excavation, snow removal, and material mixing.

[0014] The vehicle body 2 supports the work machine 4. The vehicle body 2 includes a front vehicle body 21 and a rear vehicle body 22. The front vehicle body 21 is disposed forward of the rear vehicle body 22. The front vehicle body 21 and the rear vehicle body 22 are connected by an articulation mechanism 23. The front vehicle body 21 is rotatably connected to the rear vehicle body 22 via the articulation mechanism 23. The rear end of the front vehicle body 21 and the front end of the rear vehicle body 22 are connected via the articulation mechanism 23. The articulation mechanism 23 includes a center pin extending in the Z-axis direction. The rear end of the front vehicle body 21 and the rear end of the rear vehicle body 22 are connected via the center pin.

[0015] The articulation mechanism 23 includes an articulation cylinder 24. The articulation cylinder 24 connects the front vehicle body 21 and the rear vehicle body 22. As the articulation cylinder 24 expands and contracts, the front vehicle body 21 bends in the left-right direction relative to the rear vehicle body 22. As the front vehicle body 21 bends relative to the rear vehicle body 22, the traveling direction of the motor grader 1 is adjusted. The articulation cylinder 24 is, for example, a hydraulic cylinder.

[0016] The rear vehicle body 22 includes an exterior cover 25. An engine compartment is formed inside the exterior cover 25. An engine 26 is disposed in the engine compartment. The rear vehicle body 22 supports the engine 26.

[0017] The cab 3 is supported by the rear vehicle body 22. A driver sits inside the cab 3. A seat 31 on which the driver sits is disposed inside the cab 3.

[0018] The work machine 4 is supported by the front vehicle body 21. The work machine 4 has a draw bar 40, a turning circle 41, and a blade 42. The draw bar 40 is disposed below the front vehicle body 21. The front end of the draw bar 40 is connected to the front end of the front vehicle body 21 via a ball shaft portion. The front end of the draw bar 40 is supported to be swingable at the front end of the front vehicle body 21. The rear end of the draw bar 40 is supported by the front vehicle body 21 via a lift cylinder 44 and a lift cylinder 45. The draw bar 40 is suspended from the front vehicle body 21 via the lift cylinder 44 and the lift cylinder 45.

[0019] The turning circle 41 is disposed below the front vehicle body 21. The turning circle 41 is disposed below the draw bar 40. The turning circle 41 is supported on the rear end portion of the draw bar 40 so as to be able to turn.

[0020] The blade 42 is disposed between the front end and the rear end of the vehicle body 2. The blade 42 is supported by a turning circle 41. The blade 42 is supported by the drawbar 40 via the turning circle 41. The blade 42 is supported by the front vehicle body 21 via the drawbar 40. The turning circle 41 is turned to adjust the thrust angle of the blade 42. The thrust angle of the blade 42 refers to the inclination angle of the blade 42 with respect to the Y-axis.

[0021] The lift cylinders 44 and 45 support the draw bar 40. The lift cylinders 44 and 45 support the blade 42 via the draw bar 40. The lift cylinders 44 and 45 extend and retract, so that the rear end of the draw bar 40 moves in the vertical direction relative to the front vehicle body 21. That is, the lift cylinders 44 and 45 extend and retract to adjust the heights of the draw bar 40 and the blade 42. The draw bar 40 swings in a rotational direction about the Y-axis as the lift cylinders 44 and 45 extend and retract by different amounts. The blade 42 can move in both the Z-axis direction and the rotational direction about the Y-axis.

[0022] The wheels 5 support the vehicle body 2. The wheels 5 include a front wheel 51 and a rear wheel 52. The front wheel 51 is disposed forward of the rear wheel 52. In the front-to-rear direction, the blade 42 is disposed between the front wheel 51 and the rear wheel 52. The front wheel 51 is disposed forward of the blade 42. The rear wheel 52 is disposed rearward of the blade 42.

[0023] The front wheels 51 are attached to the front body 21. The front body 21 rotatably supports the front wheels 51. The front wheels 51 are supported at the front end portion of the front body 21. One front wheel 51 is disposed on the left side of the front body 21, and the other is disposed on the right side of the front body 21. That is, two front wheels 51 are provided.

[0024] The rear wheels 52 are attached to the rear body 22. The rear body 22 rotatably supports the rear wheels 52. Two rear wheels 52 are disposed on the left side of the rear body 22, and two are disposed on the right side of the rear body 22. That is, four rear wheels 52 are provided. The rear wheels 52 rotate based on power generated by the engine 26.

[0025] In the embodiment, the X-axis is parallel to the rotation axis of the front wheels 51. The Z-axis is perpendicular to the contact surface of the front wheels 51 that contacts the ground. When the motor grader 1 travels in a straight line, the rotation axis of the front wheels 51 and the rotation axis of the rear wheels 52 are parallel to each other.

[0026] The work machine operating device 6 and the travel operating device 7 are each disposed inside the cab 4. The work machine operating device 6 and the travel operating device 7 are each operated by the driver.

[0027] The work machine operating device 6 generates an operation signal for operating the work machine 4. An example of the work machine operating device 6 is a work machine operating lever. When the driver operates the work machine operating device 6, the work machine operating device 6 generates an operation signal. The operation signal of the work machine operating device 6 is transmitted to the work machine control device 8. The work machine control device 8 outputs a control command for operating the work machine 4 based on the operation signal of the work machine operating device 6. The control command output from the work machine control device 8 includes a control command for operating at least one of the lift cylinder 44 and the lift cylinder 45. The blade 42 of the work machine 4 moves in at least one of the Z-axis direction and the rotational direction about the Y-axis based on the control command output from the work machine control device 8.

[0028] The travel operation device 7 generates an operation signal for driving, braking, and turning the motor grader 1. Examples of the travel operation device 7 include a steering wheel, a gear shift lever, an accelerator pedal, a brake pedal, and an inching pedal. When the driver operates the travel operation device 7, the travel operation device 7 generates an operation signal. The operation signal of the travel operation device 7 is transmitted to the travel control device 9. The travel control device 9 outputs a control command for driving, braking, and turning the motor grader 1 based on the operation signal of the travel operation device 7. The control command output from the travel control device 9 for driving the motor grader 1 includes a control command for increasing the output of the engine 26. The control command output from the travel control device 9 for braking the motor grader 1 includes a control command for operating a brake device (not shown). The control command output from the travel control device 9 for turning the motor grader 1 includes a control command for operating the articulated cylinder 24. The motor grader 1 performs at least one of driving, braking, and turning based on the control command output from the travel control device 9.

[0029] <Perimeter monitoring system> The periphery monitoring system 10 monitors the periphery of the motor grader 1. The periphery monitoring system 10 includes a camera system 11, a radar system 12, a periphery monitoring monitor 13, and a monitor control device 14.

[0030] The camera system 11 has multiple cameras. The cameras have an optical system and an image sensor. Examples of the image sensor include a CCD (Couple Charged Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The cameras are provided on the vehicle body 2 of the motor grader 1. The cameras capture images of the periphery of the motor grader 1 to obtain images of the periphery of the motor grader 1. The cameras capture images of at least the periphery of the vehicle body 2. In the following description, images captured by the cameras of the camera system 11 will be referred to as camera images as appropriate.

[0031] The radar system 12 has a plurality of radars. The radars are provided on the vehicle body 2 of the motor grader 1. The radars detect obstacles around the motor grader 1 in a non-contact manner.

[0032] Fig. 3 is a top view showing a schematic diagram of the camera system 11 according to the embodiment. As shown in Figs. 1, 2, and 3, the camera system 11 has a first camera 111 provided on the front vehicle body 21, a second camera 112 provided on the front vehicle body 21, a third camera 113 provided on the rear vehicle body 22, a fourth camera 114 provided on the rear vehicle body 22, and a fifth camera 115 provided on the rear vehicle body 22. The first camera 111 and the second camera 112 capture an image in front of the front vehicle body 21. The third camera 113, the fourth camera 114, and the fifth camera 115 capture an image behind the rear vehicle body 22.

[0033] The first camera 111 is provided on the left side of the front end of the front vehicle body 21. The first camera 111 captures an image of the left front of the front vehicle body 21. An imaging range M1 of the first camera 111 is defined on the left front of the front vehicle body 21.

[0034] The second camera 112 is provided on the right side of the front end portion of the front vehicle body 21. The second camera 112 captures an image of the right front of the front vehicle body 21. An imaging range M2 of the second camera 112 is defined on the right front of the front vehicle body 21.

[0035] The third camera 113 is provided on the left side of the rear vehicle body 22. The third camera 113 images the left side of the rear vehicle body 22. An imaging range M3 of the third camera 113 is defined to the left side of the rear vehicle body 22.

[0036] The fourth camera 114 is provided on the right side of the rear vehicle body 22. The fourth camera 114 captures an image to the right of the rear vehicle body 22. An imaging range M4 of the fourth camera 114 is defined to the right of the rear vehicle body 22.

[0037] The fifth camera 115 is provided at the rear end of the rear vehicle body 22. The fifth camera 115 captures an image behind the rear vehicle body 22. An imaging range M5 of the fifth camera 115 is defined behind the rear vehicle body 22.

[0038] At least a portion of the imaging range M1 and the imaging range M2 overlap. At least a portion of the imaging range M3 and the imaging range M5 overlap. At least a portion of the imaging range M4 and the imaging range M5 overlap.

[0039] A non-imaging area NR1 that is not imaged by the camera system 11 is provided around the cab 3. The non-imaging area NR1 is defined on the left front side of the cab 3 and on the right front side of the cab 3, respectively.

[0040] 4 is a top view diagrammatically illustrating the radar system 12 according to the embodiment. As shown in FIG. 1, FIG. 2, and FIG. 4, the radar system 12 has a first radar 121 provided on the front vehicle body 21, a second radar 122 provided on the front vehicle body 21, a third radar 123 provided on the rear vehicle body 22, a fourth radar 124 provided on the rear vehicle body 22, and a fifth radar 125 provided on the rear vehicle body 22.

[0041] The first radar 121 is provided on the left side of the front end portion of the front vehicle body 21. The first radar 121 detects the left front of the front vehicle body 21. A detection range D1 of the first radar 121 is defined on the left front of the front vehicle body 21.

[0042] The second radar 122 is provided on the right side of the front end portion of the front vehicle body 21. The second radar 122 detects the right front of the front vehicle body 21. A detection range D2 of the second radar 122 is defined on the right front of the front vehicle body 21.

[0043] The third radar 123 is provided on the left side of the rear vehicle body 22. The third radar 123 detects the left side of the rear vehicle body 22. A detection range D3 of the third radar 123 is defined to the left side of the rear vehicle body 22.

[0044] The fourth radar 124 is provided on the right side of the rear vehicle body 22. The fourth radar 124 detects the right side of the rear vehicle body 22. A detection range D4 of the fourth radar 124 is defined to the right side of the rear vehicle body 22.

[0045] The fifth radar 125 is provided at the rear end of the rear vehicle body 22. The fifth radar 125 detects the rear of the rear vehicle body 22. A detection range D5 of the fifth radar 125 is defined behind the rear vehicle body 22.

[0046] At least a portion of the detection range D1 and the detection range D2 overlap. At least a portion of the detection range D3 and the detection range D5 overlap. At least a portion of the detection range D4 and the detection range D5 overlap.

[0047] A non-detection range NR2 that is not detected by the radar system 12 is provided around the cab 3. The non-detection range NR2 is defined on the left front side of the cab 3 and on the right front side of the cab 3, respectively.

[0048] 5 is a functional block diagram showing the motor grader 1 according to the embodiment. The periphery monitoring system 10 includes a camera system 11, a radar system 12, a periphery monitoring monitor 13, and a monitor control device 14.

[0049] The monitor control device 14 includes a computer system. The monitor control device 14 has a camera image acquisition unit 141, a first display image generation unit 142, a second display image generation unit 143, a boundary image generation unit 144, a symbol image generation unit 145, a detection data acquisition unit 146, and a display control unit 147. The monitor control device 14 may include a single computer system, or a plurality of computer systems may cooperate with each other to function as the monitor control device 14.

[0050] The camera image acquisition unit 141 acquires camera images from the camera system 11. The camera image acquisition unit 141 acquires a first camera image showing the situation to the left front of the front vehicle body 21 from the first camera 111. The camera image acquisition unit 141 acquires a second camera image showing the situation to the right front of the front vehicle body 21 from the second camera 112. The camera image acquisition unit 141 acquires a third camera image showing the situation to the left of the rear vehicle body 22 from the third camera 113. The camera image acquisition unit 141 acquires a fourth camera image showing the situation to the right of the rear vehicle body 22 from the fourth camera 114. The camera image acquisition unit 141 acquires a fifth camera image showing the situation behind the rear vehicle body 22 from the fifth camera 115.

[0051] The first display image generation unit 142 generates a first display image IM1 showing a first portion of the periphery of the motor grader 1 based on a first camera image of the periphery of the motor grader 1 captured by the first camera 111 and a second camera image of the periphery of the motor grader 1 captured by the second camera 112. In the embodiment, the first display image IM1 includes a front portion of the periphery of the motor grader 1. The first display image IM1 includes the periphery of the front vehicle body 21.

[0052] The first display image IM1 is a first viewpoint image seen from a first viewpoint different from the positions of the first camera 111 and the second camera 112. In the embodiment, the first display image generating unit 142 generates a panoramic image showing the periphery of the front vehicle body 21 based on the first camera image and the second camera image.

[0053] A panoramic image is an image generated by synthesizing a plurality of camera images acquired by a plurality of cameras with images in a plurality of viewing directions with a reference viewpoint related to the generation of the panoramic image as the viewing position. For example, a panoramic image of the entire circumference is an image in which images corresponding to a plurality of viewing directions in the range of 0° to +180° and the range of 0° to -180° centered on the reference viewpoint are continuously connected. In the case of a panoramic image of the entire circumference, an image of any viewing direction in the range of 360° is obtained. Note that the panoramic image does not have to correspond to the entire circumference. For example, an image corresponding to a viewing direction in the range of 0° to +90° and the range of 0° to -90° (a 180° panoramic image) may be used. In addition, the panoramic image may be an image in which images corresponding to a plurality of viewing directions obtained by rotating the viewing direction in the horizontal direction are consecutively obtained, or an image in which images corresponding to a plurality of viewing directions obtained by rotating the viewing direction in the vertical direction are consecutively obtained. In addition, the panoramic image may be an image in which images corresponding to a plurality of viewing directions obtained by changing the viewing direction three-dimensionally are consecutively obtained, like a celestial sphere image.

[0054] In the embodiment, the panoramic image is generated so as to connect the right edge of the first camera image and the left edge of the second camera image.

[0055] In the following description, the first display image IM1 will be referred to as the panoramic image IM1 where appropriate.

[0056] The second display image generation unit 143 generates a second display image IM2 showing a second portion of the periphery of the motor grader 1 based on a third camera image of the periphery of the motor grader 1 captured by the third camera 113, a fourth camera image of the periphery of the motor grader 1 captured by the fourth camera 114, and a fifth camera image of the periphery of the motor grader 1 captured by the fifth camera 115. In the embodiment, the second display image IM2 includes a rear portion of the periphery of the motor grader 1. The second display image IM2 includes the periphery of the rear vehicle body 22.

[0057] Second display image IM2 is a second viewpoint image seen from a second viewpoint different from the positions of third camera 113, fourth camera 114, and fifth camera 115. In the embodiment, second display image generating unit 143 generates an overhead image showing the periphery of rear vehicle body 22 based on the third camera image, the fourth camera image, and the fifth camera image.

[0058] The overhead image refers to an image generated by converting a plurality of camera images captured by a plurality of cameras into images from an upper viewpoint for generating the overhead image and synthesizing the images.

[0059] The second display image IM2 is an image in a display form different from that of the first display image IM1 For example, the second display image IM2 is an image viewed from a different viewpoint from that of the first display image IM1.

[0060] In the following description, the second display image IM2 will be referred to as the overhead image IM2 where appropriate.

[0061] The boundary image generating unit 144 generates a boundary image BI to be placed between the panoramic image IM1 and the overhead image IM2.

[0062] The boundary image BI refers to an image displayed at the boundary between the panoramic image IM1 and the overhead image IM2 in order to clearly distinguish between the panoramic image IM1 and the overhead image IM2.

[0063] The symbol image generating unit 145 generates a symbol image SI indicating the motor grader 1. In the embodiment, the symbol image generating unit 145 changes the symbol image SI based on an operation signal from the traveling operation device 7.

[0064] The symbol image SI refers to an image simulating the motor grader 1 as viewed from above.

[0065] The detection data acquisition unit 146 acquires radar detection data from the radar system 12 .

[0066] The display control unit 147 causes the display screen of the perimeter monitoring monitor 13 to display the panoramic image IM1, the overhead image IM2, the boundary image BI, and the symbol image SI.

[0067] <Periphery monitoring monitor> 6 is a diagram showing the periphery monitoring monitor 13 according to the embodiment. The periphery monitoring monitor 13 displays the periphery of the motor grader 1. The periphery monitoring monitor 13 displays at least the periphery of the vehicle body 2. As shown in FIG. 6, the periphery monitoring monitor 13 has a display unit 131.

[0068] The display control unit 147 displays the single camera image IS in the first area 133A of the display screen 133 of the display unit 131. The display control unit 147 displays the composite image CI in the second area 133B of the display screen 133 of the display unit 131. In the embodiment, the second area 133B is defined to the left of the first area 133A.

[0069] The single camera image IS refers to a camera image of a part of the surroundings of the motor grader 1 acquired by at least one of the multiple cameras of the camera system 11. The single camera image IS includes at least one of a first camera image acquired by the first camera 111 showing the situation to the left front of the motor grader 1, a second camera image acquired by the second camera 112 showing the situation to the right front of the motor grader 1, a third camera image acquired by the third camera 113 showing the situation to the left of the motor grader 1, a fourth camera image acquired by the fourth camera 114 showing the situation to the right of the motor grader 1, and a fifth camera image acquired by the fifth camera 115 showing the situation to the rear of the motor grader 1.

[0070] In the example shown in FIG. 6, the single camera image IS displayed in the first area 133A is a fifth camera image captured by the fifth camera 115 and showing the situation behind the motor grader 1.

[0071] A reference line GD1 is displayed in the first area 133A together with the single camera image IS. The reference line GD1 indicates a reference for the distance from the vehicle body 2. In the embodiment, the reference line GD1 indicates a distance from an outer edge of the vehicle body 2.

[0072] 7 is a diagram showing a composite image CI according to the embodiment. As shown in Fig. 6 and Fig. 7, the composite image CI includes a panoramic image IM1, an overhead image IM2, a boundary image BI, and a symbol image SI.

[0073] The display control unit 147 displays a panoramic image IM1 in a part of the composite image CI on the display screen 133. The display control unit 147 displays an overhead image IM2 in a part of the composite image CI on the display screen 133. The display control unit 147 displays a boundary image BI on the display screen 133 at the boundary between the panoramic image IM1 and the second display image IM2.

[0074] The display control unit 147 displays the symbol image SI in the central region of the composite image CI on the display screen 133. The display control unit 147 displays the panoramic image IM1, the overhead image IM2, and the boundary image BI around the symbol image SI.

[0075] The panoramic image IM1 is placed at the top of the composite image CI, the overhead image IM2 is placed at the bottom of the composite image CI, and the boundary image BI is displayed on the side of the symbol image SI.

[0076] In the embodiment, the border image BI is displayed on the side of the articulation mechanism 23S of the symbol image SI. The border image BI is arranged so as to extend in the left-right direction in the composite image CI. The border image BI is displayed in a strip shape on each of the left and right sides of the symbol image SI.

[0077] The boundary image BI may be displayed between the front wheels of the symbol image SI and the cab on the display screen 133. The boundary image BI may be displayed in a predetermined area below the panoramic image IM1 displayed in the composite image CI on the display screen 133. The boundary image BI may be displayed in a predetermined area above the overhead image IM2 displayed in the composite image CI on the display screen 133. The boundary image BI may be displayed in a predetermined area of ​​the composite image CI on the display screen 133. The predetermined area may be defined by an image coordinate system with the top left pixel of the composite image CI as the origin.

[0078] As described above, a non-detection range NR2 that is not captured by the camera system 11 is generated in a part of the periphery of the cab 3. For the non-detection range NR2, the panoramic image IM1 and the overhead image IM2 are not generated. That is, the area corresponding to the non-detection range NR2 on the display screen 133 is a non-display area where the panoramic image IM1 and the overhead image IM2 are not displayed. The boundary image BI is generated so as to cover the non-display area on the display screen 133 where the panoramic image IM1 and the overhead image IM2 are not displayed.

[0079] The symbol image SI clarifies the positional relationship between the motor grader 1 and its surroundings.

[0080] Along with the composite image CI, a reference line GD2 is displayed in the second area 133B. Like the reference line GD1, the reference line GD2 indicates a guide of the distance from the vehicle body 2. In the embodiment, the reference line GD2 is displayed so as to be superimposed on the overhead image IM2. The reference line GD2 is arranged around the symbol image SI in the overhead image IM2. Note that the reference line GD2 does not have to be arranged.

[0081] When an obstacle OB is detected by the radar system 12, the display control unit 147 may display a marker MK so as to overlap the obstacle OB displayed in the overhead image IM2 on the display screen 133. The marker MK functions as a symbol image that emphasizes the obstacle OB on the display screen 133.

[0082] An operation unit (not shown) of the periphery monitoring monitor 13 has a plurality of switches arranged in the cab 3. Specific functions are assigned to the plurality of switches. When the driver operates the switch, an operation signal for the specific function is generated.

[0083] Fig. 8 is a diagram showing a composite image CI according to the embodiment. In the embodiment, the symbol image generating unit 145 changes the symbol image SI based on an operation signal from the traveling operation device 7. As shown in Fig. 8, when the front vehicle body 21 is bent relative to the rear vehicle body 22 by the operation of the traveling operation device 7, the symbol image generating unit 145 generates the symbol image SI based on the operation signal from the traveling operation device 7 so that the front vehicle body 21S is also bent relative to the rear vehicle body 22S in the symbol image SI.

[0084] <Synthetic image generation method> 9 is a flowchart showing a synthetic image generating method according to the embodiment. When the motor grader 1 is turned on, the periphery monitoring system 10 is started up.

[0085] The camera of the camera system 11 captures images of the periphery of the motor grader 1. The camera image acquisition unit 141 acquires camera images from the camera (step SP1).

[0086] The first display image generation unit 142 generates a panoramic image IM1 showing the front part of the surroundings of the motor grader 1 based on the first camera image of the surroundings of the motor grader 1 captured by the first camera 111 and the second camera image of the surroundings of the motor grader 1 captured by the second camera 112 (step SP2).

[0087] The second display image generation unit 143 generates an overhead image IM2 showing the rear part of the surroundings of the motor grader 1 based on the third camera image of the surroundings of the motor grader 1 captured by the third camera 113, the fourth camera image of the surroundings of the motor grader 1 captured by the fourth camera 114, and the fifth camera image of the surroundings of the motor grader 1 captured by the fifth camera 115 (step SP3).

[0088] The boundary image generating unit 144 generates a boundary image BI. The symbol image generating unit 145 generates a symbol image SI based on an operation signal from the driving operation device 7.

[0089] The display control unit 147 causes the symbol image SI to be displayed on the display screen 133 of the display unit 131. Furthermore, the display control unit 147 causes the panoramic image IM1, the overhead image IM2, and the boundary image BI to be displayed around the symbol image SI (step SP4).

[0090] <Computer System> FIG. 10 is a block diagram showing a computer system 1000 according to an embodiment. Each of the above-mentioned monitor control device 14, the work machine control device 8, and the travel control device 9 includes a computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 1003, and an interface 1004 including an input / output circuit. For example, the function of the monitor control device 14 is stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003, expands it in the main memory 1002, and executes the above-mentioned processing according to the computer program. The computer program may be distributed to the computer system 1000 via a network.

[0091] In accordance with the above-described embodiment, the computer program or computer system 1000 can perform the following operations: generate a first display image IM1 showing the surroundings of the motor grader 1 based on a camera image of a camera that captures the surroundings of the motor grader 1; generate a second display image IM2 having a display form different from the first display image IM1 based on camera images of multiple cameras that capture the surroundings of the motor grader 1; display the first display image IM1 on a part of the display screen 133, display the second display image IM2 on a part of the display screen 133, and generate a signal for displaying a boundary image BI at the boundary between the first display image IM1 and the second display image IM2 on the display screen 133.

[0092] <Effects> As described above, according to the embodiment, in a case where a panoramic image IM1 is displayed in a partial area of ​​the display screen 133 and an overhead image IM2 is displayed in another partial area of ​​the display screen 133, a boundary image BI is displayed at the boundary between the panoramic image IM1 and the overhead image IM2. The boundary image BI makes the boundary between the panoramic image IM1 and the overhead image IM2 clear, thereby suppressing a decrease in the visibility of each of the panoramic image IM1 and the overhead image IM2.

[0093] The panoramic image IM1 is a first viewpoint image seen from a reference viewpoint different from the positions of the first camera 111 and the second camera 112. The overhead image IM2 is a second viewpoint image seen from an upper viewpoint different from the positions of the third camera 113, the fourth camera 114, and the fifth camera 115. When the panoramic image IM1 and the overhead image IM2, which has a display form different from that of the panoramic image IM1, are displayed in parallel, the driver may find it difficult to visually recognize each of the panoramic image IM1 and the overhead image IM2. According to the embodiment, the boundary image BI makes the boundary between the panoramic image IM1 and the overhead image IM2 clear, so that a decrease in the visibility of each of the panoramic image IM1 and the overhead image IM2 is suppressed.

[0094] A symbol image SI indicating the motor grader 1 is displayed on the display screen 133. The panoramic image IM1, the overhead image IM2, and the boundary image BI are each displayed around the symbol image SI. This allows the driver to recognize the relative positions of the motor grader 1, a first portion of the periphery of the motor grader 1 indicated by the panoramic image IM1, a second portion of the periphery of the motor grader 1 indicated by the overhead image IM2, and the boundary image BI.

[0095] A first portion of the surroundings of the motor grader 1 shown by the panoramic image IM1 shows a front portion of the surroundings of the motor grader 1. A second portion of the surroundings of the motor grader 1 shown by the overhead image IM2 shows a rear portion of the surroundings of the motor grader 1. By displaying the boundary image BI to the side of the symbol image SI, the driver can recognize the relative positions of the front portion of the surroundings of the motor grader 1 and the rear portion of the surroundings of the motor grader 1, and can clearly recognize the boundary between the panoramic image IM1 and the overhead image IM2.

[0096] The motor grader 1 has a front vehicle body 21, a rear vehicle body 22, and an articulation mechanism 23 that connects the front vehicle body 21 and the rear vehicle body 22. The first camera 111 and the second camera 112 are each provided on the front vehicle body 21. The third camera 113, the fourth camera 114, and the fifth camera 115 are each provided on the rear vehicle body 22. This allows the periphery monitoring system 10 to monitor the peripheries of each of the front vehicle body 21 and the rear vehicle body 22.

[0097] The first portion of the periphery of the motor grader 1 shown in the panoramic image IM1 shows the periphery of the front body 21. As a result, for example, when the motor grader 1 enters an intersection with poor visibility, the driver can properly recognize the condition of the intersection and the roads that make up the intersection based on the panoramic image IM1.

[0098] A second portion of the periphery of the motor grader 1 shown in the overhead image IM2 shows the periphery of the rear vehicle body 22. This allows the driver to properly recognize the state of obstacles around the rear vehicle body 22 based on the overhead image IM2.

[0099] The boundary image BI is displayed on the side of the articulating mechanism 23S of the symbol image SI. This allows the driver to recognize the relative positions of the periphery of the front body 21 and the periphery of the rear body 22, and to clearly recognize the boundary between the panoramic image IM1 and the overhead image IM2.

[0100] The boundary image BI is generated so as to cover a non-display area where the panoramic image IM1 and the overhead image IM2 are not displayed on the display screen 133. Since the non-display area corresponding to the non-imaging range NR1 where a camera image cannot be obtained is covered with the boundary image BI, the driver can clearly recognize the boundary between the panoramic image IM1 and the overhead image IM2.

[0101] [Second embodiment] The perimeter monitoring system 10 according to the first embodiment generates a boundary image BI so as to cover a non-display area corresponding to the non-image capture area NR1. In contrast, the perimeter monitoring system 10 according to the second embodiment displays the boundary image BI at the boundary between the panoramic image IM1 and the overhead image IM2 when the non-image capture area NR1 does not occur.

[0102] As in the first embodiment, the periphery monitoring system 10 according to the second embodiment has a first camera 111 that acquires a first camera image for generating a first display image IM1 and a second camera 112 that acquires a second camera image. The periphery monitoring system 10 according to the second embodiment has a plurality of cameras that capture other imaging ranges around the motor grader 1 that are different from the imaging range M1 of the first camera 111 and the imaging range M2 of the second camera 112. That is, the plurality of cameras are arranged so as to capture a portion different from the portion of the periphery of the motor grader 1 shown by the panoramic image IM1. Note that the plurality of cameras may be arranged so as to capture the entire 360° periphery of the motor grader 1.

[0103] A second display image generating unit 143 according to the second embodiment uses camera images from a plurality of cameras to generate an overhead image showing a part different from the part around the motor grader 1 shown in the panoramic image IM1.

[0104] The display control unit 147 according to the second embodiment displays a symbol image SI on the display screen 133 of the display unit 131. The display control unit 147 also displays a panoramic image IM1, an overhead image IM2, and a boundary image BI around the symbol image SI. The panoramic image IM1 is disposed at the top of the composite image CI. The overhead image IM2 is disposed at the bottom of the panoramic image IM1. The boundary image BI is displayed on the side of the symbol image SI so as to cover a portion of the overhead image IM2. Note that the boundary image BI may be disposed so as to cover a portion of the panoramic image IM1, or may be disposed so as to cover a portion of the panoramic image IM1 and a portion of the overhead image IM2.

[0105] <Effects> The surroundings monitoring system 10 of the second embodiment displays a boundary image BI so as to cover a portion of the overhead image IM2, thereby making it possible to clearly recognize the boundary between the panoramic image IM1 and the overhead image IM2 when no non-image capture range NR1 occurs.

[0106] [Other embodiments] Fig. 11 is a diagram showing a composite image CI according to the embodiment. In the above-described embodiment, the boundary image BI is strip-shaped. As shown in Fig. 11, the boundary image BI may be line-shaped. Also, the boundary image BI may be an image filled with a predetermined color such as black.

[0107] Fig. 12 is a diagram showing a composite image CI according to an embodiment. As shown in Fig. 12, a panoramic image IM1 may be surrounded by a frame image FI1. The frame image FI1 is displayed so as to follow the periphery of the panoramic image IM1. A part of the frame image FI1 may function as a boundary image BI.

[0108] Fig. 13 is a diagram showing a composite image CI according to an embodiment. As shown in Fig. 13, a panoramic image IM1 may be surrounded by a frame image FI1, and an overhead image IM2 may be surrounded by a frame image FI2. The frame image FI2 is displayed so as to follow the peripheral portion of the overhead image IM2. The color of the frame image FI1 is different from the color of the frame image FI2. A part of the frame image FI1 and a part of the frame image FI2 may function as a boundary image BI.

[0109] In the above-described embodiment, the boundary image BI may be curved, triangular, or polygonal, such as a pentagon.

[0110] In the above-described embodiment, a part of the overhead image IM2 displayed to the side of the symbol image SI may function as the boundary image BI. A part of the overhead image IM2 displayed to the side of the symbol image SI may be displayed semi-transparently, and the semi-transparent display part may function as the boundary image BI.

[0111] In the above-described embodiment, the boundary image generating unit 144 generates the boundary image BI. In other embodiments, the boundary image generating unit 144 may not generate the boundary image BI. For example, the boundary image BI may be stored in advance in the display control unit 147, and the display control unit 147 may display the pre-stored boundary image BI.

[0112] In the above-described embodiment, the perimeter monitoring system 10 may not include the perimeter monitoring monitor 13 and the radar system 12. For example, the perimeter monitoring system 10 may output a signal to be displayed on a monitor. Also, instead of the radar system 12, an obstacle may be detected by image analysis of the camera image of the camera system 11. Also, an obstacle may not be detected. Also, some components of the perimeter monitoring system 10 may be mounted inside the motor grader 1, and other components may be provided outside the motor grader 1. For example, the perimeter monitoring monitor 13 of the perimeter monitoring system 10 may be disposed in a remote control room provided in a remote location of the motor grader 1.

[0113] In the above-described embodiment, the display control unit 147 may generate signals for displaying on the display screen 133 the panoramic image IM1, the overhead image IM2, the boundary image BI, and the symbol image SI.

[0114] In the above embodiment, the first camera 111 and the second camera 112 are provided on the front vehicle body 21. The number of cameras provided on the front vehicle body 21 may be any number of three or more.

[0115] In the above-described embodiment, the installation positions of the cameras (111, 112, 113, 114, 115) and radars (121, 122, 123, 124, 125) are not limited to the installation positions described in the above-described embodiment, and may be any installation positions.

[0116] In the above-described embodiment, the imaging range M1 and the imaging range M2 do not have to overlap, the imaging range M3 and the imaging range M5 do not have to overlap, and the imaging range M4 and the imaging range M5 do not have to overlap.

[0117] In the above embodiment, the third camera 113, the fourth camera 114, and the fifth camera 115 are provided on the rear vehicle body 22. The number of cameras provided on the front vehicle body 21 may be two, or any number of cameras equal to or greater than four.

[0118] In the above-described embodiment, the camera system 11 may be configured with one camera having a wide viewing angle.

[0119] In the above-described embodiment, the first display image IM1 may be an overhead image and the second display image IM2 may be a panoramic image. Both the first display image IM1 and the second display image IM2 may be panoramic images. Both the first display image IM1 and the second display image IM2 may be overhead images.

[0120] In the above-described embodiment, one or both of the first display image IM1 and the second display image IM2 may be a single camera image.

[0121] In the above-described embodiment, the work machine 1 may be an articulated dump truck having an articulated mechanism. The work machine 1 may also be a wheel loader having an articulated mechanism and a work implement. The work machine 1 may also be a hydraulic excavator or bulldozer not having an articulated mechanism, or may also be a rigid dump truck. [Explanation of symbols]

[0122] 1...motor grader (working machine), 2...body, 3...cab, 4...working machine, 5...wheel, 6...working machine operating device, 7...traveling operating device, 8...working machine control device, 9...traveling control device, 10...periphery monitoring system, 11...camera system, 12...radar system, 13...periphery monitoring monitor, 14...monitor control device, 21...front body, 22...rear body, 23...articulating mechanism, 24...articulating cylinder, 25...exterior cover, 26...engine, 31...seat, 40...drawbar, 41...turning circle, 42...blade, 44...lift cylinder, 45...lift cylinder, 51...front wheel, 52...rear wheel, 111...first camera, 112...second camera, 113...third camera, 114...fourth camera, 115...fifth camera, 121...first radar, 122...second radar, 123...second 3 radar, 124...4th radar, 125...5th radar, 131...display unit, 133...display screen, 141...camera image acquisition unit, 142...1st display image generation unit, 143...2nd display image generation unit, 144...boundary image generation unit, 145...symbol image generation unit, 146...detection data acquisition unit, 147...display control unit, BI...boundary image, CI...synthetic image, D1...detection range, D2...detection range, D3...detection range , D4...detection range, D5...detection range, FI1...frame image, FI2...frame image, GD1...guideline, GD2...guideline, IM1...first displayed image (panoramic image), IM2...second displayed image (bird's-eye view image), IS...single camera image, M1...imaging range, M2...imaging range, M3...imaging range, M4...imaging range, M5...imaging range, NR1...non-imaging range, NR2...non-detection range, SI...symbol image.

Claims

1. A camera image acquisition unit that acquires camera images from a plurality of cameras for imaging the periphery of a work machine; a first display image generating unit that generates a first display image including a front portion of the periphery of the work machine based on the camera image; a second display image generating unit that generates a second display image different from the first display image, the second display image including a rear portion of the periphery of the work machine, based on the camera image; a display control unit that generates a signal for displaying a symbol image indicating the work machine on a display screen, displaying the first display image and the second display image around the symbol image, and displaying a boundary image disposed between the first display image and the second display image. Periphery monitoring system for work machines.

2. The display form different from the first display image is an image viewed from a different viewpoint from the first display image. The periphery monitoring system for a work machine according to claim 1.

3. the first display image is a panoramic image; The second display image is an overhead image. The surroundings monitoring system for a work machine according to claim 1 or 2.

4. the cameras include a first camera that acquires a first camera image for generating the first display image, a second camera that acquires a second camera image, a third camera that acquires a third camera image for generating the second display image, and a fourth camera that acquires a fourth camera image, The work machine has a front vehicle body, a rear vehicle body, and an articulation mechanism that connects the front vehicle body and the rear vehicle body, the first camera and the second camera are provided on the front vehicle body, The third camera and the fourth camera are provided on the rear vehicle body. The periphery monitoring system for a work machine according to claim 1.

5. the first display image includes the periphery of the front vehicle body, the second display image includes the periphery of the rear vehicle body, the border image is displayed on a side of the articulation mechanism of the symbol image; The surroundings monitoring system for a work machine according to claim 4.

6. the first display image is a first viewpoint image seen from a first viewpoint different from the position of the first camera and the position of the second camera, the second display image is a second viewpoint image seen from a second viewpoint different from the position of the third camera and the position of the fourth camera; The surroundings monitoring system for a work machine according to claim 4 or 5.

7. the boundary image is generated so as to cover a non-display area on the display screen where the first display image and the second display image are not displayed. The surroundings monitoring system for a work machine according to any one of claims 1 to 6.

8. A work machine, A work machine for performing work at a work site; A vehicle body supporting the working machine; A plurality of cameras for capturing images of the periphery of the vehicle body; a monitor for displaying an image captured by the camera; a processor; The processor, Acquiring a camera image from the camera; generating a first display image including a front portion of the periphery of the vehicle body based on the acquired camera image; generating a second display image different from the first display image, the second display image including a rear portion of the periphery of the vehicle body, based on the acquired camera image; displaying a symbol image indicating the work machine on the monitor, displaying the first display image and the second display image around the symbol image, and displaying a boundary image disposed between the first display image and the second display image; Working machinery.

9. The vehicle body is supported by front and rear wheels. The vehicle body includes a front vehicle body and a rear vehicle body, The work machine has a blade disposed between the front wheels and the rear wheels, The front wheels are disposed forward of the blade, and the rear wheels are disposed rearward of the blade.

9. A work machine according to claim 8.

10. At least one of the plurality of cameras is provided on the front vehicle body and captures an image in front of the front vehicle body, At least one of the plurality of cameras is provided on the rear vehicle body and captures an image behind the rear vehicle body.

10. A work machine according to claim 9.

11. the cameras include a first camera and a second camera provided on the front vehicle body for capturing an image of a front area of ​​the front vehicle body, and a third camera and a fourth camera provided on the rear vehicle body for capturing an image of a rear area of ​​the rear vehicle body, The processor generates the first display image based on camera images captured by the first camera and the second camera, and generates the second display image based on camera images captured by the third camera and the fourth camera.

11. A work machine according to claim 10.

12. Acquiring camera images from a plurality of cameras for imaging an area around a work machine; generating a first display image including a front portion of a periphery of the work machine based on the captured camera image; generating a second display image different from the first display image based on the acquired camera image and including a rear portion of the surroundings of the work machine; generating a signal for displaying a symbol image indicating the work machine on a display screen, displaying the first display image and the second display image around the symbol image, and displaying a boundary image disposed between the first display image and the second display image. A method for monitoring the periphery of a work machine.

13. The display form different from the first display image is an image viewed from a different viewpoint from the first display image. The method for monitoring the periphery of a work machine according to claim 12.

14. the first display image is a panoramic image; The second display image is an overhead image. A method for monitoring the periphery of a work machine according to claim 12 or 13.

15. the cameras include a first camera that acquires a first camera image for generating the first display image, a second camera that acquires a second camera image, a third camera that acquires a third camera image for generating the second display image, and a fourth camera that acquires a fourth camera image, The work machine has a front vehicle body, a rear vehicle body, and an articulation mechanism that connects the front vehicle body and the rear vehicle body, the first camera and the second camera are provided on the front vehicle body, The third camera and the fourth camera are provided on the rear vehicle body. The method for monitoring the periphery of a work machine according to claim 12.

16. the first display image includes the periphery of the front vehicle body, the second display image includes the periphery of the rear vehicle body, the border image is displayed on a side of the articulation mechanism of the symbol image; The method for monitoring the periphery of a work machine according to claim 15.

17. the boundary image is generated so as to cover a non-display area on the display screen where the first display image and the second display image are not displayed. The method for monitoring the periphery of a work machine according to any one of claims 12 to 16.

Images