Display control device and remote control device

Abstract

To provide an appropriate graphic display in a series of work machine movements.SOLUTION: This display control device comprises a three-dimensional information acquisition unit that acquires topographic information indicating a three-dimensional shape of a topography of a work target area, a bucket position calculation unit that acquires posture information regarding the posture of a work machine and calculates the position of the bucket within the work target area based on the posture information, an AR image generation unit that generates an AR image to be displayed on the display superimposed on the image of the work target area based on the topographical information and the position of the bucket. The AR image generation unit generates a first figure that highlights and shows the position of at least a portion of the bucket as the AR image, and a second figure that highlights a part of a second area adjacent to the first area, with an area including a portion of the first figure projected in the vertical direction or front-back direction of the work target area as the first area.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a display control device and a remote control device. [Background technology]

[0002] In a work machine control system that enables remote operation of a work machine, a display control device is proposed in Patent Document 1, for example, that displays an image of at least a portion of the work machine (such as a shovel) on a display and superimposes on this image a graphic display of a figure (AR image) or the like that assists the operator in operating the work machine. Such a graphic display is useful for grasping the sense of distance between the work machine and the ground, etc.

[0003] With conventional display control devices, when an image and a graphic are displayed overlapping each other, the image may be obscured by the graphic, which may in turn hinder the user's ability to grasp the sense of distance. Furthermore, the work of a work machine includes not only excavation operations but also a series of operations such as turning and moving, but conventional display control devices do not provide appropriate work support for such a series of operations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-160741 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a display control device and a remote control device that are capable of providing appropriate graphic displays for work support during a series of operations of a work machine. [Means for solving the problem]

[0006] In order to solve the above problems, a display control device according to the present invention is a display control device that acquires an image of a work area for a work machine equipped with a bucket and displays it on a display, and includes: a three-dimensional information acquisition unit that acquires terrain information indicating the three-dimensional shape of the terrain of the work area; a bucket position calculation unit that acquires attitude information related to the attitude of the work machine and calculates the position of the bucket within the work area based on the attitude information; and an AR image generation unit that generates an AR image to be displayed on the display, based on the terrain information and the bucket position. The AR image generation unit generates, as the AR image, a first graphic that highlights the position of at least a portion of the bucket, and a second graphic that highlights a portion of a second area adjacent to a first area that includes a portion of the first graphic projected vertically or longitudinally onto the work area. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an appropriate graphic display for work support during a series of operations of a work machine, thereby improving the work efficiency of the work machine. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating the configuration of a display control device according to a first embodiment and a working machine 100 to which the display control device is applied. [Figure 2] 2 is a schematic diagram illustrating the configuration of sensors S1 to S3 mounted on the work machine 100. FIG. [Figure 3] 2 is a schematic diagram illustrating an example of the detailed configuration of a remote control unit 300. FIG. [Figure 4] 2 is a block diagram illustrating an example of the detailed configuration of an image generating unit 103 mounted on the work machine 100. FIG. [Figure 5] 5 is a block diagram illustrating the detailed configuration of a remote control unit 300 corresponding to the video generation unit 103 in FIG. 4. FIG. [Figure 6]10 is a block diagram illustrating another example of the detailed configuration of the image generating unit 103 mounted on the work machine 100. FIG. [Figure 7] FIG. 7 is a block diagram illustrating the detailed configuration of a remote control unit 300 corresponding to the video generation unit 103 in FIG. 6. [Figure 8] 6 is a flowchart illustrating a procedure for generating a superimposed image including an AR display when the image generating unit 103 and the driving / display control device of FIGS. 4 and 5 are employed. [Figure 9] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 10] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 11] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 12] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 13] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 14] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 15] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 16] An example of an AR image displayed in a superimposed image in the first embodiment will be described. [Figure 17] FIG. 10 is a block diagram illustrating an example of the detailed configuration of a video generation unit 103 of a display control device according to a second embodiment. [Figure 18] FIG. 10 is a schematic diagram illustrating the operation of the second embodiment. [Figure 19] FIG. 10 is a schematic diagram illustrating the operation of the second embodiment. [Figure 20] FIG. 10 is a schematic diagram illustrating the operation of the second embodiment. [Figure 21]10 is a flowchart illustrating a procedure for generating a superimposed image including an AR image in the second embodiment. [Figure 22] FIG. 10 is a block diagram illustrating an example of the detailed configuration of a video generation unit 103 of a display control device according to a third embodiment. [Figure 23] An example of an AR image displayed in a superimposed image in a display control device according to the third embodiment will be described. [Figure 24] An example of an AR image displayed in a superimposed image in a display control device according to the third embodiment will be described. [Figure 25] FIG. 10 is a schematic diagram illustrating a modified example. [Figure 26] FIG. 10 is a schematic diagram illustrating a modified example. [Figure 27] FIG. 10 is a schematic diagram illustrating a modified example. [Figure 28] 1A and 1B are conceptual diagrams for explaining switching of the display state of an AR image in a display control device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0010] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to implement the present disclosure, it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0011] [First embodiment] 1 and 2, the configuration of a display control device according to a first embodiment and a work machine (construction machine) 100 to which the display control device is applied will be described. As an example, this work machine 100 is a backhoe-type shovel in which a bucket 113 faces an operation room OR, and the operation room OR is configured to be rotatable relative to the main body, so that the bucket 113 can be moved to a work area. The operation of the work machine 100 excavates the ground L, which is the work area, to form, for example, a plane L0, a plane L1, a slope L2, etc. The soil excavated by the work machine 100 is discharged (discharged) into a vessel 200, such as a dump truck, as the work machine 100 rotates and the bucket 113 moves to the vessel 200.

[0012] As an example, work machine 100, which is a backhoe-type shovel, has a lower traveling body 101, an upper rotating body 102 supported and transported by lower traveling body 101, an operation room OR mounted on upper rotating body 102, an image generation unit 103, a communication device 104, various sensors S1 to S4, an antenna 107, a boom 111, an arm 112, and a bucket 113. The boom 111, arm 112, and bucket 113 are driven by an engine, a hydraulic cylinder, etc. (not shown), thereby performing excavation in the ground L, which is the work area.

[0013] Furthermore, the lower traveling body 101 is driven by a drive unit (not shown), which allows the upper rotating body 102 to move forward, backward, left and right. The upper rotating body 102 is configured to be able to rotate relative to the lower traveling body 101, and is driven by a drive unit (not shown), which controls the direction and angle of rotation. The work machine 100 is configured to be unmanned and remotely controlled by the remote control unit 300, and there is no need for an operator to board the control room OR.

[0014] The upper rotating body 102 is equipped with an image generation unit 103 and a communication device 104. The image generation unit 103 generates images of the ground, which is the work area, and the bucket 113 using various sensors S1 to S4 including image pickup elements. The image generation unit 103 also generates an AR image that illustrates the current position of the bucket 113 and the position of the ground to which the bucket 113 is about to move, and generates a superimposed image by superimposing this on the actual image. The communication device 104 is configured to transmit the acquired image data to the remote control unit 300 and to receive control information from the remote control unit 300.

[0015] Sensors S1 to S4 are disposed on the upper rotating body 102 to detect the conditions around the work machine 100. As shown in FIG. 2, as an example, sensor S2 can be configured as an imaging device having a CMOS sensor or the like that captures an image of a field of view SR2 ahead of the work machine 100, or an object detection sensor such as a LiDAR or ultrasonic sensor that detects the presence and distance of an object. Similarly, sensors S1 and S3 detect fields of view SR1 and SR3 to the sides of the work machine 100. The fields of view are not limited to these three directions, and full periphery detection may be performed by adjusting the installation position, angle, and number of sensors according to the angle of view. Sensor S4 may be an attitude sensor that detects the attitude of the work machine 100. Attitude sensor S4 can be configured, for example, as a gyro sensor, tilt sensor, or acceleration sensor, or a combination of these. The imaging elements that make up sensors S1 to S3 may be configured as a stereo camera capable of stereoscopic vision, or as a monocular camera. Furthermore, the example shown in FIG. 2 is merely an example, and the present disclosure is not limited to this.

[0016] An example of the detailed configuration of the remote control unit 300 will be described with reference to Fig. 3. The remote control unit 300 includes, for example, a driver's seat 301, an operation control unit 302, an operation rod 303, a display 304, a driving / display control device 305, and a communication device 306.

[0017] The driver's seat 301 is a chair on which the operator sits, and an operation control unit 302 and an operation stalk 303 are provided on the side of the seat. In addition to the operation stalk 303, various levers, switches, pedals, etc. may be arranged, and the operation of these may be reflected in the operation of the work machine 100. When the operation stalk 303, etc. is operated, a corresponding operation signal is generated in the operation control unit 302 and transmitted to the operation / display control device 305. The operation / display control device 305 generates a control signal for driving the work machine 100 in accordance with the received operation signal, and transmits the control signal to the work machine 100 via the communication device 306. The operation / display control device 305 also functions as a display control device that controls the display of a superimposed image containing an AR image on the display 304, as will be described later.

[0018] Display 304 is a display device disposed in front of driver's seat 301, and displays bucket 113, the ground surface of the work area, and an AR image (described later). As shown in Fig. 3, display 304 can be configured to include multiple display areas (display screens) with appropriate depression angles so as to realistically represent a three-dimensional space. Display 304 is not limited to this, and may be configured with only one display area, or may be a head-mounted display.

[0019] An example of the detailed configuration of the image generation unit 103 mounted on the work machine 100 will be described with reference to Fig. 4. The image generation unit 103 in Fig. 4 includes, as an example, a coordinate system integration processing unit 1031, a bucket position calculation unit 1032, a corresponding ground position calculation unit 1033, an AR image generation unit 1034 (support information generation unit), an image superimposition unit 1035, and an image compression unit 1036. The image generation unit 103 may be a general-purpose computer and may be composed of, for example, a CPU 151 (processor), ROM 152, RAM 153, a hard disk drive 154, an input device 155, and a display 156 (display unit). The CPU 151 is one aspect of a processor, and may be a GPU or other semiconductor device capable of arithmetic processing, or a combination thereof.

[0020] The coordinate system integration processing unit 1031 executes a process (sensor fusion) that integrates the coordinate systems of detection signals obtained by multiple sensors S1 to S4 (as an example, sensors S1 to S3 are three-dimensional information acquisition devices such as LiDAR and image capture devices (CMOS sensors, etc.), and sensor S4 is an attitude sensor) into a single video signal. Specifically, correction information for the coordinate systems of each sensor is calculated according to the results of the integration process, and this correction information is added to or applied to the detection signals of each sensor. In other words, the coordinate system integration processing unit 1031 functions as a shape information acquisition unit that acquires shape information that indicates the three-dimensional shape of the ground, which is the work target area of ​​the work machine 100.

[0021] The bucket position calculation unit 1032 acquires information (attitude information) relating to the attitude of the work machine 100 from the attitude sensor S4, and calculates the position of the bucket 113 within the work target area based on the attitude information. In this embodiment, the bucket position calculation unit 1032 calculates the position of the bucket 113 based on data indicating the results of the integration processing output from the coordinate system integration processing unit 1031 and a detection signal from the attitude sensor S4.

[0022] The corresponding ground surface position calculation unit 1033 calculates the position of the ground surface that the bucket 113 is about to excavate (corresponding ground surface) based on data indicating the results of the integration processing output from the coordinate system integration processing unit 1031, data calculated by the bucket position calculation unit 1032, and position information obtained from sensors S1 to S3 serving as three-dimensional information acquisition devices. In this specification, the "corresponding ground surface" refers to the area that the bucket 113 is expected to actually contact and excavate. Taking into account excavation errors, it is also possible to define the corresponding ground surface as an area slightly larger than the area of ​​the ground surface that is expected to actually be excavated. The corresponding ground surface position can be calculated, for example, from the average height of the corresponding ground surface in shape information that indicates the three-dimensional shape of the ground surface that exists vertically downward from the position of the bucket 113.

[0023] The AR image generation unit 1034 generates an AR image showing the positions of the bucket position data and the corresponding ground position data obtained. The AR image is a highlight image of a part of an actual image of the work machine 100, work target (ground, etc.), etc., that is displayed superimposed on the image to support work by the work machine 100. The image superimposition unit 1035 generates a superimposed image by superimposing the AR image on the image data obtained by the sensors (imaging devices) S1 to S3. The image compression unit 1036 compresses the generated superimposed image using a predetermined image compression method and outputs it to the communication device 104. Well-known methods such as MPEG, H.264, and MotionJPEG can be used as the image compression method, but the method is not limited to these.

[0024] The configuration of the remote control unit 300 will be described in detail with reference to Fig. 5. The superimposed image data described above is compressed by the communication device 104 in the work machine 100, and is received by the communication device 306 in the remote control unit 300 via wireless communication via a wireless base station (not shown), and then received by the operation / display control device 305. As shown in Fig. 5, the operation / display control device 305 is equipped with, for example, a video decompression unit 3051, an information addition / display switching unit 3052, and a display control unit 3053 as a configuration for controlling the display of the superimposed image containing the AR image. The operation / display control device 305 also has a function of transmitting control signals from the operation control unit 302 from the communication device 306, thereby controlling various operations of the work machine 100, but details of this will be omitted. The operation / display control device 305 can be realized by a general-purpose computer equipped with a CPU or GPU, various memories, etc., and computer programs for executing the various operations described below.

[0025] The compressed superimposed image data received by the communication device 306 is first input to the video decompression unit 3051 and decompressed. The decompressed superimposed image data is input to the information addition / display switching unit 3052. The information addition / display switching unit 3052 has the function of adding an additional AR image to the currently displayed AR image, or switching to an AR image other than the currently superimposed AR image, in accordance with input from the input device (operation console) 303 or the like. The information addition / display switching unit 3052 adds or changes the AR image to be displayed, depending on the state of the work machine 100 based on operation by the remote control unit 300. The display control unit 3053 executes display control when displaying the superimposed image generated in this way on the display 304.

[0026] Another example of the detailed configuration of the image generation unit 103 mounted on the work machine 100 will be described with reference to Fig. 6. The image generation unit 103 in this example is configured to transmit image information acquired by the imaging devices S1 to S3 and three-dimensional information acquired by a three-dimensional information acquisition device (such as LiDAR) separately to the remote control unit 300 via the communication device 104.

[0027] Specifically, the video generation unit 103 in Figure 6, as an example, includes a coordinate system integration processing unit 1031, a bucket position calculation unit 1032, and a video compression unit 1036 similar to those in the first example of Figure 4, as well as coordinate system conversion units 1037 and 1038 and a three-dimensional information compression unit 1039.

[0028] Based on the integration processing result output from the coordinate system integration processing unit 1031, the coordinate system conversion unit 1037 converts the coordinate system of the detection data of the orientation sensor S4, and outputs the detection data of the orientation sensor S4 after the coordinate system conversion. Similarly, based on the integration processing result output from the coordinate system integration processing unit 1031, the coordinate system conversion unit 1038 converts the coordinate system of the detection data from the sensors S1 to S3 serving as three-dimensional information acquisition devices, and outputs the detection data of the sensors S1 to S3 after the coordinate conversion. The output data is compressed by a three-dimensional information compression unit 1039 using a predetermined image compression method.

[0029] 6, the compressed video signals obtained based on the sensors S1 to S3 (imaging devices), the compressed three-dimensional information obtained based on the sensors S1 to S3 (three-dimensional information acquisition devices), and the detection signal after coordinate system transformation obtained from the attitude sensor S4 are separately transmitted via the communication device 104 to the remote control unit 300 shown in Fig. 7. Based on these pieces of data, the remote control unit 300 shown in Fig. 7 calculates the position of the bucket 113 and the position of the corresponding ground surface.

[0030] 7, an example of the configuration of the remote control unit 300 including the operation / display control device 305 corresponding to the image generation unit 103 of FIG. 6 will be described. The operation / display control device 305 of FIG. 7 includes a three-dimensional information decompression unit 3051′, a bucket position calculation unit 1032′, a corresponding ground position calculation unit 1033′, and an AR image generation unit 1034′, in addition to the image decompression unit 3051 and information addition / display switching unit 3052 similar to those of FIG. 5. The bucket position calculation unit 1032′, the corresponding ground position calculation unit 1033′, and the AR image generation unit 1034′ have substantially the same functions as the bucket position calculation unit 1032, the corresponding ground position calculation unit 1033, and the AR image generation unit 1034 shown in FIG. 4. However, the former is provided in the operation / display control device 305, while the latter is provided in the image generation unit 103.

[0031] The video decompression unit 3051 has a function of decompressing compressed video signals received via the communication devices 104 and 306. On the other hand, the three-dimensional information decompression unit 3051′ has a function of decompressing compressed three-dimensional information received via the communication devices 104 and 306. The bucket position calculation unit 1032′ acquires bucket position data indicating the position of the bucket 113 in accordance with the decompressed three-dimensional information data and the detection signal of the attitude sensor S4. Because the decompressed three-dimensional information data and the detection signal of the attitude sensor S4 include data indicating the result of the integration process by the coordinate system integration processing unit 1031, the bucket position calculation unit 1032′ can calculate data indicating the position of the bucket 113 in the same manner as in the case of FIG. 4. The corresponding ground surface position calculation unit 1033′ calculates the position of the ground surface that the bucket 113 is attempting to excavate (corresponding ground surface) based on the data calculated by the bucket position calculation unit 1032′ and the decompressed three-dimensional information, etc.

[0032] According to the image generation unit 103 and the operation / display control device 305 shown in Figures 6 and 7, the entity that calculates the position of the bucket 113 and the corresponding ground surface is different from the case of Figures 4 and 5, but in either case, it is possible to perform accurate AR display based on the image signal obtained by integrating the coordinate systems of the detection signals of multiple sensors S1 to S4.

[0033] The procedure for generating a superimposed image including an AR display when the video generation unit 103 and the driving / display control device 305 in Figures 4 and 5 are employed will be described with reference to the flowchart in Figure 8. First, in step S11, after detection data from each of the sensors S1 to S4 is obtained, coordinate system correction information for correcting the coordinate systems of each of the detection data from the sensors S1 to S4 is calculated and obtained by integration processing in the coordinate system integration processing unit 1031.

[0034] Then, in the following step S12, using the coordinate system correction information and detection data from the attitude sensor S4, position information of the bucket 113 is calculated in the bucket position calculation unit 1032. Once the position information of the bucket 113 has been calculated, in step S13, the position of the ground surface (corresponding ground surface) on which the bucket 113 is about to move is calculated according to the position information of the bucket 113, the detection signals of the sensors S1 to S3 serving as three-dimensional information acquisition devices, and the coordinate system correction information which is the result of the coordinate system integration processing.

[0035] Next, according to the calculation results of steps S12 and S13, the distance between the bucket 113 and the corresponding ground position is calculated (step S14). According to the calculation results of steps S12 to S14, an AR image to be superimposed on the captured real video is generated in the AR image generation unit 1034 (step S15). Once the AR image is generated, the AR image data is superimposed on real video data captured by the sensors S1 to S3 (imaging devices) in the video superimposition unit 1035 to generate a superimposed image (step S16). The superimposed image is compressed in the video compression unit 1036 (step S17) and transmitted from the communication device 104 to the remote control unit 300. Note that the procedure for generating a superimposed image including an AR display when the video generation unit 103 and the operation / display control device 305 of FIGS. 6 and 7 are employed is generally similar to the above, except for the installation position of the information generator, and therefore a detailed description thereof will be omitted.

[0036] Next, an example of an AR image displayed in the superimposed image in the first embodiment will be described with reference to Fig. 9 and Fig. 10. On the display 304, together with the composite image as shown in Fig. 9, the superimposed image generated by the video superimposing unit 1035 is displayed in a form including the AR image, for example, as shown in Fig. 10.

[0037] 9 and 10, the AR image generation unit 1034 (support information generation unit) generates support information as part of an AR image, including a support graphic (first graphic) RL1 that highlights the position of a portion of the bucket 113 within the work area, and a support graphic (second graphic) RL2 that highlights a portion of a second area adjacent to a first area that includes a portion of the first graphic RL1 projected vertically or forward / backward in the work area. The video superimposition unit 1035 generates an AR image that displays the support information superimposed on a video of the work area.

[0038] For example, the AR image in the first embodiment includes a line-shaped graphic RL1 (first graphic) that highlights the tip position (toe position) of the bucket 113 during excavation work by the work machine 100, a line-shaped graphic RL2 (second graphic) that highlights the position of the ground surface Lg (corresponding ground surface Lg (first area)) along which the bucket 113 is about to move, and a line-shaped graphic RL3 that connects the two graphics. In other words, the second graphic RL2 is two graphics that highlight portions corresponding to both ends of the first graphic RL1 on the surface of the work area when the first graphic RL1 is projected onto the work area in the vertical direction or the front-to-back direction. Furthermore, the third graphic RL3 is a line-shaped graphic that extends vertically or in the front-to-back direction and connects both ends of the first graphic RL1 and the two graphics of the second graphic RL2. If the construction machine is a backhoe-type shovel, the corresponding ground surface Lg is located directly below the current position of the bucket 113. As will be described later, when the construction machine is a loading shovel, the corresponding ground surface Lg is located in the depth direction (front-rear direction) as viewed from the current position of the bucket 113.

[0039] The relationship between the graphic RL1 and the graphic RL2 is not limited to a specific one, as long as the graphic RL1 represents the position of a portion of the bucket 113 and the graphic RL2 displays the position of the corresponding ground surface Lg in a manner that does not obstruct the visibility of the corresponding ground surface Lg. However, as shown in Fig. 10, it is preferable that the graphic RL1 is a straight line extending from the right end to the left end of the cutting edge of the bucket 113, and the graphic RL2 is a straight line figure that sandwiches the corresponding ground surface Lg, which is the lowered position of the cutting edge of the bucket 113, on both sides. This display form is preferable in that it allows the position of the cutting edge of the bucket 113 to be clearly visible, and furthermore, the position of the corresponding ground surface Lg can be confirmed without obstructing visibility.

[0040] The shapes of the figures RL1, RL2, and RL3 are not limited to the example shown in the figure, and various shapes are possible as long as they do not overlap with the corresponding ground surface Lg and make the corresponding ground surface Lg easily visible. For example, the figures RL1 and RL2 can be various shapes other than straight lines, such as wavy lines, dashed lines, double lines, rectangles, and circles. The colors of the figures RL1, RL2, and RL3 can also be changed. It is also possible for one of the figures RL1, RL2, and RL3 to be constantly lit while the others flash. Here, the corresponding ground surface Lg can be defined according to the function of the work machine 100. For example, if the work machine 100 is a backhoe-type excavator as shown in FIG. 1, the current vertical position of the bucket 113 within the work area (in this embodiment, the position directly below it in the vertical direction) can be defined as the corresponding ground surface Lg. In other words, the corresponding ground surface Lg can be defined as one of the positions that the work machine 100 can reach without swinging.

[0041] The figure RL1 (first figure) is displayed at the position of the bucket 113 calculated by the bucket position calculation unit 1032. The figure RL1 can be displayed in the superimposed image in a form that is superimposed on and highlighted with the tip of the bucket 113. Note that the figure RL1 can also be displayed in an area near the tip of the bucket 113, rather than being displayed so as to overlap the tip of the bucket 113. This makes it possible to view the tip of the bucket 113 without obstructing its view.

[0042] On the other hand, the figure RL2 (second figure) is not displayed so as to overlap the ground surface Lg (corresponding ground surface (first area)) over which the bucket 113 is about to move, but is displayed in an area (second area) near the corresponding ground surface Lg. In other words, the figure RL2 is displayed around the corresponding ground surface Lg, thereby indicating the position of the corresponding ground surface Lg. By displaying the figure RL2 around the corresponding ground surface Lg and avoiding displaying the figure RL2 so as to overlap the corresponding ground surface Lg, the operator of the remote control unit 300 can clearly recognize the positional relationship between the bucket 113 and the corresponding ground surface Lg, while also being able to clearly observe the corresponding ground surface Lg on the display 304. Preferably, the figure RL2 is displayed on both sides of the corresponding ground surface Lg so as to sandwich the corresponding ground surface Lg, but this is not limited to this.

[0043] Furthermore, figure RL3 (third figure) is displayed as a line-like figure extending vertically so as to connect the end of figure RL1 with the end of figure RL2. When bucket 113 excavates level ground, figures RL1 and RL2 extend substantially horizontally, while figure RL3 extends substantially vertically, intersecting these. As shown in Figures 11 and 12, when the position of bucket 113 moves, the display positions of figures RL1 to RL3 also change. However, in either case, figure RL2 is displayed around the corresponding ground Lg along which bucket 113 is about to move, without overlapping it.

[0044] 13 and 14, an example of a method for defining the corresponding ground surface Lg (first region, non-display region of the support graphic) will be described. In the case where the work machine 100 is a backhoe-type excavator, and the upper rotating body 102 does not rotate, and only the boom 111, arm 112, and bucket 113 rotate about the rotation axis, the possible region of the corresponding ground surface Lg is defined as a band-shaped region L02 along the plane on which the boom 111, arm 112, and bucket 113 rotate. This band region L02 has, for example, a width corresponding to the width of the bucket 113, and a length corresponding to the range in the depth direction (front-to-back direction) over which the bucket 113 can rotate.

[0045] The AR image generation unit 1034 calculates the position of this strip area L02, and because there is a possibility that a ground surface Lg corresponding to this strip area L02 exists, the AR image generation unit 1034 does not display the figure RL2 in this strip area L02, but displays the figure RL2 only in the adjacent strip areas L01 and L03. Note that, as shown in Fig. 14, if there is a wall surface L4 that rises up so as to intersect with the ground surface L0, strip areas L02', L01', and L03' can be defined along this wall surface L4.

[0046] In the example of FIG. 10 , the figure RL2 is displayed at a position sandwiching the corresponding ground surface Lg in the horizontal direction. However, for example, based on a control signal from the operation / display control device 305, it may be determined that the position of the corresponding ground surface Lg is limited to a part of the band area L02. In that case, as shown in FIG. 15 , the figure RL2 may be displayed in areas A3 and A5 sandwiching the corresponding ground surface Lg in the longitudinal direction (vertical direction) of the band area L02. Alternatively, as shown in FIG. 16 , in order to clearly indicate the corresponding ground surface Lg, the figure RL2 may be displayed in one of the surrounding areas of the corresponding area Lg (for example, the upper area A3, the lower left area B5, and the lower right area C5). In short, display control is performed so that the corresponding ground surface Lg, which is the area where the cutting edge of the bucket 113 is likely to reach, and the figure RL2 are not displayed overlapping each other. In other words, the non-display area (first area) of the support figure RL2, which is the corresponding ground surface Lg, is, for example, an area that includes at least the vertical position of the bucket 113 in the work area, and may be, for example, the entire or part of a band-shaped area (band area L02) that has a width corresponding to the width of the bucket 113 and extends in the depth direction. When the non-display area is part of band area L02, for example, the non-display area may be a rectangular area (the area between areas A3 and A5 in band area L02) that has a width corresponding to the width of the bucket 113 and a length corresponding to the length of the bucket 113 in the depth direction, or may be an area in front of the bucket 113 that has a width corresponding to the width of the bucket (the area in front of area A3 in band area L02).

[0047] As described above, according to the first embodiment, the AR image is generated so that the figure RL2 is displayed in a nearby area that does not overlap with the corresponding ground surface Lg. This makes it possible to clearly recognize the position to which the bucket 113 is about to move without obstructing its visibility.

[0048] [Second embodiment] Next, a display control device according to a second embodiment will be described with reference to Figs. 17 to 21. The overall configuration of the display control device according to the second embodiment is similar to that of the first embodiment, so a duplicated description will be omitted. The second embodiment differs from the first embodiment in the configuration of the image generation unit 103. Specifically, the second embodiment differs from the first embodiment in that the image generation unit 103 is provided with a three-dimensional information storage unit 1044 that stores three-dimensional information detected by sensors S1 to S3 as three-dimensional information acquisition devices.

[0049] As shown in Figure 18, when slope L3 is excavated and the cutting edge of bucket 113 moves to the position of figure RL2, an error may occur in the display position of figure RL2 due to the excavated soil and sand accumulated in bucket 113, as shown in Figure 19. In this case, the shape of slope L3 is stored in advance as data in three-dimensional information storage unit 1044, and the position of figure RL2 can be corrected using this data, as shown in Figure 20.

[0050] With reference to the flowchart in FIG. 21, the procedure for generating a superimposed image including an AR image in the second embodiment will be described. Steps S11 to S14 are the same as those in the first embodiment (FIG. 8). Next, in step S21, it is determined whether the distance between the bucket 113 and the corresponding ground surface Lg is shorter than a specified distance. If YES, the position of the corresponding ground surface corresponding to the position of the bucket 113 is calculated using the terrain information stored in the three-dimensional information storage unit 1044. If NO, the obtained position information of the bucket 113 and terrain information according to the position of the corresponding ground surface Lg are newly stored in the three-dimensional information storage unit 1044 (step S22). Thereafter, steps S15 to S17 are executed in the same manner as in the first embodiment.

[0051] As described above, according to the second embodiment, errors in the graphic display position due to the excavated soil and sand deposited in the bucket 113 can be prevented, and stable work can be performed regardless of the shape of the excavated soil and sand.

[0052] [Third embodiment] Next, a display control device according to a third embodiment will be described with reference to FIG. 22. The overall configuration of the display control device according to the third embodiment is the same as that of the first embodiment, and therefore a duplicated description will be omitted. The third embodiment differs from the first embodiment in the configuration of the image generation unit 103. Specifically, the image generation unit 103 is configured to display an AR image that supports operation when the bucket 113 turns and moves toward the vessel 200. In FIG. 22, the same components as those in the first embodiment (FIG. 4) are assigned the same reference numerals, and a duplicated description thereof will be omitted. The configuration of FIG. 22 and the configuration of FIG. 4 may be combined and mounted on a single image generation unit 103.

[0053] The image generation unit 103 in FIG. 22 includes a vessel position calculation unit 1050 in addition to the same components as those in the first embodiment. The vessel position calculation unit 1050 calculates the position of the vessel 200 (for example, the upper end) based on data indicating the results of the integration process in the coordinate system integration processing unit 1031 and position information obtained from the sensors S1 to S3 serving as three-dimensional information acquisition devices. The AR image generation unit 1034 is configured to generate an AR image for grasping the positional relationship between the position of the vessel 200 and the bucket 113. Note that the vessel position calculation unit 1050 may detect, in addition to the position of the vessel, the position of other objects with which the bucket 113 may collide when the bucket 113 swings. Examples of other objects include soil and sand accumulated in the work area, vehicles such as bulldozers, buildings, workers, and other objects that may hinder work.

[0054] 23 and 24 illustrate an example of an AR image displayed in a superimposed image in the display control device of the third embodiment. On the display 304, a superimposed image generated by the video superimposing unit 1035 is displayed together with a composite image as shown in FIG. 23, including an AR image, as shown in FIG. 24, for example. As an example, a figure RL4 indicating the position of the bottom end of the bucket 113 and a line-shaped figure RL5 highlighting the position of the top end of the vessel 200 can be superimposed and displayed as an AR image. That is, in this third embodiment, when the vessel 200 is present in the work target area, the AR image generating unit 1034 generates, as an AR image, a figure RL5 highlighting the position of at least a portion of the vessel 200. Furthermore, if a vessel 200 exists across the first area and the second area, the AR image generation unit 1034 sets the upper end portion of the vessel 200 when the first figure RL1 is projected as a new first area, sets the upper end portion of the bucket 113 adjacent to the set first area as a new second area, and generates a figure RL5 that highlights the set second area.

[0055] As described above, according to the third embodiment, the operation to avoid a situation in which the bucket hits a dump vessel loaded with earth and sand can be easily performed, and work can be carried out safely.

[0056] [Variations] In the above embodiment, the case where the work machine 100 is a so-called backhoe-type shovel has been described. However, a similar configuration can also be adopted when the work machine 100 is a loading shovel with a bucket facing forward. That is, as shown in FIGS. 25 and 26, the AR image generation unit 1034 (support information generation unit) displays a line-shaped graphic RL5 that highlights the position of the cutting edge of the bucket 113 of the loading shovel, and the area above the bucket 113 in the fore-and-aft direction (depth direction) in the work target area as a first area, which is a non-display area for the support graphic. In this embodiment, the AR image generation unit 1034 displays a graphic RL6 in the vicinity of the ground position corresponding to where the cutting edge of the bucket 113 is expected to move in the fore-and-aft direction, avoiding that position. As shown in FIG. 27, when the bucket 113 is swinging, an AR image can be displayed that shows a graphic RL7 that highlights the position of the bottom end of the bucket 113 and a graphic RL8 that highlights the position of the upper end of the vessel 200.

[0057] 28 is a conceptual diagram illustrating switching of the display state of an AR image in the display control device of the above-described embodiment. The AR image generation unit 1034 (support information generation unit) may be configured to switch the display form of the support information according to the work state of the work machine 100. More specifically, the work machine 100 moves to the excavation site (S31), waits (S32), and then drives the bucket 113 to perform excavation (S33). After excavation, the work machine 100 swings (forward) (S34) and dumps earth into the vessel 200 (S35). After dumping is complete, the bucket 113 swings (returns) (S36) back to the excavation site again, and excavation continues. The display state is switched in this manner: while excavation is in progress, an AR image showing the positional relationship between the bucket 113 and the corresponding ground surface Lg is displayed; and when swinging, an AR image showing the positional relationship between the bucket 113 and the vessel 200 is displayed. These display state switching settings can be stored for each user and read out at the time of operation, allowing for customized settings that are easy to operate for each user. Also, since the brightness and contrast of captured images change depending on the weather, time, and shooting direction (such as front lighting or back lighting), it is effective to improve visibility by adjusting the brightness and contrast of the superimposed AR image (lowering the brightness for dark images such as at night, and increasing the brightness for bright images such as during the day). These brightness setting values ​​can also be stored for each user and read out at the time of operation.

[0058] As described above, the display control device of the present disclosure is a display control device that acquires an image of a work area for a work machine equipped with a bucket and displays it on a display, and includes a three-dimensional information acquisition unit that acquires topographical information indicating the three-dimensional shape of the topography of the work area, a bucket position calculation unit (1032) that acquires attitude information related to the attitude of the work machine from the work machine and calculates the position of the bucket within the work area based on the attitude information, and an AR image generation unit (1034) that generates an AR image to be displayed on the display superimposed on the image of the work area based on the three-dimensional information and the bucket position. The AR image generation unit generates, as the AR image, a first graphic that highlights the position of at least a portion of the bucket, and a second graphic that highlights a portion of a second area adjacent to a first area that includes a portion of the first graphic projected vertically or longitudinally onto the work area. When generating an AR image to be displayed on the display superimposed on the image of the work area, the AR image generation unit (1034) generates, as the AR image, a first graphic that highlights the position of at least a portion of the bucket, and a second graphic that highlights a portion of a second area adjacent to a first area that includes a portion of the first graphic projected onto the work area in the vertical or longitudinal direction. This makes it possible to provide an appropriate graphic display for work support during a series of operations of the work machine, thereby improving the work efficiency of the work machine.

[0059] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. [Explanation of symbols]

[0060] 100...work machine (construction machine), 101...undercarriage, 102...upper rotating body, 103...image generation unit, 104...communication device, 107...antenna, 111...boom, 112...arm, 113...bucket, 200...vessel, 300...remote control unit, 301...driver's seat, 302...operation control unit, 303...input device (operation shaft), 304...display, 305...operation / display control device, 306...communication device, 1031...coordinate system integration processing unit, 1032, 1032'...bucket position calculation unit, 1033, 1033'...corresponding ground position calculation unit, 1034, 1034'...AR image generation unit, 1035...image superposition unit, 1036...image compression unit, 1037...coordinate system conversion unit, 1039...Three-dimensional information compression unit, 1044...Three-dimensional information storage unit, 1050...Bessel position calculation unit, 3051...Image decompression unit, 3051'...Three-dimensional information decompression unit, 3052...Information addition / display switching unit, 3053...Display control unit, L0, L1, L2...Ground, L01, L01', L02, L02', L03, L03'...Band area, L4...Wall surface, Lg...Corresponding ground, OR...Operation room, RL1 to 8...Figure, S1 to S4...Sensors.

Claims

1. A display control device that acquires an image of a work area in a work machine equipped with a bucket and displays it on a display, a three-dimensional information acquisition unit that acquires topographical information indicating the three-dimensional shape of the topography of the work area; a bucket position calculation unit that acquires posture information related to the posture of the work machine and calculates the position of the bucket within the work target area based on the posture information; an AR image generation unit that generates an AR image to be displayed on the display by superimposing it on the image of the work area based on the topographical information and the position of the bucket; Equipped with The AR image generation unit generates the AR image as a first graphic that highlights the position of at least a portion of the bucket; a second graphic that shows a part of the first graphic projected onto the work area in the vertical direction or the front-to-back direction as a non-display area, and highlights a part of an area adjacent to the non-display area; A display control device characterized by generating a display.

2. The AR image generation unit As the first graphic, a line-shaped graphic is generated that highlights the position of the tip of the bucket; and The display control device described in claim 1, characterized in that as the second figure, two figures are generated that highlight and show the portions corresponding to both ends of the first figure on the surface of the work area when the first figure is projected in the vertical direction or the front-to-back direction of the work area.

3. The AR image generation unit The display control device according to claim 2, characterized in that a line-shaped third figure extending in the vertical direction or the front-to-back direction that connects both ends of the first figure and the two figures of the second figure is generated as the AR image.

4. The AR image generation unit 2. The display control device according to claim 1, wherein a line-shaped graphic indicating the position of the bottom end of the bucket is generated as the first graphic.

5. an object position calculation unit that acquires detection information of an object present around the work machine and calculates the position of the object based on the detection information; The display control device according to claim 4 , wherein, when the object is present in the work area, the AR image generation unit generates, as the AR image, a fourth figure that highlights a position of at least a part of the object.

6. The display control device according to claim 5, characterized in that, when the object exists across the non-display area and the area in which the second figure is displayed, the AR image generation unit sets the upper end portion of the object when the first figure is projected as a new non-display area, and generates as the fourth figure a figure that highlights the upper end portion of the object adjacent to the set non-display area.

7. The display control device according to claim 1 , wherein the AR image generation unit switches the display form of the AR image depending on the working state of the work machine.

8. A display control device according to any one of claims 1 to 7; a control device that includes the display, is provided at a position remote from the work machine, and controls the work machine while communicating with the work machine.

Images