Display control system and display control method

Abstract

To provide a display control system for supplementing lack of sensation which an operator is not able to recognize because of remote control.SOLUTION: A captured image acquiring part acquires images captured by an imaging device mounted on a working vehicle. A supplementary image acquiring part acquires supplementary images that are images for supplementing lack of sensation recognized during riding on the work vehicle, which is information regarding to the working vehicle. A display image generation part generates display images where the supplementary images are disposed in an area reflecting a part of the working vehicle out of captured images. A display control part outputs a display signal for displaying the display images to a display device.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a display control system and a display control method for controlling a display device in a remote operator's cab for remotely operating a work vehicle. This application claims priority from Japanese Patent Application No. 2018-205905, filed on October 31, 2018, the contents of which are incorporated herein by reference. [Background technology]

[0002] There is known technology for remotely controlling a work vehicle. In order to remotely control a work vehicle, it is necessary to be able to recognize the situation around the work vehicle from outside. For this reason, a remotely controlled work vehicle is equipped with an imaging device that captures images of the surrounding situation and a communication device that transmits the captured images to the outside. This allows an operator to operate the work vehicle while visually checking the images transmitted from the work vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-168778 Summary of the Invention [Problem to be solved by the invention]

[0004] The operator recognizes the status of the work vehicle by visually checking the images and operates the work vehicle based on that status. However, when remotely operated, the operator cannot get the sensations that he or she gets when actually riding in the work vehicle, such as the tilt of the work vehicle. Therefore, while when riding in the work vehicle, the operator can infer the status of the work vehicle from these sensations and perform the work, when remotely operated, it is difficult for the operator to infer the status of the work vehicle. An object of an aspect of the present invention is to provide a display control system and a display control method that can compensate for the lack of sensation that an operator cannot obtain due to remote operation. [Means for solving the problem]

[0005] According to a first aspect of the present invention, a display control system is a display control device that controls a display device provided outside a work vehicle, and includes an image acquisition unit that acquires an image captured by an imaging device mounted on the work vehicle, a supplementary image acquisition unit that acquires a supplementary image that is an image representing information related to the work vehicle, a display image generation unit that generates a display image in which the supplementary image is placed in an area of ​​the image in which a part of the work vehicle may be captured, and a display control unit that outputs a display signal to the display device to display the display image. [Effects of the Invention]

[0006] According to the above aspect, the display control system can supplement the sensations that the operator cannot get due to remote control. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing the configuration of a remote control system according to a first embodiment. [Figure 2] 1 is an external view of a work vehicle according to a first embodiment. [Figure 3] 3 is an example of an image captured by an imaging device of a work vehicle according to the first embodiment. [Figure 4] 1 is a schematic block diagram showing the configuration of a control device for a remote operator cab according to a first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of an image cut out from an image captured by a front camera. [Figure 6] FIG. 10 is a diagram illustrating an example of a posture image. [Figure 7] FIG. 10 is a diagram illustrating an example of a peripheral image. [Figure 8]3A and 3B are diagrams illustrating examples of display images displayed on the display device according to the first embodiment. [Figure 9] 5 is a flowchart showing a method for setting layout information by the control device of the remote operator cab according to the first embodiment. [Figure 10] 4 is a flowchart showing a display control method by the control device of the remote operator cab according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment Remote Control System FIG. 1 is a schematic diagram showing the configuration of a remote control system according to the first embodiment. The remote control system 1 comprises a work vehicle 100 that is operated by remote control, and a remote operator's cab 500 for performing the remote control. The work vehicle 100 operates at a work site (for example, a mine or a quarry). The remote operator's cab 500 is provided at a location away from the work vehicle 100 (for example, in a city or within the work site). The work vehicle 100 and the remote operator's cab 500 are connected via a communication means such as the Internet. The remote control system 1 is a system for operating a work vehicle 100 using a remote operator's cab 500.

[0009] The work vehicle 100 operates in accordance with operation signals received from the remote operator's cab 500 . The remote operator's cab 500 receives operations for the work vehicle 100 through operations by the operator, and transmits operation signals to the work vehicle 100 .

[0010] <Work vehicle> FIG. 2 is an external view of the work vehicle according to the first embodiment. The work vehicle 100 according to the first embodiment is a hydraulic excavator. Note that the work vehicle 100 according to other embodiments may be a work vehicle other than a hydraulic excavator, such as a wheel loader or a bulldozer. The work vehicle 100 includes a hydraulically driven work implement 130 , a revolving body 120 that supports the work implement 130 , and a running body 110 that supports the revolving body 120 .

[0011] The work implement 130 includes a boom 131, an arm 132, and a bucket 133. The work implement 130 is driven by extension and retraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. A boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139 are attached to the boom 131, the arm 132, and the bucket 133, respectively.

[0012] The base end of the boom 131 is attached to the rotating body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip end of the boom 131 via a pin. The boom 131 and the arm 132 are an example of a support member that supports the bucket 133. The bucket 133 includes a blade for digging up earth and sand and a container for collecting the excavated earth and sand. The base end of the bucket 133 is attached to the tip of the arm 132 via a pin. The bucket 133 is an example of a work tool. In other embodiments, an attachment other than the bucket 133 may be used as the work tool.

[0013] The boom cylinder 134 is, for example, a hydraulic cylinder for driving the boom 131. A base end of the boom cylinder 134 is attached to the revolving body 120. A tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is, for example, a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 135 is attached to the boom 131. A tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is, for example, a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 136 is attached to the boom 131. A tip end of the bucket cylinder 136 is attached to the bucket 133.

[0014] The boom angle sensor 137 is attached to the boom 131 and detects the tilt angle of the boom 131 . The arm angle sensor 138 is attached to the arm 132 and detects the tilt angle of the arm 132 . The bucket angle sensor 139 is attached to the bucket 133 and detects the inclination angle of the bucket 133 . The boom angle sensor 137, arm angle sensor 138, and bucket angle sensor 139 according to the first embodiment detect the angle of inclination relative to the ground plane. Note that the angle sensors according to other embodiments are not limited to this, and may detect the angle of inclination relative to another reference plane. For example, in other embodiments, the angle sensors may detect the relative rotation angle using potentiometers provided at the base ends of the boom 131, arm 132, and bucket 133, or may detect the angle of inclination by measuring the cylinder lengths of the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 and converting the cylinder lengths into angles.

[0015] The revolving unit 120 is provided with a cab 121. The cab 121 is provided on the left side of the work implement 130. The cab 121 is provided with a front camera 122. The front camera 122 is installed at the front and upper part of the cab 121. The front camera 122 captures images of the area in front of the cab 121 through the windshield in front of the cab 121. Here, "forward" refers to the direction in which the work implement 130 is mounted on the revolving unit 120, and "rearward" refers to the opposite direction of "forward." "Side" refers to a direction intersecting the fore-and-aft direction (left-right direction). Examples of the front camera 122 include an imaging device using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The work vehicle 100 according to other embodiments may not have a cab 121 if it is driven by remote control or unmanned operation. In this case, the forward camera 122 is installed at the front of the rotating body 120 so as to face forward. FIG. 3 is an example of an image captured by the imaging device of the work vehicle according to the first embodiment. The front camera 122 captures an image of the range in which the work implement 130 and the work object in front of the cab 121 are captured. That is, the image P1 captured by the front camera 122 captures the work implement 130 and the work object in front of the cab 121, as shown in FIG. 3. Furthermore, because the cab 121 is provided on the left side of the work implement 130, a portion of the boom 131 is captured on the right side of the image P1. Note that the work implement 130 of a work vehicle 100 according to other embodiments may be mounted on the right side of the cab 121. In this case, a portion of the boom 131 is captured on the left side of the image P1. The work implement 130 is captured in the image P1 during excavation work, for example.

[0016] The work vehicle 100 is equipped with a forward camera 122 , a bucket camera 123 , a plurality of surrounding monitoring cameras 124 , a position and orientation calculator 125 , an inclination measuring device 126 , a hydraulic device 127 , and a control device 128 .

[0017] The bucket camera 123 is provided on the front side of the arm 132 and captures images of the inside of the bucket 133 . The multiple perimeter monitoring cameras 124 are each provided on the circumferential surface of the revolving unit 120 and capture images of the vicinity of the work vehicle 100. The perimeter monitoring cameras 124 are arranged so that the imaging ranges of adjacent perimeter monitoring cameras 124 overlap each other. This makes it possible to simultaneously capture images of the entire perimeter of the work vehicle 100. The perimeter monitoring cameras 124 are provided, for example, at the front, left front, right front, left rear, right rear, rear, and under the counterweight of the revolving unit 120.

[0018] The position and orientation calculator 125 calculates the position of the revolving unit 120 and the orientation in which the revolving unit 120 faces. The position and orientation calculator 125 is equipped with two receivers that receive positioning signals from artificial satellites that make up the GNSS. The two receivers are installed at different positions on the revolving unit 120. The position and orientation calculator 125 detects the position of a representative point of the revolving unit 120 in the site coordinate system (the origin of the excavator coordinate system) based on the positioning signals received by the receivers. The position and direction calculator 125 uses the positioning signals received by the two receivers to calculate the direction in which the rotating unit 120 faces as the relationship between the installation position of one receiver and the installation position of the other receiver. In other embodiments, the position and orientation calculator 125 may detect the orientation of the revolving unit 120 based on measurement values ​​from a rotary encoder or an IMU. Work vehicle 100 according to other embodiments does not need to be equipped with the position and orientation calculator 125.

[0019] The inclination measuring device 126 measures the acceleration and angular velocity of the revolving unit 120 and detects the attitude of the revolving unit 120 (e.g., roll angle, pitch angle, yaw angle) based on the measurement results. The inclination measuring device 126 is installed, for example, on the underside of the revolving unit 120. The inclination measuring device 126 may be, for example, an inertial measurement unit (IMU). The inclination measuring device 126 may be an inclinometer that detects the inclination angle without relying on the acceleration and angular velocity. Furthermore, the work vehicle 100 according to other embodiments may not be equipped with the inclination measuring device 126.

[0020] The hydraulic device 127 includes a hydraulic oil tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by power from an engine or electric motor (not shown), and supplies hydraulic oil to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and adjusts the flow rate of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 depending on the position of the spool. The spool is driven based on a control command received from the control device 128. In other words, the amount of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 is controlled by the control device 128.

[0021] The control device 128 transmits to the remote operator's cab 500 information on images captured by the front camera 122, bucket camera 123, and multiple peripheral monitoring cameras 124, the rotation speed, position, direction, and tilt angle of the rotating body 120, and the tilt angles of the boom 131, arm 132, and bucket 133. Hereinafter, the information acquired by the various sensors and cameras equipped in the work vehicle 100 and transmitted by the control device 128 will also be referred to as vehicle body information. The control device 128 receives an operation signal from the remote operator's cab 500. The control device 128 drives the work machine 130, the revolving body 120, or the traveling body 110 based on the received operation signal.

[0022] Remote Control Cabin The remote operator's cab 500 includes an operator's seat 510 , a first display device 520 , a second display device 530 , an operating device 540 , and a control device 550 . The first display device 520 is disposed in front of the driver's seat 510. The first display device 520 is located in front of the operator when the operator sits in the driver's seat 510. As shown in FIG. 1 , the first display device 520 is composed of a center display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525 arranged side by side. The left display 522 is provided to the left of the center display 521. The right display 523 is provided to the right of the center display 521. The upper display 524 is provided above the center display 521. The lower display 525 is provided below the center display 521. In other embodiments, the number of displays constituting first display device 520 is not limited to this. For example, first display device 520 may be composed of a single display. Also, first display device 520 may project an image onto a curved or spherical surface using a projector or the like.

[0023] The second display device 530 is disposed diagonally forward of the driver's seat 510. The second display device 530 displays vehicle body information (e.g., remaining fuel level, engine water temperature) transmitted from the work vehicle 100, notifications of abnormalities in the work vehicle 100, and the like. Note that in other embodiments, the position of the second display device 530 does not have to be diagonally forward of the driver's seat 510, as long as it is in a position that is visible to the operator. Furthermore, the remote driver's cab 500 according to other embodiments does not need to be equipped with the second display device 530. When the remote driver's cab 500 does not have the second display device 530, the vehicle body information and notifications described above may be displayed on the first display device 520.

[0024] The operation device 540 is disposed near the driver's seat 510. The operation device 540 is located within an operable range of the operator when the operator sits in the driver's seat 510. The operation device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and the electric pedal, the operation device 540 outputs operation signals for the boom 131, the arm 132, and the bucket 133, a rotation operation signal for the rotating unit 120, and a travel operation signal for the traveling unit 110.

[0025] The control device 550 displays the images and vehicle body information received from the work vehicle 100 on the first display device 520. In other words, the control device 550 is an example of a display control system. The display control system may be a remote operator cab 500 that includes the control device 550, or may further include the control device 128 and forward camera 122 of the work vehicle 100. The control device 550 also transmits an operation signal input to the operation device 540 to the work vehicle 100.

[0026] Remote cab control device FIG. 4 is a schematic block diagram showing the configuration of the control device of the remote operator cab according to the first embodiment. The control device 550 is a computer including a processor 910, a main memory 930, a storage 950, and an interface 970. The storage 950 stores a program. The processor 910 reads the program from the storage 950, loads it into the main memory 930, and executes processing in accordance with the program.

[0027] Examples of the storage 950 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. The storage 950 may be an internal medium directly connected to the common communication line of the control device 550, or an external medium connected to the control device 550 via an interface 970. The storage 950 is a non-transitory tangible storage medium. In other embodiments, the control device 550 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) or a semi-custom large-scale integrated circuit (LSI) such as an application-specific integrated circuit (ASIC) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor 910 may be implemented by the integrated circuit.

[0028] By executing the program, the processor 910 is provided with a vehicle body information receiving unit 911, an image cropping unit 912, an attitude image generating unit 913, an overhead image generating unit 914, a display image generating unit 915, a layout receiving unit 916, a display control unit 917, and an operation signal transmitting unit 918. Furthermore, by executing the program, the processor 910 secures a storage area for a layout storage unit 931 in the main memory 930.

[0029] The vehicle body information receiving unit 911 receives information from the work vehicle 100, such as images captured by the front camera 122, the bucket camera 123, and the multiple periphery monitoring cameras 124, the rotation speed, position, direction, and tilt angle of the rotating unit 120, and the tilt angles of the boom 131, the arm 132, and the bucket 133. In other words, the vehicle body information receiving unit 911 is an example of a captured image acquiring unit. The image captured by the bucket camera 123 will be referred to as an inside-bucket image P4 below. The bucket interior image P4 is an example of a supplemental image that compensates for the lack of sensation obtained when riding in the work vehicle 100. In other words, the vehicle body information receiving unit 911 is an example of a supplemental image acquisition unit. By actually riding in the work vehicle 100, the operator can sense even slight changes in the inclination of the work vehicle 100. When the operator operates the work vehicle 100 and soil is placed in the bucket 133, the work vehicle 100 tilts forward due to the weight of the soil. This allows the operator to estimate the amount of soil that has been placed in the bucket 133. On the other hand, in the case of remote operation, the operator cannot sense changes in the inclination of the work vehicle 100. However, by visually checking the bucket interior image P4, the operator can estimate the amount of soil that has been placed in the bucket 133. In other words, the bucket interior image P4 can compensate for the lack of even slight changes in the inclination of the work vehicle 100.

[0030] FIG. 5 is a diagram showing an example of an image cut out from an image P1 captured by the front camera. The image cropping unit 912 crops out a center image P11 to be displayed on the center display 521, a left image P12 to be displayed on the left display 522, a right image P13 to be displayed on the right display 523, an upper image P14 to be displayed on the upper display 524, and a lower image P15 to be displayed on the lower display 525 from the image P1 captured by the forward camera 122 and received by the vehicle body information receiving unit 911. The center image P11 corresponds to the area of ​​the image P1 that shows the view in front of the driver's cab. The left image P12 corresponds to the area of ​​the image P1 that shows the view to the left of the driver's cab. The right image P13 corresponds to the area of ​​the image P1 that shows the view to the right of the driver's cab. In other words, the left image P12 and the right image P13 show the view to the side of the driver's cab. The upper image P14 corresponds to the area of ​​the image P1 that shows the view above the driver's cab. The lower image P15 corresponds to the area of ​​the image P1 that shows the view below the cab. As shown in FIG. 5, the side portion of the base end of the boom 131 is reflected in most of the right image P13. In other words, the base end of the work implement 130 is reflected in most of the right image P13. Hereinafter, the area in the image that shows part of the work vehicle 100 is also referred to as the "area in which part of the work vehicle 100 may be reflected." In other words, the "area in which part of the work vehicle 100 may be reflected" is not limited to the area in which part of the work vehicle 100 is always reflected, regardless of the attitude of the work vehicle 100. The "area in which a portion of the work vehicle 100 may be captured" includes any one of the following: an area in the display image in which a portion of the work vehicle 100 actually appears; an area in the display image in which a portion of the work vehicle 100 appears for the majority of the time during work; an area in the display image in which a portion of the work vehicle 100 may appear; and a display image in which a work vehicle appears in the majority of the area of ​​a plurality of displays. The area in which a portion of the work vehicle 100 may appear includes an area corresponding to the movable range of the work implement 130, for example, an area in which a portion of the work vehicle 100 may appear during excavation work. Note that if the first display device 520 is configured from a single display, the image cropping unit 912 does not need to crop the image. Note that the center image P11, upper image P14, and lower image P15 are areas in the image P1 in which the bucket 133, which is a work implement, may appear.On the other hand, the left image P12 and the right image P13 are areas different from the area in which the bucket 133 can appear. Furthermore, the left image P12 and the right image P13 are areas in which the side of the boom 131, which is a support member, can appear.

[0031] FIG. 6 is a diagram showing an example of a posture image of the work vehicle 100. As shown in FIG. The posture image generation unit 913 generates a posture image P2 representing the current posture of the work implement 130 based on information about the inclination angles of the boom 131, arm 132, and bucket 133 received by the vehicle body information receiving unit 911. The posture image P2 includes a work vehicle image P21 depicting the side of the work vehicle 100 and a grid P22 that serves as a length guide. In the posture image P2, a revolving unit image P21a representing the revolving unit 120 is drawn at a fixed position. The posture image generation unit 913 draws a boom image P21b representing the boom 131 by tilting it by the inclination angle of the boom 131 so that its base end is located at the position of the pin in the revolving unit image P21a. The posture image generation unit 913 draws an arm image P21c representing the arm 132 by tilting it by the inclination angle of the arm 132 so that its base end is located at the position of the pin in the boom image P21b. The posture image generator 913 tilts the bucket image P21d representing the bucket 133 by the tilt angle of the bucket 133 and renders it so that the base end is positioned at the position of the pin in the arm image P21c. The revolving unit image P21a, boom image P21b, arm image P21c, and bucket image P21d may be stored in advance in the storage 950, for example. Note that although the revolving unit image P21a is rendered at a fixed position in the posture image P2 according to the first embodiment, in other embodiments the revolving unit image P21a may be tilted based on the measurement value of the inclination measuring device 126. The posture image P2 is an example of a supplementary image that compensates for sensory deficiencies that may occur when riding the work vehicle 100. The posture image generation unit 913 acquires the posture image P2 by generating the posture image P2. In other words, the posture image generation unit 913 is an example of a supplementary image acquisition unit. Note that in this specification, "acquire" means to obtain a new value. For example, "acquire" includes receiving a value, accepting input of a value, reading a value from a storage device, calculating another value from a value, and generating a value. By actually riding the work vehicle 100, the operator can recognize the depth of the field of view ahead of the work vehicle 100. Because the work implement 130 is visible in the field of view ahead of the work vehicle 100, the operator can estimate the position and posture of the work implement 130 by recognizing the depth of the work implement 130. However, depth information is missing from the image captured by the front camera 122. Therefore, in the case of remote operation, the operator cannot recognize the depth of the work implement 130. On the other hand, the operator can recognize the position and posture of the work implement 130 by visually checking the posture image P2. In other words, the posture image P2 can make up for the lack of depth information. Note that posture images according to other embodiments do not need to include the grid P22.

[0032] FIG. 7 is a diagram showing an example of a peripheral image. 7 based on images captured by the multiple perimeter monitoring cameras 124 received by the vehicle body information receiving unit 911. For example, the overhead image generating unit 914 generates the overhead image P3 based on images captured by the perimeter monitoring cameras 124 provided at the front, left front, right front, left rear, right rear, and rear of the rotating unit 120. The overhead image generating unit 914 transforms the images captured by the perimeter monitoring cameras 124 based on a transformation formula determined in advance based on the arrangement of the perimeter monitoring cameras 124 to generate a partial overhead image P31, and places the partial overhead image P31 at a predetermined position on the overhead image P3. The partial overhead image P31 is a top-view image that constitutes a portion of the overhead image P3. The overhead image generating unit 914 then renders a work vehicle image P32 depicting the top surface of the work vehicle 100 at the center of the overhead image P3. The overhead image P3 is an example of a supplemental image that compensates for sensory deficiencies that can be experienced while riding in the work vehicle 100. In other words, the overhead image generation unit 914 is an example of a supplemental image acquisition unit. When the operator is actually riding in the work vehicle 100, he or she can recognize the surroundings of the work vehicle 100 through a window or mirror provided on the side of the driver's seat 510. On the other hand, in the case of remote operation, the displayed range of the image displayed on the first display device 520 does not change even if the operator changes his or her posture. For example, the field of view when the operator peers out the window cannot be reproduced by the image displayed on the first display device 520. Therefore, it is difficult for the operator to recognize the surroundings of the work vehicle 100. On the other hand, the operator can recognize the surroundings of the work implement 130 by viewing the overhead image P3. Note that in other embodiments, instead of the overhead image P3, an image captured by the periphery monitoring camera 124 or an image captured by a fixed camera may be used as a supplemental image.

[0033] FIG. 8 is a diagram showing an example of a display image displayed on the display device according to the first embodiment. The display image generation unit 915 generates a display right image P13a by arranging the posture image P2, the overhead image P3, and the bucket interior image P4 in the arrangement area R of the right image P13 cut out by the image cutout unit 912 in accordance with the layout information stored in the layout storage unit 931. The arrangement area R is an area in the right image P13 in which a portion of the work vehicle 100, including the boom 131, may be captured. In the example shown in FIG. 8 , the arrangement area R is an area overlapping an area R1 corresponding to the movable area of ​​the work implement 130 and an area R2 in which a portion of the revolving unit 120 is always captured. The layout information is information that indicates the layout of multiple supplemental images in the arrangement area R. For example, the layout information may be information that specifies the arrangement order of the multiple supplemental images, or information that specifies the arrangement coordinates of each of the multiple pieces of supplemental information.

[0034] The layout receiving unit 916 receives input or changes to layout information through an operator's operation via a tablet device (not shown) or the like.

[0035] The display control unit 917 causes the center display 521 to display the center image P11. The left display 522 to display the left image P12. The control device 550 causes the right display 523 to display the display right image P13a. The control device 550 causes the upper display 524 to display the upper image P14. The control device 550 causes the lower display 525 to display the lower image P15. In other words, the display control unit 917 causes the supplemental image to be displayed in the arrangement area of ​​the first display device 520, separately from the vehicle body information displayed by the second display device 530.

[0036] The operation signal transmitter 918 generates an operation signal based on the operation of the operation device 540 by the operator, and transmits the operation signal to the work vehicle 100 .

[0037] <<Display control method for remote driver's cab>> FIG. 9 is a flowchart showing a method for setting layout information by the control device of the remote operator cab according to the first embodiment. Before remotely operating the work vehicle 100, the operator operates a tablet terminal (not shown) or the like in advance to input the layout of the supplemental information in the placement area R of the right display 523. The layout receiving unit 916 of the control device 550 receives the input of the layout information (step S01) and stores the input layout information in the main memory 930 (step S02). Note that in other embodiments, the layout of the supplemental information may be fixed. In this case, the control device 550 does not need to be provided with the layout receiving unit 916, and does not need to perform the processing of FIG. 9.

[0038] 10 is a flowchart showing a display control method by the control device of the remote operator's cab according to the first embodiment. When remote operation of the work vehicle 100 is started, the control device 550 executes the following display control at a predetermined cycle. The vehicle body information receiving unit 911 receives vehicle body information from the control device 128 of the work vehicle 100 (step S11). Next, the image cutting out unit 912 cuts out a center image P11, a left image P12, a right image P13, an upper image P14, and a lower image P15 from the image P1 captured by the front camera 122 from the received vehicle body information (step S12).

[0039] The posture image generation unit 913 generates a posture image P2 representing the current posture of the work machine 130 based on the inclination angles of the boom 131, arm 132, and bucket 133 from the received vehicle body information (step S13). The overhead image generation unit 914 generates an overhead image P3 based on images captured by the multiple perimeter monitoring cameras 124 from the received vehicle body information (step S14).

[0040] The display image generation unit 915 generates a display right image P13a by arranging the posture image P2, the overhead image P3, and the bucket interior image P4 in the arrangement area R of the right image P13 in accordance with the layout information stored in the layout storage unit 931 (step S15). The display control unit 917 generates a display signal for displaying the center image P11, the left image P12, the display right image P13a, the upper image P14, and the lower image P15 on the first display device 520, and transmits the display signal to the first display device 520 (step S16).

[0041] Actions and Effects In this way, the control device 550 of the remote operator's cab 500 according to the first embodiment displays, on the first display device 520, a display image in which a supplementary image for compensating for the lack of sensations that the operator does not get while riding in the work vehicle 100 is arranged in an area of ​​the image captured by the forward camera 122 where a portion of the work vehicle 100 may be captured. This allows the operator to remotely operate the work vehicle 100 while compensating for the lack of sensations that the operator does not get due to remote operation. For example, by visually checking the bucket interior image P4, which is a supplementary image, the operator can estimate the amount of soil and sand contained in the bucket 133, even if the operator cannot sense changes in the inclination of the work vehicle 100 due to remote operation. Furthermore, by visually checking the attitude image P2, the operator can recognize the position and attitude of the work implement 130, even if the operator cannot recognize the depth of the work implement 130 due to remote operation. Furthermore, by visually checking the overhead image P3, the operator can recognize the situation around the work implement 130, even if the operator cannot change his / her position to look out the window. Furthermore, by displaying the layout image in an area where a portion of the work vehicle 100 can be captured, the layout image can be displayed without obscuring the work target in the displayed image. This prevents the layout image from interfering with the operation of the work vehicle 100 by the operator.

[0042] In particular, in the first embodiment, the layout image is placed in an area in the display image where the work implement 130 may appear. When the cab 121 and the work implement 130 are side by side, the work implement 130 will appear in the image captured by the front camera 122 and will block the work target. Therefore, the control device 550 sets the area in the display image where the work implement 130 may appear as the layout area R, thereby ensuring a wide layout area R for placing the supplemental image. Note that other embodiments are not limited to this, and the layout area R may also be an area where other parts of the work vehicle 100 may appear, such as the area in the upper image P14 where the ceiling of the cab 121 is visible.

[0043] Furthermore, the control device 550 according to the first embodiment receives instructions for the layout of multiple supplemental images and arranges the supplemental images in accordance with the instructions. This allows the operator to display supplemental information in a preferred layout on the first display device 520. For example, the operator can arrange supplemental images that are frequently checked at a position at about the same height as the operator's line of sight, and arrange supplemental images that are less frequently checked at a position away from the operator's line of sight. Note that other embodiments are not limited to this, and multiple supplemental images may be arranged based on a fixed layout. In this case, it is preferable that at least one supplemental image is arranged at a position at the same height as the operator's line of sight as designed.

[0044] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design modifications and the like are possible. For example, in the above-described embodiment, the posture image P2, the overhead image P3, and the bucket interior image P4 are displayed as supplemental images, but this is not limited to this. For example, in other embodiments, other supplemental images may be displayed, such as an inclinometer that indicates the inclination of the work vehicle 100, or a cutting edge indicator that indicates whether the cutting edge of the bucket 133 of the work vehicle 100 has come into contact with the work object. The inclinometer is a supplemental image that supplements the lack of a sense of the inclination of the work vehicle 100. The cutting edge indicator is a supplemental image that supplements the lack of a sense of the impact that is applied to the work vehicle 100 when the bucket 133 comes into contact with the work object. Also, for example, in other embodiments, the number of supplementary images does not have to be three. For example, the number of supplementary images according to other embodiments may be one or two, or may be four or more. Also, only some of the supplementary images may be displayed.

[0045] Furthermore, in the embodiment described above, the control device 550 generates a supplemental image based on the vehicle information received from the work vehicle 100, but this is not limited to this. For example, in other embodiments, the control device 128 of the work vehicle 100 or an external server device may generate a supplemental image, which the control device 550 may receive and display.

[0046] Furthermore, in the above-described embodiment, the display control device is implemented in the remote operation system 1, but this is not limited to this. For example, in another embodiment, the display control device may be applied to a radio control system that operates the work vehicle 100 via wireless communication from a position outside the work vehicle 100 where the work vehicle 100 can be seen. When the display control device is applied to a radio control system, the control device may be equipped with a display device. [Industrial Applicability]

[0047] According to the above aspect, the display control system can supplement the sensations that the operator cannot get due to remote control. [Explanation of symbols]

[0048] 1...Remote operation system 100...Work vehicle 130...Work machine 120...Swing body 110...Traveling body 122...Forward camera 123...Bucket camera 124...Periphery monitoring camera 125...Position and orientation calculator 126...Inclinometer 127...Hydraulic device 128...Control device 500...Remote driver's cab 510...Driver's seat 520...First display device 530...Second display device 540...Operation device 550...Control device 911...Vehicle body information receiving unit 912...Image extraction unit 913...Attitude image generating unit 914...Bird's-eye view image generating unit 915...Display image generating unit 916...Layout receiving unit 917...Display control unit 918...Operation signal transmitting unit 921...Layout storage unit

Claims

1. A display control system for controlling a display device provided on the outside of a work vehicle equipped with a work implement having a work tool and a support member for supporting the work tool, an image acquisition unit mounted on the work vehicle that acquires images captured by a plurality of imaging devices; a display control unit that outputs a display signal to the display device to cause the display device to display an overhead image of the surroundings of the work vehicle, the overhead image being created by transforming and combining the images captured by the multiple imaging devices, in a smaller size than the captured image and superimposed on the captured image; A display control system comprising:

2. a layout storage unit that stores layout information that determines an arrangement of the overhead image on the display device; the display control unit generates the display signal based on the layout information. The display control system according to claim 1 .

3. the display control unit outputs a display signal to the display device to cause the display device to display the posture image of the work vehicle and the captured image.

3. The display control system according to claim 1.

4. The display control unit outputs a display signal to the display device to cause the display device to display the bucket interior image and the captured image.

3. The display control system according to claim 1.

5. A display control method for controlling a display device provided on the outside of a work vehicle equipped with a work implement having a work tool and a support member that supports the work tool, comprising: acquiring images captured by a plurality of imaging devices mounted on the work vehicle; a step of displaying, on the display device, an overhead image showing the surroundings of the work vehicle, the overhead image being created by transforming and combining the images captured by the plurality of imaging devices, the overhead image being smaller than the captured image and being superimposed on the captured image; A display control method comprising:

Images