Control system for work machine and control method for work machine

Abstract

This control system 1 for a work machine comprises a plurality of imaging devices 128 that capture images of a site where a work machine 100 comprising a work apparatus performs construction, a communication device 129 that transmits the images captured by the imaging devices 128 to a remote operation room where the work machine 100 is remotely operated, a display control unit 221 that displays the images captured by the imaging devices 128 on a display device 200 disposed in the remote operation room, and a work machine control unit 17 that controls the movement of the work machine 100, wherein: the display control unit 221 changes and displays at least one image among the images displayed on the display device 200; and the work machine control unit 17 controls the movement of the work machine 100 when the at least one image among the images displayed on the display device 200 is changed.

Description

Control System and Control Method of Working Machine

[0001] The present disclosure relates to a control system and a control method of a working machine.

[0002] In the technical field related to working machines, in a working machine equipped with a front working device as disclosed in Patent Document 1, when evaluating the communication status of wireless communication from a remote device and restricting the operation of the working machine according to the evaluation result, a technique is known in which the restriction of the operation of the working machine is relaxed according to the detection result of a state quantity detection device that detects a state quantity related to the operation state of the working device.

[0003] Japanese Patent Application Laid-Open No. 2022-145114

[0004] In the remote operation of a working machine, images captured by a plurality of imaging devices arranged on the working machine are displayed on a display device in a remote operation room. In the remote operation room, an operator operates the working machine while checking the periphery of the working machine by changing the images displayed on the display device as needed. When changing the images, if the images are not changed smoothly, it becomes difficult for the operator to check the periphery of the working machine.

[0005] An object of the present disclosure is to ensure safety when changing an image displayed on a display device in a remote operation room for remotely operating a working machine.

[0006] According to the present disclosure, there is provided a control system of a working machine, including: a plurality of imaging devices for imaging a site where the working machine equipped with a working implement performs construction; a communication device for transmitting an image captured by the imaging device to a remote operation room for remotely operating the working machine; a display control unit for causing the image captured by the imaging device to be displayed on a display device arranged in the remote operation room; and a working machine control unit for controlling the operation of the working machine, wherein the display control unit changes and displays at least one of the images displayed on the display device, and the working machine control unit controls the operation of the working machine when at least one of the images displayed on the display device is changed.

[0007] According to this disclosure, a method for controlling a work machine is provided, which includes: capturing images of a work site where a work machine equipped with a work implement is performing work using multiple imaging devices; transmitting the images captured by the imaging devices to a remote control room from which the work machine is operated remotely; displaying the images captured by the imaging devices on a display device located in the remote control room; and controlling the operation of the work machine, wherein the control system of the work machine modifies and displays at least one of the images displayed on the display device, and controls the operation of the work machine when at least one of the images displayed on the display device is modified.

[0008] According to this disclosure, safety can be ensured when changing the image displayed on a display device in a remote control room that remotely operates a work machine.

[0009] Figure 1 is a schematic diagram showing an example of a work machine. Figure 2 is a block diagram of the control system of the work machine according to the embodiment. Figure 3 is a block diagram of the controller of the work machine. Figure 4 is a block diagram of the computer system according to the embodiment. Figure 5 is a schematic diagram illustrating an example of a display mode. Figure 6 is a schematic diagram illustrating another example of a display mode. Figure 7 is a schematic diagram illustrating another example of a display mode. Figure 8 is a flowchart illustrating the control method of the work machine according to the embodiment.

[0010] The embodiments described below will be explained with reference to the drawings, but the disclosure is not limited thereto. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0011] [Embodiment] <Control System for Work Machinery> Figure 1 is a schematic diagram showing an example of work machine. In this embodiment, work machine 100 is a bulldozer. The control system 1 for work machine 100 allows an operator to remotely control work machine 100, which is located at the work site, from a remote control room located away from the work site. The control system 1 controls the operation of work machine 100 when at least one image displayed on the display device in the remote control room is changed.

[0012] In this embodiment, the control of the operation of the work machine 100 includes controlling the movement of the work machine 100 or changing the work mode. The work mode is, for example, an economy mode for operating the work machine 100 in an energy-saving manner, or a power mode selected when performing high-load work. The control of the movement of the work machine 100 includes stopping or decelerating. Deceleration also includes changing the gear ratio of the transmission. Stopping includes automatic braking.

[0013] In this embodiment, the image refers to both a moving image and a still image.

[0014] <Remote Control Room> Figure 2 is a block diagram showing the control system of a work machine according to the embodiment. The remote control room is located away from the work machine 100. The remote control room contains equipment for remotely operating the work machine 100. The remote control room is equipped with a display device 200, a remote control device 210, and a remote controller 220.

[0015] The display device 200 is installed in a management office or similar location, away from the work machine 100. The display device 200 is positioned independently of the work machine 100. Based on construction data, including a design surface cross-section showing the target shape of the work machine 100's construction target, the display device 200 displays an image of the design surface showing the target shape of the work machine 100's construction target. Based on video data, the display device 200 displays a real-time captured image of the work machine 100's construction target. Based on operation data and vehicle data, the display device 200 displays an image showing the real-time position of the cutting edge of the excavation work machine 110 or ripper work machine 114 of the work machine 100 relative to the construction target. The display device 200 includes a flat panel display such as a liquid crystal display or an organic EL display. The display device 200 may be a touch panel type. The display device 200 may also be a curved monitor or a projector.

[0016] The remote control device 210 consists of various control levers for remotely operating the work machine 100. The remote control device 210 is operated by an operator seated in a control seat located in the remote control room. The operator operates the remote control device 210 while viewing the display screen of the display device 200.

[0017] Operation data indicating the operations performed on each operating lever of the remote control device 210 is transmitted to the controller 10 via the communication system through the remote controller 220, the base station 250, and the communication device 129 of the work machine 100.

[0018] The remote control device 210 includes a work lever operated for the operation of the excavation work machine 110 and the ripper work machine 114, and a travel pedal operated for the operation of the travel device 102.

[0019] The remote controller 220 is connected to the display device 200 and the remote control device 210. The remote controller 220 includes a numerical processing unit (processor) such as a CPU (Central Processing Unit).

[0020] The remote controller 220 receives video data and vehicle data from the work machine 100 via the base station 250 through the communication system. The remote controller 220 transmits operation data indicating operations performed on the remote control device 210 to the controller 10 of the work machine 100 via the base station 250 through the communication system. The remote controller 220 transmits delay status data, which allows detection of interruptions or delays in the video display on the display device 200 of the remote control device 210, to the controller 10 of the work machine 100 via the base station 250 through the communication system. The remote controller 220 receives construction data from the server device 230 via the base station 250. The remote controller 220 transmits video data, vehicle data, and construction data to the display device 200.

[0021] The delay status data may be, for example, data that can detect the communication status (delay status) of video data received from the work machine 100 via the base station 250 by the communication system. The delay status data may also be, for example, data that can detect the interval of time between video data receptions.

[0022] The delay status data may be, for example, a video signal displayed on the display device 200.

[0023] The delay status data may, for example, be video data of the remote control room, captured by an imaging device (not shown) located in the remote control room, which captures the video displayed on the display device 200.

[0024] The remote controller 220 includes a display control unit 221. The display control unit 221 of the remote controller 220 will be described later.

[0025] <Server Device> The server device 230 is a construction data supply device that provides construction data to the control system 1 of the work machine 100. The server device 230 is installed in a location away from the work machine 100, such as a construction data design office. The server device 230 generates construction data and transmits it to the base station 250.

[0026] <Base Station> The base station 250 is located away from the work machine 100. The base station 250 connects the work machine 100, which is located at the work site, with the server device 230 and remote controller 220, which are located away from the work site, via a communication system, enabling data communication.

[0027] The base station 250 receives video data and vehicle data from the work machine 100 via the communication system. The base station 250 receives construction data from the server device 230. The base station 250 transmits the video data, vehicle data, and construction data to the remote controller 220.

[0028] The base station 250 receives operation data and delay status data from the remote controller 220. The base station 250 transmits the operation data and delay status data to the work machine 100 via the communication system.

[0029] <Work Machinery> As shown in Figure 1, the work machine 100 comprises a vehicle body 101, a traveling device 102, an excavation work machine 110, and a ripper work machine 114.

[0030] The running gear 102 supports the vehicle body 101 and moves. The running gear 102 has a pair of tracks 103. The work machine 100 moves as the tracks 103 rotate.

[0031] The excavation work machine 110 performs excavation work, soil pushing work, or leveling work on the target area. The excavation work machine 110 is attached to the vehicle body 101. At least a part of the excavation work machine 110 is positioned in front of the vehicle body 101. The excavation work machine 110 has an excavation blade 111, a lift frame 112, a tilt cylinder 111a, and a lift cylinder 111b.

[0032] The drilling blade 111 is positioned in front of the vehicle body 101.

[0033] The lift frame 112 supports the drilling blade 111. One end of the lift frame 112 is connected to the back of the drilling blade 111 via a rotating mechanism. The other end of the lift frame 112 is connected to the vehicle body 101 via a rotating mechanism. Alternatively, the other end of the lift frame 112 may be connected to the traveling device 102 via a rotating mechanism.

[0034] The tilt cylinder 111a and the lift cylinder 111b operate the drilling blade 111. The tilt cylinder 111a drives the drilling blade 111 to tilt. The lift cylinder 111b drives the drilling blade 111 to lift. The tilt operation of the drilling blade 111 refers to the operation of changing the tilt angle of the drilling blade 111. The lift operation of the drilling blade 111 refers to the operation of moving the drilling blade 111 in the vertical direction.

[0035] One end of the tilt cylinder 111a is connected to the back of the drilling blade 111 via a rotation mechanism. The other end of the tilt cylinder 111a is connected to the upper surface of the lift frame 112. The drilling blade 111 tilts as the tilt cylinder 111a extends and retracts.

[0036] One end of the lift cylinder 111b is connected to the lift frame 112 via a rotating mechanism. The other end of the lift cylinder 111b is connected to the vehicle body 101 via a rotating mechanism. The extension and retraction of the lift cylinder 111b causes the drilling blade 111 to lift.

[0037] The ripper implement 114 performs ripping work on the work target. The work target of the ripper implement 114 includes the ground at the work site. The ripper implement 114 is attached to the vehicle body 101. At least a portion of the ripper implement 114 is positioned at the rear of the vehicle body 101.

[0038] The ripper work machine 114 includes a shank 115, a ripper arm 116, a tilt cylinder 115a, a lift cylinder 115b, and a beam 117.

[0039] The shank 115 is positioned at the rear of the vehicle body 101.

[0040] The ripper arm 116 supports the shank 115. The ripper arm 116 connects the vehicle body 101 and the shank 115. One end of the ripper arm 116 is connected to the rear of the vehicle body 101 via a pivot mechanism. The other end of the ripper arm 116 is connected to the beam 117.

[0041] The beam 117 is rotatably connected to the ripper arm 116. The shank 115 is connected to the ripper arm 116 via the beam 117.

[0042] The tilt cylinder 115a and the lift cylinder 115b operate the shank 115. The tilt cylinder 115a and the lift cylinder 115b are connected to the vehicle body 101. The tilt cylinder 115a drives the shank 115 to tilt. The lift cylinder 115b drives the shank 115 to lift. The tilt operation of the shank 115 refers to the operation of changing the tilt angle of the shank 115. The lift operation of the shank 115 refers to the operation of moving the shank 115 in the vertical direction.

[0043] One end of the tilt cylinder 115a is connected to the beam 117 via a rotation mechanism. The other end of the tilt cylinder 115a is connected to the rear part of the vehicle body 101. When the tilt cylinder 115a expands and contracts, the shank 115 tilts.

[0044] One end of the lift cylinder 115b is connected to the beam 117 via a rotation mechanism. The other end of the lift cylinder 115b is connected to the rear part of the vehicle body 101. When the lift cylinder 115b expands and contracts, the shank 115 lifts.

[0045] When there is no need to distinguish between the tilt cylinder 111a, the lift cylinder 111b, the tilt cylinder 115a, and the lift cylinder 115b, they are collectively referred to as hydraulic cylinders.

[0046] The main valves 120 of the excavation work machine 110 and the ripper work machine 114 are connected to a hydraulic pump (not shown). The hydraulic pump supplies hydraulic oil to each of the cylinders via the main valve 120. The main valve 120 has a spool. When the spool of the main valve 120 moves, the direction and flow rate of the hydraulic oil supplied from the hydraulic pump to the hydraulic cylinder are adjusted.

[0047] The main valve 120 can extend or contract the tilt cylinder 111a, for example, by adjusting the direction of the hydraulic oil supplied to the tilt cylinder 111a based on a solenoid control command from the controller 10. The main valve 120 can adjust the operating speed of the tilt cylinder 111a by adjusting the flow rate of the hydraulic oil supplied to the tilt cylinder 111a based on a solenoid control command from the controller 10. The same applies to the lift cylinder 111b, the tilt cylinder 115a, and the lift cylinder 115b.

[0048] The GNSS receiver 121 detects the position of the working machine 100 in the global coordinate system by using GNSS (Global Navigation Satellite System). The position of the working machine 100 in the global coordinate system is referred to as vehicle body position data. The GNSS receiver 121 transmits the vehicle body position data to the controller 10.

[0049] The GNSS antenna 122 is a GNSS antenna arranged on the working machine 100. The GNSS antenna 122 is at least one GNSS antenna.

[0050] The working machine attitude sensor 125 detects the attitudes of the excavation working machine 110 and the ripper working machine 114 in the local coordinate system respectively. The attitude of the excavation working machine 110 includes the angle of the excavation working machine 110. The attitude of the ripper working machine 114 includes the angle of the ripper working machine 114. The working machine attitude sensor 125 transmits the detected working machine attitude data to the controller 10.

[0051] In the embodiment, the working machine attitude sensor 125 is a stroke sensor arranged on the hydraulic cylinder. The hydraulic cylinder has a cylinder tube, a piston that moves inside the cylinder tube, and a rod connected to the piston. The stroke sensor detects the stroke length of the hydraulic cylinder indicating the moving distance of the rod. The stroke length refers to the moving distance of the rod from the stroke end of the hydraulic cylinder. The stroke end refers to the end position of the movable range of the rod. That is, the stroke end refers to the position of the rod when the hydraulic cylinder is in the most contracted state or the position of the rod when the hydraulic cylinder is in the most extended state. Note that the working machine attitude sensor 125 may be constituted by an inertial measurement unit (IMU: Inertial Measurement Unit).

[0052] The working machine attitude sensor 125 includes a stroke sensor arranged on the tilt cylinder 111a. The working machine attitude sensor 125 detects the stroke length of the tilt cylinder 111a.

[0053] The work machine attitude sensor 125 includes a stroke sensor located on the lift cylinder 111b. The work machine attitude sensor 125 detects the stroke length of the lift cylinder 111b.

[0054] The workpiece attitude sensor 125 includes a stroke sensor positioned on the tilt cylinder 115a. The workpiece attitude sensor 125 detects the stroke length of the tilt cylinder 115a.

[0055] The work machine attitude sensor 125 includes a stroke sensor located on the lift cylinder 115b. The work machine attitude sensor 125 detects the stroke length of the lift cylinder 115b.

[0056] The vehicle attitude sensor 126 detects the attitude of the vehicle body 101 in the local coordinate system. The attitude of the vehicle body 101 includes the roll angle and pitch angle, which are the inclination angles of the vehicle body 101 with respect to the horizontal plane. The vehicle attitude sensor 126 includes an inertial measurement unit (IMU) installed on the vehicle body 101. The vehicle attitude sensor 126 transmits the detected vehicle attitude data to the controller 10.

[0057] The imaging device 128 captures images of the work site where the work machine 100 is performing work. The imaging device 128 captures images of the work site and acquires images of the work site. The imaging device 128 is, for example, positioned on the vehicle body 101. The video data of the images captured by the imaging device 128 is transmitted to the video controller 20.

[0058] The imaging device 128 includes an optical system and an image sensor that receives light that has passed through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0059] The imaging device 128 consists of multiple cameras. In this embodiment, the imaging device 128 includes a front camera 128a, a rear camera 128b, a left camera 128c, a right camera 128d, a left blade camera 128e, a right blade camera 128f, a left ripper camera 128g, and a right ripper camera 128h.

[0060] The front camera 128a is an imaging device that captures images of the area in front of the work machine 100. The front camera 128a is positioned, for example, on the upper front of the vehicle body 101, facing forward.

[0061] The rear camera 128b is an imaging device that captures images of the area behind the work machine 100. The rear camera 128b is positioned, for example, below and behind the vehicle body 101, facing backward.

[0062] The left-side camera 128c is an imaging device that captures images of the left side of the work machine 100. The left-side camera 128c is positioned, for example, on the upper left rear of the vehicle body 101, facing left.

[0063] The right-side camera 128d is an imaging device that captures images of the area to the right of the work machine 100. The right-side camera 128d is positioned, for example, to the right of the upper right rear of the vehicle body 101, facing to the right.

[0064] The left blade camera 128e is an imaging device that captures images of the left side of the excavation blade 111 of the excavation work machine 110. The left blade camera 128e is positioned, for example, on the upper left rear of the vehicle body 101, facing forward.

[0065] The blade right camera 128f is an imaging device that captures images of the area to the right of the drilling blade 111 of the drilling work machine 110. The blade right camera 128f is positioned, for example, facing forward to the upper right rear of the vehicle body 101.

[0066] The left ripper camera 128g is an imaging device that captures images of the left side of the ripper work machine 114. The left ripper camera 128g is positioned, for example, facing rearward at the lower left rear of the vehicle body 101.

[0067] The ripper right camera 128h is an imaging device that captures images to the right of the ripper work machine 114. The ripper right camera 128h is positioned, for example, facing rearward, at the lower right rear of the vehicle body 101.

[0068] The communication device 129 receives video data of images captured by the imaging device 128. The communication device 129 receives vehicle data from the controller 10. The communication device 129 transmits the video data and vehicle data to the remote controller 220 in the remote control room via the base station 250 using the communication system.

[0069] The communication device 129 receives operation data and delay status data from the remote controller 220 via the base station 250 through the communication system. The communication device 129 also receives construction data from the server device 230 via the base station 250 through the communication system. The communication device 129 transmits the operation data, construction data, and delay status data to the controller 10.

[0070] <Video Controller> The video controller 20 includes a numerical processing unit (processor) such as a CPU. The video controller 20 is located in the work machine 100. The video controller 20 transmits video data acquired from the imaging device 128 to the communication device 129.

[0071] <Controller> Figure 3 is a block diagram of the controller of the work machine. The controller 10 includes a video status detection unit 11, a transmission status detection unit 13, a stop determination unit 15, a work machine control unit 17, and a storage unit 19.

[0072] Figure 4 is a block diagram illustrating a computer system according to an embodiment. The controller 10, video controller 20, and remote controller 220 include a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the controller 10, video controller 20, and remote controller 220 described above are stored as programs in the storage 1003. The processor 1001 reads the programs from the storage 1003, loads them into the main memory 1002, and executes the above-mentioned processes according to the programs. The programs may be distributed to the computer system 1000 via a network.

[0073] Returning to Figure 2, the controller 10 receives vehicle position data from the GNSS receiver 121. The controller 10 receives work equipment attitude data from the work equipment attitude sensor 125. The controller 10 receives vehicle attitude data from the vehicle attitude sensor 126. The controller 10 transmits vehicle data, including vehicle position data, work equipment attitude data, and vehicle attitude data, to the communication device 129. The controller 10 receives operation data, construction data, and delay status data from the communication device 129. The controller 10 transmits a solenoid control command to the main valve 120.

[0074] The work machine 100 operates based on operation data from a remote location. The controller 10 operates the work machine 100 based on operation data acquired from the remote control device 210 via a communication system. The operation of the work machine 100 includes at least one of the following: the operation of the travel device 102, the operation of the excavation work machine 110, and the operation of the ripper work machine 114.

[0075] Returning to Figure 3, the video status detection unit 11 detects the update status (display status) of the video on the display device 200 located in the remote control room. In this embodiment, the video status detection unit 11 detects the delay status of the video update when the video displayed on the display device 200 is updated. From the delay status data, the video status detection unit 11 detects whether the display of the video on the display device 200 is interrupted or delayed. In this embodiment, the delay status detection process is performed by the transmission status detection unit 13.

[0076] For example, if the delay status data is data indicating the interval between video data reception times, the delay status detection process is performed in the transmission status detection unit 13, which will be described later, in this embodiment.

[0077] The video status detection unit 11 may, for example, detect a delay if the delay status data is a video signal, and the video signal is not updated when it is above the first time threshold and below the second time threshold, and detect an interruption if it is above the second time threshold and is not updated. The first time threshold is a value smaller than the second time threshold. The first time threshold and the second time threshold can be set arbitrarily.

[0078] The video status detection unit 11 may, for example, detect a delay if the video data of the remote control room, captured by an imaging device (not shown) located in the remote control room, is equal to or greater than the first time threshold and less than the second time threshold, and detect an interruption if it is equal to or greater than the second time threshold and does not change. The first time threshold is a value smaller than the second time threshold. The first and second time thresholds can be set arbitrarily.

[0079] The transmission status detection unit 13 detects the data transmission status between the work machine 100 and the remote control room. The transmission status detection unit 13 also detects data interruptions or delays between the work machine 100 and the remote control room.

[0080] An example of how the transmission status is detected by the transmission status detection unit 13 will be described when the transmitted data is an image, but it is not limited to this. The same method can be applied to data other than images. The transmission status detection unit 13 reconstructs an encoded image from the packet received by the receiving side (in this case, the remote control room) and identifies the transmission status of the image based on the encoded image and the encoded image stored by the transmitting side (in this case, the work machine 100). The transmission status detection unit 13 detects the transmission status of the data based on the delay time, which is the difference between the transmission time stored by the data transmitting side (work machine 100 or the remote control room) and the time when the data was reconstructed by the receiving side. The transmission status detection unit 13 compares the encoded image reconstructed from the received packet with the encoded image stored by the transmitting side and identifies an encoded image that is identical to the received encoded image. The difference between the transmission time stored by the transmitting side in association with the encoded image and the time when the encoded image was reconstructed from the received packet is calculated as the round-trip delay time of the image. The transmission status detection unit 13 calculates the one-side delay time of the image by dividing the round-trip delay time of the image by 2.

[0081] Image one-way delay refers to the delay time from when an image is captured by the imaging device 128 until it is displayed on the display device 200. Image one-way delay is information that represents the transmission status of the image. If no packet loss occurs, the image one-way delay is equal to the one-way delay of a single packet. If packet retransmission occurs due to packet loss, the worse the communication environment, the longer the image one-way delay will be compared to the one-way delay of a single packet.

[0082] The transmission status detection unit 13 determines that transmission is good, or in other words, there is no interruption or delay, if the calculated one-sided delay time of the image is less than the first delay threshold. The transmission status detection unit 13 determines that there is a delay if the calculated one-sided delay time of the image is greater than or equal to the first delay threshold and less than the second delay threshold. The transmission status detection unit 13 determines that transmission is interrupted if the calculated one-sided delay time of the image is greater than or equal to the second delay threshold. The first delay threshold is a value smaller than the second delay threshold. The first and second delay thresholds can be set arbitrarily.

[0083] The work machine control unit 17 outputs various control signals for automatically controlling the work machine 100. The work machine control unit 17 controls the work machine 100 based on construction data, vehicle position data, vehicle data, and operation data.

[0084] The work machine control unit 17 controls the operation of the work machine 100, for example, by controlling its movement. When at least one of the images displayed on the display device 200 is changed, the work machine control unit 17 controls the operation of the work machine 100, for example, by controlling its movement.

[0085] In this embodiment, when the work machine control unit 17 changes at least one of the images displayed on the display device 200, if it determines that the image has been interrupted or delayed, it may control the operation of the work machine 100, for example, by controlling it to move.

[0086] In this embodiment, when the work machine control unit 17 determines that the transmission status is interrupted or delayed when changing at least one of the images displayed on the display device 200, it may control the operation of the work machine 100, for example, by controlling it to move.

[0087] In this embodiment, the work machine control unit 17 may activate the parking lock after stopping the work machine 100. By activating the parking lock after stopping the work machine 100, the operator must release the parking lock in order to start the work machine 100 again. This prevents the operator from unintentionally releasing the automatic brake on the work machine 100. Alternatively, another lock that stops the work machine 100 may be activated, and the operator may release that other lock in order to start the work machine 100 again.

[0088] In this embodiment, the work machine control unit 17 may control the movement of the work machine 100 if at least one of the images displayed on the display device 200 changes while the work machine 100 is in motion, even if it is not determined that the image or transmission status has been interrupted or delayed.

[0089] The storage unit 19 stores work machine posture data. The storage unit 19 stores vehicle body posture data. The storage unit 19 stores construction data, including design plane cross-sections, acquired from the server device 230. The storage unit 19 stores operation data acquired from the display device 200 located in the remote control room.

[0090] <Display Control Unit of Remote Controller> The display control unit 221 of the remote controller 220 controls the display of various images on the display device 200 in the remote control room. Based on the construction data, the display control unit 221 generates a display signal to display an image showing the cross-section of the design plane on the display device 200. For example, the display control unit 221 generates a display signal to display an image related to the guidance of the work machine 100, along with an image of the work machine 100, based on the construction data.

[0091] The display control unit 221 causes the image captured by the imaging device 128 to be displayed on the display device 200 located in the remote control room.

[0092] The display control unit 221 changes and displays at least one of the images displayed on the display device 200. The display control unit 221 may change and display at least one of the images displayed on the display device 200 based on, for example, an operator's operation of the remote control device 210 in the remote control room. The display control unit 221 may also automatically change and display at least one of the images displayed on the display device 200 in accordance with the operation of the work machine 100.

[0093] In this embodiment, the display control unit 221 has set a combination of multiple images to be displayed simultaneously on the display device 200 as a display mode, and may change the display on a display mode basis.

[0094] The display mode is set according to the operation of the work machine 100, for example.

[0095] The display modes will be explained using Figures 5 to 7. Figure 5 is a schematic diagram illustrating one example of a display mode. Figure 6 is a schematic diagram illustrating another example of a display mode. Figure 7 is a schematic diagram illustrating another example of a display mode. Figure 5 is an example of an image combination for the "display mode during driving and stopping". The image for the "display mode during driving and stopping" consists of a front image A11 captured by the front camera 128a, a rear image A12 captured by the rear camera 128b, a left image A13 captured by the left camera 128c, and a right image A14 captured by the right camera 128d. On the display screen, the front image A11 is located at the top center, the rear image A12 is located at the bottom center, the left image A13 is located on the left, and the right image A14 is located on the right.

[0096] Figure 6 shows an example of an image combination for the "Excavation Display Mode". The "Excavation Display Mode" image consists of a front image A11 captured by the front camera 128a, a rear image A12 captured by the rear camera 128b, a left blade image A15 captured by the left blade camera 128e, and a right blade image A16 captured by the right blade camera 128f. On the display screen, the front image A11 is located in the upper left, the rear image A12 is located in the lower right, the left blade image A15 is located in the lower left, and the right blade image A16 is located in the lower center.

[0097] Figure 7 shows an example of an image combination for the "display mode during ripping." The image for the "display mode during ripping" consists of a front image A11 captured by the front camera 128a, a rear image A12 captured by the rear camera 128b, a left ripper point image A17 captured by the left ripper camera 128g, and a right ripper point image A18 captured by the right ripper camera 128h. On the display screen, the front image A11 is located in the upper center, the rear image A12 is located in the lower right, the left ripper point image A17 is located in the lower left, and the right ripper point image A18 is located in the lower center.

[0098] In this embodiment, the display control unit 221 may display an indication that the video is interrupted or delayed if the video is interrupted or delayed.

[0099] An indication that the video is interrupted or delayed may be, for example, by filling the entire display screen of the display device 200 or the display area where the video is displayed with a single color such as green or blue.

[0100] An indication that the video is interrupted or delayed may be, for example, a message image or icon image indicating that the video is interrupted or delayed superimposed on the display screen of the display device 200.

[0101] <Control Method> Figure 8 is a flowchart showing the control method of the work machine according to the embodiment. When the work machine 100 is turned on, the control system 1 of the work machine 100 is activated. When the control system 1 of the work machine 100 is activated, various data are transmitted and received between the work machine 100 and the remote control room via the communication system. When the control system 1 of the work machine 100 is activated and an operation to change the image on the display device 200 in the remote control room is detected, the processing shown in the flowchart in Figure 8 is started.

[0102] The controller 10 detects the update status (display status) of the video on the display device 200 in the remote control room using the video status detection unit 11 (step ST11). The controller 10 then proceeds to step ST12.

[0103] The controller 10 detects the data transmission status between the work machine 100 and the remote control room using the transmission status detection unit 13 (step ST12). The controller 10 then proceeds to step ST13.

[0104] The controller 10 determines whether the video signal has been interrupted or delayed (step ST13). If the controller 10 determines that the video signal has been interrupted or delayed (Yes in step ST13), it proceeds to step ST14. If the controller 10 does not determine that the video signal has been interrupted or delayed (No in step ST13), it terminates the process of this flowchart.

[0105] If the controller determines that the video signal has been interrupted or delayed (Yes in step ST13), the controller 10 causes the work machine control unit 17 to control the operation of the work machine 100. As part of controlling the operation of the work machine 100, for example, the automatic brake is activated (step ST14). Note that the controller 10 may activate the automatic brake of the work machine 100 regardless of the determination result of whether the video signal has been interrupted or delayed.

[0106] Thus, the control method for the work machine 100 executed by the control system 1 of the work machine 100 includes: capturing images of the work site where the work machine 100 equipped with the work machine is performing work using a plurality of imaging devices 128; transmitting the images captured by the imaging devices 128 to a remote control room from which the work machine 100 is operated remotely; displaying the images captured by the imaging devices 128 on a display device 200; and controlling the operation of the work machine 100. The control method for the work machine 100 includes changing and displaying at least one of the images displayed on the display device 200, and when at least one of the images displayed on the display device 200 is changed, the movement of the work machine 100 is controlled.

[0107] <Effects> As described above, in this embodiment, when at least one of the images displayed on the display device 200 located in the remote control room is changed, the operation of the work machine 100 can be controlled, for example, by controlling its movement. According to this embodiment, when an image is changed, if the image is not changed smoothly and it becomes difficult for the operator to check the area around the work machine 100, the work machine 100 can be prevented from operating, for example, by moving. Thus, according to this embodiment, the safety of the work machine 100 can be ensured. Furthermore, according to this embodiment, when an image is changed, the work machine 100 can be prevented from operating, for example, by moving.

[0108] In this embodiment, when changing at least one image among those displayed on the display device 200, if it is determined that the image has been interrupted or delayed, the operation of the work machine 100 can be controlled, for example, by controlling its movement. According to this embodiment, when it is difficult for the operator to check the surroundings of the work machine 100 when changing the image, the operation of the work machine 100 can be suppressed, for example, by preventing it from moving.

[0109] In this embodiment, when changing at least one image among those displayed on the display device 200, if it is determined that the transmission status is interrupted or delayed, the operation of the work machine 100 can be controlled, for example, by controlling its movement. According to this embodiment, when it is difficult for the operator to check the surroundings of the work machine 100 when changing an image, the operation of the work machine 100 can be suppressed, for example, by preventing it from moving.

[0110] In this embodiment, the operation of the work machine 100 can be controlled by stopping or slowing down its movement. According to this embodiment, when it is difficult for the operator to check the area around the work machine 100 when the image is changed, the safety of the work machine 100 can be ensured.

[0111] In this embodiment, a combination of multiple images to be displayed simultaneously on the display device 200 is set as a display mode, and the display can be changed on a display mode basis. According to this embodiment, the operator can easily change the display.

[0112] In this embodiment, the parking lock can be activated after the work machine 100 has stopped. According to this embodiment, it is possible to avoid the operator unintentionally releasing the automatic brake on the work machine 100. According to this embodiment, the safety of the work machine 100 can be ensured.

[0113] In this embodiment, when the work machine 100 is in motion, the movement of the work machine 100 can be controlled if at least one of the images displayed on the display device 200 is changed.

[0114] [Modification] In the above embodiment, the server device 230 does not need to generate construction data and transmit it to the work machine 100 via the base station 250. In this case, the display control unit 221 does not need to generate a display signal based on the construction data to display an image showing the cross-section of the design plane on the display device 200.

[0115] The display control unit 221 may synthesize images captured by multiple cameras to generate an overhead view image showing the surroundings of the work machine 100. In this case, the display control unit 221 may change at least one of the images displayed on the display device 200 to the overhead view image. Alternatively, the display control unit 221 may change the overhead view image displayed on the display device 200 to an image captured by at least one camera. When the overhead view image displayed on the display device 200 is changed to an image captured by at least one camera, the work machine control unit 17 may control the operation of the work machine 100 if it determines that the image or transmission status has been interrupted or delayed, or even if it does not determine that the transmission status has been interrupted or delayed.

[0116] In the embodiment described above, the imaging device 128 includes a front camera 128a, a rear camera 128b, a left camera 128c, a right camera 128d, a left blade camera 128e, a right blade camera 128f, a left ripper camera 128g, and a right ripper camera 128h. However, the number and placement of the imaging devices may be changed as appropriate. For example, instead of the left ripper camera 128g and the right ripper camera 128h, a camera for imaging the side of the ripper and a camera for imaging the work object of the ripper work machine 114 may be provided.

[0117] In the above-described embodiment, the imaging device 128 may be a single camera. In this case, the display control unit 221 can change the range to be displayed on the display device 200 from the imaging range determined by the field of view. The display control unit 221 can change from a first display range, which indicates a predetermined range from the imaging range determined by the field of view, to a second display range that is different from the first display range. The work machine control unit 17 may control the operation of the work machine 100 when the display range of the image displayed on the display device 200 is changed, or when it is determined that the image or transmission status has been interrupted or delayed, or even when it is not determined that the transmission status has been interrupted or delayed.

[0118] In the above-described embodiment, the display control unit 221 can change the position of at least one image among the images displayed on the display device 200. For example, the position of the front image A11 captured by the front camera 128a can be changed from the upper center to the upper left of the display screen. The image whose position can be changed is not limited to the front image A11. The number of images whose position can be changed may be two or more. The positions that can be changed are not limited to the above example. When the position of an image displayed on the display device 200 is changed, the work machine control unit 17 may control the operation of the work machine 100 when it is determined that the image or transmission status has been interrupted or delayed, or even when it is not determined that the transmission status has been interrupted or delayed.

[0119] In the embodiment described above, the remote controller 220 transmits operation data to the base station 250. However, a controller other than the remote controller 220 installed in the remote control room, such as a remote control controller, may transmit the operation data to the base station 250.

[0120] In the above description, the processing of the video situation detection unit 11 was explained as being implemented in the controller 10 of the work machine 100, but it is not limited to this. The processing of the video situation detection unit 11 may also be implemented in the remote controller 220.

[0121] In the embodiments described above, the work machine is not limited to a bulldozer. The work machine can be a hydraulic excavator, a dump truck, a wheel loader, or a control system for other work machines.

[0122] 1...Control system, 10...Controller, 11...Video status detection unit, 13...Transmission status detection unit, 15...Stop determination unit, 17...Work machine control unit, 19...Storage unit, 20...Video controller, 100...Work machine, 101...Body, 102...Running gear, 103...Tracks, 110...Excavating work machine, 111...Excavating blade, 111a...Tilt cylinder, 111b...Lift cylinder, 112...Lift frame, 114...Ripper work machine, 115...Sha 115a...Tilt cylinder, 115b...Lift cylinder, 116...Ripper arm, 117...Beam, 120...Main valve, 121...GNSS receiver, 122...GNSS antenna, 125...Work equipment attitude sensor, 126...Vehicle attitude sensor, 128...Imaging device, 129...Communication device, 200...Display device, 210...Remote control device, 220...Remote controller, 221...Display control unit, 230...Server device, 250...Base station.

Claims

1. A control system for a work machine comprising: an imaging device that takes images of the work site where the work machine is performing work; a communication device that transmits the images taken by the imaging device to a remote control room from which the work machine is operated remotely; a display control unit that displays the images taken by the imaging device on a display device located in the remote control room; and a work machine control unit that controls the operation of the work machine, wherein the display control unit changes and displays the image shown on the display device, and the work machine control unit controls the operation of the work machine when the image shown on the display device is changed.

2. The control system for a work machine according to claim 1, wherein the work machine control unit controls the operation of the work machine when it determines that at least one of the images displayed on the display device has been interrupted or delayed.

3. A control system for a work machine according to claim 2, comprising: a transmission status detection unit for detecting the data transmission status between the work machine and the remote control room, wherein the work machine control unit controls the operation of the work machine when it determines that the transmission status is interrupted or delayed when changing at least one of the images displayed on the display device.

4. The control of the operation of the work machine is to stop or decelerate, as described in claim 1.

5. The control system for a work machine according to claim 1, wherein the display control unit has set a combination of a plurality of images to be displayed simultaneously on the display device as a display mode, and changes the display on a display mode basis.

6. The control system for a work machine according to claim 1, wherein the work machine control unit activates a parking lock after the work machine has stopped.

7. The control system for a work machine according to claim 1, wherein the work machine control unit controls the movement of the work machine when at least one of the images displayed on the display device is changed while the work machine is in motion.

8. The control system for the work machine according to claim 1, wherein a plurality of imaging devices are arranged.

9. The control system for a work machine according to claim 1, wherein the work machine is a bulldozer.

10. A method for controlling a work machine, comprising: capturing images of a work site where a work machine equipped with a work implement is performing work using multiple imaging devices; transmitting the images captured by the imaging devices to a remote control room from which the work machine is operated remotely; displaying the images captured by the imaging devices on a display device located in the remote control room; and controlling the operation of the work machine, wherein the control system for the work machine modifies and displays at least one of the images displayed on the display device, and controls the operation of the work machine when at least one of the images displayed on the display device is modified.

11. A method for controlling a work machine according to claim 10, wherein when it is determined that at least one of the images displayed on the display device has been interrupted or delayed, the operation of the work machine is controlled.

12. A method for controlling a work machine according to claim 11, comprising detecting the data transmission status between the work machine and the remote control room, and controlling the movement of the work machine when it is determined that the transmission status is interrupted or delayed when changing at least one of the images displayed on the display device.

13. The control method for a work machine according to claim 10, wherein the control of the operation of the work machine is to stop or decelerate its movement.

14. A control method for a work machine according to claim 10, wherein a combination of multiple images to be displayed simultaneously on the display device is set as a display mode, and the display is changed on a display mode basis.

15. A method for controlling a work machine according to claim 10, wherein a parking lock is activated after the work machine has stopped.

16. A method for controlling a work machine according to claim 10, wherein the operation of the work machine is controlled when at least one of the images displayed on the display device is changed while the work machine is in motion.

17. The method for controlling a work machine according to claim 10, wherein the work machine is a bulldozer.

Images