Control device and remote operation system of construction machine

Abstract

To provide a control device and a remote operation system of a construction machine to transport necessary images in high resolution and high frame rate through determining a camera with high priority on the basis of work details for the remote-controlled construction machine acquiring images of surroundings of a vehicle captured with a plurality of cameras.SOLUTION: A control device (21) of a construction machine (100), which is remotely controlled on the basis of an operation signal received from a remote operation device (30), transports images of surroundings of a vehicle captured by a plurality of on-vehicle cameras (17) and comprises: a nail-tip distance calculation section (213) for calculating a first nail-tip distance (La) between a bucket nail-tip (10a) and a design surface (42) of the construction machine; and an image processing section (214) for setting priority of images captured by the on-vehicle cameras according to the first nail-tip distance and operation amount of the work machine by the remote control device to adjust the resolution and frame rate of captured images captured by the plurality of on-vehicle cameras according to the priority.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a control device and a remote control system for a construction machine. [Background technology]

[0002] BACKGROUND ART With the development of wireless communication technology, it has become possible to transmit large amounts of data, and as a result, there has been active development of remote control technology for controlling automobiles, construction machinery, and the like from a remote location.

[0003] In remote control, a remote control device consisting of a monitor, control device, and communication terminal is installed at a remote location, and images from a camera attached to the aircraft that captures the area around the aircraft are sent to the control device and displayed on the monitor. The operator operates the control device while watching the monitor image. Control signals obtained from the control device are sent to the aircraft, and the aircraft is moved in response to the control signals. This is a common configuration.

[0004] One of the challenges facing remote control is the need to increase the volume, speed, and stability of the transmission volume and speed of video and control signals. If the amount of data that can be transmitted is small, the resolution of the video seen by the operator deteriorates, making it difficult to recognize the situation around the aircraft. If the transmission speed of video and control signals is slow, the operator will feel a delay between their own operations and the aircraft's movements, which can cause stress when operating the aircraft and increase the risk of operational errors. Furthermore, because transmission speeds vary depending on the line users and weather, it is also necessary to process transmitted data in accordance with line conditions. Various companies are conducting various studies to address the remote control issues mentioned above.

[0005] Patent Document 1 targets a hydraulic excavator with a turning function that supports remote control, and performs data processing using control signals for operations such as turning and traveling, which involve large amounts of image data, such as uniformly lowering the resolution and frame rate of the transmitted video when turning or traveling.

[0006] Patent Document 2 targets a vehicle that can be remotely controlled, and performs data processing by selecting a high-priority camera that needs to maintain high resolution and a high frame rate from three cameras mounted on the vehicle based on steering and blinker operation of the remote control device, the operator's gaze point, and obstacle detection, and then lowering the resolution and frame rate of the lower-priority camera to maintain data transmission from the higher-priority camera. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6581844 specification [Patent Document 2] Japanese Patent Publication No. 2023-012821 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology in Patent Document 1 uniformly reduces the amount of data transmitted, which can lead to a significant deterioration in resolution and stuttering due to a drop in frame rate, potentially hindering the transmission of information necessary for the operator.The technology in Patent Document 2 is superior to the technology in Patent Document 1 in that it estimates the information necessary for the operator and selectively limits the amount of data transmitted, but because it is a processing method specialized for driving automobiles, it is difficult to apply to construction machinery.

[0009] Therefore, the present invention aims to transmit the necessary images at high resolution and high frame rate in a remotely controlled construction machine that uses multiple cameras to capture the area around the vehicle by determining the camera with the highest priority based on the work content. [Means for solving the problem]

[0010] The control device for a construction machine of the present invention that solves the above problems comprises: A control device for a construction machine, comprising a control device that transmits images of the surroundings of the vehicle body captured by a plurality of on-board cameras to a remote control device and remotely controls the construction machine based on operation signals received from the remote control device, The control device a toe distance calculation unit that calculates a first toe distance between the bucket toe of the construction machine and a predetermined construction surface; an image processing unit that sets a priority for the images captured by the multiple on-board cameras according to the first toe distance and the amount of operation of the construction machine by the remote control device, and adjusts the resolution and frame rate of the images captured by the multiple on-board cameras according to the priority that is set to be higher as the first toe distance is shorter and the amount of operation is smaller; The present invention is characterized by having the following. [Effects of the Invention]

[0011] According to the present invention, a control device for a construction machine capable of transmitting necessary images at high resolution and a high frame rate in a remotely controlled construction machine that uses multiple cameras to capture images of the area around the vehicle body is provided.

[0012] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a left side view of the construction machine according to the first embodiment. [Figure 2] 1 is an overhead view of a construction machine according to a first embodiment. [Figure 3] FIG. 1 is a system configuration diagram of a construction machine according to a first embodiment. [Figure 4] FIG. 1 is a system configuration diagram of a remote control device according to a first embodiment. [Figure 5] 4 is a process flow for generating a camera image performed by the construction machine system according to the first embodiment. [Figure 6]10 is a graph showing the relationship between the operation amount and the actuator speed. [Figure 7] 10 is a graph showing the relationship between resolution and frame rate and the required transmission speed. [Figure 8] 10 is an example of a display on the remote control device when the toe distance is equal to or greater than a threshold value in the first embodiment. [Figure 9] 10 is an example of a display on the remote control device when the toe distance is equal to or less than a threshold value in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a modification of the first embodiment. [Figure 11] FIG. 10 is a diagram for explaining a situation in which determination is made on both the design surface and the current topography in the second embodiment. [Figure 12] 10 is a flowchart showing a process for determining both the design surface and the current topography in the second embodiment. [Figure 13] FIG. 10 is a diagram for explaining priority. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings, taking an example in which the embodiment of the present invention is applied to a remote control system for a construction machine.

[0015] [First embodiment] FIG. 1 shows a left side view of a hydraulic excavator according to the first embodiment. In this embodiment, a system for remotely operating a hydraulic excavator 100, which is an example of a construction machine, will be described. Note that the construction machine to which the present invention is applied is not limited to a hydraulic excavator, and may also include other construction machines such as a wheel loader.

[0016] As shown in FIG. 1 , the hydraulic excavator 100 has a lower traveling body 1, an upper rotating body 2 rotatably mounted on the lower traveling body 1, a cab 3 mounted at the front of the upper rotating body 2, a front working implement 4 mounted on the front side of the upper rotating body 2, and a counterweight 5 mounted at the rear of the upper rotating body 2.

[0017] The front working implement 4 has a boom 6, an arm 8, and a bucket 10, each of which is rotated by a corresponding hydraulic cylinder. The boom 6 is rotatably mounted on the front of the upper rotating body 2, and rotates when a hydraulic cylinder 7 for operating the boom extends and retracts. The arm 8 is rotatably mounted at the tip of the boom 6, and rotates when a hydraulic cylinder 9 for operating the arm extends and retracts. The bucket 10 is rotatably mounted at the tip of the arm 8, and rotates when a hydraulic cylinder 11 for operating the bucket extends and retracts.

[0018] A GNSS antenna 14 and a communication device 16 are attached to the upper rotating body 2. The GNSS antenna 14 receives signals transmitted from GNSS satellites and has two GNSS antennas, 14a and 14b. The GNSS antenna 14 is generally attached to the top surface of the upper rotating body 2 so that radio waves from the GNSS satellites are not blocked by other structures.

[0019] A swing device 18 that swings the upper swing body relative to the lower traveling body 1 is attached to the connecting portion between the upper swing body 2 and the lower traveling body 1. A traveling device 19 that allows the hydraulic excavator 100 to travel is attached to the lower traveling body 1. The swing device 18 and the traveling device 19 use hydraulic equipment such as a hydraulic motor that is driven by hydraulic pressure.

[0020] The hydraulic excavator 100 is configured to be able to select and switch between manned operation, in which the excavator is operated by an operator in the cab 3, and remote operation, in which the excavator is operated by an operator from a remote location away from the excavator 100.

[0021] FIG. 2 shows an overhead view of the construction machine in the first embodiment. On the upper rotating body 2, on-board cameras 17 are attached to the front, left, and right rear of the vehicle body to capture images of the surroundings of the hydraulic excavator 100. The multiple on-board cameras 17 capture images from an imaging range 171 of the front on-board camera, an imaging range 172 of the right on-board camera, an imaging range 173 of the left on-board camera, and an imaging range 174 of the rear on-board camera, respectively.

[0022] FIG. 3 shows a system configuration diagram of a hydraulic excavator according to the first embodiment. The hydraulic excavator 100 has a controller (construction machinery control device) 21 that controls each part, and the input side of the controller 21 is connected to an attitude detector 13, a GNSS antenna 14, multiple on-board cameras 17, a GNSS receiver 20, and a communication device 16.

[0023] The attitude detector 13 is attached to the bucket link 12, the arm 8, the smooth surface of the boom 6, and the upper rotating body 2, and detects the angle of each with respect to the ground. The attitude detector 13 uses, for example, a stroke sensor, an angle sensor, an IMU (Inertial Measurement Unit), etc.

[0024] The GNSS receiver 20 calculates the antenna azimuth angle from the position of the GNSS antenna 14a and the direction vector connecting the GNSS antennas 14a and 14b, based on signals received by the GNSS antenna 14a from GNSS (Global Navigation Satellite System) satellites and correction information acquired by the communication device 16. Furthermore, when parameters for converting from the global coordinate system to the on-site coordinate system are input from the communication device 16, the GNSS receiver 20 converts the position and direction vector into the on-site coordinate system.

[0025] The communication device 16 wirelessly communicates with external devices using a wireless communication network that has a communication standard such as 5G or Wi-Fi (registered trademark). The communication device 16 transmits to the remote control device 30 design surface data, parameters for converting from a global coordinate system to a site coordinate system, RTK (Real Time Kinematic) correction information, received information on operation signals from the remote control device 30, and images from the in-vehicle camera 17.

[0026] An engine 22, a hydraulic pump 23, a control valve 24, and a display device 25 are connected to the output side of the controller 21. The engine 22 is formed of, for example, an internal combustion engine, and is controlled based on control signals from the controller 21. The hydraulic pump 23 is driven by the engine and sends hydraulic oil to the control valve 24 and actuators such as the hydraulic cylinder 7 for boom operation. Based on control signals from the controller 21, the control valve 24 distributes and sends the hydraulic oil to the hydraulic cylinders 7, 9, and 11, the swing device 18, and the traveling device 19, thereby driving the hydraulic excavator 100. The display device 25 performs processing to display images captured by the multiple on-board cameras 17 on a monitor (not shown) provided in the cab 3.

[0027] The controller 21 is configured by an information processing device having an ECU (Electronic Control Unit) consisting of a CPU, memory, etc. The controller 21 realizes the functions of each section, which will be described later, by the CPU loading a program stored in the memory of the ECU into the memory and executing it. The controller 21 realizes the functions of each section, which will be described later, as an attitude / position calculation section 211, a design surface data management section 212, a toe distance calculation section 213, an image processing section 214, and a vehicle body control section 215.

[0028] The attitude / position calculation unit 211 calculates the three-dimensional position of the bucket toe 10a of the bucket 10 from the position and antenna azimuth angle of the GNSS antenna 14a calculated by the GNSS receiver 20, and the ground angles of the boom 6, arm 8, and bucket link 12 obtained from the attitude detector 13. This three-dimensional position is calculated as a position in the on-site coordinate system.

[0029] The design surface data management unit 212 manages design surface data, which is three-dimensional surface information configured with TIN (Triangular Irregular Network) acquired from the communication device 16. The design surface data includes information on the design surface (hereinafter also referred to as the construction surface) 42 of the area where construction work will be performed. The information on the design surface 42 includes information on the shape of the ground when construction work is completed according to the design drawings.

[0030] The toe distance calculation unit 213 calculates the vertical distance from the bucket toe 10a of the bucket 10 to the design surface 42 as the toe distance (first toe distance) La based on the bucket toe position calculated by the posture / position calculation unit 211 and the design surface data.

[0031] The video processing unit 214 determines the priority of the camera image of each vehicle-mounted camera 17 based on the vertical distance from the bucket toe 10a to the design surface 42 calculated by the bucket toe calculation unit 27 and the operation signal transmitted from the remote control device 30 acquired by the communication device 16. The priority is set to be higher as the toe distance is shorter and the operation amount is smaller. The resolution and frame rate of the camera image of each vehicle-mounted camera 17 are controlled according to the priority. Specifically, the resolution and frame rate of the camera image of a vehicle-mounted camera 17 determined to have a high priority are made higher compared to the resolution and frame rate of the camera image of the other vehicle-mounted cameras 17. The resolution and frame rate of the camera image of the other vehicle-mounted cameras 17 are made lower compared to the resolution and frame rate of the camera image of a vehicle-mounted camera 17 determined to have a high priority, thereby controlling the total communication volume used for the camera image between the vehicle-mounted camera 17 and the remote control device 30 to remain unchanged.

[0032] The vehicle body control unit 215 controls the engine 22 , the hydraulic pump 23 , and the control valve 24 based on operation signals input from the remote control device 30 or the cab 3 .

[0033] FIG. 4 shows a system configuration diagram of the remote control device according to this embodiment. The remote control device 30 is disposed at a location remote from the hydraulic excavator 100. The remote control device 30 is operated by an operator, and controls the hydraulic excavator 100 in response to the operation of the remote control device 30.

[0034] The remote control device 30 includes a communication device 31, a control device 32, a display 33, and an operation device 34. The communication device 31 receives images from the on-board camera 17 sent from the hydraulic excavator 100, and transmits operation signals generated by the remote control device 30 to the hydraulic excavator 100.

[0035] The control device 32 is an electronic control device having a central processing unit, a storage unit, an input / output unit, etc. The control device 32 realizes the functions of each unit, namely, a display control unit 321 and an operation signal generation unit 322, by the central processing unit reading and executing a software program stored in the storage unit. The display control unit 321 processes the camera image acquired from the communication device 16 and generates a video signal to be sent to the display 33. The operation signal generation unit 322 processes the operation input from the operation device 34 and generates an operation signal to be sent to the communication device 16.

[0036] The display 33 displays the video signal received from the display control unit 321, and is arranged, for example, facing the seat in front of the seat where the remote control device 30 is used. The display 33 may be installed by combining four display panels that respectively display front, rear, left, and right images, or may simultaneously display front, rear, left, and right images on a single display panel.

[0037] The operation device 34 is disposed near the seat of the cab 3. The operation device 34 may be of the same specifications as the operation lever or the like installed in the cab 3 of the hydraulic excavator 100, or may be a joystick, a game pad, or a touch panel instead of the operation lever.

[0038] The above-described remote control device 30 has been described as an example of a device installed at a location distant from the hydraulic excavator 100, but the configuration is not limited to the above and may be a portable configuration such as a mobile terminal. For example, a tablet terminal with a communication function may be used as a remote control device by installing a remote control application on the tablet terminal and executing the application.

[0039] FIG. 5 shows the processing flow for generating a camera image carried out by the construction machine system in the first embodiment. This process is performed by the video processing unit 214 of the controller 21. The video processing unit 214 has thresholds for determining the priority of the camera video for the operation amounts obtained from the fingertip distance and the operation signal, respectively. Here, the case where there are five types of priorities, namely "extremely high", "high", "medium", "low", and "extremely low", will be described. Note that "extremely high" among the types of priorities corresponds to "the third priority" in the claims, "high" corresponds to "the first priority" in the claims, "low" corresponds to "the second priority" in the claims, and "extremely low" corresponds to "the fourth priority" in the claims.

[0040] When the fingertip distance La is greater than or equal to the first distance threshold Lt1 (La ≧ Lt1) (NO in S511), the priorities of each video acquired by the plurality of in-vehicle cameras 17 are made the same, and the resolution and frame rate of the video are determined. That is, the priority of the front video of the in-vehicle camera 17 that captures the front and the priority of the video of the other in-vehicle cameras 17 are each set to "medium" (S518, S519).

[0041] On the other hand, when the fingertip distance La is shorter than the first distance threshold Lt1 (La < Lt1) (YES in S511), it is determined whether the operation amount M of the hydraulic excavator 100 by the operator is greater than the operation amount threshold Mt.

[0042] Figure 6 is a graph showing the relationship between the operation amount and the actuator speed. The operation amount shown in Figure 6 is, for example, the operation lever input amount input by operating the operation lever, and the actuator speed V is the speed at which the hydraulic cylinder 11 for bucket operation, which is one of the actuators, expands and contracts. In the operation signal generation unit 322, when the operation amount M is smaller than the operation amount threshold Mt, the degree of increase (slope) of the actuator speed V with respect to the operation lever input amount is also low, and when the operation amount M becomes greater than or equal to the operation amount threshold Mt, the operation signal is generated so that the degree of increase (slope) of the actuator speed V with respect to the operation lever input amount also becomes large.

[0043] Returning to FIG. 5, when the operation amount M is greater than or equal to the operation amount threshold Mt (M≧Mt) (NO in S512), it is determined that the distance from the bucket tip 10a to the design surface 42 is large and that a speed - priority operation rather than accuracy is required. Among the plurality of in - vehicle cameras 17, the priority of the front - view video of the in - vehicle camera 17 that captures the front is set to "high" (first priority), and the priorities of the videos of the other in - vehicle cameras 17 are each set to "low" (second priority) (S516, S517).

[0044] And when the tip distance La is shorter than the first distance threshold Lt1 (La<Lt1) and the operation amount M is smaller than the operation amount threshold Mt (M<Mt) (YES in S512), it is determined that the distance from the bucket tip 10a to the design surface 42 is short and that an accuracy - priority operation rather than speed is required. The priority of the video of the front in - vehicle camera 17 is set to "extremely high" (third priority), which is extremely high, and the priorities of the videos of the other in - vehicle cameras 17 are set to "extremely low" (fourth priority), which is extremely low (S513, S514).

[0045] When the priority of each of the plurality of in - vehicle cameras 17 is determined, the resolution and frame rate of each in - vehicle camera 17 are controlled according to that priority (S515). For example, in S518 and S519, when the priorities of the videos captured by the plurality of in - vehicle cameras 17 are all set to "medium", for each in - vehicle camera 17, the resolution and frame rate of the video are controlled to their respective default setting values (normal setting values) set in advance. In this embodiment, when the priorities of all the videos are "medium", the resolution and frame rate of each video are controlled to be displayed at the same value.

[0046] And in S516 and S517, when the priority of the video of the front in - vehicle camera 17 is set to "high" and the priorities of the videos of the other in - vehicle cameras 17 are each set to "low", the resolution and frame rate of the video of the front in - vehicle camera 17 are controlled to a first setting value higher than the default setting value, and the resolution and frame rate of the videos of the other in - vehicle cameras 17 are controlled to a second setting value lower than the default setting value.

[0047] Furthermore, when S513 and S514 set the priority of the image from the front vehicle-mounted camera 17 to "very high" and the priority of the images from the other vehicle-mounted cameras 17 to "very low," the resolution and frame rate of the image from the front vehicle-mounted camera 17 are controlled to a third setting value that is even higher than the first setting value, and the resolution and frame rate of the image from the other vehicle-mounted cameras 17 are controlled to a fourth setting value that is even lower than the second setting value.

[0048] FIG. 7 is a graph showing the relationship between the resolution and frame rate and the required transmission speed. Resolution (ppi) indicates the number of pixels per inch, and the higher the number, the clearer the image can be, down to the finer details. Resolution x monitor size = total number of pixels per drawing (frame). The amount of data per pixel is 24 bits, so resolution x monitor size x 24 bits = data amount per frame. Frame rate is a number that indicates how many images are displayed per second; for example, 30 fps means that 30 images are displayed per second. The higher this value, the smoother the video will be. The amount of data per second is "resolution x monitor size x 24 bits x frame rate" (bps: bits per second). To send this video to a remote location, a transmission speed based on the resolution and frame rate is required.

[0049] For example, if the data load on the communication path increases due to fluctuations in the communication volume of other users, the transmission speed cannot be maintained, and if the transmission speed falls below the amount of data you want to transmit, frame drops will occur. If frames are dropped, the display will appear as if time has skipped, which is undesirable from a work perspective. In consideration of fluctuations in transmission speed, it is desirable to keep the data volume as low as possible within a tolerable range.

[0050] In this process, the image resolution (ppi) and frame rate (fps) of the image are increased by increasing the priority of the image from the front vehicle-mounted camera 17, but the image resolution and frame rate are decreased accordingly by decreasing the priority of the image from the other vehicle-mounted cameras 17. In other words, by assigning priority to the image captured by each vehicle-mounted camera 17 and increasing or decreasing the resolution and frame rate, an increase in the overall data volume is suppressed while transmitting image sufficient for remote control. Therefore, the required data transmission rate (bps) can be kept within the range of the actual transmission rate. In this process, the amount of data communication with the remote control device 30 when the resolution and frame rate of the image from the front vehicle-mounted camera 17 are increased and the resolution and frame rate of the image from the other vehicle-mounted cameras 17 are decreased is set to be the same as the amount of data communication with the remote control device 30 when the resolution and frame rate of the image from all vehicle-mounted cameras 17 are controlled to the normal setting.

[0051] Figures 8 and 9 show examples of display on the display of the remote operator device in the first embodiment. Figure 8(1) shows a state in which the bucket toe 10a of the bucket 10 is away from the design surface 42 by the first distance threshold Lt1 or more, and Figure 8(2) shows a state in which the state shown in Figure 8(1) is displayed on the display 33 of the remote operator device 30. Figure 9(1) shows a state in which the bucket toe 10a of the bucket 10 is closer to the design surface 42 than the first distance threshold Lt1, and Figure 9(2) shows a state in which the state shown in Figure 9(1) is displayed on the display 33 of the remote operator device 30.

[0052] The display 33 displays in real time the images captured by each vehicle-mounted camera 17, and is composed of a display 331 for the front camera image, a display 332 for the right camera image, a display 333 for the left camera image, and a display 334 for the rear camera image. Note that, although the present embodiment has been described with reference to an example in which a display is provided for each vehicle-mounted camera 17, it is also possible to display images from all vehicle-mounted cameras 17 together on a single display.

[0053] As shown in FIG. 8, when the toe distance La from the bucket toe 10a of the bucket 10 to the design surface 42 is equal to or greater than the first distance threshold Lt1, the images captured by each vehicle-mounted camera 17 have the same priority, and the resolution and frame rate of each image are the same, so the images displayed on each display 331 to 334 are displayed at the same resolution and frame rate.

[0054] 9, when the toe distance La from the bucket toe 10a of the bucket 10 to the design surface 42 is smaller than the first distance threshold Lt1, the priority of the front camera image captured by the front on-board camera 17 is high. Therefore, the resolution and frame rate of the front camera image are transmitted at a higher level than those of the camera images captured by the other on-board cameras 17. Therefore, the image displayed on the front camera image display 331 is displayed at a higher resolution and frame rate than those of the other camera image images displayed on the displays 332, 333, and 334.

[0055] FIG. 10 is a diagram illustrating a modified example. For example, in addition to the toe distance La between the bucket toe 10a and the design surface 42, a measuring device such as Lidar capable of measuring the shape of the current terrain may be mounted on the vehicle body, and the distance Lb between the current terrain and the bucket toe 10a may be used as the judgment criterion. In this modification, the priority of the images is determined using the toe distance Lb from the bucket toe 10a of the bucket 10 to the current terrain 43. The hydraulic excavator 100 is equipped with the above-mentioned measuring device 44 such as Lidar as measuring device capable of measuring the current terrain 43. The toe distance calculation unit 213 of the controller 21 calculates the vertical distance from the bucket toe 10a of the bucket 10 to the current terrain 43 as the toe distance Lb, based on the current terrain 43, which is the construction surface actually measured by the measuring device 44, and the bucket toe position calculated by the attitude / position calculation unit 211. The image processor 214 can also determine the priority of the images captured by each vehicle-mounted camera 17 using the toe distance Lb instead of the toe distance La from the design surface 42.

[0056] The remote control system for construction machinery of the present embodiment described above includes a hydraulic excavator 100 and a remote control device 30 that remotely controls the hydraulic excavator 100. The hydraulic excavator 100 includes a plurality of on-board cameras 17 that capture images of the area around the vehicle body, and a controller 21 that remotely controls the hydraulic excavator 100 based on operation signals received from the remote control device 30. The controller 21 includes a toe distance calculation unit 213 that calculates a first toe distance La between the bucket toe 10a of the hydraulic excavator 100 and the design surface 42, and an image processing unit 214 that sets priorities for the images captured by the plurality of on-board cameras 17 in accordance with the first toe distance La and the amount of operation M of the hydraulic excavator 100 by the remote control device 30, and adjusts the resolution and frame rate of the images captured by the plurality of on-board cameras 17 in accordance with the priorities.

[0057] For example, if the distance between the bucket toe 10a and the design surface 42 in the machine guidance is short, it is assumed that work is being performed that requires close attention to the bucket toe 10a, such as finishing work or aligning the start of pulling. In addition, if the front work implement 4 is being operated slowly from that state, it is assumed that final finishing work is being performed, and it can be estimated that detailed information about the area around the bucket toe 10a is required.

[0058] Therefore, in the remote operation system of this embodiment, the controller 21 determines the work content currently being performed by the operator based on the distance between the bucket toe 10a of the hydraulic excavator 100 and the design surface 42 and the amount of operation M by the operator, and accordingly sets the priority of the camera image from the front vehicle-mounted camera 17 higher than that of the other vehicle-mounted cameras 17. Based on the priority, processing is performed to lower the resolution and frame rate of the camera image from the other vehicle-mounted cameras 17 and to increase the resolution and frame rate of the camera image from the front vehicle-mounted camera 17.

[0059] The position of the bucket toe 10a is calculated using an attitude sensor 13 such as an IMU attached to the vehicle body and a position sensor 20 such as a GNSS. A toe distance La, which is the distance between the position of the bucket toe 10a and the design surface 42, is calculated from the calculated bucket toe position and design surface data. If the calculated toe distance La is equal to or less than a first distance threshold Lt1, the priority of the front imaging camera 17 is increased by one level. On the other hand, the priority of the other vehicle-mounted cameras 17 is decreased. In addition, the operation amount M of the operator is obtained from the operation device 34 of the remote control device 30, and if the operation amount M is equal to or less than an operation amount threshold Mt, the priority of the front vehicle-mounted camera 17 is further increased. The resolution and frame rate of the camera image transmitted to the remote control device 30 are changed based on the priority.

[0060] [Second embodiment] FIG. 11 is a diagram explaining the situation in which judgment is made based on both the design surface and the current terrain in the second embodiment, FIG. 12 is a flowchart for making judgment based on both the design surface and the current terrain in the second embodiment, and FIG. 13 is a diagram for explaining the priority.

[0061] A distinctive feature of this embodiment is that the priority of the image is determined by taking into consideration both the toe distance (first toe distance) La from the design surface 42 and the toe distance (second toe distance) Lb from the current terrain 43 as the distance between the bucket toe 10a and the construction surface.

[0062] The hydraulic excavator 100 performs work of excavating an existing topography 43 and leveling it to a design surface 42, for example, as shown in FIG. 11 . The existing topography 43 is measured by a measuring device 44. The design surface data is acquired from the design surface data management unit 212.

[0063] Then, when the toe distance Lb between the current terrain 43 and the bucket toe 10a is equal to or greater than the second distance threshold Lt2, which is the determination distance (No in S611), the priority of the image from the front in-vehicle camera 17 and the priority of the image from the other in-vehicle cameras 17 are set to the same priority. In this embodiment, the priority of the image from the front in-vehicle camera 17 is divided into nine levels, and to set the priority to the same level, the priority is set to 5 out of the nine levels (S623), and the priority of the image from the other in-vehicle cameras 17 is also set to 5 out of the nine levels, just like the priority of the image from the front in-vehicle camera 17 (S624). As shown in FIG. 13, the priority is set so that 5 is the median, the smaller the number, the lower the priority, and the larger the number, the higher the priority.

[0064] If the toe distance Lb is smaller than the second distance threshold Lt2 (Yes in S611), it is determined whether the toe distance La between the design surface 42 and the bucket toe 10a is smaller than the first distance threshold Lt1. If the toe distance La is equal to or greater than the first distance threshold Lt1 (No in S612), it is determined whether the lever operation amount M is smaller than the operation amount threshold Mt (S618). If the lever operation amount M is equal to or greater than the operation amount threshold Mt (No in S618), the operation prioritizes work speed, so the priority of the front image is set to 6 (S621) and the priority of the other images is set to 4 (S622). On the other hand, if the lever operation amount M is smaller than the operation amount threshold Mt (Yes in S618), the operation prioritizes work accuracy, so the priority of the front image is set to 7 (S619) and the priority of the other images is set to 3 (S620).

[0065] Furthermore, when the toe distance La is smaller than the first distance threshold Lt1 (Yes in S612), it is determined whether the lever operation amount M is smaller than the operation amount threshold Mt (S613). This operation amount threshold Mt may be the same as or different from the operation amount threshold Mt in S618. When the lever operation amount M is equal to or greater than the operation amount threshold Mt (No in S613), although the operation is near the design surface, the operation prioritizes speed, so the priority of the forward image is set to 8 (S616) and the priority of the other images is set to 2 (S617). On the other hand, when the lever operation amount M is smaller than the operation amount threshold Mt (Yes in S613), the operation prioritizes high-precision work near the design surface, so the priority of the forward image is set to 9 (S614) and the priority of the other images is set to 1 (S615).

[0066] In this way, by changing the priority of the camera image from the front on-board camera 17 and the camera images from other on-board cameras 17 according to the toe distances La, Lb from the design surface 42 or the current terrain 43, it is possible to display images appropriate to the work content on the display 33. In other words, when both the design surface 42 and the current terrain 43 exist as data, the design surface 42 is the final finishing surface, but the current terrain 43 may be in the excavation stage, so the priority is differentiated. For example, when the work is being finished when the work is close to the design surface 42, the camera image from the front on-board camera 17 has the highest priority, and when the work is close to the current terrain, the priority is lower than when the work is close to the design surface 42, but higher than in other cases.

[0067] The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0068] 1. Lower running body, 2. Upper rotating body, 3. Cab, 4. Front work implement, 5. Counterweight, 6. Boom, 7. Hydraulic cylinder for boom operation, 8. Arm, 9. Hydraulic cylinder for arm operation, 10. Bucket, 11. Hydraulic cylinder for bucket operation, 12. Bucket link, 13. Attitude detector, 14a, 14b. GNSS antenna, 16. Communication device, 17. On-board camera, 171. Imaging range of front on-board camera, 172. Imaging range of right on-board camera, 173. Imaging range of left on-board camera, 174. Imaging range of rear on-board camera, 18 Swing device, 19... Traveling device, 20... GNSS receiver, 21... Controller, 22... Engine, 23... Hydraulic pump, 24... Control valve, 25... Display device, 211... Attitude and position calculation unit, 212... Design surface data management unit, 213... Toe distance calculation unit, 214... Image processing unit, 215... Vehicle control unit, 30... Remote control device, 31... Communication device, 32... Control device, 321... Display control unit 322... Operation signal generation unit, 33... Display, 34... Operation device, 331... Display of front camera image, 332... Display of right vehicle-mounted camera image, 333... Display of left vehicle-mounted camera image, 334... Display of rear vehicle-mounted camera image, 42... Design surface, 43... Current terrain, 44... Measurement device (terrain measurement device)

Claims

1. A control device for a construction machine, comprising a control device that transmits images of the surroundings of the vehicle body captured by a plurality of on-board cameras to a remote control device and remotely controls the construction machine based on operation signals received from the remote control device, The control device a toe distance calculation unit that calculates a first toe distance between a bucket toe of the construction machine and a predetermined construction surface; an image processing unit that sets a priority for the images captured by the multiple on-board cameras according to the first toe distance and the amount of operation of the construction machine by the remote control device, and adjusts the resolution and frame rate of the images captured by the multiple on-board cameras according to the priority that is set to be higher as the first toe distance is shorter and the amount of operation is smaller; A control device for a construction machine, comprising:

2. The video processing unit includes: When the first toe distance is equal to or greater than a predetermined first distance threshold, the priority of the image captured by the front vehicle-mounted camera is set to be the same as the priority of the image captured by the other vehicle-mounted camera; A construction machine control device as described in claim 1, characterized in that when the first toe distance is smaller than the first distance threshold and the amount of operation of the construction machine by the remote control device is equal to or greater than a predetermined operation amount threshold, the priority of the image captured by the front on-board camera is set to a first priority that is higher than the priority when the distance is equal to or greater than the first distance threshold, and the priority of the image captured by the other on-board camera is set to a second priority that is lower than the priority when the distance is equal to or greater than the first distance threshold.

3. The video processing unit includes: When the priority of the image captured by the front vehicle-mounted camera is the same as the priority of the image captured by the other vehicle-mounted camera, the resolution and frame rate of the image captured by the front vehicle-mounted camera and the resolution and frame rate of the image captured by the other vehicle-mounted camera are controlled to normal preset values; A control device for a construction machine as described in claim 2, characterized in that when the priority of the image from the front vehicle-mounted camera is set to the first priority, the resolution and frame rate of the image from the front vehicle-mounted camera are controlled to a first setting value higher than the normal setting value, and the resolution and frame rate of the image from the other vehicle-mounted camera are controlled to a second setting value lower than the normal setting value.

4. The video processing unit includes: A construction machinery control device as described in claim 3, characterized in that when the first toe distance is smaller than the first distance threshold and the amount of operation of the construction machinery by the remote control device is smaller than the operation amount threshold, the priority of the image from the front vehicle-mounted camera is set to a third priority higher than the first priority, and the priority of the image from another vehicle-mounted camera is set to a fourth priority lower than the second priority.

5. The video processing unit includes: A control device for a construction machine as described in claim 4, characterized in that when the priority of the image from the front vehicle-mounted camera is set to the third priority, the resolution and frame rate of the image from the front vehicle-mounted camera are controlled to a third setting value that is even higher than the first setting value, and the resolution and frame rate of the image from the other vehicle-mounted camera are controlled to a fourth setting value that is even lower than the second setting value.

6. 6. A construction machine control device according to claim 5, characterized in that the amount of data communication with the remote control device when the resolution and frame rate of the image from the front vehicle-mounted camera are controlled to the first set value or the third set value and the resolution and frame rate of the image from the other vehicle-mounted cameras are controlled to the second set value or the fourth set value is the same as the amount of data communication with the remote control device when the resolution and frame rate of the image from all vehicle-mounted cameras are controlled to the normal set value.

Images