Remote control device, remote control method, and remote control system

Abstract

To suppress lowering of operability.SOLUTION: An image correction device includes: an image acquisition part for acquiring an image photographed by an imaging device provided on a remotely operated work machine; a vibration information acquisition part for acquiring vibration information indicating vibration of the work machine; a turning detection part for detecting that a turning body of the work machine is being turned; and a blur correction part for performing blur correction on the image on the basis of the vibration information, and invalidating blur correction in a yaw axis direction of the image during the turning.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to Remote control devices, Remote control methods, and remote operation systems.

Background Art

[0002] As disclosed in Patent Document 1, a technique for remotely operating a work machine is known. According to Patent Document 1, an image of a portion corresponding to a working tool is generated and displayed using information on the position of the working tool (bucket) and information on the position of the work target obtained from information on the distance to the work target.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In remote operation, an operator in the operation cab performs operations while viewing an image. However, since the image shakes due to the vibration of the camera fixed to the vehicle body, it may cause operator fatigue, so it is conceivable to perform blurring correction of the image. However, in that case, by performing the blurring correction, an image that should originally appear to move may be displayed as staying in place. As a result, the operability of the operator may be reduced. An object of the present disclosure is to provide an image correction device, an image correction method, and a remote operation system that can suppress a decrease in operability.

Means for Solving the Problems

[0005] According to a first aspect of the present disclosure, the image correction device includes an image acquisition unit that acquires an image captured by an imaging device provided on a remotely operated work machine; a vibration information acquisition unit that acquires vibration information indicating vibration of the work machine; a rotation detection unit that detects that the rotating body of the work machine is rotating; and a blur correction unit that corrects the blur of the image based on the vibration information and disables blur correction in the yaw axis direction of the image while it is rotating. [Effects of the Invention]

[0006] According to the above embodiment, the image correction device can suppress a decrease in operability. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram showing the configuration of the remote control system according to the first embodiment. [Figure 2] This is an external view of a work machine according to the first embodiment. [Figure 3] This is a schematic block diagram showing the configuration of the remote system according to the first embodiment. [Figure 4] This is a flowchart showing the display control method performed by the remote control device according to the first embodiment. [Figure 5] This is a schematic block diagram showing the configuration of the remote system according to the second embodiment. [Figure 6] This is a flowchart showing the display control method performed by the remote control device according to the second embodiment. [Figure 7] This is a schematic block diagram showing the configuration of the remote system according to the fourth embodiment. [Figure 8] This is a flowchart showing the method for outputting captured images performed by the work machine according to the fourth embodiment. [Figure 9] This is a schematic block diagram showing the configuration of the remote system according to the fifth embodiment. [Figure 10] This is a flowchart showing the method for outputting captured images performed by the work machine according to the fifth embodiment. [Modes for carrying out the invention]

[0008] <First Embodiment> "system" Figure 1 is a schematic diagram showing the configuration of a remote control system according to the first embodiment. The remote control system 1 comprises a work machine 100 that operates by remote control and a remote control device 500. The work machine 100 is installed at a work site (e.g., a mine, a quarry). The remote control device 500 is installed in a remote control room at a location at or away from the work site (e.g., a city, within the work site). The work machine 100 and the remote control device 500 are connected via a network such as the Internet. The remote control system 1 is a system for operating a work machine 100 using a remote control device 500.

[0009] The work machine 100 operates according to the operation signals received from the remote control device 500. When the levers and pedals of the control device 530 in the remote control room are operated by the operator, control signals for operations such as working with the work equipment, slewing, and traveling are generated. The generated control signals are transmitted to the work machine 100.

[0010] 《Work Machinery》 Figure 2 is an external view of the work machine according to the first embodiment. The work machine 100 according to the first embodiment is a hydraulic excavator. However, the work machine 100 may be a work machine other than a hydraulic excavator, such as a wheel loader or a bulldozer. The work machine 100 comprises a hydraulically operated work implement 110, a slewing body 120 that supports the work implement 110, and a traveling body 130 that supports the slewing body 120. The traveling body 130 is, for example, a track.

[0011] The revolving body 120 is provided with a cab 121. An imaging device 122 is provided above the cab 121. The imaging device 122 is installed in front of and above the inside of the cab 121. The imaging device 122 captures an image (for example, a moving image) in front of the cab 121 through the front glass on the front surface of the cab 121. Examples of the imaging device 122 include, for example, an imaging device using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that the imaging device 122 does not necessarily have to be provided in the cab 121, and it is sufficient if the imaging device 122 is provided at a position where at least the work target of the revolving body 120 and the working machine 110 can be imaged. For example, the imaging device 122 may be provided outside the cab 121, for example, it may be provided on the revolving body. Also, the imaging device 122 may be provided outside the working machine 100, that is, it may be provided at a location separate from the working machine 100.

[0012] The working machine 100 includes an imaging device 122, a turning speed sensor 123, a vibration sensor 124, and a control device 125.

[0013] The turning speed sensor 123 detects the rotational speed when the revolving body 120 turns. For example, the turning speed sensor 123 may be a rotary encoder. The vibration sensor 124 measures the acceleration and angular velocity of the revolving body 120, and detects vibration information indicating the operation of the revolving body 120 (for example, roll angle, pitch angle, yaw angle) based on the measurement results. It is assumed that the relative positional relationship between the vibration sensor 124 and the imaging device 122 is fixed. The vibration sensor 124 is installed, for example, on the lower surface of the cab 121. The vibration sensor 124 can use, for example, an inertial measurement unit (IMU). The roll angle indicates the angle around the axis in the front-rear direction of the revolving body. The pitch angle indicates the angle around the axis in the left-right direction of the revolving body. The yaw angle indicates the angle around the axis in the up-down direction of the revolving body. Note that the vibration information can also be obtained from the acceleration and angular velocity of the IMU without using the roll angle, pitch angle, yaw angle, etc. Also, the vibration sensor 124 may be arranged outside the revolving body 120 (for example, the working machine 110, etc.).

[0014] The control device 125 receives an operation signal from the remote operation device 500 via the communication unit 126 (see FIG. 3). The control device 125 drives the working machine 110, the revolving body 120, or the traveling body 130 according to the received operation signal.

[0015] 《Remote Operation Device》 As shown in FIG. 1, the remote operation device 500 includes a driver's seat 510, a display device 520, an operation device 530, and a control device 540. The display device 520 is arranged in front of the driver's seat 510. The display device 520 is located in front of the operator's eyes when the operator sits on the driver's seat 510. The display device 520 is composed of the arranged displays 521, 522, 523, 524, 525 as shown in FIG. 1. Note that the number of displays constituting the display device 520 is not limited to this. For example, as shown in FIG. 1, the display device 520 may be composed of a plurality of arranged displays, or may be composed of one large display. Also, the display device 520 may project an image onto a curved surface or a spherical surface by a projector or the like.

[0016] The control device 530 is located near the driver's seat 510. The control device 530 is located within the operator's reach when the operator is seated in the driver's seat 510. The control device 530 includes a slewing lever for slewing the slewing body 120. The control device 530 includes, for example, an electric lever and an electric pedal.

[0017] The control device 540 is an example of an image correction device. The control device 540 displays the image received from the work machine 100 on the display device 520 and transmits an operation signal representing the operation of the operation device 530 to the work machine 100.

[0018] Figure 3 is a schematic block diagram showing the configuration of the remote system according to the first embodiment. The control device 125 of the work machine 100 is a computer equipped with a processor 1250, main memory 1257, storage 1258, and image encoding device 1259. The storage 1258 stores program Q. The processor 1250 reads program Q from storage 1258, expands it into main memory 1257, and executes processing according to program Q. The control device 125 is connected to a network via a communication unit 126. The image encoding device 1259 encodes (compresses) the image captured by the imaging device 122. Note that the image encoding device 1259 may be provided separately from the control device 125.

[0019] The control device 125 associates the encoded image information with the rotation speed information of the rotating body 120 detected by the rotation speed sensor 123 and the vibration information measured by the vibration sensor 124. This synchronizes the information. The control device 125 then transmits the associated information to the remote control device 500.

[0020] The control unit 540 of the remote control device 500 is a computer comprising a processor 5100, main memory 5200, storage 5300, image decoding device 5400, and receiving unit 5500. The storage 5300 stores program P. The processor 5100 reads program P from storage 5300, loads it into main memory 5200, and executes processing according to program P. The control unit 540 is connected to a network via a communication unit 550. The receiving unit 5500 receives associated image information, rotation speed information, and vibration information via the communication unit 550. The image decoding device 5400 decodes the encoded image. Note that the image decoding device 5400 may be provided separately from the control unit 540.

[0021] The storage 5300 has a storage area. Examples of storage 5300 include HDDs, SSDs, magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memory. The storage 5300 may be an internal medium directly connected to the common communication line of the control device 540, or it may be an external medium connected to the control device 540 via an interface. The storage 5300 is a tangible, non-temporary storage medium.

[0022] The processor 5100, upon execution of program P, includes an image acquisition unit 5101, a vibration information acquisition unit 5102, a blur correction unit 5103, a speed information acquisition unit 5104, a rotation detection unit 5105, and a display control unit 5106.

[0023] The image acquisition unit 5101 acquires the image decoded by the image decoding device 5400. The image acquired by the image acquisition unit 5101 is the image acquired by the imaging device 122 of the work machine 100, encoded by the control device 125, and decoded by the image decoding device 5400.

[0024] The display control unit 5106 causes the image acquired by the image acquisition unit 5101 to be displayed on the display device 520.

[0025] The vibration information acquisition unit 5102 acquires vibration information from the work machine 100. The vibration information is detected by the vibration sensor 124.

[0026] The image stabilization unit 5103 corrects image blur on the display device 520 to prevent operator motion sickness associated with remote operation. Specifically, the image stabilization unit 5103 corrects image blur received by the image acquisition unit 5101 in accordance with vibration information acquired by the vibration information acquisition unit 5102. The image stabilization unit 5103 corrects blur in the roll axis direction, pitch axis direction, and yaw axis direction. Here, we will describe in detail the image blur correction based on vibration information. For example, the image stabilization unit 5103 corrects each of the multiple captured images (each frame) obtained by continuous shooting at a high frame rate by applying horizontal movement and rotation.

[0027] Specifically, the image stabilization unit 5103 corrects blur in the yaw direction by substituting the yaw angle into a function that pre-defined the relationship between the yaw angle and the amount of shift in the X direction. The X direction is the horizontal direction of the screen on which the captured image is displayed, and is one of the directions of blur in the captured image. The function that pre-defined the relationship between the yaw angle and the amount of shift in the X direction may be expressed as a tangent function, for example. The image stabilization unit 5103 also corrects blur in the pitch direction by substituting the pitch angle into a function that pre-defined the relationship between the pitch angle and the amount of shift in the Y direction. The Y direction is the vertical direction of the screen on which the captured image is displayed, and is another direction of blur in the captured image. The function that pre-defined the relationship between the pitch angle and the amount of shift in the Y direction may be expressed as a tangent function, for example. The image stabilization unit 5103 shifts the captured image in the X direction and Y direction by the calculated correction amount. The vibration correction unit 5103 then corrects the vibration in the direction of the roll axis by substituting the roll angle into a function that shows the relationship between the roll angle and the amount of shift in the X axis direction, and a function that shows the relationship between the roll angle and the amount of shift in the Y axis direction. The vibration correction unit 5130 performs vibration correction in three axes. The vibration correction unit 5130 may perform vibration correction using one of the yaw angle, pitch angle, and roll angle, or a combination thereof. For example, the vibration correction unit 5130 may perform vibration correction using only the yaw angle, or preferably, it may perform vibration correction using both the pitch angle and the yaw angle.

[0028] The speed information acquisition unit 5104 acquires slewing speed information indicating the slewing speed of the work machine 100. The speed information acquisition unit 5104 acquires slewing speed information based on the detection result of the slewing speed sensor 123.

[0029] The rotation detection unit 5105 detects that the work machine 100 is rotating. The rotation detection unit 5105 detects that the machine is rotating based on the rotation speed information acquired by the speed information acquisition unit 5104. For example, if rotation speed information is acquired by work with a work machine that does not involve rotation, such as excavation, it is preferable that the rotation detection unit 5105 does not detect that the machine is rotating when the rotation speed is below a threshold, so as not to consider that the machine is rotating. Also, in the case of a combined operation in which the operation of the work machine and the rotation operation are performed simultaneously, such as loading, the operation may be performed so that the rotation speed is low. In such cases, it is preferable not to detect that the machine is rotating in order to suppress a decrease in operability. For this reason, the rotation detection unit 5105 detects that the machine is rotating when the rotation speed is above a preset threshold. Alternatively, the rotation detection unit 5105 may also detect that the machine is rotating by considering that the rotation speed is above a threshold when the lever operation amount is above a predetermined operation amount.

[0030] The shake correction unit 5103 performs shake correction in three axes during operations other than rotation, such as when operating or traveling with work equipment like excavators, or during complex operations that are not considered rotation. On the other hand, the shake correction unit 5103 disables shake correction in the yaw axis direction of the image received by the image acquisition unit 5101 while the work machine 100 is rotating. Disabling shake correction in the yaw axis direction includes, for example, not performing (not executing) shake correction in the yaw axis direction, prohibiting shake correction, or disabling shake correction. In this embodiment, the shake correction unit 5103 can prevent shake correction in the yaw axis direction by substituting zero as the yaw angle.

[0031] The wobble correction unit 5103 enables wobble correction in the yaw axis direction when the work machine 100 is not rotating. Enabling wobble correction in the yaw axis direction includes, for example, performing (executing) wobble correction in the yaw axis direction, not prohibiting wobble correction, and not disabling wobble correction. The wobble correction unit 5103 may also enable yaw axis wobble correction and execute the operation to include yaw axis wobble correction when the work machine operation is involved, such as in a compound operation where the work machine operation and rotation operation are performed simultaneously.

[0032] "method" Here, we will describe the method for controlling the display of captured images performed by the remote control device 500 according to the first embodiment. Figure 4 is a flowchart showing the display control method performed by the remote control device according to the first embodiment. The control device 540, specifically its image acquisition unit 5101, acquires image information (step S1). Then, the vibration information acquisition unit 5102 acquires vibration information (step S2). Next, the speed information acquisition unit 5104 acquires slewing speed information indicating the slewing speed of the work machine 100 (step S3). The information acquired in steps S1 to S3 is associated with each other.

[0033] Then, the rotation detection unit 5105 determines whether the rotation speed indicated by the rotation speed information is above a threshold (step S4). If the rotation speed is not above the threshold (step S4: NO), the shake correction unit 5103 performs shake correction on the three axes (step S5) and proceeds to step S7.

[0034] On the other hand, if the rotation speed is above a threshold (step S4: YES), the shake correction unit 5103 performs shake correction in two axes, the pitch axis direction and the roll axis direction (step S6). Then, the display control unit 5106 displays the shake-corrected image on the display device 520 (step S7), and the process shown in Figure 4 is completed.

[0035] Action / Effect Thus, according to the first embodiment, the control device 540, which corrects image blur in the image captured by the imaging device 122 based on vibration information from the work machine 100, disables blur correction in the yaw axis direction of the image while the work machine 100 is rotating, and does not perform blur correction. This prevents the image, which should appear to be moving during rotation, from appearing to remain stationary. Therefore, it is possible to reduce the operator's discomfort during rotation and prevent a decrease in operator operability. Furthermore, when the machine is not rotating, blur correction is performed in all three axes, thus eliminating blur caused by vibrations from the work machine 100. Therefore, it is possible to suppress motion sickness in the operator during operations that do not involve rotation, such as excavation and soil removal.

[0036] Furthermore, according to the first embodiment, the control device 540 enables yaw axis direction wobble correction when not rotating. As a result, when not rotating, wobble correction can be performed in all three axes, thereby eliminating wobble caused by vibrations of the work machine 100. Therefore, motion sickness of the operator can be suppressed during operations that do not involve rotation, such as excavation and soil removal.

[0037] Furthermore, according to the first embodiment, the control device 540 detects whether the work machine 100 is rotating based on rotation speed information indicating the rotation speed of the work machine 100. This makes it possible to easily and accurately detect whether or not the work machine 100 is rotating.

[0038] Furthermore, according to the first embodiment, the control device 540 detects that the machine is rotating when the rotation speed is above a threshold. This allows the machine to operate at a low rotation speed when rotation speed information is acquired during operations that do not involve rotation, such as excavation, or in cases of combined operations where the operation of the work machine and rotation are performed simultaneously, such as loading. In such cases, the machine can be prevented from being considered as rotating, and thus motion sickness can be corrected in the yaw axis direction as well. Therefore, motion sickness during operations such as excavation and loading can be suppressed.

[0039] <Second Embodiment> In the first embodiment, the rotation detection unit 5105 detects that rotation is in progress using rotation speed information. In the second embodiment, in addition to or instead of this configuration, the rotation detection unit 5105 detects that rotation is in progress using the rotation operation amount of the operating device 530.

[0040] Figure 5 is a schematic block diagram showing the configuration of the remote system according to the second embodiment. The work machine 100 according to the second embodiment does not include the slewing speed sensor 123, which is part of the configuration of the first embodiment. Also, the control device 540 does not include the speed information acquisition unit 5104, which is part of the configuration of the first embodiment. The lever on the operating device 530 receives rotational commands from the operator. Upon receiving a rotational command, the lever outputs the amount of rotational movement indicated by the received rotational command to the control device 540.

[0041] The control device 540 includes an operation amount acquisition unit 5110. The operation amount acquisition unit 5110 acquires the operation amount for rotation output from the operating device 530. The rotation detection unit 5105 detects that rotation is in progress based on the operation amount acquired by the operation amount acquisition unit 5110. For example, if the operation amount is acquired by a rotation operation (fine operation) such as excavation or loading, the rotation detection unit 5105 does not detect that rotation is in progress if the operation amount is below a threshold, so that it is considered that rotation is in progress. In other words, the rotation detection unit 5105 detects that rotation is in progress if the operation amount is above a threshold.

[0042] The control device 540 transmits the manipulated amount acquired by the manipulated amount acquisition unit 5110 to the work machine 100 via the communication unit 550. The manipulated amount transmitted from the remote control device 500 is input to the control device 125 via the communication unit 126. As a result, the control device 125 rotates the slewing body 120 at an angle corresponding to the input manipulated amount.

[0043] Furthermore, the operation quantity acquisition unit 5110 is not limited to acquiring the operation quantity output from the operation device 530, but may also acquire the rotation angle when the work machine 100 actually rotates. To elaborate, the work machine 100 can transmit to the remote control device 500 information that associates the image captured by the imaging device 122, the detection results of the vibration sensor 124, and the rotation angle when it actually rotates. The operation quantity acquisition unit 5110 may also extract and acquire the rotation angle from this associated information.

[0044] "method" Here, we will describe the method for controlling the display of captured images performed by the remote control device 500 according to the second embodiment. Figure 6 is a flowchart showing the display control method performed by the remote control device according to the second embodiment. The image acquisition unit 5101 of the control device 540 acquires image information (step S11). Then, the vibration information acquisition unit 5102 acquires vibration information (step S12). Next, the manipulated amount acquisition unit 5110 acquires the manipulated amount of rotation (step S13). Note that the information acquired in steps S11 to S13 is related to each other.

[0045] The rotation detection unit 5105 then determines whether the amount of rotation is greater than or equal to a threshold (step S14). If the amount of rotation is not greater than or equal to the threshold (step S14: NO), the shake correction unit 5103 performs shake correction on the three axes (step S15) and proceeds to step S17.

[0046] On the other hand, if the amount of rotation operation is greater than or equal to a threshold (step S14: YES), the shake correction unit 5103 prohibits shake correction on the yaw axis and performs shake correction on the pitch axis and roll axis (step S16). Next, the display control unit 5106 displays the shake-corrected image on the display device 520 (step S17), and the process shown in Figure 6 is terminated.

[0047] In the second embodiment, if the slewing speed is above a threshold, two-axis shake correction may be performed. Specifically, in the second embodiment, the work machine 100 only needs to be equipped with a slewing speed sensor 123. The control device 540 only needs to be equipped with a speed information acquisition unit 5104. Then, in step S14, if the amount of slewing operation is not above a threshold, the speed information acquisition unit 5104 should determine whether the slewing speed of the work machine 100 is above a threshold. Furthermore, if the slewing speed is above a threshold, the shake correction unit 5103 should perform two-axis shake correction.

[0048] Action / Effect Thus, according to the second embodiment, the control device 540 detects that the machine is rotating based on the amount of rotation operation. This reduces the operator's discomfort during rotation, thereby preventing a decrease in operator operability. Furthermore, it allows for easy and accurate detection of whether or not the work machine 100 is rotating.

[0049] <Third Embodiment> In the first embodiment, when the rotating body 120 is rotating, no wobble correction is performed in the yaw axis direction. In the third embodiment, a remote control device 500 is described in which no wobble correction is performed in the yaw axis direction, regardless of whether the rotating body 120 is rotating or not.

[0050] In the third embodiment, the work machine 100 does not include the swivel speed sensor 123 as in the configuration of the first embodiment. Also, the control device 540 does not include the speed information acquisition unit 5104 and the swivel detection unit 5105 as in the configuration of the first embodiment. The blur correction unit 5103 performs blur correction in the roll axis direction and pitch axis direction of the image received by the image acquisition unit 5101 in accordance with the vibration information acquired by the vibration information acquisition unit 5102. On the other hand, the blur correction unit 5103 does not perform blur correction in the yaw axis direction of the image at all times.

[0051] "method" Here, we will describe the method for controlling the display of captured images performed by the remote control device 500 according to the third embodiment. The image acquisition unit 5101 of the control device 540 acquires image information. Then, the vibration information acquisition unit 5102 acquires vibration information. Next, the blur correction unit 5103 performs two-axis blur correction. Finally, the display control unit 5106 displays the blur-corrected image on the display device 520 and terminates the process.

[0052] Action / Effect Thus, according to the third embodiment, the control device 540 does not perform blur correction in the yaw axis direction of the image while the power is on. This makes it possible to suppress the display of an image that should appear to be moving during rotation as if it were stationary, with simple control. Therefore, it is possible to reduce the operator's discomfort during rotation and easily prevent a decrease in operator operability.

[0053] <Fourth Embodiment> In the first embodiment, shake correction is performed on the remote control device 500 side. In the fourth embodiment, shake correction is performed on the work machine 100 side.

[0054] Figure 7 is a schematic block diagram showing the configuration of the remote system according to the fourth embodiment. The control device 125 for the work machine is an example of an image correction device.

[0055] The processor 1250, upon execution of program Q, includes a vibration information acquisition unit 1251, a correction amount calculation unit 1252, a rotation detection unit 1253, a speed information acquisition unit 1254, and an image output unit 1255. The imaging device 122 includes a blur correction unit 1220.

[0056] The vibration information acquisition unit 1251 acquires vibration information detected by the vibration sensor 124. The image stabilization unit 1220 includes a mechanism that mechanically corrects image blur in accordance with vibration information acquired by the vibration information acquisition unit 1251. The image stabilization unit 1220 is an optical type that cancels out blur by moving a lens or image sensor in accordance with vibration information. Specifically, the image stabilization unit 1220 includes an actuator that drives in the yaw direction and an actuator that drives in the pitch direction, and drives each actuator in accordance with vibration information. However, the image stabilization unit 1220 is not limited to an optical type, but may also be an external type attached to the imaging device 122 such as a gimbal, or an electronic type that performs a predetermined calculation on image data received from a light-receiving element to apply correction. The electronic image stabilization unit 1220 may correct blur in the same way as the image stabilization unit 5103 of the first embodiment, for example.

[0057] The correction amount calculation unit 1252 calculates the correction amount for the shake correction unit 1220. The correction amount calculation unit 1252 corrects the vibration information generated in the shake correction unit 1220. Specifically, when turning, the angular velocity in the yaw axis direction of the vibration information is rewritten to zero, and this rewriting is not performed when not turning. The shake correction unit 1220 performs shake correction according to the vibration information corrected by the correction amount calculation unit 1252.

[0058] The rotation detection unit 1253 detects that the work machine 100 is rotating. The speed information acquisition unit 1254 acquires rotation speed information indicating the rotation speed based on the detection result of the rotation speed sensor 123. The rotation detection unit 1253 detects that the machine is rotating based on the rotation speed information acquired by the speed information acquisition unit 1254. For example, the rotation detection unit 5105 detects that the machine is rotating when the rotation speed is above a threshold.

[0059] The image output unit 1255 outputs the image captured by the imaging device 122. Specifically, the image output from the image output unit 1255 is input to the image encoding device 1259 and encoded. The encoded image is output to the communication unit 126. The communication unit 126 transmits the encoded image to the communication unit 550 of the remote control device 500.

[0060] The communication unit 550 outputs the image received from the work machine 100 to the image decoding device 5400. The image decoding device 5400 decodes the encoded image and outputs it to the image acquisition unit 5101. The display control unit 5106 displays the image acquired by the image acquisition unit 5101 on the display device 520.

[0061] "method" Here, we will describe the method for outputting captured images performed by the work machine 100 according to the fourth embodiment. Figure 8 is a flowchart showing the method for outputting captured images performed by the work machine according to the fourth embodiment. The vibration information acquisition unit 1251 acquires vibration information (step S31). The control device 125 then inputs the vibration information to the vibration correction unit 1220 (step S32). As a result, the vibration correction unit 1220 performs corrections in the pitch direction, roll direction, and yaw direction. Next, the slewing detection unit 1253 acquires slewing speed information from the speed information acquisition unit 1254, which indicates the slewing speed of the work machine 100 (step S33).

[0062] Then, the rotation detection unit 1253 determines whether the rotation speed indicated by the rotation speed information is above a threshold (step S34). If the rotation speed is not above the threshold (step S34: NO), the correction amount calculation unit 1252 proceeds to step S37.

[0063] On the other hand, if the rotation speed is above a threshold (step S34: YES), the correction amount calculation unit 1252 rewrites the angular velocity of the yaw angle in the vibration information to zero (step S35). Then, the control device 125 inputs the vibration information calculated by the correction amount calculation unit 1252 to the blur correction unit 1220 (step S36). As a result, the blur correction unit 1220 performs correction in the pitch direction but not in the yaw direction. Next, the image output unit 1255 outputs the image captured by the imaging device 122 (step S37), and the process shown in Figure 8 is completed.

[0064] Action / Effect Thus, according to the fourth embodiment, the control device 125 controls the vibration correction unit 1220 to rewrite the yaw angle angular velocity of the vibration information to zero while the work machine 100 is rotating. This prevents the image, which should appear to be moving during rotation, from appearing to remain stationary. Therefore, it is possible to reduce the operator's discomfort during rotation and prevent a decrease in operator operability. Furthermore, when the machine is not rotating, vibration correction is performed in all three axes, eliminating vibrations caused by the work machine 100. Therefore, motion sickness in the operator during operations that do not involve rotation, such as excavation and loading, can be suppressed.

[0065] Furthermore, according to the fourth embodiment, the control device 125 detects whether the work machine 100 is rotating based on rotation speed information indicating the rotation speed of the work machine 100. This makes it possible to easily and accurately detect whether or not the work machine 100 is rotating.

[0066] Furthermore, according to the fourth embodiment, the control device 125 detects that the vehicle is rotating when the rotation speed is above a threshold. This allows the vehicle to be considered as rotating when a rotation speed is obtained through rotation operations such as excavation or loading, and thus allows for correction of wobble in the yaw axis direction as well. Therefore, motion sickness during operations such as excavation and loading can be suppressed.

[0067] <Fifth Embodiment> In the fourth embodiment, the rotation detection unit 1253 detects that the machine is rotating using rotation speed information. In the fifth embodiment, in addition to or instead of the above configuration, a work machine 100 is described in which the rotation detection unit 1253 detects that the machine is rotating using the amount of rotational operation of the operating device 530.

[0068] Figure 9 is a schematic block diagram showing the configuration of the remote system according to the fifth embodiment. When the operating device 530 receives a rotation command, it outputs the received control amount to the control device 540. The control device 540 transmits the input control amount to the work machine 100 via the communication unit 550.

[0069] The control device 125 of the work machine 100 includes an operation amount acquisition unit 1256 in addition to the configuration of the fourth embodiment. The operation amount acquisition unit 1256 acquires the operation amount received by the communication unit 126. The slewing detection unit 1253 detects that the machine is slewing based on the operation amount acquired by the operation amount acquisition unit 1256. In the fifth embodiment, the work machine 100 does not include the slewing speed sensor 123, which is part of the configuration of the fourth embodiment. Also, the control device 125 does not include the speed information acquisition unit 1254, which is part of the configuration of the fourth embodiment.

[0070] "method" Here, we will describe the method for outputting captured images performed by the work machine 100 according to the fifth embodiment. Figure 10 is a flowchart showing the method for outputting captured images performed by the work machine according to the fifth embodiment. The vibration information acquisition unit 1251 acquires vibration information (step S41). Then, the control device 125 inputs the vibration information to the shake correction unit 1220 (step S42). As a result, the shake correction unit 1220 performs corrections in the pitch direction, roll direction, and yaw direction. Next, the manipulated amount acquisition unit 1256 acquires the manipulated amount for turning (step S43).

[0071] The rotation detection unit 5105 then determines whether the amount of rotation operation is greater than or equal to a threshold (step S44). If the amount of rotation operation is not greater than or equal to the threshold (step S44: NO), the process proceeds to step S48. On the other hand, if the amount of rotation operation is greater than or equal to the threshold (step S44: YES), the correction amount calculation unit 1252 rewrites the angular velocity of the yaw angle in the vibration information to zero (step S45). The control device 125 then inputs the vibration information calculated by the correction amount calculation unit 1252 to the shake correction unit 1220 (step S46). As a result, the shake correction unit 1220 performs correction in the pitch direction but not in the yaw direction.

[0072] Then, the image output unit 1255 outputs the image captured by the imaging device 122 (step S47), and the process shown in Figure 10 is completed. The output image is input to the control device 540 of the remote control device 500 via the communication units 126 and 550 and displayed on the display device 520.

[0073] In the fifth embodiment, if the slewing speed is above a threshold, the angular velocity of the yaw angle in the vibration information may be rewritten to zero. Specifically, in the fifth embodiment, the work machine 100 only needs to be equipped with a slewing speed sensor 123. The control device 125 only needs to be equipped with a speed information acquisition unit 1254. Then, in step S34, if the amount of slewing operation is not above a threshold, the speed information acquisition unit 1254 should determine whether the slewing speed of the work machine 100 is above a threshold. Furthermore, if the slewing speed is above a threshold, the correction amount calculation unit 1252 should rewrite the angular velocity of the yaw angle in the vibration information to zero.

[0074] Action / Effect Thus, according to the fifth embodiment, the control device 125 detects that the machine is rotating based on the amount of rotation operation. This reduces the operator's discomfort during rotation, thereby preventing a decrease in operator operability. Furthermore, it allows for easy and accurate detection of whether or not the work machine 100 is rotating.

[0075] <Sixth Embodiment> In the fourth embodiment, when the slewing body 120 is slewing, no wobble correction is performed in the yaw axis direction. In the sixth embodiment, a work machine 100 is described in which no wobble correction is performed in the yaw axis direction, regardless of whether the slewing body 120 is slewing or not.

[0076] In the sixth embodiment, the work machine 100 does not include the swivel speed sensor 123 as in the fourth embodiment. Also, the control device 125 does not include the swivel detection unit 1253 and the speed information acquisition unit 1254 as in the fourth embodiment. The correction amount calculation unit 1252 always rewrites the angular velocity in the yaw axis direction of the vibration information to zero.

[0077] "method" Here, we will describe the method for controlling the display of captured images performed by the remote control device 500 according to the sixth embodiment. The vibration information acquisition unit 1251 acquires vibration information. Then, it overwrites the angular velocity of the yaw angle in the vibration information to zero. Next, the control device 125 inputs the vibration information calculated by the correction amount calculation unit 1252 to the shake correction unit 1220. As a result, the shake correction unit 1220 performs corrections in the pitch direction and roll direction, but does not perform corrections in the yaw direction.

[0078] The image output unit 1255 then outputs the image captured by the imaging device 122 and terminates processing. The output image is input to the control device 540 of the remote control device 500 via the communication units 126 and 550 and displayed on the display device 520.

[0079] Action / Effect Thus, according to the sixth embodiment, the control device 125 constantly rewrites the yaw angle angular velocity in the vibration information to zero. This makes it possible to suppress the display of an image that should appear to be moving during rotation, but instead appears to be stationary, with simple control. Therefore, it is possible to reduce the operator's discomfort during rotation and easily prevent a decrease in operator operability.

[0080] <Other Embodiments> Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes are possible. For example, although the vibration correction unit corrects vibration in three axes, it may also correct vibration in two axes, yaw and pitch, or in the case of two-axis vibration correction, yaw vibration correction may not be performed during turning. In another embodiment, in a vehicle using a hydraulic drive system, vibration can be detected by pressure fluctuations of the hydraulic actuator. Specifically, vibration can be detected by monitoring pressure fluctuations of the hydraulic cylinder and hydraulic motor.

[0081] In the above-described embodiment of the control device 540 and control device 125, the case in which programs P and Q are stored in storage 5300 and 1258, respectively, has been described, but the invention is not limited to this. For example, programs P and Q may be delivered to the control device 125 and control device 540, respectively, via a communication line. In this case, the control device 125 and control device 540, upon receiving the deliveries, expand programs P and Q into main memory 5200 and 1257, respectively, and execute the above processing.

[0082] Furthermore, programs P and Q may each be used to implement some of the functions described above. For example, programs P and Q may each implement the functions described above in combination with other programs P and Q stored in storage 5300 and 1258, or in combination with other programs P and Q implemented in other devices.

[0083] Furthermore, control devices 125 and 540 may each be equipped with a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some of the functions realized by processors 5100 and 1250 may be realized by the respective PLDs.

[0084] Furthermore, control devices 125 and 540 may each comprise multiple processors 5100 and 1250, or may be composed of multiple computers.

[0085] Furthermore, although the above description describes the control device 125 as being equipped on the work machine 100, it is not limited to this configuration. The control device 125 may be equipped on an external computer device (for example, a cloud server). Also, all or part of the functional units of the control device 125 (vibration information acquisition unit 1251, correction amount calculation unit 1252, rotation detection unit 1253, speed information acquisition unit 1254, image output unit 1255, operation amount acquisition unit 1256, etc.) may be equipped on an external computer device. For example, all or part of the functional units of the control device 125 may be equipped on one computer device, or on multiple computer devices. If the functional units are equipped on an external computer device, the remote control device 500 should be configured to receive various information from the external computer device.

[0086] The same applies to the control device 540. That is, although the above description described the control device 540 as being provided in the remote control device 500, it is not limited to this configuration. The control device 540 may be provided in an external computer device (for example, a cloud server). Also, all or part of the functional units of the control device 540 (image acquisition unit 5101, vibration information acquisition unit 5102, blur correction unit 5103, speed information acquisition unit 5104, rotation detection unit 5105, display control unit 5106, operation amount acquisition unit 5110, etc.) may be provided in an external computer device. For example, all or part of the functional units of the control device 540 may be provided in one computer device, or in multiple computer devices. If each functional unit is provided in an external computer device, the display device 520 only needs to display the various information received from the external computer device. [Explanation of Symbols]

[0087] 1…Remote control system 100…Working machine 122…Imaging device 123…Swivel speed sensor 124…Vibration sensor 125…Control device 500…Remote control device 510…Driver's seat 520…Display device 530…Operating device 540…Control device 1220…Shake correction unit 1251…Vibration information acquisition unit 1252…Correction amount calculation unit 1253…Swivel detection unit 1254…Speed ​​information acquisition unit 1255…Image output unit 1256…Operation amount acquisition unit 5101…Image acquisition unit 5102…Vibration information acquisition unit 5103…Shake correction unit 5104…Speed ​​information acquisition unit 5105…Swivel detection unit 5106…Display control unit 5110…Operation amount acquisition unit

Claims

1. An image acquisition unit that acquires images captured by an imaging device installed in a remotely controlled work machine. A vibration information acquisition unit that acquires vibration information indicating the vibration of the aforementioned work machine. A rotation detection unit that detects whether the rotating body of the aforementioned work machine is rotating. A blur correction unit that corrects the blur of the image based on the vibration information. Equipped with, The aforementioned image stabilization unit is When the aforementioned rotation is in progress, blur correction is disabled in the yaw axis direction of the image, and blur correction is enabled in the roll axis direction and pitch axis direction. When not in the aforementioned turning state, all wobble corrections in the roll axis direction, pitch axis direction, and yaw axis direction are enabled. Remote control device.

2. The system further includes a speed information acquisition unit that acquires rotation speed information indicating the rotation speed of the aforementioned work machine, The rotation detection unit detects that rotation is in progress based on the rotation speed information. The remote control device according to claim 1.

3. The turning detection unit detects that the turning is in progress when the turning speed is equal to or greater than a threshold. The remote control device according to claim 2.

4. The aforementioned work machine is equipped with an operation amount acquisition unit that acquires the amount of operation related to the rotation of the operating unit, The rotation detection unit detects that rotation is in progress based on the operation amount. The remote control device according to claim 1 or 2.

5. The steps include acquiring an image captured by an imaging device installed on a remotely controlled work machine, and A step of acquiring vibration information indicating the vibration of the aforementioned work machine. A step of detecting that the rotating body of the aforementioned work machine is rotating. A motion correction step that corrects the image based on the vibration information. Execute, In the aforementioned image stabilization step, When the aforementioned rotation is in progress, blur correction is disabled in the yaw axis direction of the image, and blur correction is enabled in the roll axis direction and pitch axis direction. When not in the aforementioned turning state, all wobble corrections in the roll axis direction, pitch axis direction, and yaw axis direction are enabled. A method for remote control.

6. An image acquisition unit that acquires images captured by an imaging device installed in a remotely controlled work machine. A vibration information acquisition unit that acquires vibration information indicating the vibration of the aforementioned work machine. A rotation detection unit that detects whether the rotating body of the aforementioned work machine is rotating. A blur correction unit that corrects the blur of the image based on the vibration information. Equipped with, The aforementioned image stabilization unit is When the aforementioned rotation is in progress, blur correction is disabled in the yaw axis direction of the image, and blur correction is enabled in the roll axis direction and pitch axis direction. When not in the aforementioned turning state, all wobble corrections in the roll axis direction, pitch axis direction, and yaw axis direction are enabled. Remote control system.

Images