Remote operation support device, remote operation support system, and remote operation support method

Abstract

To provide a remote control support device etc. capable of reducing a load of arithmetic processing for causing an operator remotely controlling a work machine to recognize a physical amount such as a distance to a target object included in a work environment image acquired via an actual machine imaging device mounted on the work machine.SOLUTION: A work environment image Im1 is output by a first remote image output device 221, and an assistance image Im2 is output by a second remote image output device 222. The second remote image output device 222 is a transparent remote image output device disposed on this side of the first remote image output device 221 in a view from an operator. In order that actual space positions in each of a work environment image coordinate system and an assistance image coordinate system may match, coordinate transformation processing is applied to the work environment image coordinate system and / or the assistance image coordinate system.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for assisting an operator in remotely controlling a work machine such as a hydraulic excavator. [Background technology]

[0002] Conventionally, a work machine has been remotely operated using a remote control device, in which an operator operates the work machine while viewing an image captured by a camera mounted on the work machine on a display installed on the remote control device.

[0003] An example of such a remote control system for a work machine is disclosed in Patent Document 1, which discloses a remote control system in which a work machine equipped with a working unit is remotely controlled wirelessly by a controller provided in an operator's seat, the remote control system having a camera provided on the work machine for photographing the working unit, and a display provided in the operator's seat to which the video signal from the camera is wirelessly transmitted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-168778 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, for example, when an operator operates the operating mechanism of a work machine, the sense of depth felt by the operator through the remote image is difficult to grasp compared to the sense of depth felt by the operator actually riding the work machine, which may result in a decrease in the efficiency of work using the work machine.

[0006] It is also possible to synthesize the captured image and the auxiliary image and display the synthesized image on a display. In this case, since the two images must be synthesized at any time, there is a problem that the computational processing load increases. In addition, the image data has a large data volume, and communication volume also increases, so there is a risk of display delays. In such a case, it may be difficult for the operator to visually recognize the image in real time.

[0007] Therefore, an object of the present invention is to provide a remote control device etc. that can reduce the computational processing load required to enable an operator remotely operating a work machine to recognize physical quantities such as the distance of a target object reflected in a work environment image acquired through an actual machine imaging device mounted on the work machine. [Means for solving the problem]

[0008] The remote operation support device of the present invention comprises: A remote operation assistance device for assisting an operator in remotely operating a work machine through a remote operation device, a first support processing element that causes a first image output device mounted on a remote control device to output a work environment image corresponding to an image captured by an imaging device mounted on the work machine, the image representing the surrounding environment of the work machine, based on communication with the remote control device; a second support processing element that outputs, based on communication with the remote control device, an auxiliary image that represents physical quantities in the vicinity of the work machine recognized by an actual environment recognition device mounted on the work machine, to a second image output device that is disposed in the remote control device so as to overlap in front of the first image output device as viewed from the operator, so that the work environment image output to the first image output device can be viewed; an image processing element that performs coordinate transformation on at least one of the work environment image coordinate system and the auxiliary image coordinate system based on a real space position so that a work environment image coordinate system, which is a coordinate system of the work environment image output to the first image output device, and a coordinate system of the auxiliary image output to the second image output device at least partially coincide with each other; It is equipped with:

[0009] The remote operation support system of the present invention is composed of the remote operation support device of the present invention and the remote operation device.

[0010] According to the remote operation support device or remote operation system (hereinafter referred to as "remote operation support device, etc.") having such a configuration, a work environment image is output to the first image output device, and an auxiliary image is output to the second image output device. The second image output device is a transparent image output device arranged in front of the first image output device as seen from the operator. In addition, a coordinate conversion process is performed on the work environment image coordinate system and / or the auxiliary image coordinate system so that the real space positions in the work environment image coordinate system and the auxiliary image coordinate system match. This allows the operator to visually recognize or estimate the physical quantity of the target object in the work environment image output to the first image output device through the auxiliary image output to the second image output device. At this time, the image processing load is reduced and display delays can be prevented by the amount that the synthesis process of the work environment image and the auxiliary image is omitted.

[0011] In the remote operation support device having the above configuration, When at least one of the operating state and the environment of the work machine satisfies a first specified condition, it is preferable that the second support processing element causes the second image output device to output the auxiliary image.

[0012] According to the remote operation support device etc. having such a configuration, when the operating state and / or environment of the work machine satisfy the first specified condition, an auxiliary image is output to the second image output device. On the other hand, when the operating state and / or environment of the work machine do not satisfy the first specified condition, output of the auxiliary image in the second image output device is omitted or stopped. Therefore, unlike the case where the auxiliary image is constantly output to the second image output device, the annoyance of the operator of the remote operation device being forced to view unnecessary and non-urgent auxiliary images is reduced, and further the visibility of the work environment image of the first image output device is improved.

[0013] In the remote operation support device having the above configuration, It is preferable that the image processing element adjusts a resolution of at least one of the work environment image and the auxiliary image depending on at least one of an operating state and an environment of the work machine.

[0014] According to the remote operation support device having this configuration, the resolution of the work environment image and / or auxiliary image is adjusted according to the operating state and / or environment of the work machine. This allows the first image output device and / or the second image output device to output a work environment image and / or auxiliary image with an appropriate resolution in consideration of the operating state and / or environment of the work machine, thereby improving the visibility of the work environment image and / or auxiliary image for the operator.

[0015] In the remote operation support device having the above configuration, When at least one of the operating state of the work machine and the environment satisfies a second specified condition, it is preferable that the image processing element omits performing coordinate transformation on the work environment image coordinate system and the auxiliary image coordinate system.

[0016] According to the remote operation support device having this configuration, when the operating state and / or environment of the work machine satisfies the second specified condition, the coordinate conversion of the work environment image coordinate system and / or the auxiliary image coordinate system is omitted. Since the captured image and the auxiliary image output to each image output device are not aligned, the real space positions of the work environment image coordinate system and the auxiliary image coordinate system do not match. However, by changing the line of sight of the operator looking at the first image output device through the second image output device, both images can be viewed in a form in which the real space positions of the work environment image coordinate system and the auxiliary image coordinate system at least partially match. Since the coordinate conversion of the captured image coordinate system and the auxiliary image coordinate system is omitted, the calculation processing load is reduced and display delays can be prevented. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is an explanatory diagram regarding the configuration of a remote operation support system. [Diagram 2] FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. [Diagram 3] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 4] FIG. 2 is an explanatory diagram regarding functions of the remote operation support device. [Diagram 5] FIG. [Figure 6] FIG. 13 is an explanatory diagram regarding a work environment image and an auxiliary image (with coordinate transformation). [Figure 7] FIG. 13 is an explanatory diagram of a work environment image and an auxiliary image (without coordinate transformation). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] (Configuration of remote operation support system) The remote operation support system shown in Fig. 1 is composed of a remote operation support device 10, a remote operation device 20 and / or a work machine 40 that can be remotely operated via the remote operation device 20. Only one of the remote operation device 20 and the work machine 14 may be a component of the remote operation support device. The remote operation support device 10, the remote operation device 20 and the work machine 40 are configured to be able to communicate with each other via a network. The mutual communication network of the remote operation support device 10 and the remote operation device 20 and the mutual communication network of the remote operation support device 10 and the work machine 40 may be the same as or different from each other.

[0019] (Configuration of remote operation support device) The remote operation support device 10 is configured with one or more server computers, and includes a database 102, a first support processing element 121, a second support processing element 122, and an image processing element 124. The database 102 stores captured image data and the like. The database 102 may be configured with a database server separate from the remote operation support device 10. Each support processing element is configured with an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing (described later) on the data in accordance with the software.

[0020] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

[0021] The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes a first remote image output device 221, a second remote image output device 222, and a remote wireless communication device 224.

[0022] The remote operation mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move a hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.

[0023] The operating levers constituting the remote control mechanism 211 are arranged around a seat St on which the operator sits, as shown in Fig. 2. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or in any other form on which the operator can sit.

[0024] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator.

[0025] The first remote image output device 221 is composed of one or more displays arranged in front of the seat St. For example, as shown in Fig. 2, the first remote image output device 221 is composed of three displays arranged diagonally forward to the left and diagonally forward to the right of the seat St, namely, a first center display 2210, a first left display 2211, and a first right display 2212. Each of the first center display 2210, the first left display 2211, and the first right display 2212 is composed of, for example, a liquid crystal display having a substantially rectangular screen (image display area).

[0026] 2, the right edge of the first left display 2211 is adjacent to the left edge of the first central display 2210, such that the screens of the first central display 2210 and the first left display 2211 form an inclined angle (e.g., 120° to 150°). As shown in FIG 2, the left edge of the first right display 2212 is adjacent to the right edge of the first central display 2210, such that the screens of the first central display 2210 and the first right display 2212 form an inclined angle (e.g., 120° to 150°). The inclination angles may be the same or different.

[0027] The number of displays constituting the first remote image output device 221, and the shape, size, and arrangement of each display may be changed in various ways. For example, the screens of the first center display 2210, the first left display 2211, and the first right display 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the image output devices among the first center display 2210, the first left display 2211, and the first right display 2212 may be composed of a plurality of divided image output devices. For example, the first center display 2210 may be composed of a pair of image output devices adjacent to each other vertically, each having a substantially rectangular screen.

[0028] The second remote image output device 222 is composed of one or more displays arranged in front of the seat St so as to overlap the front side of the first remote image output device 221 with respect to the headrest of the seat St. The second remote image output device 222 is composed of three displays arranged in front of the seat St, diagonally forward to the left, and diagonally forward to the right, namely, a second center display 2220, a second left display 2221, and a second right display 2222, as shown in FIG. 2 for example. Each of the second center display 2220, the second left display 2221, and the second right display 2222 is composed of a transparent display such as a transparent organic EL display having a substantially rectangular screen (image display area). Therefore, the operator seated in the seat St can view the first remote image output device 221 arranged behind the second remote image output device 222 through the second remote image output device 222.

[0029] 2, the right edge of the second left display 2221 is adjacent to the left edge of the second central display 2220, such that the screens of the second central display 2220 and the second left display 2221 form an inclined angle (e.g., 120° to 150°). As shown in FIG 2, the left edge of the second right display 2222 is adjacent to the right edge of the second central display 2220, such that the screens of the second central display 2220 and the second right display 2222 form an inclined angle (e.g., 120° to 150°). The inclination angles may be the same or different.

[0030] The number of displays constituting the first remote image output device 221, and the shape, size, and arrangement of each display may be changed in various ways. For example, the screens of the second center display 2220, the second left display 2221, and the second right display 2222 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the image output devices among the second center display 2220, the second left display 2221, and the second right display 2222 may be composed of a plurality of divided image output devices. For example, the second center display 2220 may be composed of a pair of image output devices adjacent to each other vertically, each having a substantially rectangular screen.

[0031] The number of displays constituting the first image output device 221 and the number of displays constituting the second image output device 222 may be the same or different. For example, the first image output device 221 may be composed of three displays 2210, 2211, and 2212, and the second image output device 222 may be composed of one display 2220 arranged so as to overlap at least the first central display 2210 among the three displays 2210, 2211, and 2212 when viewed from the seat St.

[0032] (Work machine configuration) The work machine 40 comprises an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a work mechanism 440. The actual machine control device 400 and each of its components are configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

[0033] The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 410, and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A working mechanism 440 is provided in the front center of the upper rotating body 420, extending forward from the bottom as a base.

[0034] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a real machine environment recognition device 414. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside a cab 424 in the same manner as the remote control mechanism 211. A drive mechanism or robot is provided in the cab 424 for receiving a signal corresponding to the operation mode of the remote control lever and moving the real machine operation lever based on the received signal.

[0035] The actual machine imaging device 412 is installed, for example, inside the cab 424, and captures an image of the environment including at least a part of the working mechanism 440 through the front window and a pair of left and right side windows. Some or all of the front window and the side windows may be omitted.

[0036] The actual machine environment recognition device 414 is composed of sensors for detecting the physical quantities of target objects present around the work machine 40. The sensors include distance measuring sensors for measuring the distance to target objects (e.g., workers, other work machines, the ground, earth and sand, rubble, buildings and / or vehicles, etc.) present around (mainly in front of) the work machine 40. The sensors may include an infrared sensor (infrared thermometer) for measuring the temperature of the target object. The resolution of the captured image acquired through the actual machine imaging device 412 and the resolution of the distance image acquired through the actual machine environment recognition device 414, in which the pixel value represents the distance to the target object, are generally different.

[0037] The real device output interface 42 includes a real device wireless communication device 422 .

[0038] 3, a working mechanism 440 as a working mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are mounted on the working mechanism 440.

[0039] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.

[0040] (function) The functions of the remote operation support device having the above configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which processing in the branching direction is executed on the condition that the data is transmitted and / or received.

[0041] In the remote operation device 20, an environment confirmation request is transmitted to the remote operation assistance device 10 via the remote wireless communication device 224 (FIG. 4 / STEP 210). For example, a requirement for starting transmission of the environment confirmation request may be that the operator has performed a designation operation via the remote input interface 210. The "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for designating the work machine 40 that the operator intends to remotely operate.

[0042] When the environment confirmation request is received in the remote operation support device 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4 / C10).

[0043] When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (Figure 4 / C40), the actual machine control device 400 acquires an image via the actual machine imaging device 412 and acquires a distance image via the actual machine environment recognition device 414, and then transmits the captured image data representing the captured image and the distance image data representing the distance image to the remote control device 10 via the actual machine wireless communication device 422 (Figure 4 / STEP 410).

[0044] In the remote operation support device 10, when the first support processing element 121 receives captured image data and distance image data (Figure 4 / C11), the second support processing element 122 determines whether the operating state and / or environment of the work machine 40 satisfies the first specified condition (Figure 4 / STEP 110).

[0045] For example, at least one of the posture of the work mechanism 440, the posture change speed of the work mechanism 440, the travel speed and / or acceleration of the work machine 40, the load on the work mechanism 440, the drive speed, the rotation angular velocity of the upper rotating body 420 relative to the lower running body 410, and the operation mode of the remote control mechanism 211 for the work mechanism 440 corresponds to the “operating state” of the work machine 40.

[0046] The attitude of the working mechanism 440 is measured through angle sensors for measuring the hoisting angle of the boom 441 relative to the upper rotating body 420, the rotation angle of the arm 443 relative to the tip of the boom 441, and / or the rotation angle of the bucket 445 relative to the tip of the arm 443. The attitude change speed of the working mechanism 440 is calculated based on the angles measured through the angle sensors. The travel speed and / or acceleration of the working machine 40 is measured through a speed sensor and / or an acceleration sensor provided on the lower traveling body 410 and / or the upper rotating body 420. The load applied to the working mechanism 440 is estimated based on the fuel consumption of the power source (engine) of the working machine 40 measured through a fuel gauge and / or the oil pressure measured through a pressure sensor installed in a pipe that leads the hydraulic oil discharged by the pump to the working mechanism 440. The swing angular velocity of the upper rotating body 420 relative to the driving speed lower traveling body 410 is calculated based on the swing angle measured through a swing angle sensor. The operation mode of the remote control mechanism 211 of the working mechanism 440 is estimated through a pressure sensor provided on an operation lever constituting the remote control mechanism 211 and / or a tilt angle sensor of the operation lever.

[0047] Furthermore, the relative positional relationship or distance between the work machine 40 or its components (e.g., bucket 445, upper rotating body 420) and a target object (e.g., the ground, earth, sand, rubble, or a structure or another work machine to be worked on) and / or the speed of change thereof corresponds to the "environment" of the work machine 40. The distance between the work machine 40 or its components (e.g., bucket 445, upper rotating body 420) and a target object is measured, for example, via a distance measuring sensor constituting the actual environment recognition device 414 mounted at a specified location on the work machine 40.

[0048] The "first specified condition" is defined as the distance between the work machine 40 or its components and the target object being less than or equal to a specified distance, the attitude of the work mechanism 440 being a specified attitude, the attitude of the work mechanism 440 having changed from the specified attitude to an attitude other than the specified attitude, the attitude change speed of the work mechanism 440 being equal to or less than a specified speed, the load on the work mechanism 440 being equal to or less than a specified value, the advancement speed of the work machine 40 being equal to or less than a specified advancement speed, the rotation speed of the upper rotating body 420 relative to the lower running body 410 being equal to or less than a specified speed, and the operation mode of the remote control mechanism 211 for the work mechanism 440 being a specified operation mode, or a combination of these.

[0049] If it is determined that the operating state and / or environment of the work machine 40 does not satisfy the first specified condition (FIG. 4 / STEP 110...NO), the second support processing element 122 transmits work environment image data corresponding to the captured image to the remote operation device 20 (FIG. 4 / STEP 112). The work environment image data is not only the captured image data itself, but also image data representing a simulated work environment image generated based on the captured image.

[0050] When the remote operation device 20 receives work environment image data through the remote wireless communication device 224 (FIG. 4 / C21), the remote control device 200 outputs a work environment image corresponding to the work environment image data to the first remote image output device 221 (FIG. 4 / STEP 212).

[0051] 5, for example, a work environment image Im1 in which a boom 441, an arm 443, and a pile of rubble or earth and sand (the target of work by the bucket 445), which are part of the work mechanism 440, are reflected in front of the cab 424 is output to the first remote image output device 221 (particularly the first central display 2210) through a window frame that defines the cab 424. At this time, no auxiliary image is output to the second remote image output device 222, which remains transparent, and the operator can view the work environment image Im1 output to the first remote image output device 221 as it is through the second remote image output device 222.

[0052] If it is determined that the operating conditions and / or environment of the work machine 40 satisfy the first specified condition (FIG. 4 / STEP 110...YES), the second support processing element 122 further determines whether the operating conditions and / or environment of the work machine 40 satisfy a second specified condition (FIG. 4 / STEP 114).

[0053] The "second specified condition" is defined as the distance between the work machine 40 or its components and the target object being less than or equal to a specified distance, the attitude of the work mechanism 440 being a specified attitude, the attitude of the work mechanism 440 having changed from the specified attitude to an attitude other than the specified attitude, the attitude change speed of the work mechanism 440 being equal to or less than a specified speed, the load on the work mechanism 440 being equal to or less than a specified value, the advancement speed of the work machine 40 being equal to or less than a specified advancement speed, the rotation speed of the upper rotating body 420 relative to the lower running body 410 being equal to or less than a specified speed, and the operation mode of the remote control mechanism 211 for the work mechanism 440 being a specified operation mode, or a combination of these.

[0054] The second specified condition is defined as a condition different from the first specified condition. When the second specified condition is defined as the same condition as the first specified condition, the determination process (see FIG. 4 / STEP 114) may be omitted.

[0055] When it is determined that the operating state and / or environment of the work machine 40 satisfies the second specified condition (FIG. 4 / STEP 114 . . . YES), the image processing element 124 calculates the working environment image coordinate system (u 1 ,v 1 ) and the auxiliary image coordinate system (u 2 ,v 2 ) (distance image coordinate system), a coordinate transformation process is performed on the work environment image coordinate system and / or the auxiliary image coordinate system so that the real space positions of at least some of the pixels that are expected to overlap from the operator's perspective coincide (Figure 4 / STEP 116).

[0056] For example, in the coordinate conversion process, operators of a translation matrix and a rotation matrix (or an equivalent quaternion, etc.) are used to match the positions and attitudes (optical axis directions) of the real machine imaging device 412 and the real machine environment recognition device 414 in the real machine coordinate system (a coordinate system in which the position and attitude are fixed with respect to the work machine 40). The operators may be stored and held in a storage device constituting the real machine control device 400, or may be associated with the real machine identifier and registered in the database 102. The coordinate values ​​of the real space position in the real machine coordinate system are estimated based on the image coordinate values ​​and pixel values ​​(distance) of each pixel of the distance image acquired through the real machine environment recognition device 414.

[0057] Next, the second support processing element 122 transmits auxiliary image data corresponding to the distance image, in addition to the work environment image data corresponding to the captured image, to the remote operation device 20 (FIG. 4 / STEP 118). The auxiliary image data is image data created based on the distance image data, such as the distance image data itself, a mesh consisting of a plurality of spaced line segments parallel to each of the X-axis and Y-axis in the real space coordinate system or the real machine coordinate system, numbers indicating distance, and point cloud data in which perspective is expressed by large / small and / or light and shade. The auxiliary image data may include image data indicating the operating state and / or environment of the work machine 40.

[0058] When the remote operation device 20 receives work environment image data and auxiliary image data through the remote wireless communication device 224 (Figure 4 / C22), the remote control device 200 outputs a work environment image corresponding to the work environment image data to the first remote image output device 221, and outputs an auxiliary image corresponding to the auxiliary image data to the second remote image output device 222 (Figure 4 / STEP 214).

[0059] As a result, for example, as shown in Fig. 6, a work environment image Im1 in which the boom 441, arm 443, and pile of rubble or earth and sand (the target of work by the bucket 445), which are part of the work mechanism 440, are reflected in front of the cab 424 through the window frame that defines the cab 424, as in Fig. 5, is output to the first remote image output device 221. At this time, as shown in Fig. 6, an auxiliary image Im2 is output to the second remote image output device 222 (particularly, the second central display 2220).

[0060] The auxiliary image Im2 includes a mesh image Im21 configured with a plurality of line segments parallel to the X-axis and Y-axis of the real space coordinate system, such as latitude and longitude lines. The auxiliary image Im2 includes an image Im22 representing the operating state and / or environment of the work machine 40. In the second remote image output device 222, the area other than the auxiliary image Im2 remains transparent, and the operator can view the work environment image Im1 output to the first remote image output device 221 through the second remote image output device 222 with the auxiliary image Im2 superimposed in front of it. The auxiliary image Im2 may be displayed semi-transparently.

[0061] If it is determined that the operating state and / or environment of the work machine 40 does not satisfy the second specified condition (FIG. 4 / STEP 114...NO), the coordinate conversion processing by the image processing element 124 (see FIG. 4 / STEP 116) is omitted, and the second support processing element 122 transmits work environment image data corresponding to the captured image and auxiliary image data corresponding to the distance image to the remote control device 20 (FIG. 4 / STEP 118).

[0062] When the remote operation device 20 receives work environment image data and auxiliary image data through the remote wireless communication device 224 (Figure 4 / C22), the remote control device 200 outputs a work environment image corresponding to the work environment image data to the first remote image output device 221, and outputs an auxiliary image corresponding to the auxiliary image data to the second remote image output device 222 (Figure 4 / STEP 214).

[0063] 7, for example, in the same way as in Fig. 6, the work environment image Im1 is output to the first remote image output device 221, and the auxiliary image Im2 is output to the second remote image output device 222. Since the coordinate conversion process has not been executed, the real space position of the target object (e.g., the ground, earth, sand, rubble, buildings, etc. present in front of the work machine 40) shown in the mesh image Im21 and the real space position of the displayed object displayed in the work environment image Im1 are shifted, unlike in Fig. 6.

[0064] In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and transmits a remote operation command corresponding to the operation mode to the remote operation support device 10 via the remote wireless communication device 224 (FIG. 4 / STEP 220).

[0065] In the remote operation support device 10, when the second support processing element 122 receives the remote operation command, the first support processing element 121 transmits the remote operation command to the work machine 40 (FIG. 4 / C12).

[0066] In the work machine 40, when an operation command is received by the actual machine control device 400 via the actual machine wireless communication device 422 (FIG. 4 / C42), the operation of the work mechanism 440 and the like is controlled (FIG. 4 / STEP 420). For example, the work of scooping up soil in front of the work machine 40 with the bucket 445, rotating the upper rotating body 410, and dropping the soil from the bucket 445 is performed.

[0067] (Action and effect) According to the remote operation support system having such a configuration or the remote operation support device 10 which is a component of the system, the work environment image Im1 is output to the first remote image output device 221, and the auxiliary image Im2 is output to the second remote image output device 222 (see FIG. 4 / STEP 118->STEP 214, FIG. 6 and FIG. 7). The second remote image output device 222 is a transparent remote image output device arranged in front of the first remote image output device 221 as seen from the operator (see FIG. 2). Moreover, a coordinate conversion process is performed on the work environment image coordinate system and / or the auxiliary image coordinate system so that the real space positions in the work environment image coordinate system and the auxiliary image coordinate system are coincident with each other (see FIG. 4 / STEP 116).

[0068] This allows the operator to visually recognize or estimate the physical quantity of the target object in the work environment image Im1 output to the first remote image output device 221 through the auxiliary image Im2 output to the second remote image output device 222. Specifically, the mesh image Im21 allows the operator to visually recognize or estimate the real spatial distance of the target object, such as the ground reflected in the work environment image Im1, based on the work machine 40. At this time, since the synthesis process of the work environment image and the auxiliary image is omitted, the image processing load is reduced and display delays can be prevented.

[0069] When the operating state and / or environment of the work machine 40 satisfy the "first specified condition", the auxiliary image Im2 is output to the second remote image output device 222 (FIG. 4 / STEP 112...YES->STEP 118->STEP 214, see FIGS. 6 and 7). On the other hand, when the operating state and / or environment of the work machine 40 do not satisfy the "first specified condition", the output of the auxiliary image Im2 in the second remote image output device 222 is omitted or stopped (FIG. 4 / STEP 112...NO->STEP 114->STEP 212, see FIG. 5). Therefore, unlike the case where the auxiliary image Im2 is constantly output to the second remote image output device 222, the annoyance of the operator of the remote operation device 20 being forced to view unnecessary and non-urgent auxiliary images is reduced, and further the visibility of the work environment image Im1 in the first remote image output device 221 is improved.

[0070] The operating conditions and / or environment of the work machine 40 satisfy the "second specified condition". do not have In this case, the coordinate conversion process of the work environment image coordinate system and / or the auxiliary image coordinate system (see FIG. 4 / STEP 116) is omitted. Since the work environment image Im1 output to the first remote image output device 221 and the auxiliary image Im2 output to the second remote image output device 222 are not aligned, the real space positions of the work environment image coordinate system and the auxiliary image coordinate system do not match (see FIG. 7). However, by changing the line of sight of the operator facing the first remote image output device 221 through the second remote image output device 222, both images can be viewed in a form in which the real space positions of the work environment image coordinate system and the auxiliary image coordinate system at least partially match (see FIG. 6). Since the coordinate conversion of the captured image coordinate system and the auxiliary image coordinate system is omitted, the calculation processing load can be reduced and display delays can be prevented.

[0071] (Another embodiment of the present invention) In the above embodiment, the remote operation support device 10 is configured by a server computer, but in another embodiment, the remote operation support device 10 may be configured by a real machine control device 400 and / or a remote control device 200. In this case, the real machine control device 400 and / or the remote control device 200 has the functions of the first support processing element 121, the second support processing element 122, and / or the image processing element 124.

[0072] The image processing element 124 may adjust the resolution of at least one of the work environment image and the auxiliary image in accordance with at least one of the operating state and the environment of the work machine 40. According to the remote operation support device 10 configured in this manner, the resolution of the work environment image Im1 and / or the auxiliary image Im2 is adjusted in accordance with the operating state and / or the environment of the work machine 40.

[0073] For example, when the operating state and / or environment of the work machine 40 satisfies the first specified condition and / or the second specified condition, the image resolution is adjusted to be higher compared to when the first specified condition and / or the second specified condition is not satisfied. The resolution of the work environment image Im1 and / or the auxiliary image Im2 may be adjusted to be higher as the attitude change speed of the work mechanism 440, the advancement speed and / or acceleration of the work machine 40, the load on the work mechanism 440, the drive speed, the rotation angular speed of the upper rotating body 420 relative to the lower traveling body 410, and the operation speed of the remote control mechanism 211 for the work mechanism 440 are higher. The resolution of the work environment image Im1 and / or the auxiliary image Im2 may be adjusted to be higher as the interval between the work machine 40 or a component thereof (e.g., the bucket 445, the upper rotating body 420) and the target object (e.g., the ground, earth, rubble, or a building or other work machine to be worked on) is smaller and / or the approach speed between the work machine 40 or a component thereof and the target object is higher.

[0074] This allows the first remote image output device 221 and / or the second remote image output device 222 to output the work environment image Im1 and / or the auxiliary image Im2 with an appropriate resolution taking into account the operating state and / or environment of the work machine 40, thereby improving the visibility of the work environment image Im1 and / or the auxiliary image Im2 for the operator. [Explanation of symbols]

[0075] 10: Remote operation support server, 20: Remote operation device, 40: Work machine, 102: Database, 121: First support processing element, 122: Second support processing element, 124: Image processing element, 200: Remote control device, 210: Remote input interface, 211: Remote operation mechanism, 220: Remote output interface, 221: First remote image output device, 222: Second remote image output device, 400: Actual machine control device, 410: Actual machine input interface, 420: Actual machine output interface, 424: Cab (operator's compartment), 440: Work mechanism, 445: Bucket.

Claims

1. A remote operation assistance device for assisting an operator in remotely operating a work machine through a remote operation device, a first support processing element that causes a first image output device mounted on a remote control device to output a work environment image corresponding to an image captured by an imaging device mounted on the work machine, the image representing the surrounding environment of the work machine, based on communication with the remote control device; a second support processing element which is mounted on the work machine and causes a second image output device, which is arranged in the remote control device so as to overlap in front of the first image output device as viewed from the operator, to output an auxiliary image representing the distribution of distances, which is a physical quantity, in the vicinity of the work machine recognized by an actual environment recognition device constituted by a sensor which is mounted on the work machine and recognizes the distribution of distances of target objects existing in the vicinity of the work machine with respect to the work machine, based on communication with the remote control device, so that the work environment image output to the first image output device can be visually recognized; an image processing element which performs coordinate transformation on at least one of the work environment image coordinate system, which is the coordinate system of the work environment image output to the first image output device, and the auxiliary image coordinate system, which is the coordinate system of the auxiliary image output to the second image output device, so that real space positions in the work environment image coordinate system and the auxiliary image coordinate system at least partially coincide with each other for at least some pixels which are assumed to overlap as viewed by the operator; A remote operation support device comprising:

2. 2. The remote operation support device according to claim 1, When at least one of the operating state and the environment of the work machine satisfies a first specified condition, the second support processing element causes the second image output device to output the auxiliary image.

3. 3. The remote operation support device according to claim 1, A remote operation support device in which the image processing element adjusts the resolution of at least one of the work environment image and the auxiliary image depending on at least one of the operating state and the environment of the work machine.

4. In the remote operation support device according to any one of claims 1 to 3, When at least one of the operating state of the work machine and the environment does not satisfy a second specified condition, the image processing element omits performing coordinate transformation on the work environment image coordinate system and the auxiliary image coordinate system. Remote operation support device.

5. A remote operation support device according to any one of claims 1 to 4, the remote operation device, A remote operation support system comprising:

6. A remote operation assistance method for assisting an operator in remotely operating a work machine through a remote operation device, comprising: a first support processing step of causing a first image output device mounted on the remote control device to output a work environment image corresponding to an image captured by an imaging device mounted on the work machine, the image representing the surrounding environment of the work machine, based on communication with the remote control device; a second support processing step of outputting an auxiliary image representing the distribution of distances, which is a physical quantity in the vicinity of the work machine, recognized by an actual environment recognition device constituted by a sensor mounted on the work machine and configured to recognize the distribution of distances of target objects present in the vicinity of the work machine relative to the work machine, on a second image output device disposed in the remote control device so as to overlap in front of the first image output device as viewed from the operator, based on communication with the remote control device, so that the work environment image output to the first image output device can be visually recognized; an image processing step of performing coordinate transformation on at least one of the work environment image coordinate system, which is the coordinate system of the work environment image output to the first image output device, and the auxiliary image coordinate system, which is the coordinate system of the auxiliary image output to the second image output device, so that real space positions in the work environment image coordinate system and the auxiliary image coordinate system at least partially coincide with each other for at least some pixels that are assumed to overlap as viewed by the operator; A remote operation support method comprising:

Images