Remote control support system

Abstract

To provide a remote operation support system capable of giving an operator an appropriate feeling of operation.SOLUTION: A remote operation support system displays a combined image obtained by superimposing an index image corresponding to an index member arranged in the operator's cab of a work machine on an image captured by an imaging device mounted on the work machine and capable of capturing an image of the surroundings of the work machine on an output interface of a remote control device for remotely controlling the work machine.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote operation support system having an intercommunication function with each of a working machine and a remote operation device for remotely operating the working machine.

Background Art

[0002] When an operator performs an operation such as turning of a working machine, the operator checks whether there are obstacles around the working machine. For example, regarding the rear of the working machine, the operator installs a camera capable of imaging the rear of the working machine and checks for obstacles by checking an image captured by the camera.

[0003] For example, Patent Document 1 describes a rear view display device for a swing-type working vehicle that includes a camera for obtaining a rear view of an upper swing body and a monitor for displaying an image obtained by this camera, and that overlays and displays an arc-shaped distance display line centered on the swing center of the upper swing body on the image obtained by the camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, when an operator actually boards a working machine and performs an operation such as turning, as a so-called positioning reference for determining the position to stop turning, the operator uses something that comes into the field of view, such as a pillar of the cab.

[0006] Furthermore, when operating machinery remotely, the operator performs the operation while viewing an image that reflects the view from the cab. This image may not be captured by a camera mounted in the driver's seat inside the cab.

[0007] In such cases, one of the challenges for the operator is that it is difficult to get a sense of the operation because there is no reference point for positioning visible in the image.

[0008] This invention has been made in view of the above problems, and aims to provide a remote control support system that can provide operators with an appropriate sense of control. [Means for solving the problem]

[0009] To achieve this objective, the remote operation support system of the present invention is characterized by displaying a composite image, on the output interface of a remote operation device for remotely operating the work machine, which is obtained by superimposing an index image corresponding to an index member placed in the operator's cab of the work machine onto an image captured by an imaging device mounted on the work machine and capable of capturing images of the area around the work machine.

[0010] To achieve this objective, the remote control support system of the present invention, based on the positional relationship between a first imaging device located inside the operator's cab of a work machine and a second imaging device mounted on the work machine and capable of imaging the area around the work machine, and the orientation relationship between the first imaging device and the second imaging device, captures a second image in a coordinate system in which the second imaging device images in the direction along the optical axis of the second imaging device, and captures a first image in which the first imaging device images in the direction along the optical axis of the first imaging device, such that a pillar, which is an indicator member located in the operator's cab, is included in the first image. Indicator member The coordinates in the coordinate system of the first imaging device are converted to the coordinate system of the second imaging device, which images in the direction along the optical axis of the second imaging device, and the index is displayed at that position. Indicators according to the material The present invention is characterized by displaying a composite image, in which images are superimposed, on the output interface of a remote control device for remotely operating the aforementioned work machine.

[0011] In the remote operation support system of the present invention, it is preferable that the indicator image is displayed in a way that allows the region superimposed on the captured image to be visually recognized.

[0012] In this configuration, the operator can visually identify the region of the captured image superimposed on the indicator image.

[0013] In the remote operation support system of the present invention, it is preferable that the indicator image is displayed at a size corresponding to the magnification of the captured image.

[0014] In this configuration, a reference image is displayed according to the magnification of the captured image. This makes it possible for the operator to obtain an appropriate sense of distance from the composite image.

[0015] In the remote operation support system of the present invention, it is preferable to display the composite image on the output interface when a rotation operation is initiated on the lower traveling body of the upper rotating body on which the actual machine imaging device is mounted in the work machine.

[0016] In this configuration, a composite image is displayed when a rotation operation is initiated on the work machine, making it possible to display the composite image when the operator needs to see an indicator image.

[0017] In the remote operation support system of the present invention, it is preferable to stop displaying the composite image on the output interface when the rotation operation of the lower traveling body of the upper rotating body on which the actual machine imaging device is mounted in the work machine is completed.

[0018] In this configuration, the display of the composite image stops when the rotation operation of the work machine is completed, thus eliminating the annoyance caused by the indicator image being displayed indefinitely.

[0019] The remote operation support composite system of the present invention is characterized by being composed of the above-described remote operation support system and at least one of the work machine and the remote operation device.

[0020] According to such an aspect, it is possible to configure a composite system in cooperation with at least one of the work machine and the remote operation device, and the usability can be improved.

[0021] The remote operation support method of the present invention includes a step of displaying, on an output interface of a remote operation device for remotely operating the work machine, a composite image in which an index image corresponding to an index member arranged in the cab of the work machine is superimposed on an imaging image captured by an imaging device mounted on the work machine and capable of imaging the surroundings of the work machine.

[0022] According to such an aspect, since the step of displaying the composite image on the output interface of the remote operation device for remotely operating the work machine is included, the operator can visually recognize the composite image, and it is possible to impart an appropriate operation feeling.

Brief Description of the Drawings

[0023] [Figure 1] It is an explanatory diagram of the configuration of a work support system as an embodiment of the present invention. [Figure 2] It is an explanatory diagram regarding the configuration of the remote operation device in FIG. 1. [Figure 3] It is an explanatory diagram regarding the configuration of the work machine in FIG. 1. [Figure 4] It is an explanatory diagram regarding the environment around the work machine in FIG. 1. [Figure 5] It is an explanatory diagram showing an example of an imaging image captured by the first actual machine imaging device in FIG. 1. [Figure 6] It is an explanatory diagram showing an example of an imaging image captured by the second actual machine imaging device in FIG. 1. [Figure 7] It is a flowchart showing the work support process of the work support system. [Figure 8]Figure 7 is a flowchart showing the subroutine for the composite image generation process. [Figure 9] This is an explanatory diagram showing how the composite image is displayed. [Figure 10] This is an explanatory diagram showing other ways of displaying the composite image. [Modes for carrying out the invention] [Examples]

[0024] (Configuration of the remote control support system) The work support system shown in Figure 1, as one embodiment of the present invention, consists of a work support server 10 for supporting the remote operation of multiple work machines 40 by a remote control device 20. The work support server 10, the remote control device 20, and the multiple work machines 40 are configured to communicate with each other via a common or separate network. In this figure, the work support server 10 is shown to be able to connect to multiple work machines 40, but the work support server 10 only needs to be able to connect to one or more work machines 40, and there is no limit to the number of machines it can connect to.

[0025] In this embodiment, the work support server 10 is described as a remote operation support system having mutual communication functions with a plurality of work machines 40 and a remote operation device 20 for remotely operating one of the plurality of work machines 40 that is to be remotely operated.

[0026] In the present invention, the term "acquiring" various types of information by the constituent elements (hardware) encompasses all arithmetic processes for preparing various types of information in a form usable in subsequent arithmetic processes, including receiving such information, reading or retrieving such information from internal storage devices (e.g., memory) and / or external storage devices (e.g., external database servers), and performing arithmetic processing on the received, read, retrieved, etc., information to calculate, estimate, predict, identify, etc.

[0027] (Configuration of the support server) The work support server 10 includes a remote operation support device 100 containing support processing elements 101, a database 110, and a server wireless communication device 122 for communication over a network.

[0028] The support processing element 101 is composed of an arithmetic processing unit (a single-core processor or a multi-core processor or processor cores comprising the same), reads necessary data and software from a storage device such as memory, and performs the arithmetic processing described later on the data in accordance with the software.

[0029] Database 110 stores and holds image data captured by imaging devices such as the first actual machine imaging device 412a and the second actual machine imaging device 412b, as well as position data for superimposing an index image onto the second image, which will be described later. Database 110 may be configured by a separate database server from the work support server 10.

[0030] (Configuration of the remote control device) The remote control device 20 comprises a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of an arithmetic processing unit (single-core processor or multi-core processor or processor cores comprising them), reads necessary data and software from a storage device such as memory, and performs arithmetic processing on said data in accordance with said software.

[0031] The remote input interface 210 is equipped with a remote control mechanism 211.

[0032] The remote control mechanism 211 includes a travel control device, a slewing control device, a boom control device, an arm control device, a bucket control device, and a hydraulic lock lever (shut-off lever) for switching the operator's operation acceptance state. Each control device has an operating lever that receives rotational operation. The operating lever (travel lever) of the travel control device is operated to move the lower travel body 430 of the work machine 40, which has a pair of left and right crawlers. The travel lever may also serve as a travel pedal. For example, a travel pedal may be provided fixed to the base or lower end of the travel lever. The operating lever (slewing lever) of the slewing control device is operated to move the hydraulic slewing motor that constitutes the slewing mechanism 440 of the work machine 40. The operating lever (boom lever) of the boom control device is operated to move the boom cylinder 462 of the work machine 40. The operating lever (arm lever) of the arm control device is operated to move the arm cylinder 464 of the work machine 40. The operating lever (bucket lever) of the bucket control device is operated to move the bucket cylinder 466 of the work machine 40. The shut-off lever outputs a signal to operate the work machine 40 depending on whether it is switched ON or OFF. For example, in the non-receiving state, no signal to operate the work machine 40 is output even if the operator operates it.

[0033] Each operating lever constituting the remote control mechanism 211 is arranged around the seat St on which the operator OP1 sits, for example, as shown in Figure 2. The seat St may take the form of a high-back chair with armrests, but it may also be any form of seating on which the operator OP1 can sit, such as a low-back chair without a headrest, or a chair without a backrest.

[0034] A pair of left and right travel levers 2110, corresponding to the left and right crawlers, are arranged side by side in front of the seat St. One operating lever may serve multiple purposes. For example, the left operating lever 2111, located in front of the left frame of the seat St shown in Figure 2, may function as an arm lever when operated in the forward / backward direction and as a slewing lever when operated in the left / right direction. Similarly, the right operating lever 2112, located in front of the right frame of the seat St shown in Figure 2, may function as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever pattern may be arbitrarily changed by the operating instructions of operator OP1.

[0035] The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222 for communication over a network.

[0036] The image output device 221 consists of a central image output device 2210, a left image output device 2211, a right image output device 2212, and a lower image output device 2213, which has a roughly rectangular screen and is positioned in front of, diagonally to the left and diagonally to the right of the sheet St, as shown in Figure 2, for example. The shape and size of the screens (image display areas) of the central image output device 2210, the left image output device 2211, the right image output device 2212, and the lower image output device 2213 may be the same or different.

[0037] The screens of the central image output device 2210, the left image output device 2211, the right image output device 2212, and the lower image output device 2213 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the image output devices among the central image output device 2210, the left image output device 2211, the right image output device 2212, and the lower image output device 2213 may be composed of multiple divided image output devices. For example, the central image output device 2210 may be composed of a pair of vertically adjacent image output devices having substantially rectangular screens.

[0038] (Configuration of the work machine) As shown in Figure 1, the work machine 40 includes a machine control device 400, a machine input interface 410, a machine output interface 420, and an operating mechanism 460.

[0039] The actual machine control device 400 is composed of an arithmetic processing unit (a single-core processor or a multi-core processor or processor cores comprising such a unit), reads necessary data and software from a storage device such as memory, and performs arithmetic processing on said data in accordance with said software.

[0040] The work machine 40 is, for example, a hydraulic, electric, or hybrid-drive crawler excavator (work machine) and, as shown in Figure 3, a side view of the work machine 40, comprises a lower traveling body 430 and an upper rotating body 450 that is rotatably mounted on the lower traveling body 430 via a slewing mechanism 440. A cab 454 (operator's cabin) is provided on the front left side of the upper rotating body 450. An operating mechanism 460 is provided on the front center of the upper rotating body 450.

[0041] The actual machine input interface 41 includes an actual machine operation mechanism 411, a first actual machine imaging device 412a, a second actual machine imaging device 412b, and a rear imaging device 412c. The actual machine operation mechanism 411 includes a number of actual machine operation levers arranged around the seat located inside the cab 454, similar to the remote control mechanism 211.

[0042] The work machine 40 is equipped with a drive mechanism or robot in the cab 454 that receives a signal corresponding to the operation of the remote control lever and moves the actual machine control lever based on the received signal.

[0043] The first machine imaging device 412a (hereinafter also referred to as the main camera) is installed, for example, inside the cab 454. The first machine imaging device 412a images the environment including at least a part of the operating mechanism 460 through the front window and a pair of left and right side windows. The first machine imaging device 412a also images the environment including at least a part of the attachment bucket 465.

[0044] As shown in Figure 4, a top view of the work machine 40, the first machine imaging device 412a images a predetermined imaging range AR1 in front of the cab 454, and images from inside the cab 454 along the optical axis AX1 with a field of view from the right end R1 to the left end L1. The imaging range AR1 is the range imaged in the left-right field of view from the right end R1 to the left end L1 in the first machine imaging device 412a which is oriented horizontally as shown in Figure 3. The imaging range AR1 shown in Figure 4 assumes a horizontal ground surface, so the range along the optical axis AX1 is not shown, but if there is an upward slope in front of the work machine 40, the imaging range AR1 can be determined by the line connecting the right end R1 and the left end L1.

[0045] Furthermore, as shown in Figure 5, the first actual machine imaging device 412a images a predetermined imaging range AR1, for example, including at least a portion of the front window FW of the cab 454, the window frame WF which serves as an indicator member arranged around the side windows SW1 and SW2, and the travel lever 2110.

[0046] The second machine imaging device 412b (hereinafter also referred to as the cab camera) is installed, for example, on the roof of the cab 454 on the front side of the work machine 40 (the front side of the cab 454), and is capable of imaging the real space around the work machine 40. As shown in Figure 4, the second machine imaging device 412b images the area in front of the work machine 40 (the front of the cab 454) from the roof of the cab 454 along the optical axis AX2 with a field of view from the right end R2 to the left end L2. The second machine imaging device 412b has a movable mechanism that allows its orientation to be changed in the vertical and horizontal directions, and in Figure 3, the second machine imaging device 412b is positioned facing downwards. Therefore, the imaging range AR2 shown in Figure 4 indicates the range imaged by the second machine imaging device 412b with a field of view in the horizontal direction from the right end R2 to the left end L2. It is preferable that the first and second imaging devices 412a and 412b are arranged such that at least a portion of their imaging ranges overlap.

[0047] As shown in Figure 6, the second machine imaging device 412b is capable of imaging the environment including at least a part of the operating mechanism 460. The second machine imaging device 412b images the environment including, for example, at least a part of the bucket 465, which is an attachment. Figure 6 is an image taken when the second machine imaging device 412b is oriented downwards as shown in Figure 3, and the ground in front of and below the work machine 40 is visible behind the operating mechanism 460. The imaging range AR2 of the second machine imaging device 412b, which overlaps with the imaging range AR1 of the first machine imaging device 412a, preferably includes, for example, at least a part of the bucket 465. The second machine imaging device 412b may be installed in a location other than the cab 454 of the work machine 40, for example, in a location where it can be fixed on the upper slewing body 450. Furthermore, the second actual machine imaging device 412b does not necessarily have to be installed on the work machine 40; it may be installed on an aircraft such as a drone, or it may be a fixed camera fixed to a support column or the like installed around the work site.

[0048] Furthermore, the first actual imaging device 412a and the second actual imaging device 412b may be added as appropriate depending on the implementation.

[0049] The actual machine input interface 410 includes a group of actual machine state sensors (not shown). The group of actual machine state sensors (not shown) consists of angle sensors for measuring the rotation angle (elevation angle) of the boom 461 relative to the upper slewing body 450, the rotation angle of the arm 463 relative to the boom 461, and the rotation angle of the bucket 465 relative to the arm 463, respectively; a slewing angle sensor for measuring the slewing angle of the upper slewing body 450 relative to the lower traveling body 430; an external force sensor for measuring the external force acting on the bucket 465; a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper slewing body 450; and position information acquisition sensors such as GNSS (Global Navigation Satellite System).

[0050] The actual device output interface 420 is equipped with an actual device wireless communication device 422 for communication over a network.

[0051] The operating mechanism 460 comprises a boom 461 mounted on the upper slewing body 450 so as to be able to be raised and lowered, an arm 463 rotatably connected to the tip of the boom 461, and a bucket 465 rotatably connected to the tip of the arm 463. The operating mechanism 460 is equipped with a boom cylinder 462, an arm cylinder 464, and a bucket cylinder 466, which are composed of extendable hydraulic cylinders. In addition to the bucket 465, various attachments such as nibblers, cutters, and magnets may be used as working parts.

[0052] The boom cylinder 462 is interposed between the boom 461 and the upper slewing body 450 so as to extend and retract when supplied with hydraulic fluid, causing the boom 461 to rotate in the luffing direction. The arm cylinder 464 is interposed between the arm 463 and the boom 461 so as to extend and retract when supplied with hydraulic fluid, causing the arm 463 to rotate around a horizontal axis relative to the boom 461. The bucket cylinder 466 is interposed between the bucket 465 and the arm 463 so as to extend and retract when supplied with hydraulic fluid, causing the bucket 465 to rotate around a horizontal axis relative to the arm 463.

[0053] The work support processing method of the work support system, which is achieved through the cooperation of the above-mentioned work support server 10, remote control device 20, first actual imaging device 412a, and second actual imaging device 412b, will be explained using the flowchart shown in Figure 7.

[0054] The work support system starts work support processing triggered by the activation of the remote control device 20. When the remote control device 20 is activated by the operator OP, for example, the remote control device 20 transmits an operation signal to the image display system 10 (STEP 211). This operation signal should be transmitted as appropriate depending on the processing after the activation of the remote control device 20, for example, when the operator OP selects a work machine 40 to be connected to (cooperated with) the remote control device 20, or when the operator OP switches the shut-off lever from the ON state to the OFF state.

[0055] When the work support server 10 receives an operation signal, it requests the second actual imaging device 412b to transmit an image (STEP 111).

[0056] When the second actual imaging device 412b receives an image transmission request, it transmits the captured image to the work support server 10 (STEP 511).

[0057] When the support processing element 101 of the work support server 10 receives an image (second image) captured by the second actual machine imaging device 412b, it generates a composite image in which an index image corresponding to an index member placed in the cab 454 of the work machine 40 is superimposed (STEP 112).

[0058] The support processing element 101 of the work support server 10 transmits the composite image generated in STEP 112 to the remote control device 20 (STEP 113).

[0059] When the remote control device 20 receives the composite image, it displays the composite image on the image output device 221 of the output interface 220, as shown in Figure 9. In other words, the support processing element 101 of the work support server 10 transmits the composite image to the remote control device 20, causing the output interface 220 of the remote control device 20 to display the composite image.

[0060] Furthermore, the composite image can be displayed on any of the image output devices of the image output device 221. For example, it may be displayed on the central image output device 2210, the left image output device 2211, the right image output device 2212, or the lower image output device 2213. In addition, the composite image may be displayed across multiple image output devices.

[0061] In this way, a composite image in which the indicator image is superimposed on the second captured image is displayed on the output interface 220. For example, the operator OP1 of the work machine 40 can operate the work machine 40 while visually checking the indicator image. Therefore, it is possible to provide an appropriate sense of operation to operator OP1 who operates the work machine 40 from a viewpoint outside the cab 454.

[0062] This section describes the subroutine for the composite image generation process in STEP 112, performed by the support processing element 101 of the work support server 10.

[0063] The support processing element 101 of the work support server 10 starts the composite image generation process triggered by the reception of the second captured image.

[0064] The support processing element 101 of the work support server 10 determines the position in which the index image is superimposed on the second captured image (STEP 121).

[0065] Specifically, the support processing element 101 of the work support server 10 sets the display position of the indicator image by reading position data for superimposing the indicator image onto the second captured image, which has been previously stored in the storage device (database 110), from the storage device (database 110). This position data is generated, for example, by the support processing element 101 determining the position to superimpose the indicator member onto the second captured image by performing a coordinate transformation on at least a portion of the image area in which the indicator member is captured in the image captured by the first actual machine imaging device 412a (first captured image) based on the position and orientation of the work machine 40 of the first actual machine imaging device 412a and the position and orientation of the work machine 40 of the second actual machine imaging device 412b, thereby determining the position in which the indicator member is superimposed onto the second captured image.

[0066] Specifically, the support processing element 101 of the work support server 10 acquires the position and orientation of the first machine imaging device 412a, which is installed in the cab 454 of the work machine 40, and the second machine imaging device 412b, which is installed on the roof of the cab 454.

[0067] Next, the support processing element 101 creates a translation matrix to transform the coordinate system of the first actual imaging device 412a to the coordinate system of the second actual imaging device 412b based on the acquired positional relationship between the first actual imaging device 412a and the second actual imaging device 412b. The support processing element 101 then creates a rotation matrix to transform the coordinate system of the first actual imaging device 412a to the coordinate system of the second actual imaging device 412b based on the acquired attitude relationship between the first actual imaging device 412a and the second actual imaging device 412b.

[0068] The support processing element 101 obtains the coordinates of, for example, the A-pillar, which is an indicator member, in the coordinate system of the first captured image. The support processing element 101 transforms the obtained coordinates of the A-pillar using a pre-created translation matrix and rotation matrix to determine the position where the indicator image is superimposed in the second captured image, and stores it as position data in a storage device (database 110).

[0069] Furthermore, the process of determining the position in which the indicator image is superimposed on the second captured image may be performed by having operator OP1 specify the position in the second captured image where the indicator image is superimposed. Specifically, operator OP1 sets the display position by, for example, operating the remote input interface 210 to the remote control device 20 to input the display position. When determining this display position, the display position of the indicator image superimposed on the second captured image may be modified by operator OP1 via the remote input interface 210 based on position data read from the storage device (database 110).

[0070] The support processing element 101 selects a pre-specified indicator image as the image to be superimposed (STEP 122). The indicator image can have a shape corresponding to the indicator member. For example, if the indicator member is an A-pillar, the indicator image may have the same shape as the indicator member, or it may be a line extending vertically in the second captured image, a rectangular shape, etc. Multiple indicator images may also be superimposed on the second captured image. For example, two images, one of the A-pillar and one of the travel lever 2110, may both be superimposed on the second captured image as indicator images. In this way, by superimposing the travel lever 2110 as an indicator image, positioning (movement operation to the target position of the movement operation) in the vicinity of the lower part of the second captured image, for example, when moving the work machine 40 in the forward and backward direction, or when moving the bucket 465 in the vertical direction, can be made easier.

[0071] Furthermore, if the indicator member is an A-pillar, the indicator image only needs to display at least one of the left and right A-pillar members. The indicator image may also be in a form unrelated to the indicator member; for example, a cross shape, circle, polygon, or other geometric shape may be used.

[0072] Operator OP1, for example, operates the remote input interface 210 on the remote control device 20 to set the type of indicator image to be displayed (shape, line, figure, etc.).

[0073] The support processing element 101 adjusts the size of the indicator image superimposed on the second captured image (STEP 123). Specifically, the support processing element 101 adjusts the size of the indicator image superimposed on the second captured image based on the information obtained in the superimposition position determination process in STEP 121. For example, if the magnification of the first captured image and the second captured image are about the same, the support processing element 101 adjusts the size of the indicator image to be about the same size as the indicator member captured in the first captured image. Also, for example, if the magnification of the second captured image is higher than the magnification of the first captured image, the support processing element 101 adjusts the size of the indicator image so that the indicator member captured in the first captured image is made larger (or thicker) to match the magnification. In this way, visibility can be improved by superimposing an indicator image with a certain width.

[0074] Therefore, the indicator image can be displayed on the output interface 220 in a manner corresponding to the magnification of the second captured image. This makes it possible for operator OP1 to obtain an appropriate sense of distance from the composite image.

[0075] Furthermore, once the superposition position of the indicator image onto the second captured image (STEP 121) is determined, the process of STEP 121 is omitted unless there is an operation to change the superposition position. Also, once the indicator image to be superimposed is selected (STEP 122), the process of STEP 122 is omitted unless there is an operation to change the indicator image. In addition, once the size of the superimposed indicator image is adjusted (STEP 123), the process of STEP 123 is omitted unless there is an operation to adjust the size of the indicator image.

[0076] Furthermore, if the magnification (focal length of the lens) of the second actual imaging device 412b can be obtained in advance, the support processing element 101 may display the indicator image at a size corresponding to the magnification (focal length of the lens) of the second image. By performing the processing in this manner, the above processing can be simplified and the processing load can be reduced.

[0077] Furthermore, the indicator image should be displayed in a way that makes the area where the second image is superimposed visible. For example, the indicator image may be displayed with a different brightness than the second image. Specifically, by displaying the indicator image semi-transparently, the area of ​​the second image where the indicator image is superimposed can be made visible. Moreover, the display of the indicator image is not limited to this form; for example, by making it blink, the area of ​​the second image where the indicator image is superimposed can be made visible.

[0078] Furthermore, it is preferable that the indicator image is displayed across the vertical direction of the second captured image. By displaying the indicator image across the vertical direction in this way, the operator OP1 can more easily grasp the amount of movement in the rotation direction, thereby improving the operability (positioning) of the rotation operation.

[0079] Furthermore, the size of the indicator image may be adjusted by operator OP1. For example, operator OP1 may input the size of the indicator member in the left-right direction, the size of the indicator member in the up-down direction, and the tilt of the indicator member in the up-down direction (the left-right position of the vertical edge of the indicator image) by operating the remote input interface 210.

[0080] The support processing element 101 generates a composite image by superimposing the index image, whose size was adjusted in STEP 123, onto the position of the second image determined in STEP 121 (STEP 124).

[0081] Furthermore, the support processing element 101 should not normally display the composite image on the output interface 220, but should display the composite image on the output interface 220 when a rotation operation on the work machine 40 is initiated. Specifically, for example, the composite image should be displayed on the output interface 220 at the moment an input operation on the work machine 40 is initiated, or when a predetermined time has elapsed since the input operation on the work machine 40 was initiated. Examples of input operations include when operator OP1 grasps the lever that rotates the work machine 40, or when the operator tilts the lever.

[0082] Furthermore, the support processing element 101 should not normally display the composite image on the output interface 220, but should display the composite image on the output interface 220 at the moment the rotational movement of the work machine 40 begins, or when a predetermined time has elapsed since the rotational movement of the work machine 40 began. In this way, the composite image can be displayed at the timing required by the operator OP1.

[0083] Furthermore, the support processing element 101 may be configured to stop displaying the composite image on the output interface 220 when the rotation operation for the work machine 40 is completed. Specifically, for example, the display of the composite image on the output interface 220 may be stopped at the moment when the input operation for the work machine 40 is completed, or when a predetermined time has elapsed after the input operation for the work machine 40 has been completed. An example of an input operation is when operator OP1 releases the lever that rotates the work machine 40.

[0084] Furthermore, it is advisable to stop displaying the composite image on the output interface 220 at the moment the rotational movement of the work machine 40 is completed, or when a predetermined time has elapsed after the rotational movement of the work machine 40 is completed. This prevents the composite image from being displayed indefinitely after the rotational operation is completed.

[0085] Furthermore, the indicator images may be superimposed on the second captured image, with both the first indicator image (before the rotation operation) and the second indicator image (during the rotation operation). For example, before the rotation operation, a composite image is displayed where the second indicator image is shown on the second captured image based on position data read from the storage device (database 110), and the first indicator image, which is the same shape and size as the second indicator image, is displayed overlapping the second indicator image. During the rotation operation, the second indicator image is shown on the second captured image based on position data read from the storage device (database 110), and a composite image is displayed where the first indicator image moves on the second captured image in the opposite direction to the rotation direction according to the amount of rotation relative to before the rotation operation. In this way, by superimposing both the first and second indicator images on the second captured image, the amount of rotation of the work machine 40 can be easily observed.

[0086] In particular, in operations such as loading, where the equipment is moved back and forth by repeatedly rotating around a fixed position, operator OP1 can perform the rotation operation using the first indicator image as a target. Therefore, the operability of the rotation operation can be improved. Alternatively, only the first indicator image may be superimposed on the second captured image.

[0087] Furthermore, in the above-described embodiment, the image output device 221 was described as displaying a composite image on the output interface 220. However, the image output device 221 may also display the first captured image on the output interface 220. For example, the image output device 221 may display at least one of the composite image and the first captured image on the output interface 220. In this embodiment as well, the positioning reference is reflected in both the first captured image in which the indicator member of the cab 454 is reflected, and the composite image in which the indicator image is superimposed on the second captured image. As a result, the operator OP1 can more easily grasp the amount of movement in the rotation direction, and the operability (positioning) of the rotation operation can be improved.

[0088] Furthermore, in the above-described embodiment, the ground surface was shown in the first and second captured images. However, it is not necessary for the ground surface to be shown in the first and second captured images, and the method can be applied to captured images at various shooting angles. For example, as shown in Figure 10, the window frame FW of the roof of the cab 454 may be used as an indicator member and superimposed on the second captured image.

[0089] This configuration improves visibility, for example, when there are many vertical movement operations, such as when the boom 461 is frequently raised and lowered. Furthermore, since the window frame FW is superimposed on the second captured image as an indicator member, the operator OP1 can easily understand that the shooting angle is not downward. [Examples]

[0090] In the above embodiment, the process in STEP 121 of the composite image generation process, which determines the position in which the reference image is superimposed on the second captured image, was performed by reading position data for superimposing the reference image on the second captured image, which had been previously stored in the storage device (database 110), from the storage device (database 110). However, this position data may be generated based on the size and positional relationship of the reference object in a predetermined direction in the first captured image, and the size of the reference object in the same predetermined direction in the second captured image.

[0091] Specifically, the support processing element 101 acquires, for example, the size of a reference object captured in the first image in a first predetermined direction, the distance between the reference object and the indicator member in the first predetermined direction, and the size of a reference object captured in the second image in a second predetermined direction corresponding to the first predetermined direction.

[0092] Specifically, as shown in Figure 5, if the reference object is the bucket 465 of the work machine 40, and the first predetermined direction is the left-right direction of the first captured image, the support processing element 101 obtains the left-right size D1 of the bucket 465 and the distance D2 from the endpoint of the bucket 465 to the A-pillar in the left-right direction.

[0093] The support processing element 101 acquires the size in a second predetermined direction that corresponds to the first predetermined direction of the reference object captured in the second captured image. Specifically, as shown in Figure 6, if the second predetermined direction is the left-right direction in the second captured image, the support processing element 101 acquires the left-right size D3 of the bucket 465.

[0094] The support processing element 101 determines the position for superimposing the indicator image so that the relationship between the size D1 of the reference object in a first predetermined direction and the distance D2 between the reference object and the indicator member in the first predetermined direction, which are captured in the first image, is aligned with the position of the indicator member in a second predetermined direction, and stores this position data in a storage device (database 110).

[0095] Specifically, the support processing element 101 determines the position in which the indicator image will be superimposed by deriving the size of the indicator image to be displayed in the second captured image, the line width of the indicator image, etc., from the relationship between D1, D2, and D3 described above.

[0096] According to this configuration, the position in which the index image is superimposed on the second image can be determined with simple processing, thereby reducing the processing load.

[0097] In the above description of the embodiments, an example was described in which the indicator image is superimposed on the second captured image. However, the indicator image may also be superimposed on the first captured image. For example, if the magnification of the camera of the first actual imaging device 412a is increased, the indicator member will no longer be visible in the first captured image. In such a case, the indicator image may be superimposed on the first captured image to generate a composite image.

[0098] Furthermore, in the above description of the embodiment, it was explained that the work support processing is performed by the work support server 10. However, the work support processing may also be performed by the remote control device 20 or by the work machine 40.

[0099] Furthermore, the database 110 was used as an example to describe a storage device that stores positional data for superimposing an index image onto the second captured image. However, the storage device is not limited to the database 110; any device that can store and retrieve such positional data is acceptable. For example, it may be a storage device other than the database 110 installed in the work support server 10. Alternatively, it may be a storage device installed in a device other than the work support server 10, such as the remote control device 20 or the work machine 40, or a storage device installed in a different device from these. [Explanation of Symbols]

[0100] 10. Work support server 101...Support processing elements 20. Remote control device 220... Output Interface 40...Working machinery

Claims

1. A remote control support system that displays, on the output interface of a remote control device for remotely controlling the work machine, a composite image in which an indicator image corresponding to the indicator member is superimposed on a second image captured by the second image device in a coordinate system that captures the direction along the optical axis of the second image device, based on the positional relationship between a first image device located inside the operator's cab of the work machine and a second image device mounted on the work machine and capable of capturing images around the work machine, and the orientation relationship between the first image device and the second image device.

2. A remote control support system that displays a composite image on the output interface of a remote control device for remotely controlling the work machine, based on the positional relationship between a first imaging device located inside the operator's cab of the work machine and a second imaging device mounted on the work machine and capable of imaging the area around the work machine, and the orientation relationship between the first imaging device and the second imaging device, in which a second image captured by the second imaging device in a coordinate system that images in the direction along the optical axis of the second imaging device, superimposed an index image corresponding to the index member in a first image captured by the first imaging device in a coordinate system that images in the direction along the optical axis of the first imaging device, such that the window frame, which is an index member located in the operator's cab, is included in the first image captured by the first imaging device in a coordinate system that images in the direction along the optical axis of the first imaging device, is converted to a display position in the coordinate system of the second imaging device that images in the direction along the optical axis of the second imaging device.

3. A remote control support system that displays on the output interface of a remote control device for remotely controlling the work machine a composite image in which an indicator image corresponding to the indicator member is superimposed on a second image captured by the second image device in a coordinate system that captures the direction along the optical axis of the second image device, based on the positional relationship between a first image device located inside the operator's cab of the work machine and a second image device mounted on the work machine and capable of capturing images around the work machine, and the orientation relationship between the first image device and the second image device.

4. In the remote operation support system according to any one of claims 1 to 3, The aforementioned indicator image is a remote operation support system that displays the superimposed region of the second image in a visible manner.

5. In the remote operation support system according to any one of claims 1 to 4, The aforementioned indicator image is displayed in a size corresponding to the magnification of the second captured image in the remote operation support system.

6. In the remote operation support system according to any one of claims 1 to 5, A remote operation support system that displays the composite image on the output interface when an operation to rotate the upper rotating body relative to the lower traveling body is initiated via a rotation operation device provided in the remote operation device and operated to move the rotation mechanism in the work machine comprising a lower traveling body, an upper rotating body mounted on the lower traveling body via a rotation mechanism, and a second imaging device mounted on the upper rotating body.

7. In the remote operation support system according to any one of claims 1 to 5, A remote operation support system that stops the display of the composite image on the output interface when the operation to rotate the upper rotating body relative to the lower traveling body in the work machine comprising a lower traveling body, an upper rotating body mounted on the lower traveling body so as to be rotatable via a swivel mechanism, and a second imaging device mounted on the upper rotating body is terminated via a swivel operation device provided in the remote operation device and operated to move the swivel mechanism.

8. A remote control support composite system comprising the remote control support system according to any one of claims 1 to 7, and at least one of the work machine and the remote control device.

9. A remote operation support method that includes the step of displaying on the output interface of a remote operation device for remotely operating the work machine a composite image in which an indicator image corresponding to the indicator member is superimposed on a second image captured by the second image device in a coordinate system that captures the direction along the optical axis of the second image device, based on the positional relationship between a first image device arranged inside the operator's cab of the work machine and a second image device mounted on the work machine and capable of capturing images around the work machine, and the orientation relationship between the first image device and the second image device.

10. A remote operation support method that includes the step of displaying on the output interface of a remote operation device for remotely operating the work machine a composite image in which an indicator image corresponding to the indicator member is superimposed on a second image captured by the second image device in a coordinate system that captures the direction along the optical axis of the second image device, based on the positional relationship between a first image device arranged inside the operator's cab of the work machine and a second image device mounted on the work machine and capable of capturing images around the work machine, and the orientation relationship between the first image device and the second image device.

11. A remote operation support method that includes the step of displaying on the output interface of a remote operation device for remotely operating the work machine a composite image in which an indicator image corresponding to the indicator member is superimposed on a second image captured by the second image device in a coordinate system that captures the direction along the optical axis of the second image device, based on the positional relationship between a first image device located inside the operator's cab of the work machine and a second image device mounted on the work machine and capable of capturing images around the work machine, and the orientation relationship between the first image device and the second image device.

Images