Work support system, and work support server

Abstract

To provide a remote control system capable of confirming whether or not a work machine operating according to an operator's operation matches a work machine selected by the operator.SOLUTION: A remote control system has a mutual communication function of communicating with each of a plurality of work machines and a remote control device for remotely controlling a cooperating work machine, which is a remote operation target among the plurality of work machines. The remote control system includes: a first support process element that acquires a captured image captured by an actual imaging device of a first work machine with which communication is established through a first communication path, and transmits the captured image to the remote operation device; and a second support processing element that recognizes that one specified operation has been performed and transmits a command signal corresponding to the one specified operation to the second work machine, wherein the one specified operation is a portion of a plurality of operations for operating the second work machine with which communication is established through a second communication path and reflects a change mode corresponding to the operation in the captured image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a remote operation system having a function of mutual communication between a plurality of work machines and a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated. [Background technology]

[0002] Conventionally, a work machine has been operated remotely without an operator riding on the work machine. For example, Patent Document 1 discloses a remote operation system for such a work machine in which a master and a slave are provided so as to be able to communicate with each other, and an operator operates an operation lever provided on the master, thereby causing a slave located in the driver's seat of the work machine to operate the operation lever of the work machine. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-176401 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when one master is used to remotely operate two or more work machines, the master and the slaves provided on each of the work machines must be connected so that they can communicate with each other.

[0005] In addition, the operator remotely controls the work machine by operating control levers etc. while viewing an image of the surrounding area of ​​the work machine to be operated on the master monitor.

[0006] However, there are cases where the image displayed on the master monitor is an image of the surroundings of a work machine that is different from the target to be operated. When an operator actually boards a work machine, the operator can check the appearance of the work machine and the state of the work site where the work machine is located before boarding, but such checks are not possible when remotely operating a work machine. For this reason, it is difficult for the operator to notice that the remote control device has been connected to an unintended work machine (a different slave). Even if an operator attempts to check using images from a camera mounted on the work machine, these images are similar and difficult to distinguish. In such a case, when remote operation is performed, there is a possibility that the operation will be performed on a work machine that the operator does not recognize as the target to be operated.

[0007] The present invention has been made in consideration of such problems, and has an object to provide a remote operation system that is capable of confirming whether a work machine operating in response to an operator's operation matches the work machine selected by the operator. [Means for solving the problem]

[0008] In order to achieve this object, the remote operation system of the present invention is a remote operation system having a mutual communication function between a plurality of work machines and a remote operation device for remotely operating a coordinated work machine which is one of the plurality of work machines to be remotely operated, and is characterized in having a first support processing element that, when a determination result of whether a designation operation of selection information specifying the coordinated work machine as a remotely operated target among the plurality of work machines has been executed in the remote operation device and the selection information has been selected in the remote operation device, acquires an image captured by an actual machine imaging device mounted on a first work machine with which communication is established with the remote operation device via a first communication path linked to the selection information, and outputs the captured image to an output interface of the remote operation device; and a second support processing element that recognizes that a designation operation has been performed which is part of a plurality of types of operations for operating a second work machine with which communication is established with the remote operation device via a second communication path linked to the selection information, and in which a change in operation corresponding to the operation is reflected in the captured image, and transmits a command signal corresponding to the designation operation to the second work machine.

[0009] According to the remote operation system of the present invention, an image captured by an actual machine imaging device mounted on a first work machine is acquired via a first communication path based on selection information that identifies a cooperating work machine, and a command signal corresponding to one specified operation is transmitted to a second work machine. Therefore, the operator can check through the captured image whether or not the first work machine operates in response to one specified operation performed on the second work machine, i.e., whether or not an operation performed on the cooperating work machine to be remotely operated has been performed on the cooperating work machine. This allows the operator to check the connection status with the work machine to be remotely operated.

[0010] In the remote operation system of the present invention, when a release condition is not satisfied, the second support processing element suspends transmission to the second work machine of a command signal responsive to another designated operation for operating the collaborative work machine, and when the release condition is satisfied, allows transmission to the second work machine of a command signal responsive to another designated operation for operating the collaborative work machine, and it is preferable that the release condition includes a release operation being performed in the remote operation device.

[0011] According to this aspect, it is possible to prevent the work machine from being operated by another designation operation until the operator has confirmed the connection status with the work machine to be remotely operated. After the operator has confirmed the connection status, the work machine can be remotely operated by an operation including the other designation operation.

[0012] In the remote operation system of the present invention, it is preferable that the second operation support processing element obtains a determination result as to whether or not the one designated operation has been executed, and, if the one designated operation has been executed, allows another designated operation for operating the collaborative work machine, and transmits a command signal corresponding to the other designated operation to the second work machine.

[0013] According to this aspect, on the condition that one designation operation has been executed, it is possible to permit another designation operation to be performed on the work machine that is the target of remote operation.

[0014] In the remote operation system of the present invention, when a release condition is not satisfied, the second support processing element suspends transmission of a command signal corresponding to another designated operation for operating the portable work machine to the second work machine, and when the release condition is satisfied, allows transmission of a command signal corresponding to another designated operation for operating the portable work machine to the second work machine, and it is preferable that the release condition includes, when the one designated operation is executed, the second support processing element obtains a determination result of whether or not there is a change in operation in the captured image corresponding to the one designated operation, and that there is a change in operation corresponding to the one designated operation.

[0015] According to this aspect, the connection status with the work machine to be remotely operated is automatically determined, thereby reducing the checking work of the operator.

[0016] In the remote operation system of the present invention, it is preferable that the first operation support processing element acquires position data of the multiple work machines to be remotely operated, and the second operation support processing element generates a display image in which each of the multiple work machines is reflected based on the multiple position data, and transmits the display image to the remote operation device.

[0017] According to this aspect, the operator can check how each of the multiple work machines to be remotely operated is reflected in the displayed image, and can therefore recognize the relative positions of each work machine at the work site. [Brief description of the drawings]

[0018] [Figure 1] 1 is a diagram illustrating the configuration of a work support complex system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is an explanatory diagram showing an example of a data structure stored in a database of a work support server. [Diagram 3] FIG. 2 is an explanatory diagram relating to a configuration of a remote control device. [Figure 4] FIG. 2 is an explanatory diagram showing an example of a screen display of a remote control device. [Diagram 5] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 6] FIG. 11 is an explanatory diagram showing an example of an image captured by a real imaging device. [Figure 7] 13 is a flowchart illustrating an example of a display process of a terminal at hand by a work support server. [Figure 8] 13 is a flowchart showing an example of a work support process performed by the work support server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (Configuration of remote control system) The work support system as one embodiment of the present invention shown in Fig. 1 is configured with a work support server 10 for supporting the remote operation of a first selected work machine 40 and a second selected work machine 50 by a remote operation device 20. The work support server 10, the remote operation device 20, the first selected work machine 40, and the second selected work machine 50 are configured to be able to communicate with each other via a common or separate network. In this figure, the work support server 10 is connectable to the first selected work machine 40 or the second selected work machine 50, but the work support server 10 only needs to be connectable to two or more work machines, and there is no limit to the number of connected machines. In addition, either the first selected work machine 40 or the second selected work machine 50 operated by the remote operation device 20 is also referred to as a cooperative work machine.

[0020] In this embodiment, the work support server 10 is described as a remote operation system having a mutual communication function with each of a plurality of selected work machines and a remote operation device for remotely operating a cooperative work machine to be remotely operated among the plurality of selected work machines. Also, in this embodiment, an example of a work support system in which a composite system is configured by the work support server 10, the remote operation device 20, a first selected work machine 40, and a second selected work machine 50 will be described.

[0021] The components (hardware) of the present invention "acquire" various information, which encompasses all types of computational processing for preparing various information in a form usable in subsequent computational processing, such as receiving the information, reading or searching the specified information from an internal storage device (e.g., a memory) and / or an external storage device (e.g., an external database server), and calculating, estimating, predicting, identifying, etc. the specified information by performing computational processing on the received, read, searched, etc. information.

[0022] (Work support server configuration) The work support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores captured image data and the like. The database 102 may be configured by a database server separate from the work support server 10.

[0023] As shown in FIG. 2, the database 102 holds first identifiers (identifier A-1, identifier B-1, identifier C-1) (the first identifier is, for example, a first IP address assigned to each piece of equipment of the work machine) that are assigned to the communication path (first communication path) of the actual machine imaging device 412 mounted on the first selected work machine 40 and the actual machine imaging device 512 mounted on the second selected work machine 50 and for identifying the work machines on the network, second identifiers (identifier A-2, identifier B-2, identifier C-2) (the second identifier is, for example, a second IP address assigned to each piece of equipment of the work machine) that are assigned to the communication path (second communication path) of the operation system of the first selected work machine 40 and the operation system of the second selected work machine 50 and for identifying the work machines on the network, and actual machine identification information including information on the work machines corresponding to each identifier assigned to the first selected work machine 40 and the second selected work machine 50. Furthermore, the database 102 stores the position information sequentially transmitted from each work machine as GNSS (global navigation satellite system) data.

[0024] Furthermore, the database 102 stores map data (not shown) of the area including each piece of location information. The map data stored in the database 102 may be stored in a plurality of pieces, each piece being categorized by work site. For example, the plurality of map data may be stored so that map data corresponding to the work site selected by the operator can be referenced.

[0025] The first support processing element 121 and the second support processing element 122 are composed of a calculation processing device (a single-core processor or a multi-core processor or a processor core that constitutes the same), read the necessary data and software from a storage device such as a memory, and perform the calculation processing described below in accordance with the software on the data.

[0026] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

[0027] The remote input interface 210 includes a remote operation mechanism 211 and a handheld terminal 2240. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, a remote wireless communication device 224, and a handheld terminal 2240. The handheld terminal 2240 serves as both the remote input interface 210 and the remote output interface 220.

[0028] The remote operation mechanism 211 includes a designated operation operating device for performing a designated operation, such as a wiper operating device for wiping off water droplets such as rainwater adhering to the outer surface of the windshield of the first selected work machine 40 or the second selected work machine 50, and a headlight operating device capable of projecting light ahead of the first selected work machine 40 or the second selected work machine 50.

[0029] It is preferable that the first specified operation is one that assists the work of the work machine. For example, the first specified operation is preferably related to an operation of a wiper device, an operation of lighting devices such as a headlight, a stop lamp, and an interior light, etc. If the captured image includes sound, the first specified operation may be related to an operation of a buzzer or an audio output device that outputs a start-up sound.

[0030] One designated operation operating device has, for example, a push button that accepts input by being pressed. When pressed, the wiper operating device (push button) starts or stops the operation of the wipers of the first selected work machine 40 or the second selected work machine 50. When pressed, the headlight operating device (push button) starts or stops the illumination of the headlights of the first selected work machine 40 or the second selected work machine 50.

[0031] The remote operation mechanism 211 also includes a travel operation device that receives operations related to the travel of the first selected work machine 40 or the second selected work machine 50, a swing operation device that receives operations related to the swing of the first selected work machine 40 or the second selected work machine 50, a boom operation device that receives operations related to the boom of the first selected work machine 40 or the second selected work machine 50, an arm operation device that receives operations related to the arm of the first selected work machine 40 or the second selected work machine 50, and a bucket operation device that receives operations related to an attachment such as a bucket of the first selected work machine 40 or the second selected work machine 50. In other words, the remote operation mechanism 211 includes an operation device for other designated operations for performing other designated operations. In this way, the other designated operations include, for example, actions that may cause interference between an object present around the coordinated work machine and the coordinated work machine when the coordinated work machine is made to perform the operation.

[0032] The other designated operating devices have an operating lever that is rotated. For example, the operating lever (travel lever) of the traveling operating device is operated to move the undercarriage 410 of the work machine 40. The traveling lever may also serve as a traveling pedal. For example, a traveling pedal fixed to the base or lower end of the traveling lever may be provided. The operating lever (swing lever) of the swing operating device is operated to move a hydraulic swing motor that constitutes the swing mechanism 430 of the work machine 40. The operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40. The operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40. The operating lever (bucket lever) of the bucket operating device is operated to move the bucket cylinder 446 of the work machine 40.

[0033] The operation levers constituting the remote operation mechanism 211 are arranged around a seat St on which the operator OP1 sits, as shown in Fig. 3. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or any other form of seating section on which the operator OP1 can sit.

[0034] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 3 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 3 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator OP1.

[0035] The handheld terminal 2240 is a display device installed diagonally forward and to the right when viewed from the seating position of the seat St, as shown in Fig. 3, for example. The handheld terminal 2420 has an input means such as a touch panel. As shown in Fig. 4, for example, the handheld terminal 2240 is capable of selecting the first selected work machine 40 or the second selected work machine 50 to be connected to the remote operation device 20 by using an input means such as a touch panel. The display screen and input surface of the handheld terminal 2420 are directed toward the seat St. The handheld terminal 2240 may be configured so that the angle of the display screen can be adjusted.

[0036] Specifically, as shown in Fig. 4, on the display screen of the remote terminal 2420, images of multiple work machines A to C that are candidates for connection are displayed in an upper column 22401, and the work machines A to C are displayed surrounded by an outer frame in a lower column 22402. A map showing the positional relationships of the multiple work machines A to C is displayed in the upper column 22401, and the unit number (actual machine identification information) and specifications, etc. of each of the work machines A to C are displayed in the lower column 22402. When the operator OP1 taps on any of the images of the work machines A to C in the upper column 22401 or the lower column 22402 to select a work machine to be connected to the remote operation device 20, the display form (color of the outer frame, thickness of the outer frame line, blinking of the work machine icon, background color, etc.) of the work machines A to C selected by the tap in the lower column 22402 is changed so as to be distinguishable from non-selected work machines.

[0037] The remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211 and a right remote image output device 2212, each having a substantially rectangular screen arranged in front of, diagonally forward left and diagonally forward right of the seat St, as shown in Fig. 3, for example. The shape and size of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211 and the right remote image output device 2212 may be the same or different. The handheld terminal 2240 is capable of selecting a first selected work machine 40 or a second selected work machine 50 to be connected to the remote operation device 20 by input means such as the handheld terminal 2240, as shown in Fig. 4, for example.

[0038] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen.

[0039] The remote audio output device 222 is composed of one or more speakers, and is composed of a central remote audio output device 2220, a left remote audio output device 2221, and a right remote audio output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively, as shown in Fig. 3. The specifications of the central remote audio output device 2220, the left remote audio output device 2221, and the right remote audio output device 2222 may be the same or different.

[0040] (Configuration of the first selected work machine) 1, the first selected work machine 40 is equipped with an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a positioning device 460. The actual machine control device 400 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

[0041] The positioning device 460 is a device that detects the position of the first selected work machine 40, and is configured, for example, by a GNSS receiver (GNNS: Global Navigation Satellite System). Position information of the first selected work machine 40 detected by the positioning device 460 is periodically transmitted to the work support server 10 and stored in the database 102.

[0042] The first selected work machine 40 is, for example, a hydraulic, electric or hybrid drive crawler excavator (construction machine) that combines hydraulic and electric drive systems, and as shown in Fig. 5, includes a crawler-type lower traveling body 410 and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.

[0043] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a real machine state sensor group 414. The real machine operation mechanism 411 includes a plurality of push buttons and a plurality of operation levers arranged around a seat arranged inside the cab 424 in the same manner as the remote control mechanism 211.

[0044] A drive mechanism or robot is provided in the cab 424 for receiving signals corresponding to the operation modes of the multiple push buttons and multiple operation levers, and for moving the actual machine operation buttons and the actual machine operation levers based on the received signals.

[0045] The actual machine imaging device 412 is installed, for example, inside the cab 424, and captures an image of the environment including at least a part of the operating mechanism 440 through the front window and a pair of left and right side windows. Some or all of the front window (or window frame) and the side windows may be omitted.

[0046] The actual machine imaging device 412 captures an image including a sticker ST bearing actual machine identification information (No. △△ in FIG. 6) corresponding to the actual machine identification information assigned to the first selected work machine 40. The sticker ST bearing the actual machine identification information is affixed, for example, to the front window of the first selected work machine 40 or the second selected work machine 50. The images captured by the actual machine imaging device 412 are sequentially transmitted to the work support server 10. Note that the actual machine identification information is not limited to the sticker ST and may be engraved, for example, on a plate or on the frame of the first selected work machine 40.

[0047] The cab 424 is provided with a wiper device for wiping off water droplets on the windshield and a headlight device capable of projecting light forward. An actual machine imaging device 412 is provided so that the captured image shows the change in the manner in which the wiper device, headlight, etc. change in response to a specified operation.

[0048] The actual machine status sensor group 414 is composed of angle sensors for measuring the rotation angle (raising and lowering angle) of the boom 441 relative to the upper rotating body 410, the rotation angle of the arm 443 relative to the boom 441, and the rotation angle of the bucket 445 relative to the arm 443, a rotation angle sensor for measuring the rotation angle of the upper rotating body 420 relative to the lower running body 410, an external force sensor for measuring the external force acting on the bucket 445, and a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper rotating body 420, etc.

[0049] The actual machine output interface 42 includes an actual machine image output device 421 and an actual machine wireless communication device 422. The actual machine image output device 421 is disposed, for example, inside the cab 424 and near the front window. The actual machine image output device 421 may be omitted.

[0050] The working mechanism 440 as an operating mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are attached to the working mechanism 440. In addition to the bucket 445, various attachments such as a nibbler, a cutter, and a magnet may be used as the working unit.

[0051] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.

[0052] (Configuration of the second selected work machine) 1, the second selected work machine 50 is equipped with an actual machine control device 500, an actual machine input interface 51, an actual machine output interface 52, and a positioning device 560. The actual machine control device 500 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

[0053] The positioning device 560 is a device that detects the position of the second selected work machine 50, and is configured, for example, by a GNSS receiver (GNNS: Global Navigation Satellite System). Position information of the second selected work machine 50 detected by the positioning device 560 is periodically transmitted to the work support server 10 and stored in the database 102.

[0054] The second work machine 40 is, for example, a hydraulic, electric or hybrid drive type crawler excavator (construction machine) that combines hydraulic and electric drive systems, and as shown in Fig. 5, includes a crawler type lower traveling body 510 and an upper rotating body 520 that is rotatably mounted on the lower traveling body 510 via a rotating mechanism 530. A cab 524 (operator's cabin) is provided on the front left side of the upper rotating body 520. A work mechanism 540 is provided in the front center of the upper rotating body 520.

[0055] The real machine input interface 51 includes a real machine operation mechanism 511, a real machine imaging device 512, and a real machine state sensor group 514. The real machine operation mechanism 511 includes a plurality of push buttons and a plurality of operation levers arranged around a seat arranged inside the cab 524 in the same manner as the remote control mechanism 211.

[0056] A drive mechanism or robot is provided in the cab 524 for receiving signals corresponding to the operation modes of the plurality of push buttons and the plurality of operation levers, and for moving the actual machine operation buttons and the actual machine operation levers based on the received signals.

[0057] The actual machine imaging device 512 is installed, for example, inside the cab 524, and images the environment including at least a part of the operating mechanism 540 through the front window and a pair of left and right side windows. The front window (or window frame) and the side windows may be omitted in part or in whole.

[0058] The actual machine imaging device 512 captures an image so as to include a sticker ST bearing the actual machine identification information assigned to the second selected work machine 50. The sticker ST bearing the actual machine identification information is affixed, for example, to the front window of the second selected work machine 50. The images captured by the actual machine imaging device 512 are sequentially transmitted to the work support server 10 and stored in the database 102. Note that the actual machine identification information is not limited to the sticker ST, and may be engraved, for example, on a plate or on the frame of the second selected work machine 50.

[0059] The cab 524 is provided with a wiper device for wiping off water droplets on the windshield and a headlight device capable of projecting light forward. An actual machine imaging device 512 is provided so that the captured image shows the change in the manner in which the wiper device, headlight, etc. change in response to a specified operation.

[0060] The actual equipment status sensor group 514 is composed of angle sensors for measuring the rotation angle (raising and lowering angle) of the boom 541 relative to the upper rotating body 510, the rotation angle of the arm 543 relative to the boom 541, and the rotation angle of the bucket 545 relative to the arm 543, a rotation angle sensor for measuring the rotation angle of the upper rotating body 520 relative to the lower running body 510, an external force sensor for measuring the external force acting on the bucket 545, and a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper rotating body 520.

[0061] The actual machine output interface 52 includes an actual machine image output device 521 and an actual machine wireless communication device 522. The actual machine image output device 521 is disposed, for example, inside the cab 524 and near the front window. The actual machine image output device 521 may be omitted.

[0062] The working mechanism 540 as an operating mechanism includes a boom 541 mounted on the upper rotating body 520 so as to be able to rise and fall, an arm 543 rotatably connected to the tip of the boom 541, and a bucket 545 rotatably connected to the tip of the arm 543. A boom cylinder 542, an arm cylinder 544, and a bucket cylinder 546 each composed of an extendable hydraulic cylinder are attached to the working mechanism 540. In addition to the bucket 545, various attachments such as a nibbler, a cutter, and a magnet may be used as the working unit.

[0063] The boom cylinder 542 is interposed between the boom 541 and the upper rotating body 520 so as to extend and retract when supplied with hydraulic oil to rotate the boom 541 in the hoisting direction. The arm cylinder 544 is interposed between the arm 543 and the boom 541 so as to extend and retract when supplied with hydraulic oil to rotate the arm 543 about a horizontal axis relative to the boom 541. The bucket cylinder 546 is interposed between the bucket 545 and the arm 543 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 545 about a horizontal axis relative to the arm 543.

[0064] (Display process of remote terminal 2240)

[0065] The display processing of the work support system achieved by cooperation between the work support server 10, the remote control device 20, and a plurality of work machines A to C (in FIG. 7, a first communicating work machine and a second communicating work machine) that are candidates for connection with the remote control device 20 will be described using the flowchart shown in FIG.

[0066] The display process is started by being triggered by the start of the remote operation device 20. When the remote operation device 20 is started by the operator OP1, the remote operation device 20 generates start-up information that is information that the remote operation device 20 has been started, and transmits this to the work support server 10 and the work machines A to C (step S201).

[0067] Each of the work machines A to C starts processing when triggered by receiving the startup information. When each of the work machines A to C receives the startup information, it periodically acquires position information from the work support server 10 (step S401), and transmits the acquired position information to the work support server 10 (step S402).

[0068] The work support server 10 starts processing with the reception of the startup information as a trigger. The work support server 10 acquires the position information transmitted from each of the work machines A to C by referring to the database 102, and generates a display image (step S101).

[0069] The display image is generated in step S101 based on the position information and map data of each of the work machines A to C. The map data is created based on map information of the work site, aerial photographs, etc. The display image is generated, for example, by superimposing icons indicating the first selected work machine 40 and the second selected work machine 50 at positions corresponding to each piece of position information on the map data. The scale of the map data can be changed as appropriate depending on the embodiment, and it is preferable that the position information of each connectable work machine is included at any scale.

[0070] The work support server 10 transmits the display image generated in step S101 to the remote operation device 20 (step S102).

[0071] When the remote control device 20 receives the display image, it outputs the display image to the upper column 22401 of the display screen of the terminal 2240 at hand, for example, as shown in FIG. 4, to display the display image (step S202), and terminates the processing, for example, when the startup of the remote control device 20 is completed.

[0072] The display process of the display image in step S202 may be performed by selectably displaying the first selected work machine 40 or the second selected work machine 50, which are candidates for connection with the remote operation device 20, in the upper column 22401 of the display screen of the handheld terminal 2240. In this way, by showing the display image from a bird's-eye view, the operator OP1 can recognize the positions in the work site of the multiple work machines A to C, which are candidates for connection.

[0073] (Work support processing) The work support processing of the work support system achieved by the cooperation of the work support server 10, remote control device 20, first selected work machine 40, and second selected work machine 50 configured as described above will be explained using the flowchart shown in FIG.

[0074] The work support system starts the work support processing, triggered by the execution of processing in step S202 of the display processing of the remote operation device 20. The operator OP1 selects, for example, the first selected work machine 40 or the second selected work machine 50, which are candidates for connection, by touching the image of the work machine displayed on the handheld terminal 2240. The remote operation device 20 determines whether the operator OP1 has selected a work machine to connect to the remote operation device 20, that is, whether a designation operation has been performed (step S211).

[0075] When it is determined that a designation operation has been performed (step S211: Y), the remote operation device 20 transmits information about the work machine selected by the operator OP1 (selection information) to the work support server 10 (step S212).

[0076] When the first support processing element 121 of the work support server 10 receives the selection information, it stores it in the database 102. The work support server 10 starts work support processing when it receives the selection information transmitted from the remote operation device 20. When the first support processing element 121 of the work support server 10 receives the selection information, it refers to the database 102, selects a work machine corresponding to the selection information based on the actual machine identification information, and requests the actual machine imaging device mounted on the first work machine (hereinafter also referred to as the first communication work machine) to transmit an image through a first communication path identified by the first identifier of the selected work machine (step S111).

[0077] The actual imaging device of the first communicating work machine starts processing when the transmission request is received as a trigger. When the actual imaging device of the first communicating work machine receives the transmission request, it transmits the captured image to the work support server 10 through the first communication path (step S411).

[0078] The first support processing element 121 of the work support server 10 acquires the captured image transmitted from the actual imaging device of the first communication work machine (step S112), transmits the captured image to the remote operation device 20, and causes the remote image output device 221 to output the captured image (step S113). Therefore, the first support processing element 121 is capable of acquiring the captured image captured by the actual imaging device mounted on the first communication work machine, and causing the remote image output device 221 of the remote operation device 20 to output the captured image.

[0079] When the remote operation device 20 receives the captured images, for example, the remote operation device 20 sequentially outputs the received captured images to the remote image output device 221. That is, the remote operation device 20 continues to output the captured images transmitted from the work support server 10 to the remote image output device 221 until the processing is completed.

[0080] Next, the remote operation device 20 displays a message or the like prompting the user to execute the one specified operation, for example, on the remote image output device 221 (step S213). Examples of messages prompting the user to execute the one specified operation include "Please turn on the wipers" and "Please turn on the headlights."

[0081] When the one specifying operation is executed, the remote operation device 20 transmits a command signal for executing the one specifying operation to the work support server 10 (step S214).

[0082] The second support processing element 122 of the work support server 10 selects a work machine corresponding to the selection information based on the actual machine identification information, and transmits a command signal for executing one specified operation transmitted from the remote control device 20 to a second work machine (hereinafter also referred to as a second communication work machine) via a second communication path identified by the second identifier of the selected work machine (step S114). Therefore, the second support processing element 122 of the work support server 10 is capable of recognizing that one specified operation has been performed that is part of multiple types of operations for operating the second communication work machine for which communication with the remote control device 20 linked to the second communication path is established by the selection information, among the multiple selected work machines, and in which a change in the operation corresponding to the operation is reflected in the captured image, and transmitting a command signal corresponding to the one specified operation to the second communication work machine.

[0083] When the second communication work machine receives the command signal regarding the execution of the one specified operation, it executes the one specified operation (step S412).

[0084] The operator OP1 can confirm whether or not a specified operation is being performed on the second communicating work machine by checking the image captured by the actual imaging device of the first communicating work machine displayed on the remote image output device 221 of the remote control device 20.

[0085] Specifically, if the first communication work machine and the second communication work machine are not the same selected work machine, the operator OP1 cannot confirm the state of one designated operation being performed by the second communication work machine in the image captured by the actual machine imaging device of the first communication work machine. On the other hand, if the first communication work machine and the second communication work machine are the same selected work machine, the operator OP1 can confirm the state of one designated operation being performed by the second communication work machine in the image captured by the actual machine imaging device of the first communication work machine.

[0086] The operator OP1, for example, touches the handheld terminal 2240 to input whether a specific operation has been performed on the second communication work machine. The remote control device 20 determines whether a specific operation has been performed on the second communication work machine based on the input by the operator (step S215).

[0087] When the remote control device 20 determines that a specified operation has been executed on the second communicating work machine (step S215: Y), it transmits good information indicating that the first communicating work machine and the second communicating work machine are the same selected work machine to the work support server 10 (step S216).

[0088] When the remote control device 20 determines that one specified operation has not been performed on the second communicating work machine (step S215: N), it transmits defect information indicating that the first communicating work machine and the second communicating work machine are not the same selected work machine to the work support server 10 (step S217).

[0089] The second support processing element 122 of the work support server 10 judges whether or not the cancellation condition for another specified operation is satisfied based on the good information or the bad information transmitted from the remote operation device 20 (step S115).

[0090] When the second support processing element 122 of the work support server 10 determines that the cancellation condition for the other designated operation is satisfied (step S115: Y), it starts accepting the other designated operation (step S116). That is, in this embodiment, the second support processing element 122 determines that a cancellation operation has been performed in the remote operation device upon receipt of good information. After starting to accept the other designated operation in step S116, the second support processing element 122 transmits a command signal corresponding to the other designated operation transmitted from the remote operation device 20 to the second communication work machine. As a result, the operation of the work mechanism and the like of the second communication work machine is controlled based on a remote operation command corresponding to the operation mode of the remote operation mechanism 211 recognized by the remote control device 200 of the remote operation device 20.

[0091] If the second support processing element 122 of the work support server 10 determines that the cancellation condition for the other specified operations is not satisfied (step S115: N), it keeps the acceptance of the other specified operations on hold (step S117) and ends the work support processing.

[0092] The operator OP1 can visually confirm the captured image in which one specified operation is being performed, displayed on the remote image output device 221. This enables the operator OP1 to confirm whether the selection information of the first selected work machine 40 or the second selected work machine 50 to which the remote operation device 20 is connected matches the first communicating work machine and the second communicating work machine.

[0093] As described above, according to the remote operation system of the present invention, an image captured by an actual machine imaging device mounted on the first communicating work machine is acquired via the first communication path based on the selection information that identifies the cooperating work machine, and a command signal corresponding to one designation operation is transmitted to the second communicating work machine. Therefore, the operator can confirm through the captured image whether the first communicating work machine operates in response to one designation operation performed on the second communicating work machine, that is, whether the operation performed on the cooperating work machine to be remotely operated has been performed on the cooperating work machine. This allows the operator to confirm the connection status with the work machine to be remotely operated.

[0094] Furthermore, by displaying the actual machine identification information together with the display image, the operator OP1 can visually check whether the actual machine identification information of the captured image displayed on the display is consistent with the actual machine identification information of the display image, thereby enabling the checking work to be performed quickly and accurately.

[0095] In the above embodiment, the work support processing is executed by the work support server 10. However, the work support processing may be executed by the remote control device 20, or may be executed by the first selected work machine 40 or the second selected work machine 50.

[0096] Also, it has been explained that the images are captured by the actual machine imaging device 412 of the first selected work machine 40 or the actual machine imaging device 512 of the second selected work machine 50. However, the images may be captured by an imaging device other than the actual machine imaging device 412 of the first selected work machine 40 or the actual machine imaging device 512 of the second selected work machine 50.

[0097] Examples of such imaging devices include a fixed-point imaging device provided in a position capable of imaging the exterior of the first selected work machine 40 or the second selected work machine 50, and an imaging device provided on an aircraft such as a drone.

[0098] When such imaging devices are used, the sticker ST on which the actual machine identification information is printed is preferably provided in a position visible from the outside of the first selected work machine 40 and the second selected work machine 50. Examples of such positions include the side of the upper revolving body 420 of the first selected work machine 40 and the upper revolving body 520 of the second selected work machine 50. It is preferable that these imaging devices are notified when a connection operation is performed on the first selected work machine 40 or the second selected work machine 50 to the remote control device 20. In this way, it is possible to start transmitting images from the imaging devices to the work support server 10. [Explanation of symbols]

[0099] 10. Work support server 121 - First support processing element 122 - Second support processing element 20. Remote control device 40. First selected work machine 50...Second selected work machine

Claims

1. A remote operation system having a mutual communication function between a plurality of work machines and a remote operation device for remotely operating a coordinated work machine, which is one of the plurality of work machines that is to be remotely operated, a first support processing element that, when a determination result of whether or not a designation operation of selection information specifying the cooperative work machine as a remotely operated target among the multiple work machines has been executed on the remote operation device and the selection information is selected on the remote operation device, acquires an image captured by an actual machine imaging device mounted on a first work machine with which communication with the remote operation device is established via a first communication path linked to the selection information, and outputs the captured image to an output interface of the remote operation device; a second support processing element that recognizes that a specified operation has been performed which is part of a plurality of types of operations for operating a second work machine, of which communication with the remote operation device is established by a second communication path linked to the selection information, and in which a change in the operation corresponding to the specified operation is reflected in the captured image, and that transmits a command signal corresponding to the specified operation to the second work machine; A remote control system comprising:

2. the second support processing element suspends transmission of a command signal corresponding to another designated operation for operating the cooperative work machine to the second work machine when a cancellation condition is not satisfied, and allows transmission of a command signal corresponding to another designated operation for operating the cooperative work machine to the second work machine when the cancellation condition is satisfied, 2. The remote control system according to claim 1, wherein the cancel condition includes a cancel operation being performed on the remote control device.

3. 2. The remote operation system according to claim 1, wherein the second operation support processing element obtains a determination result as to whether or not the one designated operation has been executed, and, if the one designated operation has been executed, allows another designated operation for operating the collaborative work machine, and transmits a command signal corresponding to the other designated operation to the second work machine.

4. the second support processing element suspends transmission of a command signal corresponding to another designated operation for operating the portable work machine to the second work machine when a cancellation condition is not satisfied, and allows transmission of a command signal corresponding to another designated operation for operating the portable work machine to the second work machine when the cancellation condition is satisfied; The remote control system according to any one of claims 1 to 3, characterized in that the release condition includes, when the one designation operation is executed, a second support processing element obtaining a judgment result as to whether or not the captured image shows a change in behavior corresponding to the one designation operation, and there being a change in behavior corresponding to the one designation operation.

5. The first operation support processing element acquires position data of the plurality of work machines to be remotely operated, The remote operation system according to any one of claims 1 to 4, characterized in that the second operation support processing element generates a display image in which each of the plurality of work machines is reflected based on the plurality of position data, and transmits the display image to the remote operation device.

Images