Work assistance system and work assistance complex system

Abstract

A remote operation system and a work assistance composite system are provided. Whether a work machine that operates in accordance with an operator's operation is consistent with a selected work machine can be confirmed. The remote operation system has a plurality of work machines and a remote operation device for remotely operating the work machines in cooperation. The remote operation system has a first assistance processing element that acquires a captured image captured by an actual machine camera of a first work machine that has established communication via a first communication path, and transmits the captured image to the remote operation device. The remote operation system has a second assistance processing element that identifies a specified operation is performed, and transmits an instruction signal corresponding to the specified operation to a second work machine, where the specified operation is, for example, a part of a plurality of operations that cause the second work machine that has established communication via a second communication path to operate, and is a specified operation in which a changed form corresponding to the operation is reflected in the captured image.

Description

Technical Field

[0001] The present invention relates to a remote operating device having multiple working machines and a remote operating device for remotely operating one of the multiple working machines that constitutes a remotely operated object, and a remote operating system having the function of enabling the remote operating device to communicate with each of the multiple working machines. Background Technology

[0002] Conventionally, operators have operated work machinery remotely without being seated in the machinery. One example of such work machinery is a remote operating system disclosed in Patent Document 1, which is configured with a master unit and a slave unit capable of communicating with each other. The operator operates a joystick on the master unit, and the slave unit, located in the driver's seat of the work machinery, operates the joystick of the work machinery.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-176401 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, when using a single host machine to remotely operate two or more machines, it is necessary to connect the host machine with the slave machines installed in each machine in a way that allows them to communicate with each other.

[0008] In addition, the operator can remotely operate the machine by observing the surrounding images of the machine on the host's display while manipulating joysticks and other controls.

[0009] However, there are situations where the image displayed on the main unit's monitor is an image of the surroundings of a different piece of machinery than the one being operated. When the operator is actually riding in the machinery, they can confirm its appearance and the surrounding environment before boarding; however, this confirmation is impossible when operating the machinery remotely. Therefore, it is difficult for the operator to notice that the remote control device is connected to an unexpected piece of machinery (a different slave machine). Even if they wanted to confirm using images from a camera mounted on the machinery, these images are generally similar and difficult to distinguish. In such cases, if remote operation is performed, the operator may operate machinery that has not been identified as the target machine.

[0010] The present invention was made in view of the above-mentioned problems, and its object is to provide a remote operating system that can confirm whether the working machine that operates according to the operator's operation is consistent with the working machine selected by the operator.

[0011] Methods for solving problems

[0012] To achieve the above objectives, the remote operating system of the present invention includes multiple working machines and a remote operating device for remotely operating one of the multiple working machines, namely a cooperative working machine, which constitutes the object of remote operation. The remote operating system has the function of enabling each of the multiple working machines to communicate with the remote operating device.

[0013] The remote operating system is characterized by having:

[0014] A first auxiliary processing element outputs a determination result and a captured image to the output interface of the remote operating device. The determination result refers to the result in the remote operating device determining whether the specified operation of the selection information of the cooperative working machine as the remote operating object among the plurality of working machines has been executed. The captured image is obtained by a real-machine shooting device mounted on the first working machine when the selection information is selected in the remote operating device. The first working machine establishes communication with the remote operating device through a first communication path associated with the selection information. A second auxiliary processing element identifies that a specified operation has been executed and sends an instruction signal corresponding to the specified operation to the second working machine. The specified operation refers to a part of a plurality of operations that cause the second working machine, which has established communication with the remote operating device through a second communication path associated with the selection information, to move. The specified operation is a specified operation in which the change of the action corresponding to the operation is reflected in the captured image.

[0015] Brief description of the attached diagram

[0016] Figure 1 This is a structural illustration of a work assistance composite system as one embodiment of the present invention.

[0017] Figure 2 This is an illustrative diagram showing an example of the data structure stored in the database of the job support server.

[0018] Figure 3 This is an explanatory diagram of the structure of the remote operation device.

[0019] Figure 4 This is an explanatory diagram showing an example of a screen display on a remotely operated device.

[0020] Figure 5 It is an explanatory diagram related to the structure of the operating machinery.

[0021] Figure 6This is an explanatory diagram showing an example of an image captured by a real camera.

[0022] Figure 7 This is a flowchart illustrating an example of display processing performed by a handheld terminal by a job support server.

[0023] Figure 8 This is a flowchart illustrating an example of job assistance processing performed by a job assistance server.

[0024] Explanation of reference numerals in the attached figures

[0025] 10: Job assistance server

[0026] 121: First auxiliary processing element,

[0027] 122: Second auxiliary processing element,

[0028] 20: Remote operating device,

[0029] 40: The first selected operating machinery.

[0030] 50: The second selected operating machinery. Detailed Implementation

[0031] (Structure of a remote operating system)

[0032] Figure 1 The work assistance system shown as an embodiment of the present invention comprises a first selected work machine 40 operated by a remote operating device 20 and a work assistance server 10 for remotely assisting the operation of a second selected work machine 50. This work assistance system is configured such that the work assistance server 10 can communicate with the remote operating device 20, the first selected work machine 40, and the second selected work machine 50 via a common or separate network. It should be noted that in this figure, the work assistance server 10 can connect to either the first selected work machine 40 or the second selected work machine 50; however, the work assistance server 10 only needs to be a server capable of connecting to two or more work machines, and the number of connected machines is not limited. Furthermore, either the first selected work machine 40 or the second selected work machine 50 operated by the remote operating device 20 is referred to as a cooperative work machine.

[0033] In this embodiment, the work assistance server 10 will be described as a remote operating system. This remote operating system has multiple selected work machines and a remote operating device. The remote operating device is used to remotely operate the cooperating work machine that constitutes the remote operation object among the multiple selected work machines. The remote operating system has the function of communicating with each of the multiple selected work machines and the remote operating device. Furthermore, in this embodiment, an example will be described using a work assistance system that is a composite system consisting of the work assistance server 10, the remote operating device 20, the first selected work machine 40, and the second selected work machine 50.

[0034] The concept of "acquiring" various information by the constituent elements (hardware) of the present invention includes the following processes: receiving the information; reading or retrieving the specified information from internal storage devices (e.g., memory) and / or external storage devices (e.g., external database servers); and performing arithmetic processing on the received information, read information, retrieved information, etc., to calculate, estimate, predict, identify, etc., in order to prepare for subsequent arithmetic processing by utilizing various information.

[0035] (Structure of the job support server)

[0036] The job assistance server 10 includes a database 102, a first auxiliary processing element 121, and a second auxiliary processing element 122. The database 102 stores and saves captured image data, etc. The database 102 can also be configured as a database server independent of the job assistance server 10.

[0037] like Figure 2As shown, database 102 stores a first identifier (identifier A-1, identifier B-1, identifier C-1) (the first identifier is, for example, the first IP address assigned to each device of the operating machinery), a second identifier (identifier A-2, identifier B-2, identifier C-2) (the second identifier is, for example, the second IP address assigned to each device of the operating machinery), and machine identification information. The first identifier is assigned to the communication path (first communication path) of the machine-mounted imaging device 412 mounted on the first selected operating machinery 40 and the machine-mounted imaging device 512 mounted on the second selected operating machinery 50, and is used to identify the operating machinery on the network. The second identifier is assigned to the communication path (second communication path) of the operating system of the first selected operating machinery 40 and the operating system of the second selected operating machinery 50, and is used to identify the operating machinery on the network. The machine identification information includes information about the operating machinery corresponding to each identifier assigned to each selected operating machinery in the first and second selected operating machinery 50. In addition, database 102 stores the location information sent sequentially from each operating machine as GNSS (global navigation satellite system) data.

[0038] Furthermore, database 102 stores map data (not shown) containing location information for various areas. It should be noted that the map data stored in database 102 can be multiple map data stored separately for each work site. For example, multiple map data can be stored in a way that allows reference to the map data corresponding to the work site selected by the operator.

[0039] The first auxiliary processing element 121 and the second auxiliary processing element 122 are composed of a computing device (a single-core processor or a multi-core processor, or a processor core constituting such processors), which reads necessary data and software from a storage device such as a memory, and performs computing processing according to the software as an object.

[0040] (Structure of the remote operation device)

[0041] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a computing device (a single-core processor or a multi-core processor, or a processor core constituting such a processor), which reads necessary data and software from storage devices such as memory, and performs computing processing according to the software based on the data.

[0042] The remote input interface 210 includes a remote operating mechanism 211 and a handheld terminal 2240. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, a remote wireless communication device 224, and a handheld terminal 2240. The handheld terminal 2240 also serves as both the remote input interface 210 and the remote output interface 220.

[0043] The remote operation mechanism 211 includes a designated operation device for performing a designated operation such as a wiper operation device and a headlight operation device. The wiper operation device wipes away rainwater or other water droplets adhering to the outer surface of the windshield of the first selected work machine 40 or the second selected work machine 50; the headlight operation device can project light in front of the first selected work machine 40 or the second selected work machine 50.

[0044] Furthermore, the designated operation is preferably a designated operation for assisting the operation of the work machinery. The designated operation is preferably related to, for example, the operation of windshield wipers, headlights, parking lights, interior lights, and other lighting devices. Additionally, if the captured image includes voice, the designated operation may also be related to the operation of a buzzer or a voice output device that outputs a start tone.

[0045] A designated operating device includes, for example, a button that receives input by pressing. By pressing the wiper operating device (button), the wipers of the first selected work machine 40 or the second selected work machine 50 are started or stopped. Similarly, by pressing the headlight operating device (button), the headlights of the first selected work machine 40 or the second selected work machine 50 are turned on or off.

[0046] Furthermore, the remote operation mechanism 211 includes: a travel operation device that handles operations related to the travel of the first selected work machine 40 or the second selected work machine 50; a slewing operation device that handles operations related to the slewing of the first selected work machine 40 or the second selected work machine 50; a boom operation device that handles operations related to the boom of the first selected work machine 40 or the second selected work machine 50; a stick operation device that handles operations related to the stick of the first selected work machine 40 or the second selected work machine 50; and a bucket operation device that handles operations related to attachments such as the bucket of the first selected work machine 40 or the second selected work machine 50. In other words, the remote operation mechanism 211 includes an operation device for performing other specified operations. Thus, for example, when a cooperating work machine is performing this operation, the other specified operation includes actions that may interfere with the cooperating work machine caused by objects present around it.

[0047] Other designated operating devices include operating levers for receiving rotational operations. For example, the operating lever (travel lever) of the travel operating device is operated to move the lower traveling body 410 of the work machinery 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever can be provided. The operating lever (slewing lever) of the slewing operating device is operated to move the hydraulic slewing motor constituting the slewing mechanism 430 of the work machinery 40. The operating lever (boom lever) of the boom operating device is operated to actuate the boom cylinder 442 of the work machinery 40. The operating lever (boom lever) of the stick operating device is operated to move the stick cylinder 444 of the work machinery 40. The operating lever (bucket stick) of the bucket operating device is operated to actuate the bucket cylinder 446 of the work machinery 40.

[0048] like Figure 3 As shown, the levers constituting the remote operation mechanism 211 are arranged around a seat St for the operator OP1 to sit on. The seat St can be a high-backed chair with armrests, a low-backed chair without a headrest, or a chair without a backrest, etc., any seating position in which the operator OP1 can sit.

[0049] A pair of left and right travel levers 2110 are arranged in front of the seat St, corresponding to the left and right tracks. One lever can function as multiple levers. For example, a... Figure 3 The left-side operating lever 2111 located in front of the left side frame of the seat St shown can function as an arm when operated in the forward / backward direction, and as a swivel lever when operated in the left / right direction. Similarly, it is set with Figure 3 The right-side operating lever 2112, located in front of the right-side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever mode can be changed arbitrarily according to the operator's instructions OP1.

[0050] For example, such as Figure 3 As shown, the handheld terminal 2240 is a display device positioned diagonally in front of the right side when viewed from the seated position of seat St. The handheld terminal 2240 includes, for example, an input unit such as a touch panel. For example, as... Figure 4 As shown, the handheld terminal 2240 can select either the first selected machine 40 or the second selected machine 50 connected to the remote operating device 20 via an input unit such as a touch panel. The display screen and input surface of the handheld terminal 2240 face the seat St side. The handheld terminal 2240 can also be configured to allow for adjustment of the display screen angle.

[0051] Specifically, such as Figure 4 As shown, on the display screen of the handheld terminal 2240, the upper bar 22401 displays images of multiple work machines A to C that are candidate work machines to be connected, and the lower bar 22402 displays work machines A to C surrounded by an outer frame. The upper bar 22401 displays a map showing the positional relationship of the multiple work machines A to C, and the lower bar 22402 displays the machine number (actual machine identification information), specifications, etc., of each work machine A to C. When the operator OP1 clicks on any of the work machines A to C images in the upper bar 22401 or the lower bar 22402 to select it as the work machine to be connected to the remote operating device 20, the display format of the work machine A to C selected by the click (the color of the outer frame, the thickness of the outer frame line, the flashing of the work machine icon, the background color, etc.) in the lower bar 22402 changes to be recognizable as a non-selected work machine.

[0052] For example, such as Figure 3 As shown, the remote image output device 221 comprises a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each with a roughly rectangular screen. The central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 are respectively positioned in front of, diagonally to the left front of, and diagonally to the right front of the seat St. The shape and size of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 can be the same or different. For example, as... Figure 4 As shown, the handheld terminal 2240 can select either the first selected machine 40 or the second selected machine 50 connected to the remote operating device 20 through its input unit.

[0053] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 can be parallel or tilted relative to the vertical direction. At least one of the image output devices 2210, 2211, and 2212 can also be composed of multiple image output devices. For example, the central remote image output device 2210 can also be composed of a pair of vertically adjacent image output devices with a generally rectangular screen shape.

[0054] The remote audio output device 222 consists of one or more speakers. For example, such as... Figure 3As shown, the remote audio output device 222 consists of a central remote audio output device 2220, a left-side remote audio output device 2221, and a right-side remote audio output device 2222, respectively located behind the seat St, behind the left armrest, and behind the right armrest. The specifications of the central remote audio output device 2220, the left-side remote audio output device 2221, and the right-side remote audio output device 2222 can be the same or different.

[0055] (The structure of the first selected machine)

[0056] like Figure 1 As shown, the first selected machine 40 includes a machine control device 400, a machine input interface 41, a machine output interface 42, and a positioning device 460. The machine control device 400 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting such a processor), which reads necessary data and software from storage devices such as memory, and performs calculations according to the software based on the data.

[0057] The positioning device 460 is a device for detecting the position of the first selected working machine 40, and may be composed of, for example, a GNSS receiver (GNSS: Global Navigation Satellite System). The position information of the first selected working machine 40 detected by the positioning device 460 is periodically sent to the work assistance server 10 and stored in the database 102.

[0058] The first selected operating machinery 40 is, for example, a hydraulic, electric, or hybrid drive tracked excavator (construction machinery). For example... Figure 5 As shown, the first selected work machine 40 includes a tracked lower traveling body 410 and an upper rotating body 420, which is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A driver's cab 424 is provided on the front left side of the upper rotating body 420. A working mechanism 440 is provided on the front center of the upper rotating body 420.

[0059] The actual machine input interface 41 includes an actual machine operation mechanism 411, an actual machine shooting device 412, and an actual machine status sensor group 414. The actual machine operation mechanism 411 has multiple buttons and multiple control levers arranged in the same manner as the remote operation mechanism 211, around the seat located inside the cab 424.

[0060] A drive mechanism or robot is installed in the cab 424. The drive mechanism or robot receives signals corresponding to the operation modes of multiple buttons and multiple levers, and drives the actual machine operation buttons and actual machine operation levers based on the received signals.

[0061] The on-board camera 412 is installed, for example, inside the cab 424, and captures images of at least a portion of the environment, including the working mechanism 440, through the front window and a pair of side windows. The front window (or window frame) and part or all of the side windows may be omitted from the captured images.

[0062] The real-time imaging device 412 includes a device containing real-time identification information (in... Figure 6 The machine identification information is captured by a label ST (No. △△), and this machine identification information corresponds to the machine identification information assigned to the first selected machine 40. The label ST containing the machine identification information is affixed, for example, to the front window of the first selected machine 40 or the second selected machine 50. The images captured by the machine identification device 412 are sequentially sent to the operation assistance server 10. It should be noted that the machine identification information is not limited to being recorded on a label ST; for example, it can also be engraved on a plate or on the frame of the first selected machine 40.

[0063] The driver's cab 424 is equipped with windshield wipers to wipe away water droplets adhering to the windshield and headlights to illuminate the road ahead. The camera device 412 is configured such that it reflects in the captured image changes due to a specified operation of the windshield wipers, headlights, etc.

[0064] The actual machine status sensor group 414 consists of an angle sensor, a slewing angle sensor, an external force sensor, and a triaxial acceleration sensor. Specifically, the angle sensor measures the rotation angle (lifting and lowering angle) of the boom 441 relative to the upper rotating body 410, the rotation angle of the stick 443 relative to the boom 441, and the rotation angle of the bucket 445 relative to the stick 443; the slewing angle sensor measures the slewing angle of the upper rotating body 420 relative to the lower traveling body 410; the external force sensor measures the external force acting on the bucket 445; and the triaxial acceleration sensor measures the triaxial acceleration acting on the upper rotating body 420.

[0065] The machine output interface 42 includes a machine image output device 421 and a machine wireless communication device 422. The machine image output device 421 is, for example, located near the front window inside the cab 424. The machine image output device 421 may also be omitted from the machine output interface 42.

[0066] The working mechanism 440, serving as the actuation mechanism, includes: a boom 441, which is mounted on the upper rotating body 420 in a lifting and lowering manner; a stick 443, which is rotatably connected to the top of the boom 441; and a bucket 445, which is rotatably connected to the top of the stick 443. The working mechanism 440 is equipped with a boom cylinder 442, a stick cylinder 444, and a bucket cylinder 446, all of which are telescopic hydraulic cylinders. In addition to the bucket 445, various attachment devices such as shears, cutters, and magnetic tools can be used as the working unit.

[0067] Boom cylinder 442 is located between boom 441 and upper slewing body 420, allowing it to extend and retract by receiving a supply of working oil, thereby causing boom 441 to rotate in the lifting direction. Stick cylinder 444 is located between stick 443 and boom 441, allowing it to extend and retract by receiving a supply of working oil, thereby causing stick 443 to rotate about a horizontal axis relative to boom 441. Bucket cylinder 446 is located between bucket 445 and stick 443, allowing it to extend and retract by receiving a supply of working oil, thereby causing bucket 445 to rotate about a horizontal axis relative to stick 443.

[0068] (Structure of the second selected operating machinery)

[0069] like Figure 1 As shown, the second selected machine 50 includes a machine control device 500, a machine input interface 51, a machine output interface 52, and a positioning device 560. The machine control device 500 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting such a processor), which reads necessary data and software from storage devices such as memory, and performs calculations according to the software based on the data.

[0070] The positioning device 560 is a device for detecting the position of the second selected machine 50, and may be, for example, a GNSS receiver (GNSS: Global Navigation Satellite System). The position information of the second selected machine 50 detected by the positioning device 560 is periodically sent to the operation assistance server 10 and stored in the database 102.

[0071] The second selected operating machinery 50 is, for example, a tracked excavator (construction machinery) with a hydraulic, electric, or hybrid drive system combining hydraulic and electric power. For example... Figure 5As shown, the second selected work machine 50 includes a tracked lower traveling body 510 and an upper rotating body 520, which is rotatably mounted on the lower traveling body 510 via a rotating mechanism 530. A driver's cab 524 is provided on the front left side of the upper rotating body 520. A working mechanism 540 is provided on the front center of the upper rotating body 520.

[0072] The actual input interface 51 includes an actual operation mechanism 511, an actual shooting device 512, and an actual status sensor group 514. The actual operation mechanism 511 has multiple buttons and multiple levers arranged in the same manner as the remote operation mechanism 211, located around the seat inside the cab 524.

[0073] A drive mechanism or robot is installed in the cab 524. The drive mechanism or robot receives signals corresponding to the operation modes of multiple buttons and multiple levers, and drives the actual machine operation buttons and actual machine operation levers based on the received signals.

[0074] The on-board camera 512 is installed, for example, inside the cab 524, and captures images of at least a portion of the environment, including the actuation mechanism 540, through the front window and a pair of side windows. Part or all of the front window (or window frame) and side windows may be omitted from the captured images.

[0075] The machine imaging device 512 takes pictures using a tag ST containing machine identification information assigned to the second selected machine 50. The tag ST, containing the machine identification information, is affixed, for example, to the front window of the second selected machine 50. The images captured by the machine imaging device 512 are sequentially sent to the operation assistance server 10 and stored in the database 102. It should be noted that the machine identification information is not limited to being recorded on a tag ST; for example, it can also be engraved on a plate or on the frame of the second selected machine 50.

[0076] The driver's cab 524 is equipped with windshield wipers to wipe away water droplets adhering to the windshield and headlights to illuminate the road ahead. The camera device 512 is configured such that it reflects in the captured image changes due to a specified operation of the windshield wipers, headlights, etc.

[0077] The actual machine status sensor group 514 consists of an angle sensor, a slewing angle sensor, an external force sensor, and a triaxial acceleration sensor. Specifically, the angle sensor measures the rotation angle (lifting and lowering angle) of the boom 541 relative to the upper rotating body 520, the rotation angle of the stick 543 relative to the boom 541, and the rotation angle of the bucket 545 relative to the stick 543; the slewing angle sensor measures the slewing angle of the upper rotating body 520 relative to the lower traveling body 510; the external force sensor measures the external force acting on the bucket 545; and the triaxial acceleration sensor measures the triaxial acceleration acting on the upper rotating body 520.

[0078] The machine output interface 52 includes a machine image output device 521 and a machine wireless communication device 522. The machine image output device 521 is, for example, located near the front window inside the cab 524. The machine image output device 521 may also be omitted from the machine output interface 52.

[0079] The working mechanism 540, serving as the actuation mechanism, includes: a boom 541, which is mounted on the upper rotating body 520 in a lifting and lowering manner; a stick 543, which is rotatably connected to the top of the boom 541; and a bucket 545, which is rotatably connected to the top of the stick 543. The working mechanism 540 is equipped with a boom cylinder 542, a stick cylinder 544, and a bucket cylinder 546, all of which are telescopic hydraulic cylinders. In addition to the bucket 545, various attachment devices such as shears, cutters, and magnetic tools can be used as the working unit.

[0080] Boom cylinder 542 is located between boom 541 and upper slewing body 520, allowing it to extend and retract by receiving a supply of working oil, thereby causing boom 541 to rotate in the lifting direction. Stick cylinder 544 is located between stick 543 and boom 541, allowing it to extend and retract by receiving a supply of working oil, thereby causing stick 543 to rotate about a horizontal axis relative to boom 541. Bucket cylinder 546 is located between bucket 545 and stick 543, allowing it to extend and retract by receiving a supply of working oil, thereby causing bucket 545 to rotate about a horizontal axis relative to stick 543.

[0081] (Display processing of the 2240 terminal at hand)

[0082] use Figure 7 The flowchart shown illustrates the display processing of the work assistance system, which uses a work assistance server 10, a remote operating device 2, and multiple work machines A to C connected to the remote operating device 20 as candidate work machines. Figure 7 The first and second communication operation machines shown in the figure function through cooperation.

[0083] The display process begins when the remote operating device 20 is activated. When operator OP1 activates the remote operating device 20, the remote operating device 20 generates activation information indicating that the remote operating device 20 has been activated, and sends it to the work assistance server 10 and the work machines A to C (step S201).

[0084] Each of the working machines A to C begins processing upon receiving a start information. Upon receiving the start information, each of the working machines A to C periodically acquires its position information (step S401) and sends the acquired position information to the work assistance server 10 (step S402).

[0085] The operation assistance server 10 begins processing upon receiving the start information. The operation assistance server 10 refers to the database 102 to obtain the position information sent from each of the operating machines A to C, and generates a display image (step S101).

[0086] In step S101, the display image is generated based on the location information of each of the operating machines A to C and map data. It should be noted that the map data is created based on map information of the work site, aerial photographs, etc. For example, the display image is generated by overlaying icons representing the first selected operating machine 40 and the second selected operating machine 50 at locations corresponding to the location information on the map data. The scale of the map data can be appropriately changed according to the implementation method, as long as it can include the location information of each connectable operating machine at any scale.

[0087] The operation assistance server 10 sends the display image generated in step S101 to the remote operation device 20 (step S102).

[0088] When the remote operating device 20 receives the displayed image, for example... Figure 4 As shown, a display image is output to the upper bar 22401 of the display screen of the handheld terminal 2240 to display the display image (step S202). For example, the process ends when the startup of the remote operation device 20 is completed.

[0089] The display processing of the image in step S202 can be displayed in the upper bar 22401 of the display screen of the handheld terminal 2240 in such a way that the first selected work machine 40 and the second selected work machine 50 connected to the remote operating device 20 can be selected. In this way, by displaying the image in a top-down manner, the operator OP1 can identify the location of the multiple work machines A to C that are connected as candidate work machines at the work site.

[0090] (Job assistance processing)

[0091] use Figure 8 The flowchart shown illustrates the work assistance process of the work assistance system, which functions through the cooperation of the work assistance server 10, remote operation device 20, first selected work machine 40, and second selected work machine 50, all of which have the above-described structure.

[0092] The operation assistance system begins execution of operation assistance processing when the processing of step S202 of the display processing of the remote operating device 20 is executed as a trigger condition. For example, operator OP1 selects either the first selected operation machine 40 or the second selected operation machine 50 as candidate operation machines to be connected by touching an image of the operation machine displayed on the handheld terminal 2240. The remote operating device 20 determines whether operator OP1 has selected the operation machine connected to the remote operating device 20, that is, whether the specified operation has been performed (step S211).

[0093] When the remote operation device 20 determines that the specified operation has been performed (step S211: Y (Yes)), it sends the information of the operating machinery selected by the operator OP1 (selection information) to the operation assistance server 10 (step S212).

[0094] When the first auxiliary processing element 121 of the job assistance server 10 receives selection information, it saves the selection information in the database 102. The job assistance server 10 starts to execute job assistance processing when it receives the selection information sent from the remote operation device 20. When the first auxiliary processing element 121 of the job assistance server 10 receives the selection information, it refers to the database 102, selects the working machine corresponding to the selection information based on the machine identification information, and requests to send an image to the machine imaging device mounted on the first working machine (hereinafter also referred to as the first communication working machine) through the first communication path identified according to the first identifier of the selected working machine (step S111).

[0095] The real-machine shooting device of the first communication operation machine starts processing upon receiving a transmission request. When the real-machine shooting device of the first communication operation machine receives the transmission request, it sends the captured image to the operation assistance server 10 through the first communication path (step S411).

[0096] The first auxiliary processing element 121 of the operation assistance server 10 acquires the captured image sent from the actual shooting device of the first communication operation machine (step S112), sends the captured image to the remote operation device 20, and outputs it to the remote image output device 221 (step S113). Therefore, the first auxiliary processing element 121 is able to acquire the captured image taken by the actual shooting device mounted on the first communication operation machine and output the captured image to the remote image output device 221 of the remote operation device 20.

[0097] When the remote operation device 20 receives a captured image, it sequentially outputs the received captured images to the remote image output device 221. That is, the remote operation device 20 continuously outputs captured images sent from the job assistance server 10 to the remote image output device 221 until the processing is completed.

[0098] Next, the remote operation device 20 displays a prompt, such as a message urging the execution of a specified operation, on the remote image output device 221 (step S213). The message urging the execution of a specified operation may include, for example, "Please turn on the windshield wipers" or "Please turn on the headlights."

[0099] When a specified operation is executed, the remote operation device 20 sends an instruction signal to the job support server 10 to execute the specified operation (step S214).

[0100] The second auxiliary processing element 122 of the work assistance server 10 sends a command signal from the remote operating device 20 to the second work machine (hereinafter also referred to as the second communication work machine) via a second communication path identified according to the second identifier of the selected work machine (step S114). The second work machine is the work machine selected based on the machine identification information corresponding to the selection information. Therefore, the second auxiliary processing element 122 of the work assistance server 10 can identify a specified operation and send the command information corresponding to the specified operation to the second communication work machine. The specified information is, for example, a part of a plurality of operations that cause the second communication work machine among a plurality of selected work machines to operate, and a specified operation that renders a variation of the action corresponding to the operation in a captured image. The second communication work machine establishes communication with the remote operating device 20 associated with the second communication path based on the selection information.

[0101] When a command signal is received relating to the execution of a specified operation, the second communication machine performs the specified operation (step S412).

[0102] Operator OP1 can confirm whether a specified operation has been performed in the second communication machine by checking the image captured by the real machine shooting device of the first communication machine displayed by the remote image output device 221 of the remote operation device 20.

[0103] Specifically, when the first and second communication operation machines are not the same selected operation machine, operator OP1 cannot confirm the specified operation being performed by the second communication operation machine in the image captured by the camera of the first communication operation machine. On the other hand, when the first and second communication operation machines are the same selected operation machine, operator OP1 can confirm the specified operation being performed by the second communication operation machine in the image captured by the camera of the first communication operation machine.

[0104] Operator OP1, for example, inputs confirmation via touch handheld terminal 2240 whether a specified operation has been performed on the second communication operation machine. Based on the operator's input, remote operation device 20 determines whether a specified operation has been performed on the second communication operation machine (step S215).

[0105] When the remote operation device 20 determines that a specified operation has been performed in the second communication operation machine (step S215: Y (Yes)), it sends a good message to the operation assistance server 10 indicating that the first communication operation machine and the second communication operation machine are the same selected operation machine (step S216).

[0106] When the remote operation device 20 determines that a specified operation has not been performed in the second communication operation machine (step S215: N (No)), it sends a negative message to the operation assistance server 10 indicating that the first communication operation machine and the second communication operation machine are not the same selected operation machine (step S217).

[0107] The second auxiliary processing element 122 of the job auxiliary server 10 determines whether the release conditions for other specified operations are met based on the good or bad information sent from the remote operation device 20 (step S115).

[0108] When the second auxiliary processing element 122 of the work assistance server 10 determines that the release condition for another specified operation is met (step S115: Y (Yes)), it begins to accept other specified operations (step S116). That is, in this embodiment, the second auxiliary processing element 122 determines whether a release operation has been performed in the remote operation device based on whether good information has been received. In addition, after the second auxiliary processing element 122 begins to accept other specified operations in step S116, it sends the instruction signal corresponding to the other specified operation sent from the remote operation device 20 to the second communication work machine. Thereby, based on the remote operation instruction corresponding to the operation mode of the remote operation mechanism 211 recognized by the remote control device 200 of the remote operation device 20, the operation of the work mechanism of the second communication work machine is controlled.

[0109] When the second auxiliary processing element 122 of the job auxiliary server 10 determines that the release condition for other specified operations is not met (step S115: N (No)), the acceptance of other specified operations is maintained (step S117), and the job auxiliary processing ends.

[0110] Operator OP1 can visually confirm the captured image displayed on the remote image output device 221, indicating that a specified operation is being performed. Thus, operator OP1 can confirm whether the first selected machine 40 or the second selected machine 50 connected to the remote operating device 20 matches the selection information, the first communication machine, and the second communication machine.

[0111] As described above, according to the remote operating system of the present invention, based on the selection information of the cooperating machinery, an image captured by a camera mounted on the first cooperating machinery is obtained via a first communication path, and a command signal corresponding to a specified operation is sent to the second cooperating machinery. Therefore, the operator can confirm, through the captured image, whether the first cooperating machinery has performed an action according to a specified operation performed on the second cooperating machinery, that is, whether the operation performed on the cooperating machinery constituting the remote operation object has been executed on that cooperating machinery. Thus, the operator can confirm the connection status with the machinery constituting the remote operation object.

[0112] In addition, by displaying the actual device identification information in the display image, operator OP1 can confirm whether the actual device identification information of the captured image displayed on the remote image output device 221 matches the actual device identification information of the displayed image visually, thus enabling quick and accurate confirmation.

[0113] In the description of the above embodiments, the work assistance processing performed by the work assistance server 10 was explained. However, the work assistance processing can be performed by the remote operation device 20, or by the first selected work machine 40 or the second selected work machine 50.

[0114] Furthermore, the above embodiments describe the acquisition of images by either the on-site imaging device 412 of the first selected machine 40 or the on-site imaging device 512 of the second selected machine 50. However, the image may also be acquired by an imaging device other than the on-site imaging device 412 of the first selected machine 40 or the on-site imaging device 512 of the second selected machine 50.

[0115] Examples of such camera devices include fixed-point shooting devices positioned to capture the appearance of the first selected machine 40 or the second selected machine 50, and shooting devices mounted on flying objects such as drones.

[0116] When using such a camera device, the tag ST containing the actual machine identification information is preferably placed in a position that can be visually confirmed from the outside of the first selected machine 40 and the second selected machine 50. Such a position could be, for example, the side of the upper rotating body 420 of the first selected machine 40 and the upper rotating body 520 of the second selected machine 50. It should be noted that this information can be communicated to the camera devices when either the first selected machine 40 or the second selected machine 50 connects to the remote operating device 20. In this way, images can be transmitted from the camera devices to the work assistance server 10.

[0117] According to the remote operating system of the present invention, based on the selection information of the cooperating machinery, an image captured by a camera mounted on the first working machinery is obtained via a first communication path, and a command signal corresponding to a specified operation is sent to the second working machinery. Therefore, the operator can determine, through the captured image, whether the first working machinery has performed an action according to a specified operation performed on the second working machinery, that is, whether the operation performed on the cooperating machinery constituting the remote operation object has been executed on that cooperating machinery. Thus, the operator can confirm the connection status with the working machinery constituting the remote operation object.

[0118] In the remote operating system of the present invention, preferably, the second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate, the release condition including the case where a release operation has been performed in the remote operating device.

[0119] In this way, while the operator is confirming the connection status with the machine being remotely operated, it is possible to prevent the machine from operating due to other specified operations. After confirming the connection status, the operator can remotely operate the machine by performing an operation that includes other specified actions.

[0120] In the remote operating system of the present invention, preferably, the second auxiliary processing element performs the following processing: obtaining a determination result of whether the specified operation has been executed; if the specified operation has been executed, allowing the execution of other specified operations to make the cooperative working machine move, and sending the instruction signal corresponding to the other specified operation to the second working machine.

[0121] In this way, other specified operations can be performed on the working machinery that constitutes the object of remote operation, provided that a specified operation has been performed.

[0122] In the remote operating system of the present invention, preferably, the second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to move; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to move. When the specified operation is executed, the release condition includes the following: the second auxiliary processing element obtains a determination result of whether there is a change in the action corresponding to the specified operation in the captured image, and the determination result is that there is a change in the action corresponding to the specified operation.

[0123] In this way, the operator's confirmation work can be reduced because the connection status with the working machinery that constitutes the object of remote operation is automatically determined.

[0124] In the remote operating system of the present invention, preferably, the first auxiliary processing element obtains the position data of the plurality of working machines constituting the remote operation object, the second auxiliary processing element generates a display image of each working machine that is reflected in the plurality of working machines based on the position data of the plurality of working machines, and sends the display image to the remote operation device.

[0125] In this way, the operator can confirm the status of each of the multiple machines that constitute the remotely operated object in the displayed image, and thus identify the relative position of each machine on the work site.

Claims

1. A remote operating system comprising multiple working machines and a remote operating device for remotely operating one of the multiple working machines, namely a cooperating working machine, which constitutes the object of remote operation; the remote operating system having the function of enabling each of the multiple working machines to communicate with the remote operating device. The remote operating system is characterized by having: The first auxiliary processing element outputs the judgment result and the captured image to the output interface of the remote operation device, wherein... The determination result refers to whether the specified operation of determining the collaborative working machine as the remotely operated object among the multiple working machines in the remote operation device has been executed. The captured image is taken by a real-machine shooting device mounted on the first working machine when the selection information is selected in the remote operation device, wherein the first working machine establishes communication with the remote operation device through a first communication path associated with the selection information; and... The second auxiliary processing element identifies a specified operation being executed and sends an instruction signal corresponding to the specified operation to the second working machine. The specified operation refers to a part of a variety of operations that cause the second working machine to establish communication with the remote operation device through a second communication path associated with the selection information, and is a specified operation in which a variation of the action corresponding to the operation is reflected in the captured image.

2. The remote operating system according to claim 1, characterized in that, The second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate. The release condition includes the case where a release operation has been performed on the remote operating device.

3. The remote operating system according to claim 1, characterized in that, The second auxiliary processing element performs the following processing: obtains a determination result of whether the specified operation has been executed; if the specified operation has been executed, allows the execution of other specified operations to make the cooperative working machine move, and sends the instruction signal corresponding to the other specified operation to the second working machine.

4. The remote operating system according to claim 1, characterized in that, The second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate. When the specified operation is performed, the release condition includes the following: the second auxiliary processing element obtains a determination result of whether there is a change in the action corresponding to the specified operation in the captured image, and the determination result is that there is a change in the action corresponding to the specified operation.

5. The remote operating system according to claim 2, characterized in that, The second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate. When the specified operation is performed, the release condition includes the following: the second auxiliary processing element obtains a determination result of whether there is a change in the action corresponding to the specified operation in the captured image, and the determination result is that there is a change in the action corresponding to the specified operation.

6. The remote operating system according to claim 3, characterized in that, The second auxiliary processing element performs the following processing: if the release condition is not met, it retains the option to send instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate; if the release condition is met, it allows the sending of instruction signals to the second working machine corresponding to other specified operations for causing the cooperating working machine to operate. When the specified operation is performed, the release condition includes the following: the second auxiliary processing element obtains a determination result of whether there is a change in the action corresponding to the specified operation in the captured image, and the determination result is that there is a change in the action corresponding to the specified operation.

7. The remote operating system according to claim 1, characterized in that, The first auxiliary processing element acquires the position data of the plurality of operating machines constituting the remotely operated object. The second auxiliary processing element generates a display image of each of the multiple operating machines based on the position data of the multiple operating machines, and sends the display image to the remote operation device.

8. The remote operating system according to claim 2, characterized in that, The first auxiliary processing element acquires the position data of the plurality of operating machines constituting the remotely operated object. The second auxiliary processing element generates a display image of each of the multiple operating machines based on the position data of the multiple operating machines, and sends the display image to the remote operation device.

9. The remote operating system according to claim 3, characterized in that, The first auxiliary processing element acquires the position data of the plurality of operating machines constituting the remotely operated object. The second auxiliary processing element generates a display image of each of the multiple operating machines based on the position data of the multiple operating machines, and sends the display image to the remote operation device.

10. The remote operating system according to claim 4, characterized in that, The first auxiliary processing element acquires the position data of the plurality of operating machines constituting the remotely operated object. The second auxiliary processing element generates a display image of each of the multiple operating machines based on the position data of the multiple operating machines, and sends the display image to the remote operation device.

Images