Remote operation assistance device and remote operation assistance method

Abstract

The object of the present invention is to provide a remote operation assistance device. When using this remote operation assistance device (100), the operator can grasp the operating feel of the working machine (40) before establishing communication between the remote operation device (20) and the working machine (40). To achieve the above object, the remote operation assistance device (100) having this configuration has a first auxiliary processing element (101). The first auxiliary processing element (101) is characterized in that it performs control by displaying a virtual environment image for simulating remote operation of the working machine (40) on a remote output interface (220).

Description

Technical Field

[0001] The remote operation assistance device of the present invention relates to a system that assists operators in remotely operating machinery. Background Technology

[0002] A remote operating system is known to consist of a remote operating device and a working machine. The remote operating device has a main operating unit that directly receives operations from the operator. The working machine is equipped with a secondary operating unit that is connected to the operator in a manner that allows communication with the operator and directly operates the operating lever of the working machine based on the amount of operation received by the operator from the operator (for example, see Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 10-252101 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] By communicating operating signals between the remote operating device and the working machinery, this remote operating system enables the auxiliary operating unit to operate the operating levers of the working machinery based on the operating input received by the operator from the operator.

[0008] However, in the period after communication is established but before the secondary control unit operates the lever state of the work machine based on the input received from the operator, the operator cannot grasp the movement of the work machine caused by the lever driven by the secondary control unit, relative to the input performed by the operator on the primary control unit. As a result, during the period from communication establishment to the operator's mastery of the operating feel, the operator operates the primary control unit overly cautiously, leading to a decrease in workability.

[0009] Therefore, in view of the above background, the object of the present invention is to provide a remote operating device and a remote operating assistance system that enables the operator to grasp the operating feel of the working machinery before establishing communication between the remote operating device and the working machinery.

[0010] Methods for solving problems

[0011] This invention relates to a remote operation auxiliary device for simulating remote operation of engineering machinery.

[0012] The remote operation assistance device is characterized in that it has a first auxiliary processing element.

[0013] The first auxiliary processing element performs the following first auxiliary processing: obtaining first specified operation information and information related to the operation of the remote operating mechanism, i.e., virtual operation instruction information; based on the first specified operation information and the virtual operation instruction information, generating an image that virtually displays the posture of the working machine changing, i.e., a virtual environment image; and sending the virtual environment image to the remote output interface, wherein the first specified operation information is information related to the input of the remote operating mechanism operated to simulate the remote operation of the working machine.

[0014] (Effects)

[0015] According to the remote operation assistance device configured as described, the first auxiliary processing element performs the following processing: acquiring information related to the input of the remote operating mechanism operated to simulate the remote operation of the work machinery, namely, first specified operation information, and information related to the operation of the remote operating mechanism, namely, virtual operation instruction information; based on the first specified operation information and the virtual operation instruction information, generating an image that virtually displays the posture change of the work machinery, namely, a virtual environment image, and sending the virtual environment image to a remote output interface. Thus, before actually remotely operating the work machinery to be remotely operated, the operator can virtually experience (simulate) the operation of the work machinery, thereby mastering its operating characteristics. Furthermore, even after actually remotely operating the work machinery, the operator can still virtually experience (simulate) its operation, thus reviewing its operating characteristics.

[0016] Brief description of the attached diagram

[0017] Figure 1 This is an explanatory diagram of the configuration of a remote operation assistance system as one embodiment of a remote operation assistance device having the configuration of the present invention.

[0018] Figure 2 This is an explanatory diagram of the structure of the remote operation device.

[0019] Figure 3 It is an explanatory diagram of the structure of the operating machinery.

[0020] Figure 4 This is an explanatory diagram regarding the function of a first auxiliary processing element as an embodiment of a remote operation assistance device having the configuration of the present invention.

[0021] Figure 5 This is an explanatory diagram of the function of a second auxiliary processing element as an embodiment of a remote operation assistance device having the configuration of the present invention.

[0022] Figure 6 This is an explanatory diagram showing the display form of a virtual environment image as an embodiment of a remote operation assistance device having the configuration of the present invention.

[0023] Figure 7 This is an explanatory diagram showing the display form of a virtual environment image related to the bending and stretching of a working mechanism in one embodiment of a remote operation assistance device having the configuration of the present invention as another embodiment.

[0024] Figure 8 This is an explanatory diagram showing the display form of a virtual environment image related to the rotation of the upper rotating body having the configuration of the present invention as another embodiment. Detailed Implementation

[0025] (Composition of a remote operation assistance system)

[0026] As having Figure 1 The remote operation assistance system of one embodiment of the remote operation assistance device of the present invention shown comprises a remote operation assistance server 10 and a remote operation device 20 for remotely operating a work machine 40. The remote operation assistance server 10, the remote operation device 20, and the work machine 40 are configured to communicate with each other via a network. The communication network between the remote operation assistance server 10 and the remote operation device 20 and the communication network between the remote operation assistance server 10 and the work machine 40 may be the same communication network or a different communication network.

[0027] In addition, the number of operating machines 40 can be one or more. Furthermore, the types of operating machines 40 can be either the same model or different types of operating machines.

[0028] In addition, remote operation refers to the following concept: the operator does not ride on the work machine 40, but operates the work machine 40 from a position separate from the work machine 40.

[0029] Additionally, "operator" refers to the following concept: a person who operates the remote control device 20 to control the work machinery 40.

[0030] (Composition of a remote operation auxiliary server)

[0031] The remote operation assistance server 10 includes a remote operation assistance device 100, a database 110, and a server wireless communication device 122. The remote operation assistance device 100 includes a first auxiliary processing element 101 and a second auxiliary processing element 102. Each auxiliary processing element is composed of a computing processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and performs computational processing according to the software, as described later, on the object of the data.

[0032] Database 110 stores and saves captured image data, etc. Additionally, database 110 can also store and save information related to the attributes of the working machine 40 and the intended tasks to be performed by the working machine 40. Furthermore, database 110 can store and save information related to the general operating characteristics of the working machine 40. Moreover, database 110 can store and save information related to the general operating characteristics of the working machine 40 and information related to the general tasks to be performed using the working machine 40 in association. Additionally, database 110 can store and save information related to operating characteristics specific to the working machine 40. Furthermore, database 110 can store and save information related to the operating characteristics specific to the working machine 40 and information related to the specific tasks to be performed using the working machine 40 in association.

[0033] (Structure of the remote operation device)

[0034] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting such a processor), and reads necessary data and software from a storage device such as memory, and performs computational processing according to the software based on the data. The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222.

[0035] The remote operating mechanism 211 includes a travel operating device, a slewing operating device, a boom operating device, a stick operating device, and a bucket operating device. Each operating device has an operating lever that accepts rotational operation. The operating lever (travel lever) of the travel operating device is operated to move the lower travel body 450, which is a part of the working mechanism 440 of the work machinery 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever can also be provided. The operating lever (slewing lever) of the slewing operating device is operated to move the hydraulic slewing motor of the slewing mechanism 430, which is a part of the working mechanism 440 of the work machinery 40. The operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machinery 40. The operating lever (stick lever) of the stick operating device is operated to move the stick cylinder 444 of the work machinery 40. The operating lever (bucket lever) of the bucket operating device is operated to activate the bucket cylinder 446 of the working machine 40.

[0036] The operating levers constituting the remote operating mechanism 211 are, for example, Figure 2 As shown, it is arranged around the seat St for the operator to sit on. The seat St can be in the form of a high-backed chair with armrests, a low-backed chair without a headrest, or a chair without a backrest, or any other form for the operator to sit on.

[0037] A pair of left and right travel levers 2110 are arranged side by side in front of the seat St, corresponding to the left and right tracks. One lever can function as multiple levers. For example, it can be used to control... Figure 2 The left-side operating lever 2111, located in front of the left side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a swivel lever when operated in the left / right direction. Similarly, it can be used for... Figure 2 The right-side operating lever 2112, located in front of the right-side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever mode can also be changed arbitrarily according to the operator's instructions.

[0038] For example, such as Figure 2 As shown, the image output device 221 consists of a central image output device 2210, a left image output device 2211, and a right image output device 2212, each with a roughly rectangular screen, respectively positioned in front of, to the left diagonally in front of, and to the right diagonally in front of the seat St. The shape and size of the screens (image display areas) of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be the same or different.

[0039] like Figure 2 As shown, with the screen of the central image output device 2210 and the screen of the left image output device 2211 tilted at an angle θ1 (e.g., 120°≤θ1≤150°), the right edge of the left image output device 2211 is adjacent to the left edge of the central image output device 2210. Figure 2 As shown, the screen of the central image output device 2210 and the screen of the right image output device 2212 are tilted at an angle θ2 (e.g., 120°≤θ2≤150°), and the left edge of the right image output device 2212 is adjacent to the right edge of the central image output device 2210. The tilt angles θ1 and θ2 can be the same or different.

[0040] The images of the central image output device 2210, the left image output device 2211, and the right image output device 2212 can be either parallel or tilted relative to the vertical direction. At least one of the image output devices 2210, 2211, and 2212 can also be composed of multiple image output devices. For example, the central image output device 2210 can be composed of a pair of vertically adjacent image output devices with a generally rectangular image shape. The image output devices 221 (central image output device 2210, left image output device 2211, and right image output device 2212) can also further include a speaker (sound output device).

[0041] (Structure of the operating machinery)

[0042] The work machinery 40 refers to a work vehicle capable of operating at a work site. The work machinery 40 includes at least one type of engineering machinery capable of bulldozing or excavating. Engineering machinery includes at least one type of hydraulic excavator with a bucket and bulldozer with a bulldozing blade. The work machinery 40 includes a machine control device 400, a machine input interface 410, a machine output interface 420, and a working mechanism 440. The machine control device 400 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and executes computational processing according to the software based on the data.

[0043] Construction machinery 40, for example, is a tracked excavator (construction machinery), such as... Figure 3As shown, the tracked excavator includes a tracked lower traveling body 450 and an upper slewing body 460 rotatably mounted on the lower traveling body 450 via a slewing mechanism 430. A cab 424 (driver's cab) is provided on the front left side of the upper slewing body 460. A working mechanism 440 is provided at the front center of the upper slewing body 460.

[0044] The machine input interface 410 includes a machine operation mechanism 411 and a machine imaging device 412. The machine operation mechanism 411 has multiple control levers arranged similarly to the remote operation mechanism 211 around the seat inside the cab 424. A drive mechanism or robot is installed in the cab 424, receives signals corresponding to the operation mode of the remote control levers, and actuates the machine control levers based on the received signals. The machine imaging device 412 is installed, for example, inside the cab 424, and captures images of the environment, including at least a portion of the working mechanism 440, through a front window and a pair of side windows located on the front side of the cab 424, divided by a pair of left and right pillars 4240 (in the case of left and right, including the symbols "L" and "R"). Alternatively, some or all of the front and side windows may be omitted.

[0045] The actual output interface 420 is equipped with actual wireless communication device 422.

[0046] The working mechanism 440 includes the following attachments: a boom 441, which is mounted on the upper slewing body 460 for lifting and lowering; a stick 443, which is rotatably connected to the top end of the boom 441; and a bucket 445, which is rotatably connected to the top end of the stick 443. The working mechanism 440 is equipped with a boom cylinder 442, a stick cylinder 444, and a bucket cylinder 446, all of which are telescopic hydraulic cylinders.

[0047] Boom cylinder 442 is located between boom 441 and upper slewing body 460, allowing it to extend and retract via a supply of working oil, thereby causing boom 441 to rotate in the lifting direction. Stick cylinder 444 is located between stick 443 and boom 441, allowing it to extend and retract via a supply of working oil, thereby causing stick 443 to rotate about a horizontal axis relative to boom 441. Bucket cylinder 446 is located between bucket 445 and stick 443, allowing it to extend and retract via a supply of working oil, thereby causing bucket 445 to rotate about a horizontal axis relative to stick 443.

[0048] (Function)

[0049] use Figure 4 and Figure 5The flowchart shown illustrates the functions of the remote operation assistance system with the above-described configuration. In this flowchart, the "C●" block is a simplified description and refers to the sending and / or receiving of data, as well as a conditional branch that executes branch direction processing based on the sending and / or receiving of such data.

[0050] use Figure 4 The flowchart shown illustrates the first auxiliary process of this embodiment. The first auxiliary process is related to the simulation of remote operation of the work machinery 40.

[0051] In the remote operation device 20, as a prerequisite for the first auxiliary process of this embodiment, the remote control device 200 determines whether the operator has performed an operation (not shown) on the work machinery 40 that was intended to be remotely operated. If the determination result is negative (not shown), the remote control device 200 repeatedly performs this determination process. On the other hand, if the determination result is positive (not shown), the remote control device 200 executes the processing after step (STEP) 211.

[0052] For example, “operation of designated machine 40” refers to the concept of operations including the following: the operator selects one or more machine 40s to be operated remotely from among the multiple machine 40s displayed on the central image output device 2210 by tapping, pressing, pinching, or stroking the remote input interface 210.

[0053] In addition, as another example of "operation of designated work machinery 40", the following operation can be given: the operator selects one or more operations to be performed from multiple operations displayed on the central image output device 2210 by tapping, pressing or pressing on the remote input interface.

[0054] In this case, for example, it can be pre-specified which machine 40 will perform each task. By selecting one or more tasks, the one or more machines 40 that will be remotely operated when one or more tasks are to be performed, as well as the specified time (second specified time) as the start time of each task, can be automatically determined.

[0055] Specifically, it can be pre-defined that a hydraulic excavator located at a work site in Hiroshima City will be used for slope formation operations. By selecting the slope formation operation, the hydraulic excavator that will be remotely operated when the slope formation operation is to be carried out will be automatically determined, as well as a specified time, such as 10:00 AM, as the start time of the slope formation operation.

[0056] In the remote operation device 20, the remote control device 200 determines whether the first specified operation has been performed by the operator. Figure 4 / Step (STEP) 211). In addition, "first designated operation" is a concept that includes the following situation: the operator operates the remote operation mechanism 211 to simulate the remote operation of the work machinery 40.

[0057] In the remote operation device 20, if the determination result in step (STEP) 211 is negative ( Figure 4 / Step (STEP) 211: No), the remote control device 200 ends the process. On the other hand, if the determination result is positive ( Figure 4 Step 211: Yes, the remote control device 200 sends a first operation signal to the remote operation assistance server 10 via the remote wireless communication device 222. Figure 4 / Step (STEP) 212), the first operation signal contains information related to a request (simulation request) for simulating the operation of the working machinery 40.

[0058] The "first operation signal" is a concept that includes the following signals: signals related to simulated operation input by the operator to operate the remote input interface 210 in order to simulate remote operation of the working machinery 40. The "first operation signal" includes an identifier of at least one of the remote operation device 20 and the operator's identifier. In addition, the "first operation signal" includes an identifier of the working machinery 40 selected by the operator.

[0059] Here, the identifier of the remote operating device 20 is a concept that includes the following information: information used to identify which of the multiple existing remote operating devices 20 the remote operating auxiliary server 10 should send and receive data with.

[0060] In addition, the operator identifier is a concept that includes the following information: information used to identify which of the multiple existing operators the remote operation assistance server 10 should communicate with for sending and receiving data.

[0061] In addition, the identifier of the operating machine 40 is a concept that includes the following information: information used to identify which of the multiple existing operating machines 40 the remote operation assistance server 10 should send and receive data with.

[0062] In the remote operation assistance server 10, when the first operation signal is obtained through the server wireless communication device 122 ( Figure 4 / C10), the first auxiliary processing element 101 identifies the virtual environment status ( Figure 4 / Step (STEP) 110).

[0063] The virtual environment status includes at least an identifier for the operating machinery 40 selected by the operator. Additionally, the movement patterns of the operating machinery 40 and environmental changes in the virtual work site are identified as the virtual environment status. Then, the first auxiliary processing element 101 sends virtual environment data to the remote operating device 20 based on this virtual environment status. Figure 4 / Step (STEP) 111).

[0064] The virtual environment data includes virtual environment images or information used to generate virtual environment images. A virtual environment image is an image that virtually displays the working machinery 40 in a virtual work environment, and is an image that reflects changes in the posture of the working machinery 40 based on identified motion patterns and environmental changes. This virtual environment image is, for example, an image of the working machinery 40 represented by computer graphics (hereinafter, appropriately referred to as a simulated image, CG image, or virtual image).

[0065] In the remote operation device 20, when the remote control device 200 receives virtual environment data ( Figure 4 / C20), outputting the virtual environment image to the remote output interface 220 (image output device 221). Figure 4 / Step (STEP) 214).

[0066] Thus, for example, such as Figure 6 As shown, the image output device 221 outputs virtual environment images that can be captured by the real-machine shooting device 412 mounted on the working machine 40 that cooperates with the remote operation device 20.

[0067] The remote control device 200 determines whether there is a stop operation for virtual environment image output executed through the remote input interface 210. Figure 4 / Step (STEP) 215). The stopping operation for the virtual environment image output is, for example, the following: tapping, pressing, pinching, or stroking operations performed by the operator on the remote input interface 210 to end the simulation of the operation of the working machinery 40. If the determination result is positive ( Figure 4 Step 215: Yes, the remote control device 200 sends a stop request for virtual environment image output to the remote operation assistance server 10 via the remote wireless communication device 222. Figure 4 / Step (STEP) 216).

[0068] On the other hand, if the judgment result is negative ( Figure 4Step 215: No, the remote control device 200 identifies the operating mode of the remote operating mechanism 211. Figure 4 / Step (STEP) 218), and, through the remote wireless communication device 222, send a virtual operation command corresponding to the operation mode to the remote operation assistance server 10. Figure 4 / Step (STEP) 219).

[0069] In the remote operation assistance server 10, upon receiving a virtual operation command ( Figure 4 / C10), the first auxiliary processing element 101 identifies the virtual environment status corresponding to the virtual operation instruction ( Figure 4 / Step (STEP) 110). For example, when the operation mode of the remote operating mechanism 211 is such that the working mechanism 440 (boom 441, stick 443, bucket 445) of the working machine 40 bends and extends, the condition in which the working mechanism 440 moves in this manner is identified as the virtual environment condition in the virtual environment.

[0070] The first auxiliary processing element 101 sends a virtual environment image to the remote operation device 20 based on the virtual environment status. Figure 4 / Step (STEP) 111). Then, determine whether a request to stop the output of the virtual environment image has been received ( Figure 4 / Step (STEP) 112). If the determination result is positive ( Figure 4 Step 112: If the result is "YES", the simulation of this implementation ends. If the result is "No", then... Figure 4 Step 112: No, determine whether it is the start time of the operation the operator wants to perform, i.e., the specified time (second specified time). Figure 4 / Step (STEP) 113). If the determination result is positive ( Figure 4 Step 113: If the result is "YES", the simulation of this implementation ends. If the result is "No", then... Figure 4 Step 113: No, repeatedly process the virtual operation command after receiving it. Figure 4 / C10→STEP110→STEP111→STEP112→STEP113).

[0071] In the remote operation device 20, when virtual environment data is received via the remote wireless communication device 222 constituting the remote output interface 220 ( Figure 4 / C20), outputs a virtual environment image on the image output device 221 that constitutes the remote output interface 220. Figure 4 / Step (STEP) 214). Thus, for example, as Figure 6 As shown, corresponding to the operator's operation mode on the remote operating mechanism 211, the bending and elongation of the virtual image 441V of the boom 441, the virtual image 443V of the stick 443, and the virtual image 445V of the bucket 445 are output and displayed on the remote output interface 220 (image output device 221).

[0072] Thus, the operator can confirm the operating feel of the working mechanism 440 (boom 441, stick 443, bucket 445), that is, the relationship between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 (boom 441, stick 443, bucket 445) of the working machine 40.

[0073] In addition, such as Figure 6 As shown, when the operator operates the remote operating mechanism 211 to rotate the upper rotating body 460 of the work machine 40 to the right with reference to the operator's position, a virtual image of the background of the work site can also be output on the remote output interface 220 (image output device 221) in a direction opposite to the rotation direction (i.e., to the left with reference to the operator's position).

[0074] Therefore, the operator can confirm the operational feel of the work machinery 40 intended for remote operation, that is, the correlation between the operating mode of the operating mechanism and the working mode of the work machinery. In this case, in conjunction with the virtual image 460V of the upper rotating body 460... Figure 7 The rotation of the remote output interface 220 (image output device 221) displays the background of the work site (excavators present at the work site that are not being remotely operated, demolition machines present at the work site, cranes present at the work site, dump trucks, bulldozers, wheel loaders, sand dunes, rocks, trees, forests, buildings, and slopes, etc.). Therefore, the operator can especially confirm the operating feel of the working mechanism 440 (upper rotating body 460), that is, the correlation between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 (upper rotating body 460) of the work machinery 40.

[0075] like Figure 6As shown, the background of the virtual environment image may also include the appearance of the work site where the operator intends to perform the work. This appearance of the work site can be represented by a real-life photograph or by a computer-generated image (CG image). When the virtual environment image includes a background, the first auxiliary processing element 101 retrieves information related to the appearance of the work site from the database 110, which is associated with the attributes of the operating machinery 40 and stored in the database 110. Based on this information, it generates the background of the virtual environment image and then performs processing to display this background on the remote output interface 220. Thus, the virtual environment image of the work site where the operator intends to perform the work is output to the remote output interface 220.

[0076] For example, such as Figure 6 As shown, a virtual environment image can also be created based on images captured by a real-machine shooting device 412 installed inside the cab 424. This image is displayed in a virtual image of the front window (4240RV, 4240LV) divided by virtual images 4240V (4240RV, 4240LV) of a pair of left and right pillars 4240 located at the front of the cab 424 (in the case of distinguishing left and right, including the symbols "L" and "R") and virtual images of a pair of left and right side windows. The virtual images also display at least a portion of the virtual images of the working mechanism 440V (441V, 443V, 445V).

[0077] like Figure 7 As shown, a virtual environment image simulating the posture of the working machinery 40 in a side view can also be displayed. For example, when the operator operates the remote control mechanism 211 in a manner that causes the working mechanism 440 (boom 441, stick 443, bucket 445) of the working machinery 40 to bend and extend, Figure 7 The virtual environment image shown is output to the remote output interface 220. It can also be... Figure 6 The corner of the virtual environment image shown is displayed when viewed from inside the driver's cab 424. Figure 7 The image shown is a virtual environment image viewed from the side.

[0078] Therefore, the operator can confirm that Figure 6 The working mechanism 440 (boom 441, stick 443, bucket 445) of the operating machine 40, which is difficult to grasp in the virtual environment image when viewed from inside the cab 424, is accompanied by the movement pattern in the front and rear directions.

[0079] like Figure 8As shown, a virtual environment image simulating the posture of the work machinery 40 when viewed from above can also be displayed. For example, when the operator operates the remote control mechanism 211 to rotate the upper rotating body 460 of the work machinery 40 to the left relative to the operator's position, a virtual image 460V of the upper rotating body 460 corresponding to this operation, rotating to the left relative to the operator's position, is displayed on the remote output interface 220 (image output device 221). It is also possible to... Figure 6 The corner of the virtual environment image shown is displayed when viewed from inside the driver's cab 424. Figure 8 The image shown is a virtual environment viewed from above.

[0080] Therefore, the operator can confirm that Figure 6 The working mechanism 440 (upper rotating body 460) of the operating machine 40, which is difficult to grasp on a virtual environment image viewed from inside the cab 424, is shown with its rotation amount (rotation angle) movement.

[0081] use Figure 5 The flowchart shown illustrates the second auxiliary process of this embodiment. This second auxiliary process is used for the actual remote operation of the work machinery 40.

[0082] In the remote operation device 20, the remote control device 200 determines whether a second specified operation has been performed through the remote input interface 210. Figure 5 / Step (STEP) 221). The "second designated operation" is, for example, a tapping or pressing operation performed on the remote input interface 210 to designate the work machine 40 that the operator actually intends to remotely operate. Other examples of the "second designated operation" may include tapping or pressing operations performed on the remote input interface 210 to designate the work that the operator intends to perform. The "second designated operation" is the operation used to initiate remote operation of the work machine 40 based on the "first designated operation".

[0083] In the case where the judgment result is negative ( Figure 5 Step 221: No, the remote control device 200 repeatedly executes the determination of whether or not the first specified operation has occurred. Figure 4 / Step (STEP) 211) and subsequent processing. On the other hand, if the determination result is positive ( Figure 5 Step 221: Yes, the remote control device 200 sends an environment confirmation request to the remote operation assistance server 10 via the remote wireless communication device 222. Figure 5 / Step (STEP) 222).

[0084] In the remote operation assistance server 10, upon receiving an environmental confirmation request, the second auxiliary processing element 102 sends the environmental confirmation request to the corresponding operating machinery 40. Figure 5 / C11).

[0085] In the operating machinery 40, when an environmental confirmation request is received via the actual machine wireless communication device 422 ( Figure 5 / C40), the actual control device 400 acquires the captured image through the actual shooting device 412 ( Figure 5 / Step (STEP) 402). Additionally, the machine control device 400 sends the captured image data representing the captured image to the remote operation assistance server 10 via the machine wireless communication device 422. Figure 5 / Step (STEP) 404).

[0086] In the remote operation assistance server 10, upon receiving the captured image data ( Figure 5 / C12), the second auxiliary processing element 102 sends the captured image data to the remote operation device 20 (transmission). Figure 5 / Step (STEP) 114). The second auxiliary processing element 102 may also send environmental image data representing a virtual environmental image generated based on the captured image to the remote operation device 20 instead of sending the captured image data. At this time, the second auxiliary processing element 102 may also send the following instruction to the remote operation device 20: the instruction indicates the segmented display mode of the captured image corresponding to the captured image data on the image output device 221 (central image output device 2210, left image output device 2211, and right image output device 2212).

[0087] In the remote operation device 20, the remote control device 200 receives the captured image data via the remote wireless communication device 222. Figure 5 / C21), controls the segmented display format of the captured image corresponding to the captured image data on the three image output devices 221 (central image output device 2210, left image output device 2211, and right image output device 2212). Figure 5 / Step (STEP) 223).

[0088] In addition, as another example of controlling the output of captured images on the remote control device 200, the following example of controlling the display of captured images can be given: the captured images are not displayed separately on the central image output device 2210, the left image output device 2211, and the right image output device 2212, but are displayed only on one image output device 221 (e.g., the central image output device 2210).

[0089] In the remote operation device 20, the remote control device 200 identifies the operating mode of the remote operation mechanism 211. Figure 5 / Step (STEP) 224), and the remote control device 200 sends a remote operation command corresponding to the operation mode to the remote operation assistance server 10 via the remote wireless communication device 222. Figure 5 / Step (STEP) 225).

[0090] In the remote operation assistance server 10, upon receiving a remote operation command, the second auxiliary processing element 102 sends the remote operation command to the operating machine 40. Figure 5 / C13).

[0091] In the operating machinery 40, the actual machine control device 400 receives an operation command through the actual machine wireless communication device 422. Figure 5 / C42), controls the actions of the working mechanism 440, etc. Figure 5 / Step (STEP) 406). For example, performing the operation of scooping up soil in front of the working machine 40 by the bucket 445 and dumping the soil from the bucket 445 after the upper rotating body 460 is rotated.

[0092] (Effect)

[0093] Based on the remote operation assistance system having this configuration, the remote operation assistance server 10 constituting the remote operation assistance system, and the remote operation device 20, a virtual environment image for simulating the remote operation of the working machinery 40, generated by the first auxiliary processing element 101 of the remote operation assistance device 100, is output and displayed on the remote output interface 220 (image output device 221). The first auxiliary processing element 101 generates virtual environment data based on the working machinery identifier of the working machinery 40 selected by the operator, and outputs the virtual environment image to the remote output interface 220 according to the virtual environment data. In this simulation, the operating mode of the remote operation mechanism is identified and virtual operation commands are generated, and the virtual environment image is output based on the virtual operation commands.

[0094] In addition, the second auxiliary processing element 102 performs processing for remote operation of the machine tool 40. Therefore, before actually remotely operating the machine tool 40, the operator can grasp the operational feel of the machine tool 40 to be remotely operated, that is, the correlation between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 in the machine tool 40, through the virtual environment image displayed by the remote output interface 220 (image output device 221).

[0095] In addition, the first auxiliary processing element performs the first auxiliary processing during a predetermined period from the first specified time to the second specified time. The first specified time refers to the time when the work object is specified in STEP 211. The work object is the work machine 40 and the work performed using the work machine 40, or an object that includes at least one of the work and the work machine 40. The second specified time refers to the time when the work machine 40 becomes remotely operable via a remote operation device 20 with a remote input interface 210 and via a network (the actual time when the work machine 40 can be remotely operated).

[0096] Therefore, the operator can simulate the remote operation of the machine 40 by utilizing the time from the moment the work object is selected to the moment when the machine 40 can be remotely operated via the remote operation device 20 (until the machine 40 can be remotely operated). That is, the operator can effectively utilize the waiting time from the first specified time to the second specified time. Therefore, the workability of the work performed using the machine 40 can be improved.

[0097] Furthermore, since database 110 stores and saves information related to the general operating characteristics of the machine tool 40, even if information related to the specific operating characteristics of the machine tool 40 is not available, the operator can simulate remote operation of the machine tool 40 based on the information related to the general operating characteristics of the machine tool 40. Additionally, since database 110 stores and saves information related to the specific operating characteristics of the machine tool 40, the operator can simulate remote operation of the machine tool 40 based on the information related to the specific operating characteristics of the machine tool 40. Therefore, compared to simulations based on information related to the general operating characteristics of the machine tool 40, the operator can simulate based on more specific information.

[0098] (Other embodiments of the remote operation assistance device having the configuration of the present invention)

[0099] In the above embodiments, an example of the first auxiliary processing being performed by the remote operating device 20 and the remote operating assistance server 10 has been described, but the implementation is not limited to such an example. For example, the first auxiliary processing element 101 may also perform the following processing: obtaining information related to the quality of the communication environment between the operating machine 40, the remote operating assistance server 10, and the remote operating device 20, i.e., communication information, and displaying a virtual environment image on the remote output interface 220 based on the communication information.

[0100] "Communication information" refers to a concept containing information related to the strength of radio waves between the operating machinery 40, the remote operation assistance server 10, and the remote operation device 20, such as communication speed. The first auxiliary processing element 101 corresponds to the communication information, causing the operating machinery 40 in the virtual environment to move in a delayed manner relative to input from the operator to the remote operation mechanism 211. For example, when the operator operates the remote operation mechanism 211 to bend and extend the working mechanism 440 (boom 441, stick 443, bucket 445) of the operating machinery 40, the working mechanism 440 moves in a delayed manner relative to that operation in the virtual environment image.

[0101] The remote operation assistance device 100 associated with the remote operation assistance server 10 having this configuration is also applicable in this case. Thus, before the operator actually performs remote operation on the work machinery 40, he / she can grasp the communication status through the virtual environment image output to the remote output interface 220 (image output device 221) by operating relative to the remote operation mechanism 211.

[0102] In the above embodiment, an example of a hydraulic excavator with a bucket 445 was described as the operating machine 40, but this example is not limited. Other examples of the operating machine 40 include a demolition machine equipped with a breaking device instead of a bucket 445, a civil engineering machine equipped with a hook instead of a bucket 445, and a crane. Furthermore, multiple operating machines 40 may exist that the operator intends to operate remotely. In this case, the operator can simulate the operational feel, that is, the correlation between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 of the operating machine 40, for remote operation of different operating machines 40, thereby adapting to the correlation between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 in the operating machine 40.

[0103] For example, when an operator wants to remotely operate a small excavator immediately after operating a large excavator, the feeling of operating the small excavator, that is, the relationship between the operating mode of the operating mechanism and the movement mode of the working mechanism of the machine, is very different.

[0104] To address this, before remotely operating the mini excavator, the operator can simulate remote operation of the mini excavator to grasp the corresponding input parameters of the remote input interface 210, which relate to the operating feel of the mini excavator, i.e., the operating mode of the control mechanism and the movement mode of the working mechanism of the work machine. Therefore, when actually remotely operating the mini excavator, the operator can immediately adapt to the operating feel of the mini excavator, i.e., the relationship between the operating mode of the control mechanism and the movement mode of the working mechanism of the work machine.

[0105] The remote operation assistance device 100 associated with the remote operation assistance server 10 having this configuration is also applicable to this situation.

[0106] In the above embodiments, an example of the remote operation assistance device 100 being installed on the remote operation assistance server 10 has been described, but the implementation is not limited to this example. For example, the remote control device 200 and the physical control device 400, or one of these two devices, may perform all or part of the functions of the remote operation assistance device 100.

[0107] The remote operation assistance device 100 associated with the remote operation assistance server 10 having this configuration can also be applied in this case.

[0108] In the above embodiment, an example is shown where the working mechanism 440 includes a boom 441, stick 443, bucket 445, and upper slewing body 460, but it is not limited to such an example. Other examples of the working mechanism 440 include those including a lower traveling body 450. In this case, as another example of simulation in the above embodiment, the virtual image 450V of the lower traveling body 450 can be driven to move the virtual image 40V of the working machine 40. Thus, the operator can confirm the operational feel of the working mechanism 440 (lower traveling body 450), i.e., the correlation between the operating mode of the remote control mechanism 211 and the movement mode of the working mechanism 440 (lower traveling body 450) of the working machine 40.

[0109] The remote operation assistance device 100 associated with the remote operation assistance server 10 having this configuration can also be applied to this situation.

[0110] In the above embodiment, an example is shown where the second auxiliary processing performed by the second auxiliary processing element 102 begins at STEP 221, but this is not a limitation. Other examples of the second auxiliary processing performed by the second auxiliary processing element 102 include the following: before STEP 221, the second auxiliary processing element 102 determines whether the first auxiliary processing has been performed in the first auxiliary processing element 101. If the determination is affirmative, the second auxiliary processing element 102 performs the STEP 221 processing; if the determination is negative, the second auxiliary processing element 102 repeatedly performs the processing prior to the determination. Thus, the operator can actually remotely operate the machine 40 after completing a simulation of its remote operation. That is, the operator can actually remotely operate the machine 40 after confirming the correlation between the operating mode of the remote operating mechanism 211 and the movement mode of the working mechanism 440 of the machine 40. Therefore, compared to performing actual remote operation without simulation, the operator does not need to excessively increase their attention to begin work, thereby improving work efficiency.

[0111] The remote operation assistance device 100 associated with the remote operation assistance server 10 having this configuration can also be applied to this situation.

[0112] In the above embodiments, for example, a portable terminal, which is a tablet PC and communicates with the remote operating device 20 to form part of the remote operating device 20, may also constitute part of the remote input interface 210 and the remote output interface 220. In this case, the operator can use the portable terminal to perform inputs such as a first specified operation, a second specified operation, and a stop operation.

[0113] In the above embodiment, in the remote operation assistance server 10, as a prerequisite for the first assistance processing of this embodiment, the first assistance processing element 101 may also determine whether the work machine 40 constituting the simulation object is actually being remotely operated or whether it is planned to be used. If the work machine 40 is actually being remotely operated (not shown) or is about to be used, the first assistance processing element 101 ends the processing before STEP 110. If the work machine 40 is not actually being remotely operated (not shown) or may be about to be used, the first assistance processing element 101 performs the processing after STEP 110 (not shown).

[0114] Furthermore, in the remote operation assistance device having this configuration, it is preferable that the remote operation assistance device has a second auxiliary processing element, which performs the following second auxiliary processing: obtaining second specified operation information, and based on the second specified operation information, being able to remotely operate the work machinery, wherein the second specified operation information is information related to the input of the remote operation mechanism operated for the purpose of remotely operating the work machinery.

[0115] (Effects)

[0116] According to this configuration, the remote operation assistance device having this configuration includes a second auxiliary processing element. The second auxiliary processing element performs the following second auxiliary processing: acquiring second specified operation information, and based on this second specified operation information, enabling remote operation of the work machinery, wherein the second specified operation information is information related to the input of the remote operation mechanism operated for the purpose of remotely operating the work machinery. Thus, the operator can actually remotely operate the work machinery.

[0117] Furthermore, in a remote operation assistance device having this configuration, it is preferable that the first assistance element performs the first assistance process before the second assistance process performed by the second assistance processing element.

[0118] (Effects)

[0119] According to this configuration, the first auxiliary processing element performs the first auxiliary processing before the second auxiliary processing element performs the second auxiliary processing. Therefore, the operator can simulate the remote operation of the machine before actually performing it. That is, by performing the simulation, the operator can learn about the operating characteristics of the machine in advance. Thus, the operator can understand the operating characteristics of the machine before remotely operating it. Therefore, the operator does not need to be overly cautious when remotely operating the machine, thereby improving the operability of the work performed using the machine.

[0120] Furthermore, in a remote operation assistance device having this configuration, it is preferable that the first assistance processing element performs the first assistance processing during a predetermined period from a first specified time to a second specified time, wherein the first specified time refers to the time when a work object is specified, the work object being the work machine and the work performed using the work machine, or an object that includes at least one of the work and the work machine; the second specified time refers to the time when the work machine establishes a connection with the remote operation device having the remote operation mechanism via a network.

[0121] (Effects)

[0122] According to this configuration, the first auxiliary process is performed during a predetermined period from a first specified time to a second specified time. The first specified time refers to the time when a work object is specified, which is the work machine and the work performed using the work machine, or an object comprising at least one of the work and the work machine. The second specified time refers to the time when the work machine establishes a connection with a remote operating device having the remote operating mechanism via a network. Therefore, the operator can simulate remote operation of the work machine by selecting at least one of the work machine they plan to remotely operate or the work they plan to perform, until the remote operating device connects to the work machine. That is, the simulation can be performed using the waiting time from the time the work object is selected until the remote operating device connects to the work machine, thus improving the workability of the work performed using the work machine.

[0123] Furthermore, in the remote operation assistance device configured in this way, it is preferable that the first auxiliary processing element performs the following first auxiliary processing: obtaining the characteristic information from a database storing characteristic information, and generating an image, i.e., a virtual environment image, that virtually displays the posture change of the working machine based on the characteristic information and operation information which is related to the operation mode of the remote operation mechanism, and outputting the virtual environment image to the remote output interface, wherein the characteristic information refers to information related to the operation characteristics of the working machine designated as the remote operation object.

[0124] (Effects)

[0125] According to this configuration, the first auxiliary processing element performs the following first auxiliary processing: retrieving the characteristic information from a database storing characteristic information, and generating an image, i.e., a virtual environment image, based on the characteristic information and operation information related to the operation mode of the remote operating mechanism, virtually displaying the change in the posture of the working machine, and outputting the virtual environment image to the remote output interface, wherein the aforementioned characteristic information is information related to the operation characteristics of the working machine designated as the remote operating object. Therefore, since the virtual environment image is displayed based on the operation characteristics of the selected working machine, the operator can grasp the actions (operation characteristics) corresponding to differences in the size, reaction, etc., of the working machine.

[0126] Furthermore, in the remote operation assistance device having this configuration, it is preferable that the first assistance processing element performs the following first assistance processing: obtaining information related to the quality of the communication environment between the remote operation device and the working machinery, i.e., communication information, generating a virtual environment image based on the communication information, and outputting the virtual environment image to the remote output interface.

[0127] (Effects)

[0128] According to this structure, the first auxiliary processing element performs the following first auxiliary processing: acquiring information related to the quality of the communication environment between the remote operating device and the operating machinery, i.e., communication information; generating a virtual environment image based on the communication information; and outputting the virtual environment image to the remote output interface. Thus, a virtual environment image corresponding to the condition of the communication line is displayed on the remote output interface, allowing the operator to grasp the operating characteristics corresponding to the condition of the communication line.

[0129] Furthermore, in the remote operation assistance device having this configuration, it is preferable that the first assistance processing element performs the following first assistance processing: acquiring operation information as information related to the operation amount of the remote operation mechanism, generating an image, i.e. a virtual environment image, that virtually displays the posture of the working mechanism of the working machine changing based on the operation information, and outputting the virtual environment image to the remote output interface.

[0130] (Effects)

[0131] According to this structure, the first auxiliary processing element performs the following first auxiliary processing: acquiring operation information as information related to the operation amount of the remote operating mechanism; generating an image, i.e., a virtual environment image, that virtually displays the change in posture of the working mechanism of the operating machine based on the operation information; and outputting the virtual environment image to the remote output interface. Thus, the operator can grasp the operational characteristics related to the movements of the working mechanism of the operating machine.

[0132] Furthermore, in the remote operation assistance device of this structure, it is preferable that the first auxiliary processing element performs the following first auxiliary processing: acquiring operation information as information related to the operation amount of the remote operation mechanism, generating an image, i.e. a virtual environment image, that virtually displays the appearance of the upper rotating body of the working machine rotating, based on the operation information, and outputting the virtual environment image to the remote output interface.

[0133] (Effects)

[0134] According to this structure, the first auxiliary processing element performs the following first auxiliary processing: acquiring operation information as information related to the operation amount of the remote operating mechanism; generating an image, i.e., a virtual environment image, that virtually displays the rotation of the upper slewing body of the working machine based on the operation information; and outputting the virtual environment image to the remote output interface. Thus, when the operator performs the operation of rotating the upper slewing body, the operator can visually confirm the virtual environment image showing the rotation of the upper slewing body. Furthermore, the virtual environment image also includes the situation where the background of the work site moves and changes in conjunction with the rotation of the upper slewing body (the background flows), therefore, the operator can grasp the rotation characteristics of the working machine.

[0135] Symbol Explanation

[0136] 10… Remote operation auxiliary server, 20… Remote operation device, 40… Working machinery, 100… Remote operation auxiliary device, 101… First auxiliary processing element, 102… Second auxiliary processing element, 211… Remote operation mechanism, 220… Remote output interface, 440… Working mechanism, 460… Upper rotating body.

Claims

1. A remote operation assistance system for simulating remote operation of machinery. The remote operation assistance system is characterized by having, The remote output interface is capable of outputting images. The first auxiliary processing element generates an image, i.e., a virtual environment image, based on information related to the operation of the remote operating mechanism, namely, virtual operating command information, which virtually displays the change in the posture of the operating machinery. The first auxiliary processing element performs the following first auxiliary processing: obtaining information related to the quality of the communication environment between the remote operating device including the remote operating mechanism and the working machine, i.e., communication information; based on the communication information, causing the working machine in the virtual environment to operate with a delay relative to the input of the remote operating mechanism; generating the virtual environment image; and outputting the virtual environment image to the remote output interface.

2. The remote operation assistance system according to claim 1, characterized in that, The first auxiliary processing element generates a virtual environment image based on first specified operation information, which is information related to the input of the remote operating mechanism operated to simulate the remote operation of the work machinery.

3. The remote operation assistance system according to claim 1, characterized in that, The system includes a second auxiliary processing element that performs the following second auxiliary processing: obtaining second specified operation information and, based on the second specified operation information, being able to remotely operate the work machinery, wherein the second specified operation information is information related to the operation of the remote operating mechanism being operated for the purpose of remotely operating the work machinery.

4. The remote operation assistance system according to claim 3, characterized in that, Before the second auxiliary processing is performed by the second auxiliary processing element, the first auxiliary processing element performs the first auxiliary processing.

5. The remote operation assistance system according to claim 1, characterized in that, The first auxiliary processing element performs the first auxiliary processing during a predetermined period from a first specified time to a second specified time, wherein the first specified time refers to the time when a work object is specified, the work object being the work machine and the work performed using the work machine, or an object containing at least one of the work and the work machine; the second specified time refers to the time when the work machine establishes a connection with a remote operating device having the remote operating mechanism via a network.

6. The remote operation assistance system according to claim 1, characterized in that, The first auxiliary processing element performs the following first auxiliary processing: retrieves the characteristic information from a database storing characteristic information, and generates an image that virtually displays the posture of the working machine changing, i.e., the virtual environment image, based on the characteristic information and the operation information which is related to the operation mode of the remote operating mechanism, and outputs the virtual environment image to the remote output interface, wherein the characteristic information refers to information related to the operation characteristics of the working machine designated as the remote operating object.

7. The remote operation assistance system according to claim 1, characterized in that, The first auxiliary processing element performs the following first auxiliary processing: acquiring operation information as information related to the operation amount of the remote operating mechanism, generating an image that virtually displays the posture of the working mechanism of the operating machine changing, i.e., the virtual environment image, based on the operation information related to the operation mode of the remote operating mechanism, and outputting the virtual environment image to the remote output interface.

8. The remote operation assistance system according to claim 1, characterized in that, The first auxiliary processing element performs the following first auxiliary processing: acquiring operation information as information related to the operation amount of the remote operating mechanism, and generating an image that virtually displays the appearance of the upper rotating body of the working machine rotating, i.e., the virtual environment image, based on the operation information, and outputting it to the remote output interface.

Images