Remote operation assistance server, remote operation assistance system, and remote operation assistance method

Abstract

This invention provides a remote operation assistance server, etc. Using this technology, when switching the remotely operated object from one machine to another via a remote operation device, it is possible to enable the operator to easily respond to environmental changes of the remotely operated object. A first working environment image based on first captured image data is output on the remote output interface (220), wherein the first captured image data is obtained by the actual machine capture device (412) of the current remotely operated object of the remote operation device (20), namely the first machine. In addition, a second working environment image based on second captured image data is output on the remote output interface (220), wherein the second captured image data is obtained by the actual machine capture device of the current remotely operated object of the remote operation device (20), namely the second machine.

Description

Technical Field

[0001] This invention relates to a remote operation assistance server for assisting operators in remotely operating multiple machines through a remote operation device. Background Technology

[0002] One proposed technology involves displaying images captured by a camera (actual shooting device) mounted on construction machinery or other working machinery onto a monitor constituting a remote operation device, thereby assisting in the remote operation of the working machinery (for example, see Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-176401 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, when switching remotely operated objects from one machine to another via a remote control device, operators sometimes find it difficult to cope with the change in environment from one machine to another.

[0008] Therefore, the object of the present invention is to provide a method that enables the operator to easily respond to environmental changes of the remotely operated object when switching the remotely operated object from one machine to another via a remote operating device.

[0009] Methods for solving problems

[0010] The remote operation assistance server of the present invention is used to assist operators in remotely operating multiple machines through remote operation devices.

[0011] The remote operation assistance server has the following features:

[0012] A first auxiliary processing element identifies the current remotely operated object of the remote operating device, namely the first working machine, acquires first captured image data representing the surrounding environment of the first working machine based on communication with the first working machine, and outputs a first working environment image corresponding to the first captured image data to the output interface of the remote operating device based on communication with the remote operating device. The first captured image data is acquired by a real-machine capturing device mounted on the first working machine or a real-machine capturing device existing around the first working machine.

[0013] The second auxiliary processing element identifies the next remotely operated object of the first operating machine of the remote operating device, namely the second operating machine, acquires second captured image data representing the surrounding environment of the second operating machine based on communication with the second operating machine, and outputs the second operating environment image corresponding to the second captured image data and the first operating environment image together to the output interface of the remote operating device based on communication with the remote operating device. The second captured image data is acquired by a real machine shooting device mounted on the second operating machine or a real machine shooting device existing around the second operating machine.

[0014] According to the remote operation assistance server having this configuration, a first working environment image based on first captured image data can be output to the output interface of the remote operation device. The first captured image data is obtained by a real-machine capturing device mounted on the current remotely operated object (i.e., the first working machine) or by a real-machine capturing device present in the vicinity of the first working machine. Furthermore, a second working environment image based on second captured image data can be output to the output interface of the remote operation device. The second captured image data is obtained by a real-machine capturing device mounted on the next remotely operated object (i.e., the second working machine) or by a real-machine capturing device present in the vicinity of the second working machine.

[0015] Therefore, it is possible for an operator who is remotely operating the first machine via a remote control device to pre-identify the surrounding environment of the second machine, which will be the next machine to be remotely operated. Consequently, when switching the remotely operated object from one machine to another via the remote control device, the operator can easily respond to changes in the environment of that machine.

[0016] Brief description of the attached diagram

[0017] Figure 1 This is an explanatory diagram regarding the configuration of a remote operation assistance system as one embodiment of the present invention.

[0018] Figure 2 This is an explanatory diagram of the configuration of a remote operation device.

[0019] Figure 3 It is an explanatory diagram of the structure of the operating machinery.

[0020] Figure 4 This is an explanatory diagram of the first function of the remote operation assistance system.

[0021] Figure 5 This is an explanatory diagram of the second function of the remote operation assistance system.

[0022] Figure 6A This is the first example diagram illustrating the output methods of the first and second operating environment images.

[0023] Figure 6B This is a second example diagram illustrating the output methods of the first and second operating environment images.

[0024] Figure 6C This is the third example diagram illustrating the output methods of the first and second working environment images.

[0025] Figure 7A This is an explanatory diagram regarding the output method of the first working environment image.

[0026] Figure 7B This is an explanatory diagram regarding the output method of the second working environment image.

[0027] Figure 8 This is an explanatory diagram regarding the output methods of the first and second working environment images.

[0028] Figure 9 This is an explanatory diagram of the time-varying patterns of the first and second work environment images. Detailed Implementation

[0029] (Structure of a remote operation assistance system)

[0030] Figure 1 The remote operation assistance system shown as an embodiment of the present invention comprises a remote operation assistance server 10 and a remote operation device 20 (client) for remotely operating a work machine 40. The remote operation assistance server 10, the remote operation device 20, and the work machine 40 are configured to communicate with each other via a network. The communication network between the remote operation assistance server 10 and the remote operation device 20 and the communication network between the remote operation assistance server 10 and the work machine 40 can be the same or different communication networks.

[0031] (Composition of a remote operation auxiliary server)

[0032] The remote operation assistance server 10 includes a database 102, a first auxiliary processing element 121, and a second auxiliary processing element 122. The database 102 stores and holds captured image data, etc. The database 102 may also be configured as a database server independent of the remote operation assistance server 10. Each auxiliary processing element is composed of a computing processing device (a single-core processor or a multi-core processor, or a processor core constituting that processor), and reads necessary data and software from storage devices such as memory, and performs computational processing according to the software, described later, on the object of that data.

[0033] (Composition of the remote operation device)

[0034] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote operation device 20 functions as at least one of a "first client" and a "second client." The remote control device 200 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting such a processor), which reads necessary data and software from storage devices such as memory, and performs computational processing according to the software on the data. The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222.

[0035] The remote control mechanism 211 includes a travel control device, a slewing control device, a boom control device, a stick control device, and a bucket control device. Each control device has a lever for receiving rotational operations. The travel control device's lever (travel lever) is operated to move the lower traveling body 410 of the work machinery 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever can be provided. The slewing control device's lever (slewing lever) is operated to operate the hydraulic slewing motor constituting the slewing mechanism 430 of the work machinery 40. The boom control device's lever (boom lever) is operated to operate the boom cylinder 442 of the work machinery 40. The stick control device's lever (arm lever) is operated to operate the stick cylinder 444 of the work machinery 40. The bucket control device's lever (bucket stick) is operated to operate the bucket cylinder 446 of the work machinery 40.

[0036] The operating levers constituting the remote operating mechanism 211 are, for example, Figure 2 As shown, it is arranged around the seat St for the operator to sit on. The seat St can be in the form of a high-backed chair with armrests, or it can be in the form of a low-backed chair without a headrest, or a chair without a backrest, etc., any form of seating for the operator.

[0037] A pair of left and right travel levers 2110 are arranged side by side in front of the seat St, corresponding to the left and right tracks. One lever can function as multiple levers. For example, it can be used to control... Figure 2 The left-side operating lever 2111, located in front of the left side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a swivel lever when operated in the left / right direction. Similarly, it can be used for... Figure 2The right-side operating lever 2112, located in front of the right-side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever mode can also be changed arbitrarily according to the operator's instructions.

[0038] For example, such as Figure 2 As shown, the image output device 221 consists of a central image output device 2210, a left image output device 2211, and a right image output device 2212, each with a roughly rectangular screen, respectively positioned in front of, to the left diagonally in front of, and to the right diagonally in front of the seat St. The shape and size of the screens (image display areas) of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be the same or different.

[0039] like Figure 2 As shown, with the screen of the central image output device 2210 and the screen of the left image output device 2211 tilted at an angle θ1 (e.g., 120°≤θ1≤150°), the right edge of the left image output device 2211 is adjacent to the left edge of the central image output device 2210. Figure 2 As shown, the screen of the central image output device 2210 and the screen of the right image output device 2212 are tilted at an angle θ2 (e.g., 120°≤θ2≤150°), and the left edge of the right image output device 2212 is adjacent to the right edge of the central image output device 2210. The tilt angle θ1 and the tilt angle θ2 can be the same or different.

[0040] The images of the central image output device 2210, the left image output device 2211, and the right image output device 2212 can be either parallel or tilted relative to the vertical direction. At least one of the image output devices 2210, 2211, and 2212 can also be composed of multiple image output devices. For example, the central image output device 2210 can be composed of a pair of vertically adjacent image output devices with a generally rectangular image shape. The image output devices 2210-2212 can further include a speaker (sound output device).

[0041] (Structure of the operating machinery)

[0042] The operating machinery 40 includes a machine control device 400, a machine input interface 41, a machine output interface 42, and a working mechanism 440. The machine control device 400 is composed of a computing processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and performs computing processing according to the software based on the data.

[0043] Construction machinery 40, for example, is a tracked excavator (construction machinery), such as... Figure 3 As shown, the tracked excavator includes a tracked lower traveling body 410 and an upper slewing body 420 rotatably mounted on the lower traveling body 410 via a slewing mechanism 430. A cab 424 (driver's cab) is provided on the front left side of the upper slewing body 420. A working mechanism 440 is provided on the front center of the upper slewing body 420.

[0044] The machine input interface 41 includes a machine operation mechanism 411 and a machine imaging device 412. The machine operation mechanism 411 has multiple control levers, similarly arranged around the seat inside the cab 424 as the remote operation mechanism 211. A drive mechanism or robot is installed in the cab 424, receiving signals corresponding to the operation mode of the remote control levers, and actuating the control levers based on these received signals. The machine imaging device 412 is installed, for example, inside the cab 424, and captures images of the environment including at least a portion of the working mechanism 440 through the front window and a pair of side windows. Some or all of the front and side windows may be omitted. The machine imaging device 412 may also be arranged around the working machinery 40. The machine imaging device 412 may also be mounted on a mobile body such as an unmanned vehicle or drone that can move around the working machinery 40.

[0045] The actual output interface 42 is equipped with actual wireless communication device 422.

[0046] The working mechanism 440 includes: a boom 441, which is mounted on the upper slewing body 420 for lifting and lowering; a stick 443, which is rotatably connected to the top end of the boom 441; and a bucket 445, which is rotatably connected to the top end of the stick 443. The working mechanism 440 is equipped with a boom cylinder 442, a stick cylinder 444, and a bucket cylinder 446, all of which are telescopic hydraulic cylinders.

[0047] Boom cylinder 442 is located between boom 441 and upper slewing body 420, allowing it to extend and retract by receiving a supply of working oil, thereby causing boom 441 to rotate in the lifting direction. Stick cylinder 444 is located between stick 443 and boom 441, allowing it to extend and retract by receiving a supply of working oil, thereby causing stick 443 to rotate about a horizontal axis relative to boom 441. Bucket cylinder 446 is located between bucket 445 and stick 443, allowing it to extend and retract by receiving a supply of working oil, thereby causing bucket 445 to rotate about a horizontal axis relative to stick 443.

[0048] (Primary Function)

[0049] use Figure 4 The flowchart shown illustrates the first function of the remote operation assistance system with the above-described configuration. In this flowchart, the term "C●" is used for simplification and refers to the sending and / or receiving of data, as well as a conditional branch that executes branch-direction processing based on the sending and / or receiving of such data.

[0050] In the remote operation device 20, which serves as the first remote operation device, it is determined whether the operator has performed the first specified operation through the remote input interface 210. Figure 4 / Step (STEP) 211). "First specified operation" is, for example, an operation such as tapping or sliding performed by the operator on the remote input interface 210 to specify the work machinery 40 intended for remote operation.

[0051] In the case where the judgment result is negative ( Figure 4 / Step (STEP) 211…No (NO)), repeatedly perform the determination process of whether or not the first specified operation has been performed ( Figure 4 / Step 211). On the other hand, if the determination result is positive ( Figure 4 / Step 211…Yes (YES)), in the remote operation device 20, which is the first remote operation device, it is determined whether the operator has performed the second specified operation through the remote input interface 210 ( Figure 4 / Step (STEP) 212). The "second designated operation" is, for example, a touch, tap, or swipe operation performed on the remote input interface 210 to set a schedule that includes the start and / or end times of remote operation of the work machinery 40 specified by the first designated operation. This schedule can also be pre-registered in the database 102. The schedule can also be set by a terminal device such as a smartphone, tablet, or personal computer.

[0052] In the case where the judgment result is negative ( Figure 4 / Step 212…No (NO)), then the determination of whether or not the first specified operation has occurred is repeated ( Figure 4 The processing after step 211). On the other hand, if the determination result is positive ( Figure 4 / Step 212…Yes (YES)), send the specified operation information to the remote operation assistance server 10 via the remote wireless communication device 222. Figure 4 / Step (STEP) 213). The specified operation information includes: a remote identifier for identifying the remote operating device 20; a physical identifier for identifying the operating machinery 40 specified by the first specified operation; and a schedule including the start and end times of the remote operation set by the second specified operation.

[0053] Through the first and second designation operations, not only can one working machine 40 be designated, but multiple working machines 40 can also be designated. In this embodiment, the following situation is described: the first designation operation designates two working machines 40, a first working machine and a second working machine, and the second designation operation sets the predetermined end time (first predetermined end time t) for the first working machine. 12 The scheduled start time of the second operating machine (the second scheduled start time t) 21 ).

[0054] In the remote operation assistance server 10, upon receiving specified operation information ( Figure 4 / C110), register the specified operation information in database 102 ( Figure 4 / Step (STEP) 110).

[0055] In the remote operation device 20, a preparation completion notification indicating that remote operation is ready is sent to the remote operation assistance server 10 via the remote wireless communication device 222 according to a specified trigger. Figure 4 / Step (STEP) 214). As a designated trigger, it detects whether the operator presses the preparation completion button constituting the remote input interface 210 or whether the operator is seated in seat St, etc. However, at a predetermined time t before the second start, 21 If the specified trigger occurs more than 10 minutes earlier than the specified time, a preparation completion notification may not be sent. The preparation completion notification contains a remote identifier.

[0056] In the remote operation assistance server 10, upon receiving a preparation completion notification, the first auxiliary processing element 121 sends the preparation completion notification to the following operating machinery 40 ( Figure 4 / C111), the machine 40 is identified by a physical machine identifier stored or registered in database 102 in association with the remote identifier contained in the preparation completion notification.

[0057] In the operating machine 40, which serves as the first operating machine, a first environmental confirmation request is received via the actual machine wireless communication device 422. Figure 4 / C410), the actual control device 400 acquires the captured image as the first captured image through the actual shooting device 412. Figure 4 / Step (STEP) 410). The machine control device 400 sends first captured image data representing the first captured image to the remote operation assistance server 10 via the machine wireless communication device 422. Figure 4 / Step (STEP) 412).

[0058] In the remote operation assistance server 10, upon receiving the first captured image data ( Figure 4 / C112), the first auxiliary processing element 121 sends the first working environment image data to the remote operation device 20 ( Figure 4 / Step (STEP) 112). In addition to the first captured image data itself, the first work environment image data can also be image data representing a simulated work environment image generated based on the first captured image.

[0059] In the remote operation device 20, when the first work environment image data is received via the remote wireless communication device 222 ( Figure 4 / C210), the remote control device 200 outputs the first working environment image corresponding to the first working environment image data to the image output device 221 (C210). Figure 4 / Step (STEP) 215).

[0060] Therefore, as Figure 6A as well as Figure 6B As shown, the first operational environment image starts from the predetermined start time t during the first remote operation. 11 Until the scheduled end time t 12 It is continuously output to the image output device 221. For example, such as Figure 7A As shown, the first working environment image Img1 is output to the image output device 221. The first working environment image Img1 includes a part of the working mechanism 440 of the working machine 40, which is the first working machine, namely the boom 441, stick 443 and bucket 445.

[0061] In the remote operation device 20, the remote control device 200 identifies the operation mode of the remote operation mechanism 211. Figure 4 / Step (STEP) 216), and, through the remote wireless communication device 222, send a remote operation command corresponding to the operation mode to the remote operation assistance server 10. Figure 4 / Step (STEP) 217). Alternatively, it can be set to not recognize the operation mode before the first predetermined start time, or it can be set to not send remote operation instructions to the remote operation assistance server 10 even if the operation mode is recognized.

[0062] In the remote operation assistance server 10, upon receiving the remote operation instruction, the first auxiliary processing element 121 sends the remote operation instruction to the operating machine 40, which serves as the first operating machine. Figure 4 / C114).

[0063] In the operating machine 40, which serves as the primary operating machine, when the actual machine control device 400 receives a remote operation command via the actual machine wireless communication device 422 ( Figure 4 / C412), controls the actions of the working mechanism 440, etc. Figure 4 / Step (STEP) 414). Thus, for example, as Figure 7A As shown, the operation involves the bucket 445 scooping up soil in area O1 in front of the working machine 40, and then dumping the soil from the bucket 445 after the upper rotating body 420 rotates.

[0064] (Second function)

[0065] use Figure 5 The flowchart shown illustrates the second function of the remote operation assistance system with the above-described configuration. In this flowchart, the "C●" block is a simplified description and refers to the sending and / or receiving of data, as well as a conditional branch that executes processing in the branch direction based on the sending and / or receiving of such data.

[0066] In the remote operation auxiliary server 10, the second auxiliary processing element 122 determines whether the current time t is earlier than the predetermined start time t of the second remote operation period. 21 The time before and after the first specified period τ1 ( Figure 5 (Step 120). That is, determine the predetermined start time t from the current time t to the second remote operation period. 21 The time interval up to that point, i.e., whether the first time interval is a time interval less than or equal to the first specified period τ1. The "second remote operation period" is the period or predetermined period during which the remote operation device 20 remotely operates the second working machine.

[0067] In the case where the judgment result is negative ( Figure 5 / Step 120…No (NO)), repeat this judgment process ( Figure 5 / Step (STEP) 120).

[0068] On the other hand, if the determination result is affirmative ( Figure 5 / Step 120…Yes (YES)), the second auxiliary processing element 122 sends a second environmental confirmation request to the operating machine 40, which is identified as the second operating machine through the second physical machine identifier. Figure 5 / Step (STEP) 121).

[0069] In the operating machine 40, which serves as the second operating machine, if a second environmental confirmation request is received via the actual machine wireless communication device 422 ( Figure 5 / C420), the actual control device 400 obtains the captured image as the second captured image through the actual shooting device 412. Figure 5 / Step (STEP) 420). The actual control device 400 sends second captured image data representing the second captured image to the remote operation assistance server 10 via the actual wireless communication device 422. Figure 5 / Step (STEP) 422).

[0070] In the remote operation assistance server 10, upon receiving the second captured image data ( Figure 5 / C122), the second auxiliary processing element 122 sends the first notification information and the second working environment image data to the remote operation device 20. Figure 5 / Step 122). "Second operational environment image data" can be, in addition to the second captured image data itself, image data representing a simulated operational environment image generated based on the second captured image. "First notification information" indicates the predetermined start time t during the period from the current time t to the second remote operation. 21 The time interval up to this point is the time interval below the first specified period τ1, and it is a notification that the operation of the second working machine is about to begin, and, at t=t 21 At time τ1, the first notification message is sent. If another operator is operating the second work machine during the first remote operation period, and that other operator's work ends earlier than the scheduled end time, the notification can also be sent at a time earlier than the scheduled start time t of the second remote operation period. 21 An earlier "first notification message" is sent. This occurs when another operator is operating the second machine during the first remote operation, and the delay in that other operator's operation of the second machine causes the scheduled start time t to be reached. 21 In the case that it is also postponed, at t=t 21 At time -τ1, the first notification message is sent.

[0071] In the remote operation device 20, when the first notification information and the second work environment image data are received via the remote wireless communication device 222 ( Figure 5 / C222), the remote control device 200 outputs the first notification information to the image output device 221 ( ...). Figure 5 / Step (STEP) 220). Additionally, in this case, the remote control device 200 outputs a second work environment image corresponding to the second work environment image data to the image output device 221 ( Figure 5 / Step (STEP) 221).

[0072] Therefore, as Figure 6A As shown, the second working environment image is from time t=t 21 -τ1 to the predetermined end time t of the first remote operation period 12 The image is continuously output to the image output device 221 during the specified period. That is, as... Figure 6A As shown, from time t = t 21 -τ1 to the predetermined end time t of the first remote operation period 12 During the period up to this point, in addition to continuously outputting the first working environment image, the image output device 221 also continuously outputs the second working environment image. For example, such as Figure 7B As shown, a second working environment image Img2 is output on the image output device 221. The second working environment image Img2 includes a part of the working mechanism 440 of the working machine 40, which is the second working machine, namely the boom 441, the stick 443 and the bucket 445.

[0073] Additionally, at the predetermined end time t of the first remote operation period 12 The scheduled start time t during the second remote operation period 21 If the time interval is slightly shorter than the first specified period τ1, such as Figure 6B As shown, the second working environment image Img2 starts from time t=t 21 -τ1 is continuously output to the image output device 221. In this case, as... Figure 6B As shown, the period during which both the first working environment image Img1 and the second working environment image Img2 can be output to the image output device 221 is shorter than the first specified period τ1.

[0074] The output of the second work environment image to the image output device 221 can be arbitrarily controlled. For example, a widget of the second work environment image can be displayed at the end of the image output device 221, and the second work environment image can be output to the image output device 221 when the operator unfolds the widget using the button constituting the remote input interface 210. Alternatively, after the second work environment image is displayed, the operator can operate the button constituting the remote input interface 210 to prevent the output of the second work environment image. In short, it is possible to check the status of the second work machine while operating the first work machine.

[0075] It should be noted that, as Figure 6C As shown, at the predetermined end time t during the first remote operation period 12 The scheduled end time t during the second remote operation period 21 If the time interval is greater than or equal to the first specified period τ1, the first environmental image Img1 and the second environmental image Img2 will not be displayed on the image output device 221 simultaneously.

[0076] First working environment image Img1 (reference) Figure 7A ) and the second working environment image Img2 (reference) Figure 7B The images can also be output separately to the image output device 221 (e.g., two of the image output devices: the central image output device 2210, the left image output device 2211, and the right image output device 2212) or output separately to the image output device 221. Figure 8 As shown, the first working environment image Img1 (reference) Figure 7A ) and the second working environment image Img2 (reference) Figure 7B The images can also be output to the image output device 221 (e.g., one of the central image output device 2210, the left image output device 2211, and the right image output device 2212). In this case, the size and configuration of the second working environment image Img2 can also be displayed so as not to overlap with the working mechanism 440 projected into the first working environment image Img1.

[0077] The first and second work environment images can also be output to the image output device 221 in the following manner: the second auxiliary processing element 122, based on communication with the remote operation device 20, outputs the image at a predetermined time at the end of the output period of the first work environment image (e.g., referring to...). Figure 6A and Figure 6B The moment t 12The second working environment image is displayed on the image output device 221 constituting the remote output interface 220 in a manner that it is relatively enlarged compared to the first working environment image. For example, as Figure 9 As shown, it can be controlled to be t at time t. i-1 →t i →t i+1 During the change process, the size of the second working environment image Img2, which overlaps with the first working environment image Img1, gradually increases.

[0078] In the remote operation device 20, which serves as the first remote operation device, it is determined whether the operator has performed a third specified operation through the remote input interface 210. Figure 5 / Step (STEP) 222). The "third designated operation" is, for example, an operation performed by the operator on the remote input interface 210, such as tapping or sliding, to switch the work machine 40 that the operator wants to remotely operate from the first work machine to the second work machine.

[0079] In the case where the judgment result is negative ( Figure 5 / Step 222…No (NO)), the series of processes ends. On the other hand, if the determination result is positive ( Figure 5 / Step 222…Yes (YES)), in the remote operation device 20, the remote control device 200 identifies the operation mode of the remote operation mechanism 211 ( Figure 5 / Step (STEP) 223), and, through the remote wireless communication device 222, send a remote operation command corresponding to the operation mode to the remote operation assistance server 10. Figure 5 / Step (STEP) 224).

[0080] like Figure 6A and Figure 6B As shown, the predetermined start time t during the second remote operation is... 21 Subsequently, the second working environment image is continuously output to the image output device 221 of the remote operation device 20. The "second remote operation period" is the period or predetermined period during which the remote operation device 20 remotely operates the second working machine. Depending on the operation mode of the remote operation mechanism 211 on the remote operation device 20, the operation of the working mechanism 440, etc., of the working machine 40, which is the second working machine, is controlled.

[0081] In the remote operation assistance server 10, upon receiving the remote operation instruction, the first auxiliary processing element 121 sends the remote operation instruction to the operating machine 40, which serves as the second operating machine. Figure 5 / C124).

[0082] In the operating machine 40, which serves as the second operating machine, when the actual machine control device 400 receives an operating command via the actual machine wireless communication device 422 ( Figure 5 / C422), controls the actions of the operating mechanism 440, etc. Figure 5 / Step 424). Thus, for example, as Figure 7B As shown, the operation involves the bucket 445 scooping up the soil pile in area O2 located in front of the working machine 40, and dumping the soil from the bucket 445 after the upper rotating body 420 rotates.

[0083] (Effect)

[0084] Based on the remote operation assistance system having this configuration or the remote operation assistance server 10 constituting the remote operation assistance system, a first working environment image can be output to the image output device 221 constituting the remote output interface 220 (see reference). Figure 4 Step 410 → Step 412 → C112 → 112 → C210 → Step 215 and Figure 7A The first working environment image is an image based on the first captured image data, which is obtained by a real machine shooting device 412 mounted on the working machine 40, which is the current remote operation object of the remote operation device 20 (or located around the working machine 40).

[0085] In addition, it can output the second operating environment image to the image output device 221 that constitutes the remote output interface 220 (see reference). Figure 5 Step 420 → Step 422 → C122 → C122 → C222 → … → Step 221 Figure 7B and Figure 8 The second working environment image is an image based on the second captured image data, which is obtained by a real-machine shooting device mounted on the second working machine 40, which is the next remote operating object of the remote operating device 20 (or located around the working machine 40).

[0086] The second auxiliary processing element 122 sets a predetermined time t from the current time t to the start of the second remote operation period for remotely operating the second machine tool via the remote operation device 20. 21 The first time interval is a necessary condition that the time interval is less than or equal to the first specified period τ1, so that the second working environment image and the first working environment image are output together to the remote output interface 220 of the remote operation device 20.

[0087] Based on communication with the remote operating device 20, the second auxiliary processing element 122 outputs a first notification message indicating that the first time interval is a time interval less than or equal to a first specified period τ1 to the remote output interface 220 of the remote operating device 20.

[0088] The predetermined time t from the current time t to the start of the second remote operation period. 21 The necessary condition is that the time interval up to the specified period τ1 is a time interval less than or equal to the first specified period, so that the second working environment image and the first working environment image are output together to the image output device 221 of the remote output interface 220 constituting the remote operation device 20 (see reference). Figure 5 Step 120…Yes (YES) →…→Step 224 Figure 6A and Figure 6B ).

[0089] Therefore, it is possible for an operator who is remotely operating the first machine via the remote operating device 20 to pre-identify the surrounding environment of the second machine, which will be the next target for remote operation (see reference). Figure 6A / momentt 11 ~time t 12 During the period Figure 6B / momentt 12 -τ1~Time t 12 During the period, refer to Figure 7B as well as Figure 8 Therefore, when switching the remotely operated object from one machine 40 to another via the remote operation device 20, the operator can easily respond to changes in the environment of the remotely operated object.

[0090] For example, it can prevent the second working environment image from being output to the remote output interface 220 for too long along with the first working environment image. Therefore, it can prevent the operator's attention to the first working environment image from being diminished by the second working environment image, and it can make it easier for the operator to respond to environmental changes of the remotely operated object.

[0091] Based on communication with the remote operating device 20, the second auxiliary processing element 122 outputs the first operating environment image and the second operating environment image to the remote output interface 220 of the remote operating device 20 in such a way that the second operating environment image is relatively larger than the first operating environment image as the predetermined start time of the second remote operation period approaches.

[0092] According to the remote operation assistance server 10 with this configuration, as the predetermined end time of the output period of the first work environment image approaches, the second work environment image is output on the remote output interface 220 of the remote operation device 20 in a manner that the second work environment image is relatively larger than the first work environment image in a continuous or phased manner. Therefore, when switching the remotely operated object from one work machine to another via the remote operation device 20, the operator's attention can be gradually shifted from one work machine to another, resulting in a smooth switching of the remotely operated object.

[0093] (Other embodiments of the present invention)

[0094] At the scheduled start time t during the second remote operation 21 In the case of notifying the first notification information at the earliest possible time (refer to) Figure 5 / C222), can also be used at t=t 21 The "first notification information" and "second operating environment image data" are received before time -τ1. Alternatively, only the "first notification information" can be output on the image output device 221 (see reference). Figure 5 / Step 220), and at t=t 21 At time -τ1, in addition to outputting the first operating environment image that has been continuously output on the image output device 221, a second operating environment image (refer to...) is also output. Figure 5 / Step 221). Alternatively, only the "first notification information" can be output on the image output device 221, and the operator can remain on the device until t=t. 21 When the button constituting the remote input interface 210 is operated during the period up to time -τ1, a second working environment image is output to the image output device 221. Alternatively, after the second working environment image is output to the image output device 221, the operator can operate the button constituting the remote input interface 210 to prevent the second working environment image from being output.

[0095] In the remote operation device 20, which serves as a second remote operation device, when another operator is remotely operating a second work machine, a predetermined start time t is set at which the operator begins remote operation, anticipating a delay in the other operator's work. 21 Even if it is postponed, it can also be done at t=t 21 -τ1 receives both the "first notification information" and the "second working environment image data". Alternatively, only the "first notification information" can be output on the image output device 221 (see reference). Figure 5 / step 220), and outputs a message indicating inoperability or delay on the image output device 221 (see reference). Figure 5(Step 221). Alternatively, when another operator finishes their work, a second work environment image may be output in addition to the first work environment image that has been output so far on the image output device 221. Alternatively, when the operator operates the button constituting the remote input interface 210, the second work environment image may only be output to the image output device 221 when only the "first notification information" is output on the image output device 221. Furthermore, after the second work environment image has been output to the image output device 221, the operator may operate the button constituting the remote input interface 210 to prevent the second work environment image from being output.

[0096] Based on communication with the remote operating device 20, the second auxiliary processing element 122 also outputs the second working environment image along with the first working environment image to the remote output interface 220 of the remote operating device 20 after the first remote operation period ends.

[0097] In this case, even if the timing of the first designated operation is delayed, the period during which the first and second work environment images are output to the remote output interface 220 is ensured after the first remote operation period ends. Therefore, when the remotely operated object is switched from one work machine to another via the remote operation device 20, the operator can easily cope with the environmental changes of the remotely operated object.

[0098] Symbol Explanation

[0099] 10…Remote operation auxiliary server; 20…Remote operation device; 40…Working machinery; 102…Database; 121…First auxiliary processing element; 122…Second auxiliary processing element; 200…Remote control device; 210…Remote input interface; 211…Remote operation mechanism; 220…Remote output interface; 221…Image output device; 400…Machine control device; 410…Lower traveling body; 420…Upper rotating body; 424…Cab (driver's cab); 440…Working mechanism (working mechanism); 445…Bucket (working part).

Claims

1. A remote operation assistance server for assisting remote operations by an operator on a plurality of work machines via a remote operation device, the remote operation assistance server characterized by comprising: a database that registers a start scheduled time and an end scheduled time of a first remote operation period of a first work machine that is a current remote operation target of the remote operation device, and a start scheduled time and an end scheduled time of a second remote operation period of a second work machine that is a next remote operation target of the first work machine by the remote operation device; a first assistance processing element that recognizes the first work machine, acquires first captured image data representing a surrounding environment of the first work machine based on communication with the first work machine, causes a first work environment image corresponding to the first captured image data to be output to an output interface of the remote operation device based on communication with the remote operation device during a period from the start scheduled time of the first remote operation period until a time point before reaching a time point before the end scheduled time of the first remote operation period by a specified period, in a case where the period from the end scheduled time of the first remote operation period to the start scheduled time of the second remote operation period is shorter than the specified period, wherein the first captured image data is acquired by a real machine camera mounted on the first work machine or a real machine camera present in the surroundings of the first work machine; and a second assistance processing element that recognizes the second work machine, acquires second captured image data representing a surrounding environment of the second work machine based on communication with the second work machine before the end scheduled time of the first remote operation period, causes a second work environment image corresponding to the second captured image data to be output to the output interface of the remote operation device together with the first work environment image based on communication with the remote operation device during the specified period, wherein the second captured image data is acquired by a real machine camera mounted on the second work machine or a real machine camera present in the surroundings of the second work machine.

2. The remote operation assistance server according to claim 1, characterized in that the second assistance processing element causes the second work environment image to be output to the output interface of the remote operation device together with the first work environment image with a first time interval from a current time to the start scheduled time of the second remote operation period being a time interval that is shorter than the specified period as a necessary condition.

3. The remote operation assistance server according to claim 2, characterized in that the second assistance processing element causes first notification information indicating that the first time interval is a time interval that is shorter than the specified period to be output to the output interface of the remote operation device based on communication with the remote operation device.

4. The remote operation assistance server according to claim 3, characterized in that The second auxiliary processing element causes the second work environment image to be output to the output interface of the remote operation device together with the first work environment image based on communication with the remote operation device also after the end of the first remote operation period.

5. The remote operation assistance server according to any one of claims 1 to 4, characterized in that, The second auxiliary processing element causes the first work environment image and the second work environment image to be output on the output interface of the remote operation device in a manner that the second work environment image relatively increases compared to the first work environment image on the output interface of the remote operation device as the start scheduled time of the second remote operation period approaches based on communication with the remote operation device.

6. A teleoperation assistance system characterized by comprising: The remote operation assistance server according to any one of claims 1 to 5 and the client constitute.

7. A remote operation assistance method for assisting an operator in remote operations by a remote operation device on a plurality of work machines, The remote operation assistance method is characterized by performing: a process of registering, in a database, a start scheduled time and an end scheduled time of a first remote operation period of a current remote operation object of the remote operation device, which is a first work machine, and a start scheduled time and an end scheduled time of a second remote operation period of a next remote operation object of the remote operation device, which is a second work machine; a first auxiliary process of identifying the first work machine, acquiring first captured image data representing a surrounding environment of the first work machine based on communication with the first work machine, and causing a first work environment image corresponding to the first captured image data to be output to an output interface of the remote operation device based on communication with the remote operation device from the start scheduled time of the first remote operation period until a time before reaching a time before the end scheduled time of the first remote operation period by a specified period, wherein the first captured image data is acquired by a real machine capturing device mounted on the first work machine or a real machine capturing device present in the surroundings of the first work machine; and a second auxiliary process of identifying the second work machine, acquiring second captured image data representing a surrounding environment of the second work machine based on communication with the second work machine before the end scheduled time of the first remote operation period, and causing a second work environment image corresponding to the second captured image data to be output to the output interface of the remote operation device together with the first work environment image based on communication with the remote operation device during the specified period, wherein the second captured image data is acquired by a real machine capturing device mounted on the second work machine or a real machine capturing device present in the surroundings of the second work machine.

Images