Method of remote control, terminal device and program

Abstract

To provide a method of remote control, a terminal device, and a program that can support operators in carrying out remote operation.SOLUTION: The method of remote control according to the present disclosure includes steps of outputting predicted area information, indicating the area of an image in which a moving object is predicted to appear, based on delay information indicating the time between the image being captured and displayed, and the object's position as reflected in the image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a method, a terminal device, and a program for remote control. [Background technology]

[0002] In recent years, various types of autonomous vehicle services have been put into practical use, and remote control systems that can remotely monitor or operate these vehicles are being developed. In these remote control systems, when an autonomous vehicle requests assistance through remote operation, an operator in a remote control room can provide assistance such as moving the autonomous vehicle by remotely operating the vehicle while viewing images captured by a camera mounted on the autonomous vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-313588 [Patent Document 2] Patent No. 7070802 [Patent Document 3] Japanese Patent Application Publication No. 2019-156527 [Patent Document 4] Japanese Patent Application Laid-Open No. 2005-145632 [Patent Document 5] Japanese Patent Application Laid-Open No. 2007-31102 [Patent Document 6] International Publication No. 2017 / 135382 [Patent Document 7] Japanese Patent Application Publication No. 2023-79739 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in autonomous port solutions, methods are being considered for remotely coordinating the operation of cranes and autonomous trucks. The cranes use crane hooks to lift containers and other cargo that autonomous trucks will carry and transport to the designated location.

[0005] When remotely operating a crane, it is essential to accurately display the crane hook's position to prevent it from coming into contact with nearby workers or obstacles. However, due to delays in remote monitoring, the operator may not be able to accurately grasp the current position. Furthermore, because the operator is operating the crane via video, it is difficult to grasp the sense of distance, which may lead to a misperception of the position relative to surrounding workers and vehicles. Furthermore, the movement of the crane hook can be unpredictable due to the influence of wind and inertia.

[0006] An object of the present disclosure is to provide a method, a terminal device, and a program for remote control that can appropriately support an operator's remote operation. [Means for solving the problem]

[0007] In order to achieve the above object, the remote control method disclosed herein outputs predicted area information indicating an area of ​​an image in which a moving object is predicted to be present, based on delay information indicating the time from when the image is captured to when it is drawn and the position of a moving object reflected in the image. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to appropriately support the operator's remote operation. Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this specification. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a remote control system according to an embodiment. [Figure 2] FIG. 2 is an activity diagram showing an example of the overall operation flow of the remote control system according to the embodiment. [Figure 3] FIG. 3 is an activity diagram showing an example of the overall operation flow of the remote control system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of each of a plurality of devices included in the remote control system according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a hardware configuration of the control device according to the embodiment. [Figure 6] FIG. 6 is a table for explaining delay information according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining the position of the crane hook according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of a camera image according to the embodiment. [Figure 9] FIG. 9 is a table showing an example of crane hook distance calculation information according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of prediction region information according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram illustrating an example of prediction region information according to the embodiment. [Figure 12] FIG. 12 is a schematic diagram showing the positional relationship between the area of ​​the predicted area information and peripheral objects according to the embodiment. [Figure 13] FIG. 13 is a table showing an example of vehicle height information according to the embodiment. [Figure 14] FIG. 14 is a schematic diagram showing an example of a display screen according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing an example of a display screen according to the embodiment. [Figure 16] FIG. 16 is a schematic diagram showing an example of a display screen according to the embodiment. [Figure 17] FIG. 17 is a flowchart illustrating an example of processing performed by the remote operation assistance device according to the embodiment. [Figure 18]FIG. 18 is a schematic diagram showing an example of a display screen according to the first modified example. [Figure 19] FIG. 19 is a schematic diagram showing an example of a display screen according to the second modified example. [Figure 20] FIG. 20 is a diagram showing an example of the configuration of each of a plurality of devices included in a remote control system according to the third modification. [Figure 21] FIG. 21 is a table showing an example of vehicle position information according to the third modified example. [Figure 22] FIG. 22 is a schematic diagram for explaining peripheral objects according to the third modified example. [Figure 23] FIG. 23 is a schematic diagram showing an example of a display screen according to the third modified example. [Figure 24] FIG. 24 is a schematic diagram showing an example of a display screen according to the third modified example. [Figure 25] FIG. 25 is a flowchart illustrating an example of processing performed by a remote operation assistance device according to the third modified example. [Figure 26] FIG. 26 is a table for explaining an example of alert information according to the fourth modification. [Figure 27] FIG. 27 is a flowchart showing an example of processing performed by a remote operation assistance device according to the fourth modified example. [Figure 28] FIG. 28 is a schematic diagram for explaining control for restricting autonomous traveling according to the fifth modified example. [Figure 29] FIG. 29 is a schematic diagram for explaining control for restricting autonomous traveling according to the fifth modified example. [Figure 30] FIG. 30 is a schematic diagram showing an example of a display screen according to the sixth modified example. [Figure 31] FIG. 31 is a schematic diagram showing an example of a display screen according to the seventh modified example. [Figure 32] FIG. 32 is a table for explaining an example of alert information according to the eighth modification. [Figure 33] FIG. 33 is a table for explaining an example of alert information according to the ninth modification. [Figure 34]FIG. 34 is a schematic diagram showing an example of a display screen according to the eleventh modified example. [Figure 35] FIG. 35 is a flowchart showing an example of processing performed by a remote operation assistance device according to the eleventh modification. [Figure 36] FIG. 36 is a schematic diagram showing an example of a display screen according to the twelfth modified example. [Figure 37] FIG. 37 is a flowchart showing an example of processing performed by a remote operation assistance device according to the twelfth modification. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a method, a terminal device, and a program for remote control according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0011] (Embodiment) Fig. 1 is a diagram showing an example of a schematic configuration of a remote operation system 1 according to this embodiment. As shown in Fig. 1, the remote operation system 1 includes a vehicle 101, a crane 102, a remote operation support device 20, and a plurality of (two in the example of Fig. 1) terminal devices 301 and 302.

[0012] 1 illustrates one vehicle 101, but this is not a limitation and the number of vehicles 101 included in the remote operation system 1 can be changed as desired depending on design conditions, etc. Also, while FIG. 1 illustrates one crane 102, this is not a limitation and the number of cranes 102 included in the remote operation system 1 can be changed as desired depending on design conditions, etc.

[0013] Furthermore, while FIG. 1 illustrates two terminal devices 301 and 302, the number of terminal devices 30 included in the remote operation system 1 can be changed arbitrarily depending on design conditions, etc. In the following description, when the terminal device 301 and the terminal device 302 are not distinguished from each other, they will be simply referred to as the "terminal device 30." The same reference numerals will be used for the same elements constituting the terminal device 30. In this embodiment, the object remotely controlled by the terminal device 301 is the vehicle 101, and the object remotely controlled by the terminal device 302 is the crane 102.

[0014] In the example of FIG. 1, a vehicle 101, a crane 102, a remote operation support device 20, and a terminal device 30 can be connected to each other via a network 40 such as the Internet.

[0015] The vehicle 101 is an autonomous vehicle that can travel autonomously and is used to provide various services. The vehicle 101 is an example of an autonomous vehicle. The crane 102 is a mechanical device that uses a crane hook to lift a container loaded on the vehicle 101 and transports it to a predetermined location. In this embodiment, the crane 102 is a gantry crane having a portal shape. However, the crane 102 is not limited to a gantry crane. The crane hook is also referred to as a moving object.

[0016] The remote operation support device 20 is a device that supports the remote operation of the vehicle 101 and the crane 102. The terminal device 301 is a device that is operated by an operator in a remote control room who remotely operates the vehicle 10. The terminal device 301 is a device that is operated by an operator in a remote control room who remotely operates the crane 102.

[0017] 2 and 3 are activity diagrams showing an example of the overall operation flow of the remote operation system 1 according to this embodiment. Figures 2 and 3 explain the operation of packing cargo at a port. Figures 2 and 3 show a vehicle 101, a crane 102, a terminal device 301, a terminal device 302, a remote operator 303 for the vehicle 101, and a remote operator 304 for the crane 102.

[0018] In step S101, the vehicle 101 enters the loading area of ​​the port. In step S102, the vehicle 101 arrives at the loading area. In step S103, the terminal device 301 receives an arrival notification from the vehicle 101 indicating that the vehicle has arrived at the loading area. In step S104, the remote operator 303 of the vehicle 101 confirms the arrival notification received by the terminal device 301. In step S105, the remote operator 303 of the vehicle 101 confirms the camera image captured by the camera of the vehicle 101. In step S106, the terminal device 301 receives the camera image captured by the camera of the vehicle 101.

[0019] In step S107, the remote operator 303 of the vehicle 101 checks whether the position of the crane hook of the crane 102 is sufficiently high based on the camera image received by the terminal device 301. In step S108, if the position of the crane hook of the crane 102 is not sufficiently high (step S108: No), the remote operator 303 of the vehicle 101 returns to step S107 and checks again whether the position of the hook of the crane 102 is sufficiently high. On the other hand, if the position of the crane hook of the crane 102 is sufficiently high (step S108: Yes), the remote operator 303 of the vehicle 101 proceeds to step S109.

[0020] In step S109, the remote operator 303 of the vehicle 101 presses a travel start button displayed on the terminal device 301. In step S110, the terminal device 301 instructs the vehicle 101 to start traveling. In step S111, the vehicle 101 enters below the crane 102. In step S112, the vehicle 101 arrives below the crane 102.

[0021] In step S113, the remote operator 303 of the crane 102 starts remote operation of the crane 102. In step S114, the terminal device 302 transmits an operation signal to the crane 102. In step S115, the crane 102 loads the container onto the vehicle 101 based on the operation signal transmitted by the terminal device 302. In step S116, the crane 102 pulls up the crane hook of the crane 102.

[0022] In step S117, the remote operator 303 of the vehicle 101 checks the camera image captured by the camera of the vehicle 101. In step S118, the terminal device 301 receives the camera image captured by the camera of the vehicle 101.

[0023] In step S119, the remote operator 303 of the vehicle 101 checks whether the position of the crane hook of the crane 102 is sufficiently high based on the camera image received by the terminal device 301. In step S120, if the position of the crane hook of the crane 102 is not sufficiently high (step S120: No), the remote operator 303 of the vehicle 101 returns to step S119 and checks again whether the position of the hook of the crane 102 is sufficiently high. On the other hand, if the position of the crane hook of the crane 102 is sufficiently high (step S120: Yes), the remote operator 303 of the vehicle 101 proceeds to step S121.

[0024] In step S121, the remote operator 303 of the vehicle 101 presses a start driving button displayed on the terminal device 301. In step S122, the terminal device 301 instructs the vehicle 101 to start driving. In step S123, the vehicle 101 starts driving toward the destination. The remote operation system 1 of this embodiment is applied to the above-mentioned loading operation at the port.

[0025] 4 is a diagram showing an example of the configuration of each of the vehicle 101, the crane 102, the remote operation support device 20, and the terminal device 30 included in the remote operation system 1 according to the embodiment. Hereinafter, the configuration of each of the vehicle 101, the crane 102, the remote operation support device 20, and the terminal device 30 will be described with reference to FIG.

[0026] First, the configuration of the vehicle 101 will be described. As shown in Fig. 4, the vehicle 101 includes, as hardware elements, a communication device 110, a camera 120, a drive device 130, and a control device 140. Note that the hardware elements of the vehicle 101 are not limited to the configuration exemplified in Fig. 4, and the vehicle 101 may include other hardware elements.

[0027] The communication device 110 is a device that communicates with an external device (for example, the remote operation support device 20, etc.) via the network 40. The camera 120 is mounted on the vehicle 101 and is positioned so as to be able to capture an image of the area ahead of the vehicle 101. The drive device 130 is a device that drives the vehicle 101. The drive device 130 includes, for example, a wheel drive device that imparts a rotational drive force to the wheels, a steering drive device that steers the wheels, etc.

[0028] The control device 140 is a device that comprehensively controls the operation of the vehicle 101. FIG. 5 is a diagram showing an example of the hardware configuration of the control device 140 according to the embodiment. In this embodiment, the control device 140 is configured as a computer device. Note that the hardware configurations of a control device 190 included in the crane 102, a control device 230 included in the remote operation support device 20, and a control device 340 included in the terminal device 30, which will be described later, are also the same as those shown in FIG. 5.

[0029] As shown in FIG. 5, the control device 140 includes a processor 150, a ROM (Read Only Memory) 151, a RAM (Random Access Memory) 152, and a device I / F (Interface) unit 153.

[0030] The processor 150 is, for example, a CPU (Central Processing Unit). The processor 150 executes a program to comprehensively control the operation of the control device 140 and realize various functions of the control device 140. The various functions of the control device 140 will be described later.

[0031] ROM 151 is a non-volatile memory that stores various types of information including programs executed by processor 150. RAM 152 is a volatile memory that has a working area for processor 150. Device I / F unit 153 is an interface for connecting to other devices (communication device 110, camera 120, drive device 130, etc.).

[0032] Returning to FIG. 4, the functions of the control device 140 will be described. As shown in FIG. 4, the control device 140 has a vehicle characteristic information transmission unit 141, a position information acquisition unit 142, an image acquisition unit 143, a driving control unit 144, and an operation information reception unit 145. Note that the example in FIG. 4 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions of the control device 140 are not limited to these. In this embodiment, the processor 150 executes a program stored in the ROM 151 to realize the functions of each of the above-mentioned units. However, without being limited to this, some or all of these functions may be realized by dedicated hardware circuits (such as semiconductor integrated circuits).

[0033] The vehicle characteristic information transmission unit 141 transmits vehicle characteristic information relating to the vehicle characteristics of the vehicle itself to the remote operation assistance device 20. Here, the vehicle characteristics will be described. The vehicle characteristics include, for example, a vehicle ID, the vehicle height of the vehicle 101, the type of the vehicle 101, etc. The vehicle ID indicates information for identifying the vehicle 101.

[0034] In this embodiment, before the service is started, the vehicle characteristic information transmission unit 141 transmits the vehicle characteristic information of the vehicle 101. The timing of transmitting the vehicle characteristic information is not limited to before the service is started, and can be set arbitrarily. The vehicle characteristic information is included in remote operation parameter information related to the remote operation parameters of the vehicle 101.

[0035] The location information acquisition unit 142 acquires location information indicating the location of the vehicle 101. Various known techniques can be used as a method of acquiring location information, but for example, the location information acquisition unit 142 can receive GPS signals indicating radio waves transmitted from multiple GPS (Global Positioning System) satellites, calculate the location of the vehicle 101 by three-dimensional positioning based on the received GPS signals, and acquire location information indicating the location.

[0036] The image acquisition unit 143 acquires images captured by the camera 120. After the service operation starts, while the vehicle 101 is traveling, the images acquired by the image acquisition unit 143 are transmitted to the remote operation support device 20.

[0037] When the vehicle 101 is in an autonomous driving mode, which indicates a state in which the vehicle 101 is driving autonomously, the driving control unit 144 controls the vehicle 101 to drive (controls the driving unit 130) so that the vehicle 101 approaches the target position based on the target position and the position information acquired by the position information acquisition unit 142.

[0038] On the other hand, in the case of a remote control mode indicating a state in which the vehicle 101 is remotely controlled by an operator, the driving control unit 144 controls the driving of the vehicle 101 in accordance with operation information indicating information input to the terminal device 30 in accordance with an operation by the operator. In this embodiment, the driving modes of the vehicle 101 include the autonomous driving mode described above and the remote control mode described above, and the vehicle 101 basically drives in the autonomous driving mode, and drives in the remote control mode when in a state in which autonomous driving is not possible.

[0039] The operation information receiving unit 145 receives operation information transmitted from the terminal device 301 via the remote operation support device 20. The driving control unit 144 controls the driving of the vehicle 101 in accordance with the operation information received by the operation information receiving unit 145, and does not perform driving control based on the target position and position information. In other words, the driving mode of the vehicle 101 switches from the autonomous driving mode to the remote control mode.

[0040] Next, a description will be given of the configuration of the crane 102. As shown in Fig. 2, the crane 102 includes, as hardware elements, a communication device 160, a camera 170, a driving device 180, and a control device 190. Note that the hardware elements of the crane 102 are not limited to the configuration exemplified in Fig. 4, and may include other hardware elements.

[0041] The communication device 160 is a device that communicates with an external device (for example, the remote operation support device 20, etc.) via the network 40. The camera 170 is mounted on the crane 102 and is positioned so as to be able to capture an image of the area below the crane 102, for example, capturing an image of the crane hook of the crane 102. The driving device 180 is a device that drives the crane 102. The driving device 180 includes, for example, a driving device that moves the position of the crane hook of the crane 102 in the horizontal and vertical directions, a driving device that moves the crane boom of the crane 102 in accordance with the position of the container, etc.

[0042] The control device 190 is a device that comprehensively controls the operation of the crane 102. As shown in FIG. 4, the control device 190 has a crane characteristic information transmission unit 191, an image acquisition unit 192, a crane control unit 193, and an operation information reception unit 194. Note that the example in FIG. 4 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions of the control device 190 are not limited to these. In this embodiment, the processor 150 executes a program stored in the ROM 151 to realize the functions of the above-mentioned units. However, without being limited to this, some or all of these functions may be realized by dedicated hardware circuits (semiconductor integrated circuits, etc.).

[0043] The crane characteristic information transmission unit 191 transmits vehicle characteristic information relating to the crane characteristics of the crane itself to the remote operation support device 20. Here, the crane characteristics will be explained. The crane characteristics include, for example, a crane ID and the type of the crane 102. The crane ID indicates information for identifying the crane 102.

[0044] In this embodiment, before service operation, the crane characteristic information transmission unit 191 transmits the crane characteristic information of the crane 102. The timing of transmitting the crane characteristic information is not limited to before service operation and can be set arbitrarily. The crane characteristic information is included in remote operation parameter information related to the remote operation parameters of the crane 102.

[0045] The image acquisition unit 192 acquires images captured by the camera 170. Furthermore, while the crane 102 is in operation after the start of service operation, the images acquired by the image acquisition unit 192 are transmitted to the remote operation support device 20.

[0046] In a remote operation mode indicating a state in which the crane 102 is remotely operated by an operator, the crane control unit 193 performs control to operate the crane 102 in accordance with operation information indicating information input to the terminal device 302 in accordance with operation by the operator. The control to operate the crane 102 is, for example, control to operate the position of the crane hook or crane boom of the crane 102.

[0047] The operation information receiving unit 194 receives operation information transmitted from the terminal device 302 via the remote operation support device 20 .

[0048] Next, the configuration of the terminal device 301 will be described. As shown in Fig. 4, the terminal device 301 includes, as hardware elements, a communication device 310, a display device 320, an operation device 330, and a control device 340. Note that the hardware elements of the terminal device 301 are not limited to the configuration exemplified in Fig. 2, and may include other hardware elements.

[0049] The communication device 310 is a device that communicates with an external device (for example, the remote operation support device 20, etc.) via the network 40. The display device 320 is a device that displays various types of information and is configured, for example, with a liquid crystal display, etc. The operation device 330 is a device with which an operator performs various operations.

[0050] The control device 340 is a device that comprehensively controls the operation of the terminal device 301. In this embodiment, the control device 340 is configured as a computer device, and has the same hardware configuration as that shown in FIG. 5. Next, the functions of the control device 340 will be described. As shown in FIG. 4, the control device 340 has an information receiving unit 341, a display control unit 342, an operation information transmitting unit 343, and a time information transmitting unit 344. Note that, although the example in FIG. 4 illustrates only the functions necessary for explaining the main parts of this embodiment, the functions of the control device 340 are not limited to these.

[0051] In this embodiment, the functions of the above-described units are realized by the processor 150 executing a program stored in the ROM 151. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0052] The information receiving unit 341 receives various types of information transmitted from the remote operation assistance device 20. For example, the information receiving unit 341 can receive information such as a remote operation request (described later), assistance information (described later), and a camera image captured by the camera 120 of the vehicle 101 from the remote operation assistance device 20. Hereinafter, the camera image captured by the camera 120 of the vehicle 101 may also be simply referred to as an image.

[0053] The display control unit 342 controls the display of various types of information on the display device 320. For example, the display control unit 342 can control the display of predicted area information (described later) on the display device 320, can control the display of alert information (described later) on the display device 320, and can also control the display of camera images captured by the camera 120 of the vehicle 101 on the display device 320.

[0054] For example, in the remote control mode, the camera image captured by the camera 120 of the vehicle 101 is transmitted to the terminal device 301 via the remote control support device 20, and the display control unit 342 displays the camera image received from the remote control support device 20 on the display device 320, thereby allowing the operator to remotely control the vehicle 101 while checking the driving status of the vehicle 101.

[0055] The operation information transmitting unit 343 transmits operation information input in response to an operator's operation of the operation device 330 to the remote operation assistance apparatus 20. For example, the operator operates the operation device 330 to start remote operation of the vehicle 101, and the operation information transmitting unit 343 can transmit the operation information input in response to the operator's operation of the operation device 330 to the remote operation assistance apparatus 20.

[0056] The time information transmission unit 344 transmits time information indicating the time at which the camera image captured by the camera 120 of the vehicle 101 received from the remote operation support device 20 is displayed on the display device 320 to the remote operation support device 20. The time information includes a terminal device ID for identifying the terminal device 301.

[0057] Next, we will explain the configuration of the terminal device 302. As shown in Fig. 4, the terminal device 302 includes, as hardware elements, a communication device 350, a display device 360, an operation device 370, and a control device 380. Note that the hardware elements of the terminal device 302 are not limited to the configuration exemplified in Fig. 4, and may include other hardware elements.

[0058] The communication device 350 is a device that communicates with an external device (for example, the remote operation support device 20, etc.) via the network 40. The display device 360 ​​is a device that displays various types of information and is configured, for example, with a liquid crystal display, etc. The operation device 370 is a device with which an operator performs various operations.

[0059] The control device 380 is a device that comprehensively controls the operation of the terminal device 302. In this embodiment, the control device 380 is configured as a computer device, and has the same hardware configuration as that shown in FIG. 5. Next, the functions of the control device 380 will be described. As shown in FIG. 4, the control device 380 has an information receiving unit 381, a display control unit 382, ​​an operation information transmitting unit 383, and a time information transmitting unit 384. Note that, although the example in FIG. 4 illustrates only the functions necessary for explaining the main parts of this embodiment, the functions of the control device 380 are not limited to these.

[0060] In this embodiment, the functions of the above-described units are realized by the processor 150 executing a program stored in the ROM 151. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0061] The information receiving unit 381 receives various types of information transmitted from the remote operation support device 20. For example, the information receiving unit 381 can receive information such as a camera image captured by the camera 170 of the crane 102 from the remote operation support device 20. Hereinafter, the camera image captured by the camera 170 of the crane 102 may also be simply referred to as an image.

[0062] The display control unit 382 controls the display of various information on the display device 360. For example, For example, the display control unit 382 can control the display of predicted area information, which will be described later, on the display device 360, can control the display of alert information, which will be described later, on the display device 360, and can also control the display of camera images captured by the camera 170 of the crane 102 on the display device 360.

[0063] For example, in the remote operation mode, the camera image captured by the camera 170 of the crane 102 is transmitted to the terminal device 302 via the remote operation support device 20, and the display control unit 382 displays the camera image received from the remote operation support device 20 on the display device 360, thereby allowing the operator to remotely operate the crane 102 while checking the position status of the crane hook of the crane 102.

[0064] The operation information transmitting unit 383 transmits operation information input in response to the operator's operation of the operation device 370 to the remote operation support apparatus 20. For example, the operator operates the operation device 370 to remotely control the crane 102 while checking the camera image captured by the camera 170 of the crane 102 displayed on the display device 360, and the operation information transmitting unit 383 can transmit the operation information input in response to the operator's operation of the operation device 370 to the remote operation support apparatus 20.

[0065] The time information transmitting unit 384 transmits time information indicating the time at which the camera image captured by the camera 170 of the crane 102 received from the remote operation support device 20 is displayed on the display device 360 ​​to the remote operation support device 20. The time information includes a terminal device ID for identifying the terminal device 302.

[0066] Next, we will explain the configuration of the remote operation support device 20. The remote operation support device 20 includes, as hardware elements, a communication device 210, a storage unit 220, and a control device 230. Note that the hardware elements of the remote operation support device 20 are not limited to the configuration exemplified in Fig. 4, and may include other hardware elements.

[0067] The communication device 210 is a device that communicates with external devices (for example, the vehicle 101, the crane 102, the terminal device 30, etc.) via the network 40. The storage unit 220 stores vehicle characteristic information in association with each vehicle 101. The storage unit 220 also stores vehicle characteristic information in association with each crane 102.

[0068] The control device 230 is a device that comprehensively controls the operation of the remote operation support device 20. In this embodiment, the control device 230 is configured as a computer device, and has the same hardware configuration as that shown in FIG.

[0069] Next, a description will be given of the functions of the control device 230 of the remote operation support device 20. As shown in Fig. 4, the control device 230 has a receiving unit 231, a delay information calculation unit 232, a crane hook position calculation unit 233, a predicted area information calculation unit 234, a determination unit 235, an alert information generation unit 236, a support control unit 237, an output unit 238, and a remote information transmission / reception unit 239. Note that, although the example in Fig. 4 illustrates only the functions necessary for explaining the main parts of this embodiment, the functions of the control device 230 are not limited to these.

[0070] In this embodiment, the functions of the above-described units are realized by the processor 150 executing a program stored in the ROM 151. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0071] As described above, the receiving unit 231 receives the vehicle characteristic information transmitted by the vehicle 101. Then, the receiving unit 231 associates the vehicle ID and the vehicle characteristic information received from the vehicle 101 with each other and stores them in the storage unit 220. Also, as described above, the receiving unit 231 receives the crane characteristic information transmitted by the crane 102. Then, the receiving unit 231 associates the crane ID and the vehicle characteristic information received from the crane 102 with each other and stores them in the storage unit 220.

[0072] Furthermore, as described above, the receiving unit 231 receives the camera image captured by the camera 120 and transmitted by the vehicle 101. Then, the receiving unit 231 associates the vehicle ID received from the vehicle 101 with the camera image and stores them in the storage unit 220. Also, as described above, the receiving unit 231 receives the camera image captured by the camera 170 and transmitted by the crane 102. Then, the receiving unit 231 associates the crane ID received from the crane 102 with the camera image and stores them in the storage unit 220.

[0073] Furthermore, as described above, the receiving unit 231 receives the time information transmitted by the terminal device 301 and the terminal device 302. Then, the receiving unit 231 stores the terminal device IDs and the time information received from the terminal device 301 and the terminal device 302 in association with each other in the storage unit 220.

[0074] In this embodiment, the receiving unit 231 has a function of receiving vehicle characteristic information, a function of storing the vehicle characteristic information in the memory unit 220, a function of receiving crane characteristic information, a function of storing the crane characteristic information in the memory unit 220, a function of receiving camera images, a function of storing the camera images in the memory unit 202, a function of receiving time information, and a function of storing the time information in the memory unit 202. However, this is not limited to this, and for example, the function of receiving vehicle characteristic information, the function of storing the vehicle characteristic information in the memory unit 220, the function of receiving crane characteristic information, the function of storing the crane characteristic information in the memory unit 220, the function of receiving images, the function of storing the images in the memory unit 202, the function of receiving time information, and the function of storing the time information in the memory unit 202 may each be provided separately.

[0075] The delay information calculation unit 232 calculates delay information indicating the time from when an image is captured until when it is drawn. Specifically, the delay information calculation unit 232 calculates delay information indicating the delay time required for the image to be drawn on the display device 320 of the terminal device 301, starting from the time when the image is captured by the camera 120 of the vehicle 101. The delay information calculation unit 232 calculates the delay information by taking the difference between the time included in the time information and the time included in the image information. The delay information will now be described with reference to FIG. 6. FIG. 6 is a table for explaining the delay information according to the embodiment.

[0076] Table T1 shown in Fig. 6 shows a video frame ID indicating an identifier of a camera image, a video acquisition time indicating the time the camera image was acquired, and a video rendering time indicating the time the image was rendered on the display device 320 of the terminal device 301. The delay information calculation unit 232 acquires the video acquisition time and the video rendering time for each video frame ID, and calculates the delay information by taking the difference between the video rendering time and the video acquisition time. For example, the delay information shown in Fig. 6 is [video rendering time (= yyyy-mm-ddThh:mm:21Z)] - [video acquisition time (= yyyy-mm-ddThh:mm:01Z)] = 20 seconds.

[0077] In the above description, the delay information calculation unit 232 calculates the delay information using the time when the camera image from the camera 120 of the vehicle 101 is acquired and the time when the image is drawn on the display device 320 of the terminal device 301. However, the delay information calculation unit 232 may calculate the delay information using the time when the camera image from the camera 170 of the crane 102 is acquired and the time when the image is drawn on the display device 360 ​​of the terminal device 302.

[0078] 4, the crane hook position calculation unit 233 calculates crane hook position information indicating the position of the crane hook using the camera image captured by the camera 120 of the vehicle 101. Here, a method for calculating the crane hook position information will be described with reference to FIGS. 7 and 8.

[0079] Fig. 7 is a schematic diagram for explaining the position of the crane hook according to the embodiment. Fig. 7 shows crane hook area 51, crane hook center position 52, vehicle 101, crane 102, camera 120, optical axis center C1 indicating the center of the optical axis of camera 120, distance R [m] indicating the distance from the position of camera 120 as the base point to crane hook center position 52, and azimuth angle θ [deg] from the position on the camera image of optical axis center C1 as the origin to crane hook center position 52 with respect to optical axis center C1.

[0080] Fig. 8 is a schematic diagram showing an example of a camera image according to the embodiment. A camera image 61 shown in Fig. 8 shows a crane hook area 51, a center position 52 of the crane hook, an azimuth angle θ, and an azimuth angle Φ [deg] from a position on the camera image where the position of the optical axis center C1 is the origin to the center position 52 of the crane hook relative to the optical axis center C1.

[0081] 7 and the drawings described below, the mutually perpendicular X-axis, Y-axis, and Z-axis respectively refer to the traveling direction of the vehicle 101, the horizontal direction of the crane boom of the crane 102, and the up-down direction of the crane hook of the crane 102. Furthermore, in the following description, when simply referred to as the X-direction, Y-direction, or Z-direction, it refers to the respective axial direction and includes the two opposite directions.

[0082] Furthermore, if the direction is identified as the positive direction of the X axis, it is the direction in which the vehicle 101 approaches the crane 102; if the direction is identified as the positive direction of the Y axis, it is the direction in which the crane boom of the crane 102 moves toward the vehicle 101; and if the direction is identified as the positive direction of the Z axis, it is the direction in which the crane hook of the crane 102 moves from below to above.

[0083] Furthermore, when the negative direction of the X axis is identified, it is a backward direction in which the vehicle 101 moves backward toward the crane 102; when the negative direction of the Y axis is identified, it is a direction in which the crane boom of the crane 102 moves away from the vehicle 101; and when the negative direction of the Z axis is identified, it is a direction in which the crane hook of the crane 102 moves from above to below.

[0084] First, the crane hook position calculation unit 233 identifies the area 51 of the crane hook of the crane 102 from the camera image captured by the camera 120 transmitted by the vehicle 101 and received by the receiving unit 231. Here, the crane hook area 51 refers to the area of ​​the crane hook and the container lifted by the crane hook. The crane hook position calculation unit 233 identifies the area 51 of the crane hook of the crane 102 from the camera image 61 using a known image recognition technique (see FIG. 8).

[0085] Furthermore, the crane hook position calculation unit 233 sets the center of the identified crane hook area to crane hook center position 52. Furthermore, the crane hook position calculation unit 233 calculates the relative orientation (θ, Φ) of the crane hook with respect to the optical axis center C1 from the position when the position of the optical axis center C1 on the image is set as the origin. Then, using the crane hook distance calculation information stored in the storage unit 220, the crane hook position calculation unit 233 calculates the distance R from the camera 120 of the vehicle 101 to the crane hook center position 52 from the crane hook relative orientation (θ, Φ) and the width W of the crane hook area 51 on the camera image 61.

[0086] Here, the crane hook distance calculation information will be explained using Fig. 9. Fig. 9 is a table showing an example of the crane hook distance calculation information according to the embodiment. Table T2 shown in Fig. 9 is a table in which the relative orientation (θ, Φ), the width W of the crane hook area 51 on the camera image 61, and the distance R are associated with each other. As a result, the crane hook position calculation unit 233 calculates the distance to the camera 120 of the vehicle 101 from the size of the crane hook area 51.

[0087] Returning to Figure 7, the crane hook position calculation unit 233 calculates the three-dimensional center position 52 (X, Y, Z) [unit: m] of the crane hook from the elevation angle, azimuth angle, and distance when the crane hook is viewed from the camera 120 of the vehicle 101 using the following formula. X=x+Rcos(θ+ω)cos(Φ+ψ) Y=y+Rcos(θ+ω)sin(Φ+ψ) Z=z+Rsin(θ+ω)

[0088] Here, the three-dimensional position of the camera 120 of the vehicle 101 is defined as (x, y, z) [unit: m]. In addition, in the above-mentioned Cartesian coordinate system, the azimuth angle of the camera 120 of the vehicle 101 is defined as ψ [deg], and the elevation angle of the camera 120 of the vehicle 101 is defined as ω [deg]. The azimuth angle ψ is defined as a positive rotation in the Y-axis direction when the X-axis is 0 degrees. The elevation angle ω [deg] is defined as a positive rotation in the Z-axis direction when the X-axis is 0 degrees. The coordinate system consisting of the distance R, the azimuth angle ψ, and the elevation angle ω is also referred to as a polar coordinate system centered on the camera 120 of the vehicle 101.

[0089] The center position 52 of the crane hook may be calculated from the results of image processing such as extraction of the crane area by deep learning.

[0090] 4, the predicted area information calculation unit 234 calculates predicted area information indicating an area in the image where it is predicted that the hook may be present, based on delay information indicating the time from when the image is captured until it is drawn, and the position of the crane hook that is the remotely operated target and is reflected in the image. Specifically, the predicted area information calculation unit 234 calculates predicted area information indicating an area in the image where it is predicted that the crane hook may be present, based on the delay information calculated by the delay information calculation unit 232 and the position of the crane hook included in the crane hook position information calculated by the crane hook position calculation unit 233.

[0091] For example, the prediction area information calculation unit 234 acquires the delay information calculated by the delay information calculation unit 232. The prediction area information calculation unit 234 also acquires the position (X, Y, Z) of the crane hook calculated by the crane hook position calculation unit 233. Furthermore, the prediction area information calculation unit 234 applies a known Kalman filter to acquire error ellipsoids in multiple confidence intervals centered on the estimated position of the crane hook after the delay time included in the delay information. Note that, when crane hook control information such as hoisting speed information indicating the hoisting speed of the crane hook can be acquired, the prediction area information calculation unit 234 may calculate a predicted value for the presence of the crane hook from the control information and correct the estimated position of the crane hook using the calculated predicted value.

[0092] Then, the prediction area information calculation unit 234 calculates an error ellipsoid in a polar coordinate system centered on the camera 120 of the vehicle 101, which corresponds to the plurality of acquired error ellipsoids, using the plurality of acquired error ellipsoids and the distance R calculated by the crane hook position calculation unit 233. The error ellipsoid in the polar coordinate system centered on the camera 120 of the vehicle 101 is an example of the above-mentioned prediction area information. Here, the prediction area information will be described with reference to FIG. 10.

[0093] Fig. 10 is a schematic diagram showing an example of prediction area information according to the embodiment. Fig. 10 is a schematic diagram in which multiple pieces of prediction area information are superimposed on the schematic diagram shown in Fig. 7. Fig. 10 shows the crane hook area 51, the vehicle 101, the crane 102, the camera 120, the optical axis center C1, the distance R [m], the prediction area information A1, and the prediction area information A2. The prediction area information A1 and the prediction area information A2 are both error ellipsoids in multiple confidence intervals centered on the estimated position of the crane hook after the delay time included in the delay information in a polar coordinate system centered on the camera 120 of the vehicle 101.

[0094] Here, the confidence interval indicates, for example, the probability that the position of the crane hook is within that interval. The size of the error ellipsoid varies depending on the probability. The relationship between the confidence interval and the error ellipsoid is that the larger the confidence interval, the larger the size of the ellipsoid. In other words, the larger the probability value of the confidence interval, the greater the rate at which the ellipsoid grows relative to the rate at which the probability value of the confidence interval increases. For example, the size of the error ellipsoid for a 95% confidence interval is twice the size of the error ellipsoid for a 68% confidence interval. Also, for example, the size of the error ellipsoid for a 99.5% confidence interval is 1.5 times the size of the error ellipsoid for a 95% confidence interval.

[0095] The predicted area information A1 shown in FIG. 10 is set to a lower probability than the predicted area information A2, so the size of the predicted area information A1 is smaller than the size of the predicted area information A2. In other words, the predicted area information A2 indicates an area where the probability of the crane hook being present is higher than the area indicated by the predicted area information A1. The predicted area information A1 is an example of first predicted area information. The predicted area information A2 is an example of second predicted area information.

[0096] Returning to Fig. 4, the prediction area information calculation unit 234 calculates an error ellipsoid on the camera image using the calculated error ellipsoid in a polar coordinate system centered on the camera 120 of the vehicle 101. The error ellipsoid on the camera image is an example of the above-mentioned prediction area information. Here, the prediction area information will be described with reference to Fig. 11.

[0097] Fig. 11 is a schematic diagram showing an example of prediction area information according to an embodiment. Fig. 11 is a schematic diagram in which a plurality of pieces of prediction area information are superimposed on the schematic diagram shown in Fig. 8. A camera image 61 shown in Fig. 11 shows prediction area information A3 and prediction area information A4. The prediction area information A3 corresponds to the prediction area information A1 shown in Fig. 10. The prediction area information A4 corresponds to the prediction area information A2 shown in Fig. 10. Furthermore, the prediction area information A3 and prediction area information A4 superimposed on the camera image 61 reflect the time from when the image is captured to when it is drawn, and since they lag behind the actual crane hook position, it can be seen that there is a discrepancy between the position of the ellipsoid center and the position of the crane hook.

[0098] 4, the determination unit 235 determines whether there is a possibility of collision between the area indicated by the prediction area information and a peripheral object. Specifically, the determination unit 235 determines whether there is a possibility of collision between the area indicated by the prediction area information calculated by the prediction area information calculation unit 234 and a peripheral object. Here, the possibility of collision will be described with reference to FIG. 12.

[0099] Fig. 12 is a schematic diagram showing the positional relationship between the area of ​​the predicted area information and peripheral objects according to the embodiment. Fig. 12 corresponds to the schematic diagram shown in Fig. 10, and shows the crane hook area 51, the vehicle 101, the crane 102, the camera 120, the predicted area information A1, the predicted area information A2, the distance H1 between the lowest point of the area of ​​the predicted area information A1 and the ground, the distance H2 between the lowest point of the area of ​​the predicted area information A2 and the ground, and the vehicle height H3 of the vehicle 101. The distance H1 is greater than the distance H2. In Fig. 12, the peripheral object will be described as the vehicle 101, but the peripheral object is not limited thereto.

[0100] For example, the determination unit 235 determines the possibility of a collision between the vehicle 101 and the area indicated by the prediction area information, using the distance from the ground of the area indicated by the prediction area information and the vehicle height of the vehicle 101. Specifically, the determination unit 235 acquires the prediction area information A1 and the prediction area information A2 calculated by the prediction area information calculation unit 234. The determination unit 235 also calculates the distance between the lowest point of each piece of acquired prediction area information and the ground. Furthermore, the determination unit 235 acquires vehicle height information related to the vehicle height of the vehicle 101 stored in the storage unit 220. FIG. 13 is a table illustrating an example of vehicle height information according to the embodiment. Table T3 illustrated in FIG. 13 associates vehicle IDs with vehicle heights.

[0101] 12, the determination unit 235 uses the distance from the ground of the area indicated by the prediction area information and the vehicle height of the vehicle 101 to determine whether there is a possibility of collision between the area indicated by the prediction area information and the vehicle 101. Specifically, if the distance H1 is smaller than the vehicle height H3, the determination unit 235 determines that there is a possibility of collision between the area indicated by the prediction area information A1 and the vehicle 101. Furthermore, if the distance H1 is greater than the vehicle height H3, the determination unit 235 determines that there is no possibility of collision between the area indicated by the prediction area information A1 and the vehicle 101.

[0102] Furthermore, if the distance H2 is smaller than the vehicle height H3, the determination unit 235 determines that there is a possibility of a collision between the area indicated by the prediction area information A2 and the vehicle 101. If the distance H2 is greater than the vehicle height H3, the determination unit 235 determines that there is no possibility of a collision between the area indicated by the prediction area information A2 and the vehicle 101.

[0103] Returning to FIG. 4, when the determination unit 235 determines that there is a possibility of collision between the area indicated by the prediction area information and a surrounding object, the alert information generation unit 236 generates alert information indicating that there is a possibility of collision. For example, when the distance H1 shown in FIG. 12 is smaller than the vehicle height H3, the alert information includes a warning sentence such as "[Warning] High possibility of collision." When the distance H2 shown in FIG. 12 is smaller than the vehicle height H3, the alert information includes a warning sentence such as "Possibility of collision." The alert information differs depending on each prediction area information because the prediction area information A1 is set to have a higher probability than the prediction area information A2.

[0104] The assistance control unit 237 performs control according to the positional relationship between the area indicated by the predicted area information and a peripheral object indicating an object around the crane hook. Specifically, the assistance control unit 237 restricts remote operation when there is a possibility of a collision between the area indicated by the predicted area information and the peripheral object. For example, when the peripheral object is a vehicle 101, the assistance control unit 237 performs control to restrict remote operation by prohibiting the vehicle 101 from entering below the crane hook.

[0105] Furthermore, the assist control unit 237 performs different control depending on the positional relationship between the area indicated by the first predicted area information and the surrounding objects than the control depending on the positional relationship between the area indicated by the second predicted area information and the surrounding objects. Specifically, the assist control unit 237 restricts remote operation when there is a possibility of collision between the vehicle 101 and the area indicated by predicted area information A1 shown in Fig. 12. Furthermore, the assist control unit 237 outputs a warning when there is a possibility of collision between the vehicle 101 and the area indicated by predicted area information A2 shown in Fig. 12.

[0106] The output unit 238 outputs the predicted area information calculated by the predicted area information calculation unit 234 to the terminal device 301. The output unit 238 also outputs the alert information generated by the alert information generation unit 236 to the terminal device 301. Furthermore, the output unit 238 causes the assistance control unit 237 to output to the terminal device 301 control information corresponding to control in accordance with the positional relationship between the area indicated by the predicted area information and peripheral objects indicating objects around the crane hook.

[0107] Here, various types of information output by the output unit 238 will be described with reference to Figs. 14, 15, and 16. Figs. 14, 15, and 16 are schematic diagrams showing examples of display screens according to an embodiment. The predicted information in Figs. 14, 15, and 16 corresponds to the predicted information shown in Fig. 11. The display screens in Figs. 14, 15, and 16 are schematic diagrams viewed from the positive direction of the X-axis, which is the traveling direction of the vehicle 101.

[0108] 14 displays predicted area information A3, predicted area information A4, a message M1 indicating the vehicle ID "51" of vehicle 101, a message M2 indicating the current status of vehicle 101, and a message M3 indicating whether vehicle 101 needs to enter under the crane hook. Display screen 321 shows that there is no possibility of a collision between the area indicated by the predicted area information and vehicle 101, and there is no alert information.

[0109] Therefore, the message M2 is an example including "Waiting," which is an example of a sentence indicating a state in which the vehicle 101 is waiting to enter under the crane hook of the vehicle 101. The message M3 is an example including "Entry permitted," which is an example of a sentence indicating a state in which the vehicle 101 is able to enter under the crane hook. Here, the operator can remotely operate the vehicle 101 by pressing the message M3 indicating "Entry permitted."

[0110] 15 shows a message M4 in which the message M3 has been updated on the display screen 321 shown in Fig. 14. The display screen 322 indicates that there is a possibility of a collision between the area indicated by the predicted area information A4 and the vehicle 101, and alert information exists. Therefore, the message M4 is an example including "Possible collision" and "Stop", which are examples of warning sentences indicating that the vehicle 101 cannot enter under the crane hook. Here, the message M4 indicates a state in which remote operation has been restricted by the assistance control unit 237.

[0111] 16 shows a message M5 in which the message M3 has been updated on the display screen 321 shown in FIG. 14. The display screen 323 indicates that there is a possibility of a collision between the area indicated by the predicted area information A3 and the vehicle 101, and alert information is present. Therefore, the message M5 is an example including "[Warning] High possibility of collision," which is an example of a warning message indicating that the vehicle 101 cannot enter under the crane hook. Here, the message M5 indicates a state in which "Entry Permitted" in the message M3 shown in FIG. 14 has been deactivated, and remote operation by the operator is not being accepted, and the message M5 indicates a state in which remote operation has been restricted by the assistance control unit 237.

[0112] 4, the remote information transmitting / receiving unit 239 transmits and receives remote information indicating information used for remotely operating the vehicle 101. The remote information includes, for example, operation information transmitted from the terminal device 301, images captured by the camera 120 of the vehicle 101, and other information.

[0113] For example, the remote information transmitting / receiving unit 239 can receive operation information transmitted from the terminal device 301 and transmit the received operation information to the vehicle 101. Furthermore, as described above, while the vehicle 101 is traveling after the service operation has started, images captured by the camera 120 of the vehicle 101 are sent to the remote operation assistance device 20, so the remote information transmitting / receiving unit 239 can also transmit images received from the vehicle 101 to the terminal device 301.

[0114] Furthermore, the remote information transmitting / receiving unit 239 transmits and receives remote information indicating information used for remotely operating the crane 102. The remote information includes, for example, operation information transmitted from the terminal device 302, images captured by the camera 170 of the crane 102, and other information.

[0115] For example, the remote information transmitting / receiving unit 239 can receive operation information transmitted from the terminal device 302 and transmit the received operation information to the crane 102. Furthermore, as described above, while the crane 102 is in operation after the start of service operation, images captured by the camera 170 of the crane 102 are sent to the remote operation support device 20, so the remote information transmitting / receiving unit 239 can also transmit images received from the crane 102 to the terminal device 302.

[0116] Next, an example of an operation procedure of the remote operation support device 20 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing an example of processing of the remote operation support device 20 according to the embodiment.

[0117] The receiving unit 231 receives the camera image captured by the camera 120 of the vehicle 101 and transmitted therefrom (step S21). The delay information calculation unit 232 calculates delay information indicating the delay time required for the image to be displayed on the display device 320 of the terminal device 301, based on the time when the image was captured by the camera 120 of the vehicle 101 (step S22). The crane hook position calculation unit 233 calculates crane hook position information indicating the position of the crane hook using the camera image captured by the camera 120 of the vehicle 101 (step S23).

[0118] The predicted area information calculation unit 234 calculates predicted area information indicating an area in the image where it is predicted that the crane hook may be present, based on the delay information calculated by the delay information calculation unit 232 and the crane hook position included in the crane hook position information calculated by the crane hook position calculation unit 233 (step S24). The output unit 238 outputs the predicted area information calculated by the predicted area information calculation unit 234 to the terminal device 301 (step S25).

[0119] The determination unit 235 determines whether there is a possibility of collision between the area indicated by the prediction area information calculated by the prediction area information calculation unit 234 and a peripheral object (step S26). If the determination unit 35 determines that there is no possibility of collision between the area indicated by the prediction area information and a peripheral object (step S26: No), this process ends. On the other hand, if the determination unit 35 determines that there is a possibility of collision between the area indicated by the prediction area information and a peripheral object (step S26: Yes), the process proceeds to step S27.

[0120] In step S27, if the determination unit 235 determines that there is a possibility of collision between the area indicated by the predicted area information and a peripheral object, the alert information generation unit 236 generates alert information indicating that there is a possibility of collision (step S27). If there is a possibility of collision between the area indicated by the predicted area information and a peripheral object, the assistance control unit 237 restricts remote operation (step S28).

[0121] The output unit 238 outputs the alert information generated by the alert information generation unit 236 to the terminal device 301 (step S29). Furthermore, the output unit 238 outputs control information corresponding to control according to the positional relationship between the area indicated by the predicted area information and the peripheral objects indicating the objects around the crane hook to the terminal device 301 via the assistance control unit 237. Then, when step S28 ends, this routine ends.

[0122] As described above, the remote operation support device 20 of this embodiment outputs predicted area information indicating an area in the image where it is predicted that the crane hook may be present, based on delay information indicating the time from when the image is captured until when it is drawn, and the position of the crane hook of the crane to be remotely operated that is reflected in the image. In this way, by outputting the predicted area information, the remote operation support device 20 allows the operator to grasp the position of the crane hook taking into account the time from when the image is captured until it is drawn on the terminal device 301.

[0123] Furthermore, the remote operation support device 20 performs control according to the positional relationship between the area indicated by the predicted area information and the surrounding objects indicating the objects around the crane hook, and restricts remote operation if there is a possibility of a collision between the area indicated by the predicted area information and the surrounding objects. As a result, the remote operation support device 20 provides control according to the distance between the crane hook and the surrounding objects, allowing the operator to control the crane hook to prevent a collision between the crane hook and the surrounding objects. Therefore, according to this embodiment, the operator can achieve safe remote operation.

[0124] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims.

[0125] Furthermore, the effects of the embodiments described in this specification are merely examples and are not limiting, and other effects may also be achieved. Modifications will be described below.

[0126] (First Modification) The display screen in the above-described embodiment is a schematic diagram viewed from the positive direction of the X-axis, which is the traveling direction of the vehicle 101. However, this is not limited to this, and may be a schematic diagram viewed from, for example, the Y-axis. FIG. 18 is a schematic diagram showing an example of a display screen according to the first modified example. A display screen 324 shown in FIG. 18 shows the vehicle 101, the crane 102, predicted area information A1, and predicted area information A2. The predicted information shown in FIG. 18 corresponds to FIG. 12.

[0127] 18 also includes a height (5 m in FIG. 18) indicating the vehicle height of vehicle 101. Furthermore, FIG. 18 also includes "Waiting for entry permission," which is an example of a sentence indicating the current state of vehicle 101, indicating that the vehicle is waiting for entry permission to enter under the crane hook of vehicle 101.

[0128] (Second Modification) For example, the display screen may be a schematic diagram viewed from the negative direction of the X axis. Fig. 19 is a schematic diagram showing an example of a display screen according to a second modified example. A display screen 325 shown in Fig. 19 shows the crane 102, predicted area information A1, and predicted area information A2. The predicted information shown in Fig. 19 corresponds to Fig. 12. Fig. 19 also includes coordinate information (3m, 5m) indicating the position of the crane hook in two dimensions on the YZ axes, when the position of the support structure L1 that supports the crane 102 from the ground is set as the origin.

[0129] (Third Modification) For example, the remote operation support device 20 may identify surrounding objects present in the area indicated by the predicted area information based on a camera image captured by the camera 170 of the crane 102 and position information of the vehicle 101. Fig. 20 is a diagram showing an example of the configuration of each of a plurality of devices included in the remote operation system 1 according to the third modified example. The control device 230 of the remote operation support device 20 according to the third modified example has a surrounding object detection unit 240 and a surrounding object identification unit 241 in addition to the functions of the control device 230 shown in Fig. 4 described above.

[0130] The receiving unit 231 sequentially acquires position information indicating the position of the vehicle 101 and stores it in the storage unit 220. The surrounding object detection unit 240 detects object position / orientation type information including the position, orientation, and type of an object present in the area indicated by the prediction information from the camera image captured by the camera 170 of the crane 102. The object position / orientation type information is expressed as (x v ,y v ,ψ v , type), where x v ,y v is expressed as a two-dimensional position on the ground [m], and ψ v indicates the X-axis as 0 degrees, the Y-axis as positive as rotation in the azimuth angle [deg], and type indicates the type of vehicle. Based on the position information indicating the position of the vehicle 101 stored in the memory unit 220 and the object position / orientation type information detected by the peripheral object detection unit 240, the peripheral object identification unit 241 compares the position information with the object position / orientation type information, and identifies peripheral objects present in the area indicated by the prediction area information.

[0131] Fig. 21 is a table showing an example of vehicle position information according to Modification 3. Table T4 shown in Fig. 21 associates a vehicle ID with the two-dimensional position (x, y) of the vehicle 101 on the ground and an azimuth angle ψ.

[0132] Fig. 22 is a schematic diagram for explaining peripheral objects according to the third modified example. Fig. 22 shows predicted area information A5, predicted area information A6, a vehicle 1011, and a vehicle 1012. For example, the object position / orientation type information of the vehicle 1011 shown in Fig. 22 is (12.3, 34.5, 0.1, Truck), and the object position / orientation type information of the vehicle 1012 is (10.2, 30.5, 30, Truck).

[0133] In the case of Figures 21 and 22, the surrounding object identification unit 241 compares the position information indicating the position of the vehicle 101 stored in the memory unit 220 shown in Figure 21 with the object position / orientation type information detected by the surrounding object detection unit 240 in Figure 22, and identifies the surrounding object vehicle 1011 located in the area indicated by the prediction area information as "a vehicle with a vehicle ID of 10."

[0134] Then, the determination unit 235 determines whether there is a possibility of collision between the area indicated by the prediction area information and the peripheral object identified by the peripheral object identification unit 241. For example, when a peripheral object exists in the area indicated by the prediction area information, the determination unit 235 determines that there is a possibility of collision between the area indicated by the prediction area information and the peripheral object.

[0135] Furthermore, when the determination unit 235 determines that there is a possibility of a collision between the area indicated by the prediction area information and a surrounding object, the support control unit 237 restricts the remote operation of the crane boom of the crane 102. For example, as a control to restrict the remote operation, the support control unit 237 performs control to restrict the maximum speed of the boom, which is an arm portion to which a crane hook is attached and which is capable of rotating. For example, as a control to restrict the remote operation, the support control unit 237 performs control to prohibit operation of the crane boom in a direction approaching the vehicle 101.

[0136] Furthermore, when the determination unit 235 determines that there is a possibility of a collision between the area indicated by the predicted area information and a surrounding object, the assistance control unit 237 establishes a call connection between the terminal device 301 and the terminal device 302. For example, when the determination unit 235 determines that there is a possibility of a collision between the area indicated by the predicted area information and the vehicle 101, the assistance control unit 237 identifies the terminal device 301 that remotely controls the vehicle 101, and establishes a call connection between the identified terminal device 301 and the terminal device 302 that remotely controls the crane 102.

[0137] 23 and 24 are schematic diagrams showing examples of display screens according to the third modified example. Display screen 326 shown in Fig. 23 is a display screen displayed on display device 320 of terminal device 301. Display screen 327 shown in Fig. 24 is a display screen displayed on display device 360 ​​of terminal device 302. It is assumed that display screen 326 and display screen 327 are displayed on the respective terminal devices at the same time.

[0138] 23 shows a message M21 in which the message M2 has been updated from the display screen 322 shown in Fig. 15, and an icon M6 that allows communication with the terminal device 302 that supports the remote operation of the currently operating crane 102. The message M21 is an example that includes the phrase "driving," which is an example of a sentence that indicates a state in which the vehicle 101 is entering under the crane hook.

[0139] The display screen 327 shown in Figure 24 displays predicted area information A5, predicted area information A6, a message M11 indicating the crane ID of the crane 102, a message M12 indicating the current status of the crane 102, a message M13 indicating alert information for the crane 102, and an icon M6 that allows communication with a terminal device 301 that assists in the remote operation of the vehicle 101 currently in motion.

[0140] Message M12 is an example of a sentence including "in operation," which is an example of a sentence indicating that crane 102 is currently operating. Message M13 is an example of a warning sentence including "possibility of collision with car 51," which is an example of a warning sentence indicating that vehicle 101 cannot enter under the crane hook.

[0141] Fig. 25 is a flowchart showing an example of the processing of the remote operation support device 20 according to the third modified example. Note that the processing flow described below is an example, and it is possible to change the processing order, delete some of the processing, or add other processing. Note that in Fig. 25, the description of the same processing as in Fig. 17 will be omitted.

[0142] In step S31, the surrounding object identification unit 241 compares the position information indicating the position of the vehicle 101 stored in the memory unit 220 with the object position / orientation type information detected by the surrounding object detection unit 240, and identifies surrounding objects present in the area indicated by the predicted area information (step S31).

[0143] In step S32, the determination unit 235 determines whether there is a possibility of collision between the area indicated by the prediction area information and the peripheral object identified by the peripheral object identification unit 241 (step S32). If the determination unit 235 determines that there is no possibility of collision between the area indicated by the prediction area information and the peripheral object identified by the peripheral object identification unit 241 (step S32: No), this processing ends. On the other hand, if the determination unit 235 determines that there is a possibility of collision between the area indicated by the prediction area information and the peripheral object identified by the peripheral object identification unit 241 (step S32: Yes), the processing proceeds to step S27.

[0144] In step S33, if the determination unit 235 determines that there is a possibility of collision between the area indicated by the prediction area information and a surrounding object, the assistance control unit 237 restricts the remote operation of the crane boom of the crane 102 (step S33). In step S34, the assistance control unit 237 identifies the terminal device 301 that remotely controls the vehicle 101, and establishes a call connection between the identified terminal device 301 and the terminal device 302 that remotely controls the crane 102 (step S34).

[0145] (Fourth Modification) For example, the alert information generation unit 236 may modify the generated alert information according to the state of the vehicle 101. Furthermore, the warning may be changed according to the positional relationship between the area indicated by the second predicted area information and surrounding objects. Fig. 26 is a table for explaining an example of alert information according to the fourth modified example.

[0146] Table T5 shown in Figure 26 is a table that associates the status of the entering vehicle, which indicates the entering vehicle 101, alert information to be displayed on the terminal device 301 that assists in the remote operation of the entering vehicle, and alert information to be displayed on the terminal device 302 that assists in the remote operation of the crane 102.

[0147] For example, when the state of the entering vehicle is waiting to enter below the crane hook of the crane 102, the alert information generating unit 236 generates alert information for the terminal device 301 that includes a warning sentence saying, "Entry not permitted." When the state of the entering vehicle is waiting to enter below the crane hook of the crane 102, the alert information generating unit 236 generates alert information for the terminal device 302 that includes a warning sentence saying, "Please pull up the hook."

[0148] For example, when the state of the approaching vehicle is below the crane hook of the crane 102 and waiting to load a container, the alert information generating unit 236 generates alert information for the terminal device 301 that includes a warning sentence saying, "Departure is not permitted." When the state of the approaching vehicle is below the crane hook of the crane 102 and waiting to load a container, the alert information generating unit 236 generates alert information for the terminal device 302 that includes a warning sentence saying, "Operate with caution as a vehicle is approaching."

[0149] For example, if the state of the entering vehicle is traveling below the crane hook of the crane 102 or other, the alert information generating unit 236 generates alert information for the terminal device 301 that includes the warning sentence "Please stop." If the state of the entering vehicle is traveling below the crane hook of the crane 102 or other, the alert information generating unit 236 generates alert information for the terminal device 302 that includes the warning sentence "Please pull up the hook."

[0150] The alert information generating unit 236 may change the type of alert information according to the horizontal positional relationship between the approaching vehicle and the crane (for example, the distance between the approaching vehicle and the crane), without being limited to the vehicle state. Furthermore, the alert information generating unit 236 may associate information regarding output, such as whether to display or hide the generated alert information, with each of the terminal device 301 and the terminal device 302.

[0151] Fig. 27 is a flowchart showing an example of the processing of the remote operation support device according to the fourth modified example. Note that the processing flow described below is an example, and it is possible to change the processing order, delete some of the processing, or add other processing. Note that in Fig. 27, the description of the same processing as in Fig. 17 will be omitted.

[0152] In step S41, the alert information generation unit 236 may modify the generated alert information in accordance with the state of the vehicle 101. In step S42, the output unit 238 outputs the generated alert information modified by the alert information generation unit 236 to the terminal device 301 and the terminal device 302 (step S42).

[0153] (Fifth Modification) For example, in the above-described embodiment, the assistance control unit 237 restricts remote operation when there is a possibility of a collision between the area indicated by the first predicted area information and a surrounding object. In the fifth modified example, when the surrounding object is an autonomous vehicle, the assistance control unit 237 may perform control to restrict the autonomous driving by preventing the autonomous vehicle from entering the area indicated by the first predicted area information.

[0154] 28 and 29 are schematic diagrams for explaining control for restricting autonomous traveling according to Modification 5. Fig. 28 shows the state of vehicle 101 before the autonomous traveling is restricted, and Fig. 29 shows the state of vehicle 101 after the autonomous traveling is restricted.

[0155] Fig. 28 shows a vehicle 1023, predicted area information A7, predicted area information A8, and a planned approach route R1 for the vehicle 1023. Here, the predicted area information A7 is first predicted area information, and the predicted area information A8 is second predicted area information. Fig. 29 shows the vehicle 1023, predicted area information A7, predicted area information A8, and an avoidance route R2 for the vehicle 1023. Note that the vehicle 1023 shown in Figs. 28 and 29 may be a peripheral object identified by the peripheral object identification unit 241 described above.

[0156] If the vehicle 1023 travels along the planned entry route R1 shown in Fig. 28, it may enter the area of ​​the predicted area information A7 and result in a collision. In this case, the assistance control unit 237 generates an avoidance route R2 as control to limit the autonomous traveling, that is, control to avoid the vehicle 1023 entering the area indicated by the predicted area information A7. This allows the vehicle 1023 to avoid entering the area indicated by the predicted area information A7 of the vehicle 1023, and to travel along the avoidance route R2. Note that the control performed by the assistance control unit 237 to limit the autonomous traveling is not limited to generating the avoidance route R2, and may also be control to stop the vehicle 1023, for example.

[0157] (Sixth Modification) The vehicle 101 in the above-described embodiment has been described as being remotely controlled, but the vehicle 101 is not limited to this, and may be applied to a vehicle 101 equipped with a terminal device capable of communicating with the remote operation support device 20. In other words, the vehicle 101 is controlled by a driver to be able to drive, without being autonomous.

[0158] Fig. 30 is a schematic diagram showing an example of a display screen according to the sixth modified example. A display screen 328 according to the sixth modified example is a screen displayed on a terminal device that is mounted on the vehicle 101 and is capable of communicating with the remote operation support device 20. The display screen 328 shown in Fig. 30 displays a message M31 indicating the name of the operator operating the crane 102, a message M32 indicating the current status of the vehicle 101, a message M33 indicating the next destination of the vehicle 101, a schematic diagram M34 including predicted area information for the vehicle 101 and the crane 102, a schematic diagram M35 indicating the current travel route of the vehicle 101, and a message M36 indicating whether the vehicle 101 needs to enter under the crane hook.

[0159] Message M31 is an example of a sentence including "Name of operator of crane A," which is an example of a sentence indicating the name of the operator operating crane 102. Message M32 is an example of a sentence including "Vehicle status: Waiting for entry permission," which is an example of a sentence indicating the current status of vehicle 101. Message M33 is an example of a sentence including "Next destination: Under crane A," which is an example of a sentence indicating the next destination of vehicle 101. Message M36 is an example of a sentence including "Possibility of collision," which is an example of a sentence indicating whether vehicle 101 needs to enter under the crane hook.

[0160] (Seventh Modification) The crane 102 in the above-described embodiment has been described as being intended for remote operation, but the crane 102 is not limited to this, and may be applied to a crane 102 equipped with a terminal device capable of communicating with the remote operation support device 20. In other words, the crane 102 is operably controlled by an operator from an operation room provided for the crane 102.

[0161] Fig. 31 is a schematic diagram showing an example of a display screen according to the seventh modification. A display screen 329 according to the seventh modification is a screen displayed on a terminal device that is mounted in the operation room of the crane 102 and is capable of communicating with the remote operation support device 20. A display screen 328 shown in Fig. 31 displays a message M41 indicating the name of the operator operating the vehicle 101, a schematic diagram M42 indicating the current travel route of the vehicle 101, and a message M43 indicating alert information for the crane 102.

[0162] The message M41 is an example including "Vehicle 10 Operator Name," which is an example of a sentence indicating the name of the operator operating the vehicle 101. The message M43 is an example including "Possibility of Collision," which is an example of a warning sentence indicating alert information for the crane 102.

[0163] (Eighth Modification) In the above-described embodiment, the crane 102 is described as a gantry crane, but the present invention is not limited to this and may be, for example, a truck crane. If the crane 102 is a trek crane, the alert information generator 236 may modify the generated alert information in accordance with the state of the truck crane.

[0164] Fig. 32 is a table for explaining an example of alert information according to Modification 8. Table T7 shown in Fig. 32 associates the state of the crane 102 with alert information to be displayed on the terminal device 302 that supports the remote operation of the crane 102.

[0165] For example, when the state of the crane 102 is that the outriggers are not deployed, the alert information that the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "Please deploy the outriggers." When the state of the crane 102 is that the outriggers are deployed and there is load on the crane hook, the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "Be careful as a vehicle is approaching." When the state of the crane 102 is that the outriggers are deployed and there is no load on the crane hook, the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "Please pull up the hook."

[0166] (Ninth Modification) For example, the crane 102 may be a crawler crane. When the crane 102 is a crawler crane, the alert information generating unit 236 may modify the generated alert information in accordance with the state of the crawler crane.

[0167] Fig. 33 is a table for explaining an example of alert information according to Modification 9. Table T8 shown in Fig. 33 associates the state of the crane 102 with alert information to be displayed on the terminal device 302 that supports the remote operation of the crane 102.

[0168] For example, when the state of the crane 102 is that there is no load, the alert information that the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "Operate with caution as a vehicle is approaching." When the state of the crane 102 is that there is load and the value of the ground pressure meter is an abnormal value, the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "The ground surface is abnormal. Please lower the load." When the state of the crane 102 is that there is load and the value of the ground pressure meter is normal, the alert information generation unit 236 generates for the terminal device 301 includes a warning sentence saying, "Please pull up the hook."

[0169] (Tenth Modification) In the above-described embodiment, the vehicle 101 has been described as having a crane 102 as an object with which it may collide, but the present invention is not limited to this. For example, other than the crane 102, the present invention may be applied to a bollard or gate bar that prevents the vehicle 101 from entering, a shutter that stores the vehicle 101 in a parking lot, or the like.

[0170] In the case of a bollard, for example, the alert information generated by the alert information generation unit 236 for the vehicle 101 includes the sentence "The bollard is coming up." In the case of a gate bar, for example, the alert information generated by the alert information generation unit 236 for the vehicle 101 includes the sentence "The gate bar is coming down." In the case of a shutter, for example, the alert information generated by the alert information generation unit 236 for the vehicle 101 includes the sentence "The shutter is coming down."

[0171] (Eleventh Modification) For example, the remotely controlled object is not limited to the vehicle 101, but may be a boarding bridge that allows passengers and crew to board and disembark from a terminal building in an airport onto an airplane or the like.

[0172] Fig. 34 is a schematic diagram showing an example of a display screen according to the eleventh modification. The display screen 331 shown in Fig. 34 is a display screen displayed on the display device 320 of the terminal device 301. The display screen 331 shows predicted area information A9, predicted area information A10, a message M1 indicating the boarding bridge ID "51" of the boarding bridge, and an icon M6 indicating that a call can be made with a person present near the boarding bridge. Here, the predicted area information A9 is first predicted area information, and the predicted area information A10 is second predicted area information.

[0173] Message M7 on display screen 331 is an example of a sentence including "approaching airplane," which is an example of a sentence indicating the positional relationship between the tip of the boarding bridge and the airplane. Message M8 on display screen 331 is an example of a sentence including "estimated distance (shortest): 5 m," which is an example of a sentence indicating the estimated distance between the tip of the boarding bridge and the airplane door. Message M22 on display screen 331 is an example of a sentence including "moving," which is an example of a sentence indicating the operating status of the boarding bridge.

[0174] FIG. 35 is a flowchart showing an example of the process of the remote operation support apparatus 20 according to the eleventh modification.

[0175] The receiving unit 231 receives a camera image captured by a camera mounted on the tip of the boarding bridge and transmitted by the boarding bridge (step S51). The delay information calculation unit 232 calculates delay information indicating the delay time required for the image to be displayed on the display device 320 of the terminal device 301, based on the time when the image was captured by the camera on the boarding bridge (step S52). The crane hook position calculation unit 233 calculates door position information indicating the position of the airplane door using the camera image captured by the boarding bridge camera (step S53).

[0176] The predicted area information calculation unit 234 calculates predicted area information indicating an area in the image where a door is predicted to be present, based on the delay information calculated by the delay information calculation unit 232 and the door position included in the door position information calculated by the crane hook position calculation unit 233 (step S54). The output unit 238 outputs the predicted area information calculated by the predicted area information calculation unit 234 to the terminal device 301 (step S55). The surrounding object identification unit 241 uses the camera image to output distance information indicating the distance between the airplane door and the end of the boarding bridge to the terminal device 301 (step S56).

[0177] Next, the determination unit 235 determines whether there is a possibility of collision between the airplane door and the end of the boarding bridge based on the area indicated by the prediction area information and the distance information output by the peripheral object identification unit 241 (step S57). If the determination unit 235 determines that there is no possibility of collision between the airplane door and the end of the boarding bridge (step S57: No), this process ends. On the other hand, if the determination unit 235 determines that there is a possibility of collision between the airplane door and the end of the boarding bridge (step S57: Yes), the process proceeds to step S58.

[0178] In step S58, if the determination unit 235 determines that there is a possibility of collision between the airplane door and the tip of the boarding bridge, the alert information generation unit 236 generates alert information indicating that there is a possibility of collision (step S58). Subsequently, if there is a possibility of collision between the area indicated by the prediction area information and a surrounding object (for example, the airplane door), the assistance control unit 237 restricts the remote operation of the boarding bridge (step S59).

[0179] Next, the output unit 238 outputs the alert information generated by the alert information generation unit 236 to the terminal device 301 (step S60). Furthermore, the output unit 238 outputs, via the assistance control unit 237, control information corresponding to control according to the positional relationship between the area indicated by the predicted area information and a peripheral object indicating an object around the end of the boarding bridge (for example, the door of the airplane), to the terminal device 301. Then, when step S60 ends, this routine ends.

[0180] (12th Modification) For example, the remotely controlled object may be a towing car. In the eleventh modification, the possibility that the tips of both wings of the airplane being towed by the remotely controlled towing car may come into contact with a surrounding object is described. Here, an example of the surrounding object is a boarding bridge. The surrounding object is not limited to a boarding bridge.

[0181] Fig. 36 is a schematic diagram showing an example of a display screen according to the twelfth modification. The display screen 332 shown in Fig. 36 is a display screen displayed on the display device 320 of the terminal device 301. The display screen 332 shows predicted area information A11, predicted area information A12, predicted area information A13, predicted area information A14, a message M11 indicating the boarding bridge ID of the towing car, "1," and an icon M6 indicating that communication with people present in the vicinity of the towing car is possible. Here, the predicted area information A11 and the predicted area information A13 are first predicted area information, and the predicted area information A12 and the predicted area information A14 are second predicted area information.

[0182] Message M9 on display screen 332 is an example of a sentence including "approaching a boarding bridge," which is an example of a sentence indicating the positional relationship between the tips of both wings of the airplane and surrounding objects. Area M10 on display screen 331 shows the results of identifying surrounding objects that may come into contact with the tips of both wings of the airplane. Figure 37 shows the results of identifying a boarding bridge that may come into contact with the left tip of the airplane. Message M23 on display screen 332 is an example of a sentence including "moving," which is an example of a sentence indicating the operating status of a towing car.

[0183] FIG. 37 is a flowchart showing an example of the process of the remote operation support apparatus 20 according to the twelfth modification.

[0184] The receiving unit 231 receives a camera image captured by a camera mounted on the towing car and transmitted by the towing car (step S61). The delay information calculation unit 232 calculates delay information indicating the delay time required for the image to be displayed on the display device 320 of the terminal device 301, based on the time the towing car camera captured the image (step S62). The crane hook position calculation unit 233 uses the camera image captured by the towing car camera to calculate wing position information indicating the positions of the tips of the airplane's wings (step S63).

[0185] The predicted area information calculation unit 234 calculates predicted area information indicating an area in the image where the tips of the wings are predicted to be present, based on the delay information calculated by the delay information calculation unit 232 and the positions of the tips of the wings included in the wing position information calculated by the crane hook position calculation unit 233 (step S64). The output unit 238 outputs the predicted area information calculated by the predicted area information calculation unit 234 to the terminal device 301 (step S65). The surrounding object identification unit 241 identifies surrounding objects present in the area indicated by the predicted area information from the camera image received by the receiving unit 231 (step S66). Here, the surrounding object is assumed to be, for example, a boarding bridge.

[0186] Next, the determination unit 235 determines whether there is a possibility of collision between the tips of both wings of the airplane and the boarding bridge, based on the area indicated by the prediction area information and the peripheral objects identified by the peripheral object identification unit 241 (step S67). If the determination unit 235 determines that there is no possibility of collision between the tips of both wings of the airplane and the boarding bridge (step S67: No), this processing ends. On the other hand, if the determination unit 235 determines that there is a possibility of collision between the tips of both wings of the airplane and the boarding bridge (step S67: Yes), the processing proceeds to step S68.

[0187] In step S68, if the determination unit 235 determines that there is a possibility of collision between the tips of both wings of the airplane and the boarding bridge, the alert information generation unit 236 generates alert information indicating that there is a possibility of collision (step S68). Subsequently, the assistance control unit 237 restricts the remote operation of the towing car if there is a possibility of collision between the area indicated by the prediction area information and a surrounding object (for example, a boarding bridge) (step S69).

[0188] Next, the output unit 238 outputs the alert information generated by the alert information generation unit 236 to the terminal device 301 (step S70). Furthermore, the output unit 238 outputs, via the assistance control unit 237, control information corresponding to control according to the positional relationship between the area indicated by the predicted area information and surrounding objects (for example, a boarding bridge) indicating objects around the tips of both wings of the airplane to the terminal device 301. Then, when step S70 ends, this routine ends.

[0189] (13th Modification) In the above-described embodiment, the vehicle 101 is described as being intended for remote control, but the vehicle 101 is not limited to this, and may be controlled so that it can be driven by a driver who actually drives the vehicle 101 using AR (Augmented Reality) glasses or the like that can communicate with the remote control support device 20.

[0190] (14th Modification) The delay information described above is information indicating the time from when an image is captured until when it is drawn, but is not limited to this. For example, the delay information may be a delay time indicating a round-trip delay based on a transmission delay when a camera image captured by the camera 120 of the vehicle 101 is transmitted to the remote operation assistance device 20 and a transmission delay when the remote operation assistance device 20 transmits operation information to the vehicle 101.

[0191] (15th Modification) In the above-described embodiment, the vehicle 101 is described as an autonomous vehicle, but the present invention is not limited to this and may be a remotely controlled vehicle (for example, a radio-controlled vehicle).

[0192] (16th Modification) Although the above-described display form of the prediction region information displayed on the display screen is an oval display, the present invention is not limited to this, and the display form may be, for example, an arrow display.

[0193] The above-described embodiment can be arbitrarily combined with the above-described modified examples, and the above-described modified examples can also be arbitrarily combined with each other. [Explanation of symbols]

[0194] 1. Remote control system 20 Remote operation support device 101 vehicles 102 Crane 141 Vehicle characteristic information transmission unit 142 Location information acquisition unit 143 Image acquisition unit 144 Travel control unit 145 Operation information receiving unit 191 Crane characteristic information transmitter 192 Image acquisition unit 193 Crane control unit 194 Operation information receiving unit 231 Receiving unit 232 Delay information calculation unit 233 Crane hook position calculation unit 234 Prediction area information calculation unit 235 Judgment section 236 Alert Information Generation Unit 237 Support Control Unit 238 Output Section 239 Remote Information Transmitter / Receiver 301, 302 Terminal equipment 341, 381 Information Receiving Department 342, 382 Display control unit 343, 383 Operation information transmission unit 344, 384 Time information transmission unit

Claims

1. outputting predicted area information indicating an area in the image where the moving object is likely to be present, based on delay information indicating the time from when the image is captured until when the image is drawn and the position of the moving object reflected in the image; Method for remote control.

2. performing control according to a positional relationship between the area indicated by the predicted area information and a peripheral object indicating an object in the vicinity of the moving object; 2. The method for remote control according to claim 1.

3. If there is a possibility of a collision between the area indicated by the predicted area information and the surrounding object, restricting the remote operation of the crane to be remotely operated.

3. The method for remote control according to claim 2.

4. When the surrounding object is an autonomous vehicle, the control to restrict the remote operation is performed by prohibiting the autonomous vehicle from entering below a crane hook of the crane.

4. The method for remote control according to claim 3.

5. As the control to limit the remote operation, a control to limit the maximum speed of a boom, which is an arm portion to which a crane hook of the crane is attached and which is capable of rotating, is performed.

4. The method for remote control according to claim 3.

6. outputting a warning when there is a possibility of collision between the area indicated by the predicted area information and the surrounding object; 3. The method for remote control according to claim 2.

7. the prediction area information includes at least first prediction area information and second prediction area information indicating an area in which the probability that the moving object exists is lower than the area indicated by the first prediction area information; a control according to a positional relationship between the area indicated by the first prediction area information and the peripheral object and a control according to a positional relationship between the area indicated by the second prediction area information and the peripheral object are different from each other; 3. The method for remote control according to claim 2.

8. restricting the autonomous driving when there is a possibility of a collision between the area indicated by the first predicted area information and the surrounding object; 8. The method for remote control according to claim 7.

9. outputting a warning when there is a possibility of collision between the area indicated by the second prediction area information and the surrounding object; 8. The method for remote control according to claim 7.

10. When the surrounding object is an autonomous vehicle, as the control to restrict the autonomous traveling, control is performed to prevent the autonomous vehicle from entering an area indicated by the first predicted area information.

9. The method for remote control according to claim 8.

11. The warning is changed depending on a positional relationship between the area indicated by the second predicted area information and the peripheral object.

10. The method for remote control according to claim 9.

12. A terminal device operated by an operator who remotely controls a vehicle, The terminal device a display control unit that controls display of predicted area information transmitted by a remote operation assistance device that assists in remote operation of the vehicle; The predicted area information indicates an area in the image where a moving object is predicted to be present, based on delay information indicating the time from when the image is captured until when the image is drawn and the position of a moving object reflected in the image. Terminal device.

13. A terminal device operated by an operator who operates a moving object, The terminal device a display control unit that controls display of predicted area information transmitted by an operation assistance device that assists in operating the moving object; The predicted area information indicates an area in the image where the moving object is likely to exist, based on delay information indicating the time from when the image is captured until when it is drawn and the position of the moving object reflected in the image. Terminal device.

14. outputting predicted area information indicating an area in the image where the moving object is likely to be present, based on delay information indicating the time from when the image is captured until when the image is drawn and the position of the moving object reflected in the image; A program that makes a computer do something.

Images