Remote monitoring method, remote monitoring device, and program

Abstract

The present invention pertains to a remote monitoring method for remotely monitoring an autonomously-moving mobile body, wherein a first video capturing a direction in which the mobile body can move is displayed (S11), and detection of a touch operation in a first region where the first video is displayed is enabled (S12).

Description

Technical Field

[0001] The present disclosure relates to a remote monitoring method, a remote monitoring device, and a program.

Background Art

[0002] When remotely monitoring a mobile body that autonomously travels, it has been considered to give an instruction to start autonomous driving remotely by a touch operation while viewing an image.

[0003] Therefore, it is desired to improve the operability of the touch operation. For example, Patent Document 1 discloses a video monitoring system that can control the operation of a video imaging device according to a touch gesture operation of an operator.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in remote monitoring of a mobile body, for the convenience of performing a safety check by viewing an image, it is desired to increase the ratio of the image occupying the screen. Therefore, it is difficult to enlarge the display for instructions such as an instruction to start autonomous driving (for example, button display), and there is room for improvement in the operability of the touch operation. Patent Document 1 does not disclose a technique capable of improving the operability of the touch operation when it is difficult to enlarge the display.

[0006] Therefore, the present disclosure provides a remote monitoring method, a remote monitoring device, and a program that can improve the operability of the touch operation more than before.

Means for Solving the Problems

[0007] A remote monitoring method according to an aspect of the present disclosure is a remote monitoring method for remotely monitoring a mobile body that autonomously travels, and displays a first video obtained by imaging a direction in which the mobile body can move, and enables detection of a touch operation on a first area where the first video is displayed.

[0008] A remote monitoring device according to an aspect of the present disclosure is a remote monitoring device for remotely monitoring a mobile body that autonomously travels, and includes a first control unit that performs control to display a video obtained by imaging a direction in which the mobile body can move, and a second control unit that enables detection of a touch operation on an area where the video is displayed.

[0009] A program according to an aspect of the present disclosure is a program for causing a computer to execute the above-described remote monitoring method.

Advantages of the Invention

[0010] According to an aspect of the present disclosure, it is possible to realize a remote monitoring method or the like that can improve the operability of touch operations as compared with the conventional art.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Mode for Carrying Out the Invention

[0012] (Background Leading to the Present Disclosure) Prior to the description of the embodiments and the like of the present disclosure, the background leading to the present disclosure will be described with reference to FIGS. 1 and 2. FIG. 1 is a schematic diagram showing a state where operator A performs remote monitoring.

[0013] As shown in FIG. 1, operator A performs remote monitoring of the moving body while checking the video such as the traveling direction of the moving body displayed on terminal 10. Operator A performs remote monitoring of, for example, a plurality of moving bodies. The moving body is a self-driving moving body. Operator A performs remote monitoring of the self-driving moving body. Note that remote monitoring means that operator A monitors the situation of the self-moving moving body at a remote location where the moving body cannot be directly visually recognized, using the video obtained by imaging the surroundings of the moving body.

[0014]

[0015] ​As shown in FIG. 2, images in the traveling direction of the moving body (in the example of FIG. 2, three images in the forward, left, and right directions) are displayed, and operation contents (commands) to be executed on the moving body such as temporary stop and resumption of travel are displayed.

[0016] During remote monitoring, operator A can give instructions to the moving body such as temporary stop and resumption of travel as necessary. For example, by pressing a temporary stop operation button (within the dashed frame in FIG. 2) presented on the outer periphery of the display area where the image of the moving body with the moving body ID of 51 is displayed, the moving body can be made to execute a temporary stop.

[0017] The instructions given by operator A during remote monitoring are to permit the operation of the moving body or to cause it to execute a predetermined operation. The instructions given by operator A during remote monitoring are executed by simple operations such as touching an operation button without using an input device such as a steering wheel. The instructions given by operator A during remote monitoring may, for example, permit a stationary moving body to start traveling, or may permit a temporary stop because there is an obstacle ahead.

[0018] Here, for the convenience of performing a safety check by viewing the images, it is required to maximize the ratio of the images occupying the screen, and there is a problem that it is difficult to enlarge the operation buttons on the screen and the operability of the touch operation is poor.

[0019] Patent Document 1 does not disclose a technique capable of solving such problems.

[0020] Therefore, the inventors of the present application have intensively studied a remote monitoring method and the like capable of improving the operability of touch operations more than before, and have devised the following remote monitoring method and the like.

[0021] The remote monitoring method according to the first aspect of the present disclosure is a remote monitoring method for remotely monitoring a moving body that autonomously travels, and displays a first video obtained by imaging a direction in which the moving body can move, and enables detection of a touch operation on a first area where the first video is displayed.

[0022] Thereby, the area where the first video is displayed can be set as the area of the operation button, so that the area of the operation button can be expanded. Therefore, it is possible to realize a remote monitoring method capable of improving the operability of touch operations as compared with the prior art.

[0023] Further, for example, the remote monitoring method according to the second aspect of the present disclosure is the remote monitoring method according to the first aspect, and the operation content executed by touching the first area may be displayed around the first area or in the first area.

[0024] Thereby, the operation content can be displayed to the operator. The operability of the touch operation can be improved as compared with the case where the operation content is not displayed.

[0025] Further, for example, the remote monitoring method according to the third aspect is the remote monitoring method according to the first aspect, and a frame surrounding the first area may be displayed.

[0026] Thereby, since the operation area where the detection of the touch operation is effective is clearly shown, the touchable area becomes clear, and the operability of the touch operation can be improved.

[0027] Further, for example, the remote monitoring method according to the fourth aspect is the remote monitoring method according to the third aspect, and the operation content executed by touching the first area is displayed around the first area or in the first area, and the frame and the operation content may be displayed in the same display mode.

[0028] Thereby, since the frame and the operation content are displayed in the same manner, the touchable area and the operation content can be made clearer. That is, the operability of the touch operation can be further improved.

[0029] Further, for example, the remote monitoring method according to the fifth aspect is the remote monitoring method according to the fourth aspect, and the frame and the operation content may be displayed in the same color.

[0030] Thereby, the frame and the operation content can be displayed in the same color.

[0031] Further, for example, the remote monitoring method according to the sixth aspect is the remote monitoring method according to any one of the third to fifth aspects, and when a touch operation on the first region is received, the display mode of the frame may be changed.

[0032] Thereby, it is possible to notify the operator that a touch operation has been received according to the display mode of the frame.

[0033] Further, for example, the remote monitoring method according to the seventh aspect is the remote monitoring method according to any one of the third to sixth aspects, and the display mode of the frame may be changed according to the execution state of the operation content on the moving body.

[0034] Thereby, it is possible to notify the operator of the execution state of the command according to the display mode of the frame.

[0035] Further, for example, the remote monitoring method according to the eighth aspect is the remote monitoring method according to the fourth or fifth aspect, and based on the state of the moving body, it is possible to switch to invalidating the detection of the touch operation on the first region, and the display mode of the operation content may be made different according to whether the detection of the touch operation on the first region is valid or invalid.

[0036] Thereby, it is possible to notify the operator whether the touch operation is valid or invalid.

[0037] Further, for example, the remote monitoring method according to the ninth aspect is the remote monitoring method according to any one of the third to eighth aspects, and based on the state of the moving body, it is possible to switch to invalidating the detection of a touch operation on the first area, and depending on whether the detection of the touch operation on the first area is valid or invalid, the display mode of the frame may be made different.

[0038] Thereby, it is possible to notify whether the touch operation is valid or invalid by the display mode of the frame.

[0039] Further, for example, the remote monitoring method according to the tenth aspect is the remote monitoring method according to any one of the third to ninth aspects, and based on the state of the moving body, it is possible to switch to invalidating the detection of a touch operation on the first area, and depending on whether the detection of the touch operation on the first area is valid or invalid, the display mode of the first video may be made different.

[0040] Thereby, it is possible to notify whether the touch operation is valid or invalid by the display mode of the first video.

[0041] Further, for example, the remote monitoring method according to the eleventh aspect is the remote monitoring method according to any one of the first to tenth aspects, and based on the state of the moving body, the operation content associated with the first area may be switched.

[0042] Thereby, it is possible to automatically switch to a command according to the state of the moving body.

[0043] Further, for example, the remote monitoring method according to the twelfth aspect is the remote monitoring method according to any one of the first to eleventh aspects, and when a touch operation on the first area is detected, the display mode of the first video may be changed.

[0044] Thereby, it is possible to notify the operator that a touch operation has been detected by the display mode of the first video.

[0045] Also, for example, the remote monitoring method according to the 13th aspect is the remote monitoring method according to any one of the 3rd to 10th aspects, and when a touch operation on the first area is detected, the display mode of the frame may be changed.

[0046] Thereby, it is possible to notify the operator that a touch operation has been detected by the display mode of the frame.

[0047] Also, for example, the remote monitoring method according to the 14th aspect is the remote monitoring method according to any one of the 1st to 13th aspects, and the display mode of the first video may be changed according to the execution state of the remote monitoring of the moving body.

[0048] Thereby, it is possible to notify the operator of the execution state of the operation content by the display mode of the first video.

[0049] Also, for example, the remote monitoring method according to the 15th aspect is the remote monitoring method according to any one of the 1st to 14th aspects, the first video is a video obtained by imaging a first direction in which the moving body can move, and further, a second video obtained by imaging a second direction in which the moving body can move is displayed in a second area different from the first area, and path information of the moving body is acquired. When the moving direction of the moving body based on the path information is the first direction, detection of a touch operation on the first area may be enabled.

[0050] Thereby, it is possible to automatically enable detection of a touch operation on a video area according to the traveling direction of the moving body.

[0051] Also, for example, the remote monitoring method according to the 16th aspect is the remote monitoring method according to any one of the 1st to 15th aspects, there are a plurality of operation types in the touch operation, and the display mode of the first video and the operation content may be made different according to the operation type of the touch operation performed on the first area.

[0052] Accordingly, by simply changing the operation type, a plurality of display modes and commands can be executed.

[0053] Further, the remote monitoring device according to the 17th aspect is a remote monitoring device for remotely monitoring a mobile body that autonomously travels, and includes a first control unit that performs control to display an image obtained by imaging the direction in which the mobile body can move, and a second control unit that enables detection of a touch operation on the area where the image is displayed. Also, the program according to the 18th aspect is a program for causing a computer to execute the remote monitoring method according to any one of the 1st to 16th aspects.

[0054] These achieve the same effects as the above remote monitoring method.

[0055] Note that these general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a CD-ROM readable by a computer, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium. The program may be pre-stored in the recording medium or may be supplied to the recording medium via a wide area communication network including the Internet or the like.

[0056] Hereinafter, embodiments and the like will be specifically described with reference to the drawings.

[0057] Note that all of the embodiments and the like described below show comprehensive or specific examples. The numerical values, shapes, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Also, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0058] Also, each figure is a schematic diagram and is not necessarily drawn precisely. Therefore, for example, scales etc. in each figure do not necessarily match. Also, in each figure, substantially the same configurations are denoted with the same reference numerals, and duplicate explanations are omitted or simplified.

[0059] Also, in this specification, terms indicating the relationship between elements such as the same, as well as numerical values and numerical ranges, are not expressions representing only strict meanings, but are expressions meaning to include substantially equivalent ranges, for example, differences of about several % (or about 10%).

[0060] Also, in this specification, ordinal numbers such as "first" and "second" do not, unless otherwise specified, mean the number or order of components, but are used for the purpose of avoiding confusion and distinguishing between components of the same kind.

[0061] (Embodiment) Hereinafter, the remote monitoring method and the like according to this embodiment will be described with reference to FIGS. 3 to 6.

[0062] [1. Configuration of Remote Monitoring System] First, the configuration of the remote monitoring system that executes the remote monitoring method according to this embodiment will be described with reference to FIG. 3. FIG. 3 is a diagram showing the functional configuration of the remote monitoring system 1 according to this embodiment.

[0063] As shown in FIG. 3, the remote monitoring system 1 includes a terminal 10. The remote monitoring system 1 further includes a control server 20, an infrastructure camera 30, and a mobile body 40. For example, the remote monitoring system 1 includes a plurality of infrastructure cameras 30 and mobile bodies 40.

[0064] The terminal 10 is a terminal system used by an operator who remotely monitors the self-driving mobile body 40, and is communicably connected to the control server 20. The terminal 10 is configured to include computer equipment. Specifically, the terminal 10 is realized by a non-volatile memory storing a program, a volatile memory which is a temporary storage area for executing the program, an input / output port, a communication interface, a processor for executing the program, a display unit such as a liquid crystal display device, etc. Further, the terminal 10 includes an output device 11, an input device 12, and a management device 13. The output device 11, the input device 12, and the management device 13 may be realized as an integrated device, or different devices may be communicably connected to each other. The terminal 10 is an example of a remote monitoring device.

[0065] The output device 11 outputs (displays) information for the operator to perform remote monitoring. The output device 11 is configured to include, for example, a display unit. Further, the output device 11 includes a video display unit 11a, an operation content display unit 11b, and a video outer frame drawing unit 11c.

[0066] The video display unit 11a displays a video (an example of the first video) of the direction in which the mobile body 40 can move to the operator. The direction in which it can move includes, for example, the front, the right turn direction, the left turn direction, etc. when the mobile body 40 moves forward. The video display unit 11a displays a video captured by at least one of a camera mounted on the mobile body 40 and a camera arranged around the mobile body 40 to the operator. The video is a video for the operator to perform remote monitoring, and it is desirable to be displayed in a wide area on the screen. The video display unit 11a is an example of the first control unit.

[0067] The operation content display unit 11b displays the types of operation contents (commands) that are executed when the video is touch-operated. The touch operation includes at least one of touching and clicking.

[0068] The video outer frame drawing unit 11c displays a frame indicating that it is an operation area around the video displayed by the video display unit 11a. The video outer frame drawing unit 11c may display, for example, a frame in a display mode according to the operation content to be executed on the moving body 40. The video outer frame drawing unit 11c may display, for example, a frame including at least one of a color, a pattern, and a shape according to the operation content.

[0069] Note that the output device 11 only needs to include at least the video display unit 11a.

[0070] The input device 12 receives an input from the operator. Specifically, the input device 12 receives a touch operation from the operator. The input device 12 includes a video display input unit 12a.

[0071] The video display input unit 12a is arranged on the display area where the video display unit 11a displays the video, and detects the touch operation of the operator. The video display input unit 12a is configured to include, for example, a touch panel. The touch panel is arranged, for example, so as to cover the display panel that displays the video.

[0072] In the present embodiment, when the operator touches the display area of the video displayed by the video display unit 11a, the video display input unit 12a detects the touch operation. It can also be said that the video display input unit 12a enables the detection of a touch operation on the area (an example of the first area) where the video is displayed on the screen of the display panel. The video display input unit 12a is an example of the second control unit.

[0073] The management device 13 manages which command to execute on the moving body 40 when the operator performs a touch operation. The management device 13 includes an execution command management unit 13a.

[0074] The execution command management unit 13a may determine a command to be executed on the moving body 40, for example, based on a table (for example, refer to FIG. 21 described later) in which the state of the moving body (for example, vehicle state) is associated with the execution command.

[0075] Note that the terminal 10 may include a storage unit (not shown) that stores a table or the like for determining commands. The storage unit is realized by, for example, a semiconductor memory, but is not limited thereto.

[0076] The control server 20 is communicably connected to the terminal 10, the infrastructure camera 30, and the mobile body 40, and transmits and receives various types of information. The control server 20 is realized by a computer device. Specifically, the control server 20 is realized by a non-volatile memory storing a program, a volatile memory that is a temporary storage area for executing the program, an input / output port, a communication interface, a processor for executing the program, and the like. The control server 20 may be realized by, for example, a server device. Further, the control server 20 includes, as a functional configuration, a mobile body information transmission management unit 21, a command transmission unit 22, a video transmission unit 23, and a mobile body information transmission unit 24.

[0077] The mobile body information transmission management unit 21 manages which video from which mobile body 40 or which infrastructure camera 30 is transferred (displayed) to which terminal 10, or which command from which terminal 10 is transmitted to which mobile body 40. The mobile body information transmission management unit 21 manages the destination, for example, based on the destination address and the source address included in the received information.

[0078] The command transmission unit 22 transmits the command from the terminal 10 to the mobile body 40 specified based on the mobile body information transmission management unit 21.

[0079] The video transmission unit 23 transmits the video from the mobile body 40 or the infrastructure camera 30 to the terminal 10 specified based on the mobile body information transmission management unit 21.

[0080] The mobile body information transmission unit 24 transmits the information indicating the state of the mobile body 40 to the terminal 10 specified based on the mobile body information transmission management unit 21.

[0081] The infrastructure camera 30 is a fixed or movable camera installed at a fixed point around the passage along which the moving body 40 moves. The infrastructure camera 30 images obstacles and the like on the target passage. The infrastructure camera 30 is communicably connected to the control server 20 and transmits the captured video to the control server 20. The infrastructure camera 30 includes an infrastructure video acquisition unit 31.

[0082] The infrastructure video acquisition unit 31 transmits the video from the camera installed in the infrastructure to the control server 20. The infrastructure video acquisition unit 31 acquires, for example, the video captured by the camera of its own device or other infrastructure cameras, and transmits the acquired video to the control server 20. The infrastructure video acquisition unit 31 is configured to include, for example, a communication interface.

[0083] The moving body 40 is an autonomously movable moving body that is the target of remote monitoring by an operator. The moving body 40 is not particularly limited as long as it is an autonomously movable moving body, and may be, for example, an autonomously traveling vehicle (e.g., an automobile, a railway, etc.), an autonomously flying aircraft (e.g., a drone), an autonomously navigating ship, etc.

[0084] The moving body 40 is communicably connected to the control server 20. The moving body 40 is configured to include computer equipment and the like. Specifically, the moving body 40 includes a non-volatile memory storing a program, a volatile memory which is a temporary storage area for executing the program, an input / output port, a communication interface, a processor for executing the program, and the like. Further, the moving body 40 includes, as a functional configuration, a command execution unit 41, a video acquisition unit 42, and a moving body information acquisition unit 43.

[0085] The command execution unit 41 receives a command from the control server 20 and executes the received command. The command execution unit 41 controls the movement of the moving body 40 according to the command.

[0086] The image acquisition unit 42 acquires images from one or more cameras installed on the mobile body 40 and transmits them to the control server 20. The mobile body 40 is equipped with cameras capable of imaging the front (front), left and right (sides), rear, etc.

[0087] The mobile body information acquisition unit 43 acquires information such as the mode, vehicle speed, and route of the mobile body 40. The mode includes an autonomous driving mode and a remote monitoring mode (or a remote operation mode). The mode can be acquired, for example, from a mode control unit or the like mounted on the mobile body 40. The vehicle speed can be acquired from a speed sensor or the like mounted on the mobile body 40. The route can be acquired based on the destination setting by the user of the mobile body 40.

[0088] [2. Operation of the remote monitoring system] Subsequently, the operation of the remote monitoring system 1 configured as described above will be described with reference to FIGS. 4 to 6. FIG. 4 is a flowchart showing the display operation (remote monitoring method) of the terminal 10 according to the present embodiment. In FIG. 4, an example of displaying the front camera image will be described, but a side camera image or the like may be displayed instead of the front camera image. Also, the operation shown in FIG. 4 is executed by the terminal 10.

[0089] As shown in FIG. 4, the video display unit 11a draws the front camera image of the mobile body 40 that is the target of the operator's remote monitoring on the display unit (S11).

[0090] Next, the video display input unit 12a generates an operation area on the front camera image (S12). It can also be said that the video display input unit 12a enables the detection of touch operations on the area where the front camera image is displayed. Thereby, it is possible to turn on the detection of the operator's touch operation on the operation area, that is, the touch operation on the front camera image. For example, the display area and the operation area of the front camera image may overlap and have the same size. The operation area is an example of the first area.

[0091] FIG. 5 is a diagram showing a display example of the terminal 10 according to the present embodiment. FIG. 5 shows a display example when the traveling direction of the moving body 40 is forward. In FIG. 5, an example in which three videos, i.e., a front video 11aF, a right video 11aR, and a left video 11aL, are displayed is shown, but the number of videos to be displayed is not limited to three. For example, a rear video may be displayed. Further, the number of videos to be displayed may be set in advance or may be set by an operator.

[0092] As shown in FIG. 5, the output device 11 of the terminal 10 displays various types of information including videos. In addition to the videos 11aF, 11aR, and 11aL, the output device 11 displays a moving body ID (for example, 51 in FIG. 5), information identifying an operator (for example, operators A to C in FIG. 5), a traveling state (traveling or schedule traveling in FIG. 5), a service status (delivery in progress in FIG. 5), a destination setting button (destination setting in FIG. 5), a safety stop button (safety stop in FIG. 5), and the like.

[0093] The video 11aF is a video captured in front of the moving body 40, the video 11aR is a video captured on the right side of the moving body 40, and the video 11aF is a video captured on the left side of the moving body 40. The video 11aF is an example of a front camera video.

[0094] In step S11, the video display unit 11a displays, for example, the videos 11aL and 11aR in addition to the video 11aF.

[0095] In the case of FIG. 5, the touch panel included in the video display input unit 12a is configured to cover at least the entire area of the video 11aF. When the video 11aF is displayed, the video display input unit 12a switches the detection of touch operations over the entire area of the video 11aF from off to on. That is, the video display input unit 12a enables the detection of touch operations on the video 11aF.

[0096] The image 11aF is displayed for the operator to remotely monitor the moving body 40. Normally, no touch operation is performed on the image 11aF (no detection of touch operation). However, in the present disclosure, from the perspective of improving the operability of the touch operation, a touch operation on the image 11aF is detected.

[0097] Thereby, the display area of the image 11aF can be used as the operation area 12a1 of the operation button. Therefore, the operation area 12a1 can be enlarged without reducing the size of the image 11aF. The operation area 12a1 is a touch area that receives the touch operation of the operator.

[0098] In addition, in the case of FIG. 5, the touch panel included in the video display input unit 12a may be configured to cover, for example, the images 11aF, 11aL, and 11aR. In this case, for example, when the images 11aL and 11aR are displayed, the video display input unit 12a may switch the detection of the touch operation over the entire area of the images 11aL and 11aR from off to on, or may maintain off.

[0099] FIG. 6 is a flowchart showing the execution operation (remote monitoring method) of the terminal 10 according to the present embodiment. The operation shown in FIG. 6 is executed by the terminal 10.

[0100] As shown in FIG. 6, when the execution command management unit 13a detects a touch operation on the image 11aF by the video display input unit 12a (S21), it executes a process for executing the command set in the image 11aF (S22). The execution command management unit 13a transmits a command corresponding to the operation area 12a1 to the moving body 40 via the control server 20. Thereby, the moving body 40 can be made to execute a traveling operation according to the command operated by the operator.

[0101] In addition, in the present embodiment, one command is associated with the operation area 12a1. No matter at which position within the operation area 12a1 a touch operation is performed, the same command is executed.

[0102] (Modification Example 1 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 7 to 8B. In the following, the description will focus on the differences from the embodiment, and the description of the same or similar content as the embodiment will be omitted or simplified. In each modification example after Modification Example 1 of the embodiment, the configuration of each component of the remote monitoring system may be the same as that of the remote monitoring system 1 according to the embodiment, and the description will be made using the reference numerals used in the embodiment.

[0103] In this modification example, an example of presenting a command executed when the operation area is touched will be described. FIG. 7 is a flowchart showing the display operation (remote monitoring method) of the terminal 10 according to this modification example.

[0104] As shown in FIG. 7, when an operation area is generated on the front camera video by the video display input unit 12a (S12), the operation content display unit 11b of the terminal 10 presents operation content in the display area or the outer periphery of the front camera video (S31). The operation content display unit 11b presents information indicating a command set in the display area of the front camera video in the display area or the outer periphery.

[0105] FIGS. 8A and 8B are diagrams showing each display example of the terminal 10 according to this modification example. FIG. 8A shows an example of presenting operation content 11b1 on the outer periphery of the display area, and FIG. 8B shows an example of presenting operation content 11b1 in the display area. In FIGS. 8A and 8B, as an example of the operation content 11b1, a pause is indicated.

[0106] As shown in FIG. 8A, the operation content display unit 11b may present the operation content 11b1 on the outer periphery of the display area of the video 11aF. The operation content display unit 11b presents the operation content 11b1 at a position that does not overlap with the video 11aF. Thereby, the operation content 11b1 can be presented without blocking a part of the video 11aF. Also, for example, the video display input unit 12a may enable detection of a touch operation on the area where the operation content 11b1 is presented.

[0107] As shown in FIG. 8B, the operation content display unit 11b may present the operation content 11b1 within the display area of the video 11aF. The operation content display unit 11b presents the operation content 11b1 by superimposing it on the video 11aF. The operation content display unit 11b superimposes the operation content 11b1 on an area where there are considered to be few obstacles within the display area. When the moving body 40 is a vehicle, the operation content display unit 11b superimposes the operation content 11b1 on an area for displaying the sky rather than near the road.

[0108] Thereby, the operation content 11b1 can be presented without blocking a part of the video 11aF. Also, for example, even when other information or the like is presented on the outer periphery of the display area of the video 11aF and it is difficult to present the operation content 11b1, the operation content 11b1 can be presented by the video display input unit 12a.

[0109] (Modification Example 2 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 9 and 10. In this modification example, an example of presenting a frame indicating an operation area will be described. FIG. 9 is a flowchart showing the display operation (remote monitoring method) of the terminal 10 according to this modification example.

[0110] As shown in FIG. 9, when the operation content 11b1 is presented by the operation content display unit 11b (S31), the video outer frame drawing unit 11c presents a frame around the display area of the front camera video (S41). The video outer frame drawing unit 11c presents a frame for clearly indicating the operation area on the video 11aF. The frame is, for example, a frame for distinguishing between the operation area on the video 11aF and the non - operation area other than that in the display area of the display unit of the terminal 10.

[0111] FIG. 10 is a diagram showing a display example of the terminal 10 according to this modification example.

[0112] As shown in FIG. 10, the video outer frame drawing unit 11c presents a frame 11c1 that surrounds the video 11aF. The frame 11c1 may be presented in the same display mode as the operation content 11b1. For example, the frame 11c1 may be presented in the same color as the operation content 11b1. Also, the frame 11c1 is not limited to being circular, and may be a frame shape with some parts such as a broken line being interrupted.

[0113] This makes it possible to clearly show that when the display area of the video 11aF is touched, the command indicated by the operation content 11b1 is executed. For example, the operator can easily know which command is executed by touching the inside of the frame 11c1 by checking the operation content 11b1 and the frame 11c1. Therefore, according to the remote monitoring system 1, the operability of the touch operation can be improved compared to the case where such a display is not performed.

[0114] In addition, in FIG. 9, step S31 may not be executed. That is, only the frame 11c1 among the operation content 11b1 and the frame 11c1 may be presented.

[0115] (Modification Example 3 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 11 and 12. In this modification example, a remote monitoring method that can intuitively convey to the operator that the touch operation has been received when the operator touches the video 11aF will be described. FIG. 11 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification example. FIG. 12 is a diagram showing a display example of the terminal 10 according to this modification example.

[0116] As shown in FIG. 11, when the operator touches the display area of the video 11aF, the video display input unit 12a detects the touch operation on the video 11aF (S51).

[0117] Next, when the touch operation is detected by the video display input unit 12a, the video display unit 11a temporarily increases the brightness of the video 11aF in order to notify the operator that the detection has occurred (S52). Thereby, it is possible to visually convey to the operator that the touch operation has been received. Note that it is not limited to changing the brightness when the touch operation is detected, and any display mode may be changed. For example, the display mode (e.g., color, blinking, etc.) of at least one of the operation content 11b1 and the frame 11c1 may be changed.

[0118] In Fig. 12(a), temporarily increasing the brightness of the video 11aF is indicated by a dashed line.

[0119] Referring to Fig. 11 again, next, the execution command management unit 13a executes a process for executing the command set in the video 11aF (S53). The execution command management unit 13a transmits the command corresponding to the operation content 11b1 to the moving body 40 via the control server 20. In the example of Fig. 12, a command for temporarily stopping the moving body 40 is transmitted to the moving body 40.

[0120] Next, the execution command management unit 13a determines whether the command executed in step S53 has failed in the moving body 40 (S54). When the execution command management unit 13a receives information indicating the completion of the execution of the command from the moving body 40, it determines that it has not failed. When it has not received the information indicating the completion of the execution of the command from the moving body 40 within a predetermined period or when it has received information indicating that the execution of the command has failed from the moving body 40, it determines that it has failed.

[0121] When the execution command management unit 13a determines that the command execution has failed (yes in S54), the video display unit 11a temporarily increases the highlight red balance of the video 11aF in order to notify the operator of the failure (S55). Also, when the execution command management unit 13a determines that the command execution has not failed (no in S54), the video display unit 11a temporarily increases the highlight green balance of the video 11aF in order to notify the operator of the success (S56).

[0122] In FIGS. 12(b) and 12(c), changing the highlight balance of the video 11aF is shown by varying the hatching pattern.

[0123] In this way, when the terminal 10 receives a touch operation, it notifies the operator in different display modes respectively when the command execution is successful and when the command execution fails. Note that the display mode is not limited to the display modes described above as long as they are different from each other in three cases. For example, in the three cases, the display mode of at least one of the frame and the operation content may be changed. The terminal 10 may change the display mode of at least one of the frame and the operation content according to the execution state of the command corresponding to the received touch operation, for example.

[0124] (Modification Example 4 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 13 to 15. In this modification example, an example in which the display mode of the video and the executed command are changed according to the operation type of the touch operation will be described. FIG. 13 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification example.

[0125] As shown in FIG. 13, when the execution command management unit 13a detects a touch operation on the front camera video by the video display input unit 12a (S51), it detects the operation type and determines the execution command (command) (S61). The execution command management unit 13a makes the determination in step S61 based on a table showing the relationship between the operation type and the execution command.

[0126] FIG. 14 is a table showing the relationship between the operation type of the touch operation according to this modification example, the display mode of the video, and the execution command. The table shown in FIG. 14 is created in advance and stored in the storage unit of the terminal 10.

[0127] As shown in FIG. 14 in the table, the operation type of the touch operation, the change in the video display mode during the operation, and the execution command are associated with each other. For example, when a tap is performed as the touch operation, the brightness of the video is increased and decreased three times, and the running stop is executed as the execution command. Also, when a long tap is performed, the brightness gradually increases, and the call start is executed as the execution command.

[0128] Note that, in the change in the video display mode during the operation, an example of changing only the brightness is shown, but in addition to the brightness, or instead of the brightness, the color of the video may be changed.

[0129] Note that the combination of the operation type, the change in the video display mode during the operation, and the execution command is not limited to the combination shown in FIG. 14, and other combinations may be used. Also, together with the display mode of the video, or instead of the display mode of the video, the display mode of at least one of the operation content and the frame may be changed.

[0130] Referring to FIG. 13 again, the operation content display unit 11b displays the execution command name within or outside the display area of the video (S62). The operation content display unit 11b displays the execution command name determined in step S61.

[0131] Next, the video display unit 11a changes the color and brightness of the video according to the type of the execution command (S63). The video display unit 11a changes the color and brightness of the video according to the content shown in the item of the change in the video display mode during the operation shown in FIG. 14. Note that in step S63, at least one of the color and brightness of the video may be changed.

[0132] FIG. 15 is a diagram showing a display example of the terminal 10 according to this modification. The content within the < > shown in FIG. 15 indicates the operation type. In FIG. 15, the portion where the display mode of the video has been changed since before the touch operation is indicated by diagonal hatching.

[0133] As shown in FIG. 15, when a touch operation is performed, at least one of the color and brightness is changed in different display modes according to the operation type. That the display modes are different here includes that the areas where the display mode is changed are different. Also, the command (operation content 11b1) to be executed is changed according to the operation type.

[0134] Referring to FIG. 13 again, the processing after step S53 continues.

[0135] In this way, there are a plurality of operation types for the touch operation, and the terminal 10 varies the display mode of the video 11aF and the command to be executed according to the operation type of the touch operation performed on the display area of the video 11aF.

[0136] (Modification Example 5 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification will be described with reference to FIGS. 16 to 19. In this modification, an example will be described in which when a command cannot be executed according to the state of the moving body 40, the command is deactivated. FIG. 16 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification.

[0137] As shown in FIG. 16, the execution command management unit 13a of the terminal 10 acquires the vehicle state from the moving body 40 (S71). The execution command management unit 13a may, for example, acquire the vehicle state periodically. Also, the execution command management unit 13a manages the vehicle state of each moving body 40. The vehicle state is an example of the state of the moving body 40.

[0138] FIG. 17 is a diagram showing an example of the vehicle state according to this modification.

[0139] As shown in FIG. 17, the vehicle state indicates the state related to the movement of the moving body 40, and includes, for example, in motion, stopped, under remote operation, under manual operation, under emergency stop, during failure occurrence, and offline. Further, the execution command management unit 13a stores, in association with each other, the moving body ID of each moving body 40 and the vehicle state.

[0140] Referring again to FIG. 16, the execution command management unit 13a determines whether a command can be executed in the moving body 40 based on the vehicle state (S72).

[0141] FIG. 18 is a flowchart showing an example of step S72 shown in FIG. 16.

[0142] As shown in FIG. 18, the execution command management unit 13a determines whether the vehicle state is a predetermined state. In the example of FIG. 18, the execution command management unit 13a determines whether the vehicle state of the moving body 40 is in motion (for example, under autonomous driving) (S72a). In the case of FIG. 17, for the moving body 40 with the moving body ID of 51, the execution command management unit 13a determines that it is in motion, and for the other moving bodies 40, it determines that it is not in motion.

[0143] Next, when the execution command management unit 13a determines that the vehicle state is in motion (yes in S72a), it determines that the pause command is executable (S72b), and when it determines that the vehicle state is not in motion (no in S72a), it determines that the pause command is not executable (S72c). The execution command management unit 13a may execute step S72 based on a table (not shown) in which the vehicle state and the executable commands are associated with each other.

[0144] Referring again to FIG. 16, when the command is executable (yes in S72), the terminal 10 executes the processing after step S12, and when the command is not executable (no in S72), the terminal 10 executes the processing after step S73. The processing after step S73 shows the operation when the state transitions from a state where the command is executable to a state where the command is not executable.

[0145] Next, the video display input unit 12a deletes the operation area on the front camera video (S73). The video display input unit 12a invalidates the detection of touch operations in the operation area on the front camera video.

[0146] Next, the operation content display unit 11b changes the state of the operation content within or outside the display area of the front camera video (S74). The operation content display unit 11b changes to a state where it can be seen that the command is inactive. The operation content display unit 11b may display the operation content dimly.

[0147] Next, the video outer frame drawing unit 11c deletes the frame around the display area of the front camera video (S75). Thereby, it is possible to intuitively convey to the user that the operation area has been deleted in step S73.

[0148] Next, the video display unit 11a reduces the brightness of the front camera video (S76). By reducing the brightness, it is possible to effectively convey to the operator that the command is inactive.

[0149] FIG. 19 is a diagram showing a display example of the terminal 10 according to this modification. FIG. 19 shows a display example when transitioning from the display shown in FIG. 10 to a state where the command is inactive. Further, FIG. 19 shows a display example when the vehicle state is stopped.

[0150] As shown in FIGS. 10 and 19, when transitioning to a state where the command is inactive, the frame 11c1 is deleted, the operation area 12a1 is deleted, and the operation content 11b1 is displayed dimly.

[0151] Thus, when the command cannot be executed, the terminal 10 may make the command inactive and change the display state of at least one of the frame, the operation content, and the video. Making the command inactive means invalidating the detection of touch operations in the operation area corresponding to the command.

[0152] Further, the terminal 10 can switch between enabling and disabling the detection of touch operations on the display area of the video 11aF based on the vehicle state. Then, the operation content display unit 11b may change the display mode of the command display according to whether the detection of the touch operation on the display area of the video 11aF is enabled or disabled. Further, the video outer frame drawing unit 11c may change the display mode of the frame 11c1 according to whether the detection of the touch operation on the display area of the video 11aF is enabled or disabled. Further, the video display unit 11a may change the display mode of the video 11aF according to whether the detection of the touch operation on the display area of the video 11aF is enabled or disabled.

[0153] (Modification Example 6 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 20 to 22. In this modification example, an example of changing the command to be executed according to the state of the moving body 40 will be described. FIG. 20 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification example.

[0154] As shown in FIG. 20, when the execution command management unit 13a acquires the vehicle state of the moving body 40 (S71), it determines whether the vehicle state has changed (S81). The execution command management unit 13a may execute the determination in step S81 based on whether the vehicle state of the moving body 40 newly acquired in step S71 matches the vehicle state of the moving body 40 acquired most recently.

[0155] Next, when the execution command management unit 13a determines that the vehicle state has changed (yes in S81), it determines an execution command according to the changed vehicle state (S82). It can be said that the execution command management unit 13a automatically updates the execution command based on the change in the vehicle state. The execution command management unit 13a makes the determination in step S82 based on a table showing the relationship between the vehicle state and the execution command. Further, when the execution command management unit 13a determines that the vehicle state has not changed (no in S81), it ends the process.

[0156] FIG. 21 is a table showing the relationship between the vehicle state and the execution command according to this modification example. The table shown in FIG. 21 is created in advance and stored in the storage unit of the terminal 10.

[0157] As shown in FIG. 21, in the table, the vehicle state and the execution command at that time are associated with each other. When the vehicle state is "running", "temporary stop" is associated, when it is "stopped", "start driving" is associated, and when it is "in emergency stop", "release emergency stop" is associated. Note that "-" indicates that there is no command executable by the operator.

[0158] Referring to FIG. 20 again, next, the operation content display unit 11b changes the execution command of the operation area on the front camera image to the execution command determined in step S82 (S83). The operation content display unit 11b switches the command associated with the operation area based on the state of the moving body 40. It can also be said that the operation content display unit 11b updates the operation content of the execution command of the operation area.

[0159] Next, the operation content display unit 11b changes the presentation mode of the operation content according to the execution command type (S84). The operation content display unit 11b changes the presentation mode of the operation content based on, for example, a table in which the presentation mode of the operation content and the execution command type are associated. Thereby, it is possible to notify the operator that the operation content has been changed by changing the presentation mode of the operation content.

[0160] Next, the video outer frame drawing unit 11c changes the presentation mode of the frame around the display area of the video according to the execution command type (S85). The video outer frame drawing unit 11c changes the presentation mode of the frame based on, for example, a table in which the presentation mode of the frame and the execution command type are associated. Thereby, it is possible to notify the operator that the operation content has been changed by changing the presentation mode of the frame.

[0161] Note that only one of steps S84 and S85 may be executed.

[0162] FIG. 22 is a diagram showing a display example of the terminal 10 according to this modification example. FIG. 22(a) shows a display example when the vehicle state before the change (during driving), and FIG. 22(b) shows a display example when the vehicle state after the change (stopped).

[0163] When the vehicle state of the moving body 40 switches from driving to stopped, the display mode of the frame is changed from frame 11c1 to frame 11c2, and the display mode of the operation content is changed from operation content 11b1 to operation content 11b2.

[0164] Thus, when the command to be executed is changed, at least one display mode of the frame and the operation content may be changed.

[0165] (Modification Example 7 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 23 and 24. In this modification example, an example in which a command execution function is assigned to a video other than the front view will be described. FIG. 23 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification example. FIG. 24 is a diagram showing a display example of the terminal 10 according to this modification example.

[0166] As shown in FIG. 23, the processing shown in FIG. 9 is executed for each video area. In the case of FIG. 24, the processing shown in FIG. 9 is executed for each of the videos 11aF, 11aR, and 11aL. For example, different commands may be set for each of the videos 11aF, 11aR, and 11aL according to the vehicle state.

[0167] In the example of FIG. 24, an example is shown in which fixed - phrase speech commands are assigned to the left and right cameras. Specifically, in FIG. 24, an example is shown in which an operation area 12a1 for video 11aF is generated, an operation area 12a2 for video 11aL is generated, and an operation area 12a3 for video 11aR is generated. Also, in FIG. 24, an example is shown in which the operation content 11b1 for video 11aF is pause, the operation content 11b3 for video 11aL is to speak "I'll pass through", and the operation content 11b4 for video 11aR is to speak "After you, please". Further, in FIG. 24, an example is illustrated in which a frame 11c1 for operation area 12a1 is presented, a frame 11c3 for operation area 12a2 is presented, and a frame 11c4 for operation area 12a3 is presented.

[0168] Thus, for example, when a touch operation is performed on operation area 12a2, the moving body 40 makes a speech of "I'll pass through" outside the moving body 40, and when a touch operation is performed on operation area 12a3, the moving body 40 makes a speech of "After you, please" outside the moving body 40.

[0169] (Modification Example 8 of the Embodiment) Hereinafter, the remote monitoring method and the like according to this modification example will be described with reference to FIGS. 25 to 28. In this modification example, an example in which the video assignment destination of the command is changed according to the traveling direction will be described. FIG. 25 is a flowchart showing the operation (remote monitoring method) of the terminal 10 according to this modification example.

[0170] As shown in FIG. 25, the terminal 10 acquires the path information of the moving body 40 to be remotely monitored (S91). The terminal 10 acquires, for example, information such as destination setting and travel route setting input by the user of the moving body 40 into a mobile terminal such as a navigation device or a smartphone mounted on the moving body 40.

[0171] FIG. 26 is a diagram showing an example of the path information according to this modification example.

[0172] As shown in FIG. 26, the route information indicates the next behavior of each autonomous mobile body 40. The next behavior indicates how the mobile body 40 will travel next at the current time. For example, it can be seen that if the mobile body has a mobile body ID of 51, it is planned to turn right next, and if the mobile body has a mobile body ID of 60, it is planned to move forward next.

[0173] Such route information can be specified based on, for example, the set destination, the travel route to the destination, the current position of the mobile body 40, map information, and the like.

[0174] Referring to FIG. 25 again, next, the execution command management unit 13a determines the video to which the execution command is assigned based on the route information (S92). The execution command management unit 13a makes the determination in step S92 based on a table in which the route and the video to which the execution command is assigned are associated.

[0175] FIG. 27 is a table showing the relationship between the route and the command assignment destination according to this modification example. The table shown in FIG. 27 is created in advance and stored in the storage unit of the terminal 10. FIG. 27 shows an example where the command is a travel start command. For example, a table shown in FIG. 27 may be created for each command. Also, in the table shown in FIG. 27, it is assumed that the mobile body 40 is currently stopped and the operator performs a touch operation to execute the travel start command, so that the mobile body 40 starts to travel.

[0176] As shown in FIG. 27, in the table, the route and the travel start command assignment destination are associated. The route indicates the current traveling direction of the mobile body 40. Moving forward indicates that the mobile body 40 is planned to move forward from the current position. Also, turning right indicates that the mobile body 40 is planned to turn right from the current position.

[0177] For example, if the mobile body has a mobile body ID of 51, the destination of the travel start command is determined to be the right camera image. Also, if the mobile body has a mobile body ID of 60, the destination of the travel start command is determined to be the front camera image. In this way, according to the next behavior of the mobile body 40, the camera image that the operator should pay more attention to is determined as the destination of the travel start command.

[0178] Referring to FIG. 25 again, next, the video display input unit 12a sets an operation area on the assigned video (S93). The video display input unit 12a enables detection of a touch operation on the area of the touch panel (operation area) that overlaps the assigned video.

[0179] Next, the video display input unit 12a deletes the operation area on the unassigned video (S94). If an operation area is set for a video other than the video determined in step S92, the video display input unit 12a deletes the area. The video display input unit 12a disables detection of a touch operation on the area of the touch panel that does not overlap the assigned video (operation area on the unassigned video).

[0180] Next, the operation content display unit 11b presents the operation content within or outside the display area of the assigned video (S95).

[0181] Next, the operation content display unit 11b deletes the operation content within or outside the display area of the unassigned video (S96). If the operation content for a video other than the video determined in step S92 is presented, the operation content display unit 11b deletes the operation content.

[0182] Next, the video outer frame drawing unit 11c presents a frame around the display area of the assigned video (S97). The video outer frame drawing unit 11c presents, for example, a frame surrounding the video.

[0183] Next, the video outer frame drawing unit 11c deletes the frames around the display areas of the videos that have not been assigned (S98). When a frame for a video other than the video determined in step S92 is presented, the video outer frame drawing unit 11c deletes the frame.

[0184] FIG. 28 is a diagram showing a display example of the terminal 10 according to this modified example. FIG. 28 shows a display example when the vehicle state is stopped.

[0185] FIG. 28(a) shows the display when the next driving route is forward, and an operation area 12a1, an operation content 11b1, and a frame 11c1 are provided in the front camera video (video 11aF) associated with forward movement.

[0186] In the case of FIG. 28(a), the video 11aF is a video obtained by imaging the forward direction (an example of the first direction) in which the moving body 40 can move, and the videos 11aL and 11aR are videos obtained by imaging the right and left turning directions (an example of the second direction) in which the moving body 40 can move, and are videos displayed in areas different from the display area of the video 11aF.

[0187] FIG. 28(b) shows the display when the next driving route is a right turn, and an operation area 12a1, an operation content 11b1, and a frame 11c1 are provided in the right camera video (video 11aR) associated with the right turn.

[0188] Thus, since it is considered that the operator will inevitably look at the next destination, an operation area, an operation content, and a frame are provided in the video that the operator is considered to look at.

[0189] In the case of FIG. 28(b), the video 11aR is a video obtained by imaging the right turn direction (an example of the first direction) in which the moving body 40 can move, and the videos 11aL and 11aF are videos obtained by imaging the forward or left turn direction (an example of the second direction) in which the moving body 40 can move, and are videos displayed in areas different from the display area of the video 11aR.

[0190] Note that at least steps S91 to S93 may be executed in FIG. 25.

[0191] (Other Embodiments) As described above, the remote monitoring method and the like according to one or more aspects have been described based on the embodiments and the like. However, the present disclosure is not limited to these embodiments and the like. Without departing from the spirit of the present disclosure, various modifications conceived by those skilled in the art applied to this embodiment, or forms constructed by combining components in different embodiments may also be included in the present disclosure.

[0192] For example, in the above embodiments and the like, the terminal 10 may include a display unit, or may be communicably connected to an external display unit.

[0193] Also, in the above embodiments and the like, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0194] Also, the order in which each step in the flowchart is executed is for the purpose of illustration for specifically describing the present disclosure, and may be other than the above order. Also, a part of the above steps may be executed simultaneously (in parallel) with other steps, or a part of the above steps may not be executed.

[0195] Also, the division of the functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or a part of the functions may be transferred to other functional blocks. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or in time division.

[0196] Also, the terminal 10 according to the above-described embodiment or the like may be realized as a single device, or may be realized by a plurality of devices. When the terminal 10 is realized by a plurality of devices, each component included in the terminal 10 may be distributed among the plurality of devices in any manner. When the terminal 10 is realized by a plurality of devices, the communication method between the plurality of devices is not particularly limited, and may be wireless communication or may be wired communication. Also, wireless communication and wired communication may be combined between the devices.

[0197] Also, each component described in the above-described embodiment or the like may be realized as software, or typically may be realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include some or all of them. Here, LSI is mentioned, but depending on the degree of integration, it may also be referred to as IC, system LSI, super LSI, or ultra LSI. Also, the method of integrating into an integrated circuit is not limited to LSI, and it may be realized by a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After manufacturing the LSI, an FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that can reconfigure the connection or setting of circuit cells inside the LSI may be used. Furthermore, if a technology for integrating into an integrated circuit that replaces the LSI appears due to the progress of semiconductor technology or a derived alternative technology, naturally, the integration of components may be performed using that technology.

[0198] A system LSI is a super multi-functional LSI manufactured by integrating a plurality of processing units on one chip. Specifically, it is a computer system including a microprocessor, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. A computer program is stored in the ROM. When the microprocessor operates according to the computer program, the system LSI achieves its function.

[0199] In addition, one aspect of the present disclosure may be a computer program that causes a computer to execute each characteristic step included in the remote monitoring method shown in any one of FIGS. 4, 6, 7, 9, 11, 13, 16, 18, 20, 23, and 25.

[0200] Also, for example, the program may be a program for causing a computer to execute. Further, one aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and by causing the processor to execute the program, it becomes possible to cause the device to perform each of the above processes.

Industrial Applicability

[0201] The present disclosure is useful for a remote monitoring system that remotely monitors an autonomous mobile object.

Explanation of Signs

[0202] 1 Remote monitoring system 10 Terminal (remote monitoring device) 11 Output device 11a Video display unit (first control unit) 11aF, 11aL, 11aR Video 11b Operation content display unit 11b1, 11b2, 11b3, 11b4 Operation content 11c Video outer frame drawing unit 11c1, 11c2, 11c3, 11c4 Frame 12 Input device 12a Video display input unit (second control unit) 12a1, 12a2, 12a3 Operation area 13 Management device 13a Execution command management unit 20 Control server 21 Mobile object information transmission management unit 22 Command transmission unit 23 Image Transmission Unit 24 Mobile Body Information Transmission Unit 30 Infrastructure Camera 31 Infrastructure Image Acquisition Unit 40 Mobile Body 41 Command Execution Unit 42 Image Acquisition Unit 43 Mobile Body Information Acquisition Unit A Operator

Claims

1. A remote monitoring method for remotely monitoring a self-driving mobile body, comprising: displaying a first video image captured in the directions in which the mobile body can move; displaying the operation content to be executed on the mobile body around or in the first area where the first video image is displayed; enabling detection of touch operations on the area where the operation content is displayed; enabling detection of touch operations on the first area; when a touch operation on the first area is detected and when a touch operation on the area where the operation content is displayed is detected, executing the operation content on the mobile body Remote monitoring method.

2. A remote monitoring method for remotely monitoring a self-driving mobile body, comprising: displaying a first video image captured in the directions in which the mobile body can move; enabling detection of touch operations on the first area where the first video image is displayed; switching to disabling the detection of touch operations on the first area based on the state of the mobile body Remote monitoring method.

3. A remote monitoring method for remotely monitoring a self-driving mobile body, comprising: displaying a first video image captured in the directions in which the mobile body can move; enabling detection of touch operations on the first area where the first video image is displayed; displaying a frame surrounding the first area; displaying, around or in the first area, the operation content to be executed by touching the first area; displaying the frame and the operation content in the same display mode; being able to switch to disabling the detection of touch operations on the first area based on the state of the mobile body; varying the display mode of the operation content according to whether the detection of touch operations on the first area is enabled or disabled Remote monitoring method.

4. A remote monitoring method for remotely monitoring a self-driving mobile body, comprising: displaying a first video image captured in the directions in which the mobile body can move; enabling detection of touch operations on the first area where the first video image is displayed; displaying a frame surrounding the first area; being able to switch to disabling the detection of touch operations on the first area based on the state of the mobile body; varying the display mode of the frame according to whether the detection of touch operations on the first area is enabled or disabled Remote monitoring method.

5. A remote monitoring method for remotely monitoring a self-driving mobile body, comprising: displaying a first video image captured in the directions in which the mobile body can move; Enable detection of a touch operation on the first area where the first video is displayed, Display a frame surrounding the first area, Based on the state of the moving body, it is possible to switch to disabling the detection of a touch operation on the first area, Differentiate the display mode of the first video according to whether the detection of a touch operation on the first area is enabled or disabled Remote monitoring method.

6. Display the operation content executed by touching the first area around or in the first area, The remote monitoring method according to claim 4 or 5.

7. Display a frame surrounding the first area, The remote monitoring method according to claim 1 or 2.

8. Display the operation content executed by touching the first area around or in the first area, The frame and the operation content are displayed in the same display mode, The remote monitoring method according to claim 7.

9. The frame and the operation content are displayed in the same color, The remote monitoring method according to claim 8.

10. When receiving a touch operation on the first area, change the display mode of the frame, The remote monitoring method according to claim 7.

11. Change the display mode of the frame according to the execution state of the operation content on the moving body, The remote monitoring method according to claim 8.

12. Based on the state of the moving body, it is possible to switch to disabling the detection of a touch operation on the first area, Differentiate the display mode of the operation content according to whether the detection of a touch operation on the first area is enabled or disabled, The remote monitoring method according to claim 8.

13. Based on the state of the moving body, it is possible to switch to disabling the detection of a touch operation on the first area, Differentiate the display mode of the frame according to whether the detection of a touch operation on the first area is enabled or disabled, The remote monitoring method according to claim 7.

14. Based on the state of the moving body, it is possible to switch to disabling the detection of a touch operation on the first area, Differentiate the display mode of the first video according to whether the detection of a touch operation on the first area is enabled or disabled, The remote monitoring method according to claim 7.

15. Based on the state of the moving body, switch the operation content associated with the first area, The remote monitoring method according to any one of claims 1 to 5.

16. When a touch operation on the first area is detected, change the display mode of the first video. The remote monitoring method according to any one of claims 1 to 5.

17. When a touch operation on the first area is detected, change the display mode of the frame. The remote monitoring method according to claim 7.

18. Change the display mode of the first video according to the execution state of remote monitoring of the mobile body. The remote monitoring method according to any one of claims 1 to 5.

19. The first video is a video captured in a first direction in which the mobile body can move. Furthermore, display a second video captured in a second direction in which the mobile body can move in a second area different from the first area. Acquire the path information of the mobile body. When the moving direction of the mobile body based on the path information is the first direction, enable the detection of a touch operation on the first area. The remote monitoring method according to any one of claims 1 to 5.

20. There are a plurality of operation types in the touch operation. Differentiate the display mode of the first video and the operation content according to the operation type of the touch operation performed on the first area. The remote monitoring method according to claim 6.

21. A remote monitoring device for remotely monitoring a mobile body that autonomously travels, A first control unit that performs control to display an operation content to be executed on the mobile body on a video captured in a direction in which the mobile body can move, and around or in the first area where the video is displayed. A second control unit that enables the detection of a touch operation on the area where the operation content is displayed. The second control unit enables the detection of a touch operation on the first area. When a touch operation on the first area is detected and when a touch operation on the area where the operation content is displayed is detected, the operation content is executed on the mobile body. Remote monitoring device.

22. A program for causing a computer to execute the remote monitoring method according to any one of claims 1 to 5.

23. The outer periphery surrounding the operation content is in contact with the outer periphery surrounding the first area. The remote monitoring method according to claim 1.

24. Display the frame surrounding the first area where the detection of the touch operation is effective more prominently than other frames. The remote monitoring method according to claim 7.

25. Display the operation content executed by touching the first area around or in the first area. When the first region is touched, the frame and the operation content are displayed in the same display mode. The remote monitoring method according to claim 7.

26. Display the operation content executed by touching the first region around or in the first region. The frame and the operation content are displayed in the same color and are in contact. The remote monitoring method according to claim 8.

27. Further change the display mode of the operation content according to the execution state of the operation content for the mobile body. The remote monitoring method according to claim 11.

28. The execution state includes a state where the execution of the operation content is successful and a state where the execution of the operation content fails. Change the display mode of the frame depending on whether the execution of the operation content is successful or the execution of the operation content fails. The remote monitoring method according to claim 11.

29. When the execution of the operation content fails, disable the detection of the touch operation in the first region. The remote monitoring method according to claim 28.

30. Switch the display mode of the operation content according to the switching of the operation content associated with the first region. The remote monitoring method according to claim 15.

31. When a touch operation on the first region is detected, change to different display modes according to the type of the touch operation. The remote monitoring method according to claim 16.

32. A remote monitoring device for remotely monitoring a mobile body that autonomously travels, A first control unit that performs control to display a video obtained by imaging the direction in which the mobile body can move, A second control unit that enables detection of a touch operation on the region where the video is displayed, The second control unit switches to disable detection of a touch operation on the region based on the state of the mobile body. Remote monitoring device.

33. A remote monitoring device for remotely monitoring a mobile body that autonomously travels, A first control unit that performs control to display a first video obtained by imaging the direction in which the mobile body can move, A second control unit that enables detection of a touch operation on a first region where the first video is displayed, The first control unit, Displays a frame surrounding the first region, Displays the operation content executed by touching the first region around or in the first region, The frame and the operation content are displayed in the same display mode, The second control unit, Based on the state of the mobile body, can switch to disable detection of a touch operation on the first region. The first control unit varies the display mode of the operation content according to whether the detection of a touch operation on the first area is valid or invalid. Remote monitoring device.

34. A remote monitoring device for remotely monitoring a mobile body that autonomously travels, a first control unit that performs control to display a first video obtained by imaging the direction in which the mobile body can move, a second control unit that enables detection of a touch operation on a first area where the first video is displayed, the first control unit displays a frame surrounding the first area, the second control unit can switch to disabling the detection of a touch operation on the first area based on the state of the mobile body, the first control unit varies the display mode of the frame according to whether the detection of a touch operation on the first area is valid or invalid. Remote monitoring device.

35. A remote monitoring device for remotely monitoring a mobile body that autonomously travels, a first control unit that performs control to display a first video obtained by imaging the direction in which the mobile body can move, a second control unit that enables detection of a touch operation on a first area where the first video is displayed, the first control unit displays a frame surrounding the first area, the second control unit can switch to disabling the detection of a touch operation on the first area based on the state of the mobile body, the first control unit varies the display mode of the first video according to whether the detection of a touch operation on the first area is valid or invalid. Remote monitoring device.

Images