Pilot system, pilot method and program

Abstract

To provide an operating system capable of reducing discomfort caused to an operator when switching a moving object to be controlled.SOLUTION: A operating system 1 operates an operation target moving object 100 selected from a plurality of moving objects 100. The operating system 1 includes a display unit for displaying display information including images of the surroundings of the moving objects 100, and a switching unit that uses movement information indicating movements of the moving object 100 to switch an operation target from a first moving object which is a switching source to a second moving object which is a switching destination. In at least any one case of after the second moving body starts moving based on the movement information of the first moving object, after the first moving body starts moving based on the movement information of the second moving body, and after the display unit displays a video adjusted according to the movement information of the first moving object, the switching unit switches the operation target from the first moving object to the second moving object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a piloting system, a piloting method, and a program. [Background technology]

[0002] There is a known technology in which an operator remotely controls a moving body such as an autonomously moving robot via a control terminal. As a related technology, for example, Patent Document 1 discloses a remote control system that allows an operator to properly check the road surface conditions in the distance while concentrating on remotely controlling a moving body in motion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6614607 Summary of the Invention [Problem to be solved by the invention]

[0004] Assume that an operator remotely controls multiple robots via a control terminal using a system such as that disclosed in Patent Document 1. The control terminal displays images captured by the camera of the robot currently being controlled by the operator. The control terminal also displays on its screen multiple robots currently in operation and accepts the operator's selection of the robot to be controlled.

[0005] The display screen may be, for example, a list format that displays the robots in a list, or a map format that displays the current positions of the robots superimposed on a map. The operator selects a robot to be operated from the multiple operating robots and issues a switch request. The operation terminal accepts the switch request and switches the robot to be operated.

[0006] In such a case, the control terminal transmits an instruction signal to the original robot and stops controlling the remote robot currently being controlled. The control terminal also needs to temporarily terminate the video of the original robot that was being displayed on the screen, transition the screen to a robot selection screen, and have the operator select another robot. Here, if there is a discrepancy in movement or speed between the original robot and the target robot, the switching of the controlled object may cause the remote operator to feel uncomfortable. In such a case, there is a problem that every time the controlled robot is switched, the feeling of moving with the robot being operated is interrupted, and the operator cannot feel a continuous movement experience.

[0007] In consideration of the above-mentioned problems, an object of the present disclosure is to provide a control system, a control method, and a program that can reduce the discomfort felt by a pilot when switching between moving objects to be controlled. [Means for solving the problem]

[0008] The control system according to the present disclosure comprises: A control system for controlling a target moving object selected from a plurality of moving objects, comprising: A display unit that displays display information including an image of the surroundings of the moving object; A switching unit that switches the object to be controlled from a first moving body that is a switching source to a second moving body that is a switching destination, using operation information that indicates an operation of the moving body; The switching unit is After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and After the display unit displays the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body.

[0009] The steering method according to the present disclosure comprises: A computer of a control system that controls a target moving object selected from a plurality of moving objects, a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the operation target from a first moving body, which is a switching source, to a second moving body, which is a switching destination, by using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body.

[0010] The program according to the present disclosure is A program to be executed by a computer of a control system for controlling a target moving body selected from a plurality of moving bodies, a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the operation target from a first moving body, which is a switching source, to a second moving body, which is a switching destination, by using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body. Effect of the Invention

[0011] The control system, control method, and program disclosed herein can reduce the discomfort felt by the control operator when switching between moving objects to be controlled. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of a flight control system according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of use of the flight control system according to the first embodiment. [Diagram 3] 1 is a block diagram showing a configuration of a moving object according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the appearance of a moving object according to the first embodiment. [Diagram 5] FIG. 1 is a block diagram showing the configuration of a control device according to a first embodiment. [Figure 6] 1 is a diagram showing an example of the appearance of a control device according to a first embodiment. FIG. [Figure 7] FIG. 4 is a diagram showing an example of a display screen displayed by a display unit according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing an example of another display screen displayed by the display unit according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a display screen on which icons according to the first embodiment are displayed. [Figure 10] FIG. 4 is a diagram showing how the display mode of an icon changes with the movement of a moving object according to the first embodiment. [Figure 11] FIG. 4 is a diagram showing how the display mode of an icon changes with the movement of a moving object according to the first embodiment. [Figure 12] FIG. 4 is a diagram showing how the display mode of an icon changes with the movement of a moving object according to the first embodiment. [Figure 13] FIG. 4 is a diagram showing a display screen after a switching process according to the first embodiment. [Figure 14] 4 is a flowchart showing the processing performed by the control device according to the first embodiment. [Figure 15] FIG. 13 is a diagram showing an example of a display screen according to a modified example of the first embodiment. [Figure 16] FIG. 11 is a block diagram showing the configuration of a control device according to a second embodiment. [Figure 17] 11 is a flowchart showing the processing performed by the control device according to the second embodiment. [Figure 18] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer that realizes a control device, etc. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For clarity of explanation, duplicate explanations will be omitted as necessary.

[0014] <Embodiment 1> (Configuration of pilot system 1) A first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of a control system 1 according to this embodiment. As shown in Fig. 1, the control system 1 includes a plurality of moving bodies 100 and a control device 10 for controlling the plurality of moving bodies 100. The control device 10 can also be called a control system for controlling the plurality of moving bodies 100. Although three moving bodies 100 are shown in the figure, the number of moving bodies 100 is not limited to this.

[0015] The control system 1 according to this embodiment is an information processing system that enables a pilot to remotely control multiple moving bodies 100 using a control device 10. The control device 10 controls the multiple moving bodies 100. The control device 10 also receives a selection of a moving body to be controlled from the pilot among the multiple moving bodies 100.

[0016] The multiple moving objects 100 and the control device 10 are connected via a network N. The network N is, for example, a wired or wireless communication line. Although an example is shown here in which one control device 10 is connected to multiple moving objects 100, multiple control devices 10 may be provided.

[0017] 2 is a diagram showing an example of use of the control system 1 according to this embodiment. The moving body 100 is, for example, a mobile robot capable of autonomously moving within an environment. The environment may be any space in which the moving body 100 can move. The environment may include, for example, a residence, a public facility, an office, a store, or a hospital. The environment may be indoors or outdoors.

[0018] The moving body 100 is equipped with a moving means for moving within an environment. The moving means may include, for example, a drive mechanism for moving on the ground, in the air, on the water surface, or underwater. The moving body 100 is not limited to the above-mentioned mobile robot, but may be, for example, an autonomous vehicle, a drone, or the like.

[0019] As shown in FIG. 2, the pilot P and the moving body 100 are located in different locations. For example, the pilot P is at home, and the moving body 100 is located in a hospital. The example in the figure shows a wheelchair user U receiving support from the moving body 100 at the hospital. The pilot P operates the control device 10 at home or the like to control the moving body 100. The locations of the pilot P, user U, and moving body 100 are not limited to this, and may be any location.

[0020] (Configuration of moving body 100) Next, the configuration of the moving body 100 according to this embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing the configuration of the moving body 100. Fig. 4 is a diagram showing an example of the appearance of the moving body 100. As shown in Fig. 3 and Fig. 4, the moving body 100 includes a main body unit 101, an imaging unit 121, a display unit 122, a head 120, an arm unit 130, a moving unit 140, and a control device 110.

[0021] The main body 101 is a trunk portion of the moving body 100. The main body 101 is connected to the head 120, the arm 130, and the moving unit 140. The main body 101 has a built-in control device 110. The main body 101 may include an extension mechanism for extending and retracting in the vertical direction. The main body 101 may also include a rotation mechanism for rotating about the axis of the main body 101.

[0022] The image capturing unit 121 is an image capturing device that captures images of the surroundings of the moving body 100 to obtain images. The image capturing unit 121 captures images of the environment from a predetermined direction, and outputs the images obtained by capturing images to the control device 110. The predetermined direction is, for example, the direction in which the head 120 in which the image capturing unit 121 is provided faces, but is not limited to this.

[0023] The photographing unit 121 is, for example, a two-dimensional camera. The photographing unit 121 may include a three-dimensional camera. The photographing unit 121 may be, for example, a camera (RGB-D camera, stereo camera), a laser range finder, an ultrasonic sensor, a three-dimensional distance sensor, or the like.

[0024] 4, the image capturing unit 121 is provided on the head 120 of the moving object 100, but is not limited to this. In addition, the moving object 100 may be provided with a plurality of image capturing units 121.

[0025] The display unit 122 is a display device that displays video. For example, the display unit 122 displays video acquired from the control device 10. The video may be video acquired by the imaging unit 50 of the control device 10. In the example shown in FIG. 4, the display unit 122 displays an image of the face of the operator P. The display unit 122 is not limited to video, and may display still images, characters, and the like. Furthermore, the display unit 122 may display video captured by the imaging unit 121, or may display information stored in the memory unit 114.

[0026] The display unit 122 may include, for example, a liquid crystal display, an organic EL display, etc. The display unit 122 may be a touch panel having a function of an input unit that receives input when the user touches it with a finger or the like.

[0027] The head 120 is provided on the upper part of the main body 101 of the moving body 100. The head 120 includes an image capturing unit 121 and a display unit 122. The head 120 can be provided with an extension mechanism (not shown) so that it can be extended and retracted in the vertical direction. This allows the image capturing unit 121 to capture images at different heights. The head 120 may also include an extension mechanism for enabling it to be extended and retracted in the vertical direction. The head 120 may also include a rotation mechanism for rotating around the axis of the head 120. This allows the image capturing unit 121 to capture images over a wider range.

[0028] The arm unit 130 includes an arm 132 supported by the main body unit 101, and a hand 131 provided at the tip of the arm 132. The arm 132 is configured as, for example, an articulated arm. The arm 132 is provided on the front side of the main body unit 101, but is not limited thereto and may be provided on the side of the main body unit 101, etc. The arm 132 changes the position and posture of the arm 132 by operating a drive mechanism (not shown) including a motor or the like in accordance with a control signal from the control device 110. The arm 132 may be configured to include an extension mechanism so that the length of the arm 132 can be changed.

[0029] The hand 131 grasps an object or the like in accordance with a control signal from the control device 110. The hand 131 changes its position and posture by the operation of a drive mechanism (not shown) equipped with a motor or the like. The hand 131 may be, for example, a two-fingered hand equipped with a pair of grasping members like two human fingers as shown in Fig. 4, or may be a three-fingered hand or a multi-fingered hand equipped with more fingers. The hand 131 is not limited to these, and various hand effectors may be used as the hand 131.

[0030] The moving unit 140 is a moving device that moves the moving body 100 within the environment in which the moving body 100 exists. The moving unit 140 moves the moving body 100 in accordance with a control signal from the control device 110. The moving unit 140 also moves the moving body 100 in accordance with a switching control signal from the pilot device 10. The switching control signal may include operation information of the other moving body 100 that is the switching source.

[0031] Here, the motion information is information related to the motion of the moving body 100. The motion information may include, for example, information related to the motion of the moving body 100. The motion information may include, for example, parameters such as the speed, acceleration, jerk, motion direction, or vibration of the moving body 100. Depending on the configuration or motion form of the moving body 100, the motion information may include the pitch angle, roll angle, or yaw angle of the moving body 100. The moving body 100 is equipped with sensors capable of detecting these pieces of motion information, and thus can acquire various pieces of motion information as described above.

[0032] The operation information may be information about the operation of each functional unit included in the moving object 100. For example, the operation information may be the position of the image capturing unit 121 or the attitude of the image capturing unit 121. The attitude of the image capturing unit 121 is, for example, the inclination of the camera with respect to the horizontal plane.

[0033] The moving unit 140 includes, for example, a pair of two opposing drive wheels and a pair of two driven wheels that can rotate left and right. This allows the moving body 100 to move in all directions. The moving unit 140 can rotate the drive wheels to perform any travel of the moving body 100, such as forward and backward movement, left and right turning, acceleration / deceleration, and stopping. The moving unit 140 realizes the movement of the moving body 100 by operating a drive mechanism (not shown) composed of, for example, a motor based on a control signal from the control device 110.

[0034] In this embodiment, the moving body 100 is a mobile robot that travels on the ground. However, if the moving body 100 moves in a different manner, a drive mechanism suitable for the manner of movement may be provided.

[0035] The control device 110 includes an operation control unit 111 , an operation information acquisition unit 112 , a transfer unit 113 , and a storage unit 114 .

[0036] The operation control unit 111 controls the operation of each of the functional units described above. That is, the operation control unit 111 controls the operation of the main body unit 101, the imaging unit 121, the display unit 122, the head unit 120, the arm unit 130, and the moving unit 140.

[0037] The motion information acquisition unit 112 acquires motion information related to the motion of the moving body 100. As described above, the motion information may include, for example, the speed, acceleration, jerk, moving direction, vibration, position of the image capturing unit 121, or the attitude of the image capturing unit 121 of the moving body 100. The motion information acquisition unit 112 acquires motion information by acquiring detection values ​​from a sensor capable of detecting such motion information. The motion information acquisition unit 112 may acquire motion information from a sensor other than that provided in the moving body 100. For example, the motion information acquisition unit 112 may acquire an image captured by a high-precision infrastructure camera (such as a surveillance camera), an image captured by an artificial satellite, or sensor information of a mobile terminal brought into the moving body 100, and acquire motion information. The motion information acquisition unit 112 may acquire motion information using a well-known image recognition technology. The motion information acquisition unit 112 may be provided not only inside the moving body 100, but also outside the moving body 100.

[0038] The transfer unit 113 transfers the video captured by the imaging unit 121 and the motion information acquired by the motion information acquisition unit 112 to the control device 10. The transfer unit 113 may transfer the video and motion information stored in the storage unit 114. The transfer unit 113 may be configured to include a communication interface with the network N, for example. The transfer unit 113 transfers the video and motion information to the control device 10 at a predetermined timing. The predetermined timing may be a predetermined time interval. The predetermined timing may also be when a transfer request is received from the control device 10.

[0039] The storage unit 114 is a storage device that stores programs for implementing each function of the moving object 100. The storage unit 114 may also store the video captured by the imaging unit 121 and the motion information acquired by the motion information acquisition unit 112.

[0040] (Configuration of the control device 10) Next, the configuration of the control device 10 according to the present embodiment will be described with reference to FIGS. Fig. 5 is a block diagram showing the configuration of the control device 10. Fig. 6 is a diagram showing an example of the external appearance of the control device 10.

[0041] The control device 10 is a computer for performing the switching process according to this embodiment. The control device 10 may be, for example, a PC (Personal Computer), a tablet terminal, a smartphone, or a mobile phone terminal. The control device 10 includes an input unit 11, an image capture unit 50, a display unit 60, and a control device 12.

[0042] The input unit 11 is an input device that accepts input from the operator P. In Fig. 5 and Fig. 6, a headset 20, a controller 30, and a mouse 40 are shown as examples of the input unit 11. These are only examples of the input unit 11, and therefore the input unit 11 may include some of these, or may include input devices other than these.

[0043] The headset 20 includes a speaker 20a and a microphone 20b. The speaker 20a outputs sound. The microphone 20b collects the sound of the pilot P. By wearing the headset 20 on his / her head, the pilot P inputs and outputs sound to and from the control device 10 using the speaker 20a and the microphone 20b. This allows the pilot P to converse with a user in the vicinity of the moving body 100 to be controlled. The headset 20 may be used for voice input to the moving body 100. For example, the control device 10 may recognize the voice of the pilot P using well-known voice recognition technology.

[0044] The controller 30 receives operations from the operator P and performs input operations on the control device 10. The controller 30 may be, for example, a game controller used for video games and the like.

[0045] The controller 30 includes, for example, a cross key 30a, a joystick 30b, and a plurality of buttons 30c. The operator P uses the controller 30 to control the moving body 100. The mouse 40 receives an operation from the operator P and performs an input operation on the control device 10. The operator P can select an icon displayed on the display unit 60 using the input unit 11.

[0046] The imaging unit 50 is an imaging device that captures images of the surroundings of the control device 10 from a predetermined position to obtain video. The predetermined position can be a position where the operator P can be photographed. In the example shown in the figure, the imaging unit 50 is provided on the upper part of the display unit 60. In this example, the imaging unit 50 captures an area including the face area of ​​the operator P. The imaging unit 50 may be a two-dimensional camera, similar to the imaging unit 121. The imaging unit 50 outputs the video acquired by photographing to the control device 10.

[0047] The display unit 60 displays display information including an image captured around the moving body 100. The image may be an image captured by the image capturing unit 121, or an image captured by another image capturing device. The other image capturing device may be a camera provided in a mobile terminal brought into the moving body 100, or a camera provided in a wearable terminal worn by a passenger of the moving body 100. Images supplied from various image supply sources may be displayed, without being limited to these.

[0048] The display unit 60 may be a display device that displays images and the like. Like the display unit 122, the display unit 60 may be, for example, a liquid crystal display, an organic EL display, or the like. The display unit 60 may also be a touch panel equipped with the function of an input unit. The display unit 60 displays images and the like under the control of the display control unit 13. The display unit 60 displays a screen including images transferred from the moving body 100. In addition to these, the display unit 60 may use stereoscopic projection (hologram) technology to three-dimensionally display the surrounding environment of the moving body around the operator P. The display unit 60 may also project images onto the floor or wall around the operator P using technology such as project mapping.

[0049] An example of a display screen displayed on the display unit 60 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a display screen 61 displayed by the display unit 60. The display screen 61 shows the position of the pilot P and the positions of the multiple moving bodies 100A-100D on a map using icons. The pilot P can operate the control device 10 to control the moving body 100 in a remote location while remaining at home or the like.

[0050] In the following description, the present embodiment will be described using four moving bodies 100A-100D as the multiple moving bodies 100, but the number of moving bodies 100 is not limited to four. In addition, the following description will mainly be given assuming that the operator P is operating the moving body 100D (first moving body). The moving body 100D is the object to be operated in this case. In addition, in this case, the moving bodies 100A-100C are candidate moving bodies for the switching destination.

[0051] For example, suppose that pilot P is about to switch the control target from moving body 100D to moving body 100B (second moving body). Pilot P makes a request to control device 10 to switch the control target by clicking the icon of moving body 100B displayed on display screen 61. Note that the display screen 61 may display, without being limited to an icon, a list of the moving body names of moving bodies 100A to 100C, for example. Pilot P makes the request to switch by clicking the moving body name of moving body 100B from the list.

[0052] Here, another display screen displayed by the display unit 60 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of another display screen 61 displayed by the display unit 60. As shown in the figure, the display screen 61 has a piloted object display area 61a, a pilot display area 61b, and an image display area 61c.

[0053] The controlled object display area 61a is an area that displays information about the moving object 100 that is the controlled object. The controlled object display area 61a displays identification information for identifying the moving object 100 that the pilot P is currently controlling. The identification information may be, for example, an identification name or an identification number of each of the multiple moving objects 100. In the example shown in the figure, the controlled object display area 61a displays "100D being controlled." This indicates that the pilot P is controlling the moving object 100D.

[0054] The operator display area 61b is an area for displaying information related to the operator P. The operator display area 61b displays an image captured by the image capturing unit 50, for example.

[0055] The image display area 61c is an area for displaying an image transferred from the moving body 100 to be controlled. In the example of FIG. 8, the image display area 61c displays an image transferred from the moving body 100D. Not limited to the above, the display screen 61 may have various display areas related to the control of the moving body 100. For example, the display screen 61 may have buttons for starting and stopping the control, switching the object to be controlled, selecting whether or not to output audio, or selecting whether or not to output video, etc. Alternatively, these buttons, etc. may be arranged in an area other than the image display area 61c.

[0056] The pilot P controls the moving body 100D by performing control operations using the headset 20, the controller 30, the mouse 40, etc. while looking at the display screen 61. Control operations are operations for controlling the movement of the moving body 100D. Control operations may include, for example, operations related to the movement of the moving body 100. Operations related to the movement are operations to move the moving body 100 forward, backward, accelerate, decelerate, or rotate. If the moving body 100 is a drone capable of flight, control of the moving body 100 may include ascending or descending the moving body 100, etc. Control operations may also include extension and retraction of the head 120, extension and retraction of the arm unit 130, taking video, or outputting audio, etc.

[0057] The above-described example of the control device 10 is merely an example, and the configuration of the control device 10 can be changed as appropriate. For example, if the display unit 60 is a touch panel or the like equipped with an input function, the control device 10 may be configured not to include the controller 30 and the mouse 40. Also, a keyboard or the like may be used as the input unit 11. Furthermore, in cases where an image from the operator P's side is not transmitted to the moving body 100, the image capturing unit 50 may not be provided.

[0058] 7, the display unit 60 displays icons for receiving a switching request from the operator P in association with each of the multiple moving objects 100 in the image display area 61c.

[0059] When the operator P is operating the moving body 100D, the display unit 60 displays icons for accepting a switching request from the operator P in the image display area 61c in association with each of the moving bodies 100A to C that are candidates for the switching destination. The icons may be ones that imitate the external appearance of the moving body 100, as shown in Fig. 7. Here, an icon using a different display form will be described with reference to Fig. 9.

[0060] 9 is a diagram showing an example of a display screen 61 on which an icon in the shape of a door is displayed. As shown in the figure, an image display area 61c includes icons 70a to 70c associated with moving objects 100A to 100C. In the image display area 61c, icons 70a to 70c are displayed superimposed on an image captured by moving object 100D using an augmented reality technique.

[0061] 9, for example, the icons 70a to 70c may include an entrance image area indicating a virtual entrance associated with the switching destination moving object 100A to 100C. The entrance image area may be formed using, for example, an image of a door, like the icons 70a to 70c shown in the figure. The entrance image area may be, for example, a gate, a window, or a hole. Not limited to this, icons of various shapes may be used as the entrance image area.

[0062] The icons 70a to 70c are provided so that the control target can be selected from the moving bodies 100A to 100C according to the movement of the switching source moving body 100D, which is the control target. The display state of the icons 70a to 70c can be changed according to the control of the display control unit 13.

[0063] For example, suppose that the control target is switched from moving body 100D to moving body 100B. For example, when moving body 100D moves forward as controlled by operator P, the display mode of icons 70a to 70c changes with the movement of moving body 100D. Specifically, display unit 60 displays image display area 61c such that icons 70a to 70c become larger on the screen as moving body 100D moves forward.

[0064] 10 to 12 are diagrams showing how the display mode of the icons 70a to 70c changes as the moving object 100D moves. As the moving object 100D moves forward by the driver P, the display of the icons 70a to 70c changes to become larger in the order of Fig. 10, Fig. 11, and Fig. 12. Also, when the moving object 100D moves backward, the display unit 60 displays the image display area 61c so that the display of the icons 70a to 70c becomes smaller.

[0065] In this way, the display unit 60 can express that the moving object 100D is approaching the icons 70a to 70c and that the moving object 100D is moving away from the icons 70a to 70c.

[0066] The operator P controls the moving body 100D so that the moving body 100D gradually approaches the icons 70a to 70c. The operator P moves the moving body 100D further forward from the state shown in FIG. 12 so that the moving body 100D passes through the door indicated by the icon 70b.

[0067] The control device 10 calculates the distance between the moving object 100D and the icon 70b in the displayed virtual space. When the distance between the moving object 100D and the icon 70b becomes less than a predetermined threshold, the display unit 60 switches the display screen from the image of the moving object 100D to the image of the moving object 100B. This allows the display unit 60 to represent on the screen that the moving object 100D has passed through the door of the icon 70b.

[0068] In this way, when the moving body 100D moves so as to pass any of the icons 70a to 70c on the screen, the display unit 60 switches the images by displaying in the image display area 61c an image acquired by the moving body 100 corresponding to the passed icon. In this way, the pilot P can get a sense of having moved from the switching source moving body to the switching destination moving body.

[0069] 13 is a diagram showing the display screen 61 after switching. The image display area 61c shows an image captured by the image capturing unit 121 of the moving object 100B, which is the switching destination. The controlled object display area 61a shows that the current controlled object is the moving object 100B.

[0070] Returning to Fig. 5, the explanation will be continued. The control device 12 controls each functional unit of the pilot device 10. In other words, the control device 12 controls the headset 20, the controller 30, the mouse 40, the photographing unit 50, and the display unit 60. The control device 12 also includes a display control unit 13, a switching unit 14, and a storage unit 19.

[0071] The display control unit 13 generates a display screen on the display unit 60 and controls the display of the display unit 60. For example, the display control unit 13 controls the display of the above-mentioned icons. The display control unit 13 changes the display mode of the icons in accordance with the motion of the moving object 100 to be controlled. For example, the display control unit 13 changes the display mode of the icons 70a to 70c in accordance with the motion information of the moving object 100D to be controlled.

[0072] The switching unit 14 uses the operation information transferred from the moving body 100 to switch the control object from a first moving body, which is the switching source, to a second moving body, which is the switching destination.

[0073] For example, the switching unit 14 performs a switching process to switch the moving body 100 to be controlled in response to a switching request from the pilot P. The switching request is a request to switch the moving body 100 to be controlled from the moving body 100 to be controlled to another moving body 100. The switching request may include information on the moving body 100 to be switched to. The information is, for example, identification information for identifying the moving body 100. In this way, the switching unit 14 specifies the moving body 100 to be switched to.

[0074] The operator P can make a switching request in various ways. For example, as described above, the operator P moves the moving object 100D to be controlled forward to approach one of the icons 70a to 70c. The operator P makes a switching request to the moving object 100B associated with the icon 70b by operating the moving object 100D to pass through the door indicated by the icon 70b.

[0075] Without being limited to this, the pilot P may select the icon 70b by clicking the icon 70b with the mouse 40. This allows the pilot P to make a switching request to switch the control target to the moving body 100B associated with the icon 70b.

[0076] The pilot P may also make a switching request using a voice recognition technology by uttering a voice such as "move to the moving body 100B." The pilot P may also make a switching request using a head mounted display or the like. For example, an icon may be selected by linking the direction of the pilot P's face with the direction of the head 120 of the moving body 100D being operated.

[0077] Here, the switching process performed by the switching unit 14 will be described. The switching unit 14 estimates the icon to be selected based on the motion information A1 of the moving object 100D, and starts the switching process according to the estimation result. The switching unit 14 detects that the moving object 100D has approached any of the icons 70a to 70c on the screen. For example, when the distance between the moving object 100D and the icon 70b on the screen becomes less than a predetermined threshold, the switching unit 14 detects that the moving object 100D has approached the icon 70b. The predetermined threshold may be fixed in advance, or may be set to be appropriately changeable.

[0078] It is assumed that the switching unit 14 detects that the moving body 100D approaches the icon 70b. The switching unit 14 presumes that the operator P has made a request to switch from the moving body 100D to the moving body 100B.

[0079] The switching unit 14 starts the operation of the moving body 100B, which is the switching destination. Specifically, the switching unit 14 transfers the motion information A1 indicating the motion of the moving body 100D, which is the switching source, to the moving body 100B, which is the switching destination, and starts the motion based on the motion information A1. The motion information A1 may include, for example, the speed of the moving body 100D.

[0080] For example, the switching unit 14 controls the moving body 100B so that the moving body 100B moves at the same speed as the moving body 100D. Note that, at this stage, the moving body 100D remains the object to be controlled. That is, the switching unit 14 starts the motion of the moving body 100B based on the motion information A1 of the moving body 100D while maintaining the state in which the operator P is controlling the moving body 100D.

[0081] The moving body 100B performs a motion so as to reduce the difference between the motion information A2 indicating the motion of the moving body 100B and the motion information A1. For example, it is assumed that the speed of the moving body 100B is slower than the speed of the moving body 100D. The moving body 100B accelerates the moving body 100B so as to reduce the difference between the speed of the moving body 100B and the speed of the moving body 100D.

[0082] As described above, the switching unit 14 switches the control target from the moving body 100D to the moving body 100B at a predetermined timing after the moving body 100B starts to move based on the motion information A1. The switching unit 14 may determine the predetermined timing using a predetermined time interval. For example, the switching unit 14 switches the control target to the moving body 100B after a predetermined time has elapsed since the moving body 100B starts to move.

[0083] The switching unit 14 may also determine the predetermined timing in consideration of the motion status of the moving body 100B. For example, the switching unit 14 acquires motion information A1 and motion information A2, and compares the motion information A1 with the motion information A2. For example, when the difference between the motion information A1 and the motion information A2 becomes less than a predetermined threshold, the switching unit 14 switches the control target to the moving body 100B.

[0084] In this way, the motion information A2 of the moving body 100D can be made to approach the motion information A1 by the time the switching to the moving body 100B is completed. This allows the pilot P to switch the controlled object without feeling uncomfortable.

[0085] As described above, the switching unit 14 starts the switching process to switch the control target to the moving body 100B associated with the icon 70b estimated to be selected, so that the operation can be started at an earlier timing. However, the switching unit 14 may start the switching process when the icon is actually selected.

[0086] The storage unit 19 is a storage device that stores programs for implementing each function of the control device 10. The storage unit 19 may store information acquired from the moving objects 100A-100D. For example, the storage unit 19 may store images or operation information acquired from each of the moving objects 100A-100D.

[0087] (Processing performed by the control device 10) Next, the switching process performed by the control device 10 according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the process performed by the control device 10. As in the above description, the control device 10 performs a switching process to switch the control target from the moving body 100D to the moving body 100B. It is also assumed that the moving body 100B is in a switchable state. It is also assumed that the control device 10 acquires operation information from each moving body 100 at any time.

[0088] First, the display control unit 13 causes the display unit 60 to display the moving objects 100 that are candidates for switching (S11). The display control unit 13 causes, for example, the icons 70a to 70c corresponding to the moving objects 100A to 100C, respectively, to be displayed in the image display area 61c. As described with reference to Figs. 9 to 12, the icons 70a to 70c are configured to change shape in response to the movement of the moving object 100D and to be selectable when a predetermined condition is satisfied.

[0089] The switching unit 14 estimates the icon to be selected based on the motion information A1 of the moving object 100D. Specifically, the switching unit 14 detects that the moving object 100D has approached any of the icons 70a to 70c on the screen (S12). For example, the switching unit 14 detects the approach when the distance between the moving object 100D and any of the icons 70a to 70c becomes less than a predetermined value. Here, the switching unit 14 detects that the moving object 100D has approached the icon 70b.

[0090] As a result, the switching unit 14 infers that the icon 70b will be selected, and starts the switching process to switch the target to be controlled. Note that, although the switching unit 14 infers which icon will be selected and starts the switching process here, the switching unit 14 may also start the switching process in response to the selection of an icon and a switching request being made.

[0091] The switching unit 14 transfers the motion information A1 of the moving body 100D, which is the switching source, to the moving body 100B, which is the switching destination (S13). The motion information A1 is, for example, the speed and moving direction of the moving body 100D. Note that, although an example in which the switching unit 14 transfers the motion information A1 is shown here, this is not limiting. A device that transfers the motion information to another moving body and a device that switches the controlled object may be provided separately. Therefore, a device other than the control device 10 may transfer the motion information A1 to the moving body 100B.

[0092] The switching unit 14 starts the motion of the moving object 100B based on the motion information A1 (S14). The switching unit 14 controls the moving object 100B so that the motion information A2 of the moving object 100B approaches the motion information A1.

[0093] The switching unit 14 judges whether or not the icon that was estimated to be selected has actually been selected (S15). If the icon has not been selected (NO in S15), the process returns to step S12.

[0094] When the icon is selected (YES in S15), the switching unit 14 switches the control target from the moving body 100D to the moving body 100B (S16). The switching unit 14 may postpone the switching until a predetermined timing. For example, when the motion information A1 and A2 include information related to speed, the switching unit 14 may switch the control target after the difference between the speed of the moving body 100B and the speed of the moving body 100D becomes less than a predetermined value.

[0095] The display control unit 13 acquires an image captured by the moving body 100B of the switching destination, and displays the image in the image display area 61c (S17). This allows the operator P to operate the moving body 100B using the input unit 11.

[0096] As described above, according to the control system 1 of this embodiment, the control device 10 that controls multiple moving bodies 100 transfers operation information indicating an operation of a first moving body, which is a switching source, to a second moving body, which is a switching destination. After the second moving body starts an operation based on the operation information of the first moving body, the control device 10 performs a switching process to switch the control target from the first moving body to the second moving body.

[0097] In this way, the control device 10 can copy the motion of the moving body at the switching source and have the moving body at the switching destination follow suit. In addition, the control device 10 uses the icon on the display screen to estimate which moving body will be selected and starts the switching process, so switching can be performed more smoothly.

[0098] This allows the operator to feel as if he or she has traveled to another place while remaining on the same moving object. Therefore, when the operator remotely controls a moving object such as a robot in a virtual space and wanders around the virtual space, the operation system 1 allows the operator to switch the object being controlled without interrupting the feeling of moving in the virtual space.

[0099] In this way, the control system 1 according to the present embodiment can reduce the sense of discomfort felt when switching between moving objects to be controlled, and can provide the operator with the experience of continuous movement in a virtual space.

[0100] The configuration of the control system 1 described above is merely an example, and the configuration of the control system 1 may be changed as appropriate. For example, each of the control device 10 and the moving body 100 may be configured using a device in which multiple components are integrated. Also, some or all of the functions of each of the control device 10 and the moving body 100 may be integrated into the same device. For example, while the control device 10 is shown outside the moving body 100 in FIG. 1, the control device 10 may be provided inside the moving body 100. In that case, the pilot P may directly operate the control device 10 by boarding the moving body 100, for example. Alternatively, the pilot P may operate the control device 10 using an information terminal or the like capable of communicating with the control device 10. Also, each functional unit in each of the control device 10 and the moving body 100 may be distributed and processed using multiple devices, for example.

[0101] <Modification 1 of the First Embodiment> Variation 1 of the first embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of a display screen 62 according to this variation. This variation differs from the first embodiment in that the icon further includes an image captured at a moving object that is a candidate for the switching destination. The configurations of the piloting system 1 and the piloting device 10 are the same as those of the first embodiment, and therefore will not be described.

[0102] 15, the icons 70a to 70c include switching destination images 80a to 80c, respectively, on the display screen 62. The switching destination images 80a to 80c are images captured by the moving objects 100A to 100C, respectively. Note that the switching destination images 80a to 80c may be still images.

[0103] In this way, by displaying the entrance area image and the switching destination image superimposed in the image display area 61c, the pilot P can visually recognize the image of the switching destination moving body 100 from the door part of each icon. This allows the pilot P to see the space visible from moving bodies 100 other than the target to be controlled in the virtual space through the icon. In this way, the pilot P can easily select the target to be controlled.

[0104] <Modification 2 of the First Embodiment> Next, a second modification of the first embodiment will be described. In the first embodiment described above, an example has been shown in which the switching unit 14 switches the control target from the first moving body to the second moving body after the second moving body starts to move based on the motion information of the first moving body. As a result, in the first embodiment, the control target can be switched while the second moving body takes over the motion of the first moving body.

[0105] In this modification, after the first moving body starts to move based on the motion information of the second moving body, the switching unit 14 switches the target to be controlled from the first moving body to the second moving body. That is, in this modification, the first moving body matches its motion with that of the second moving body. Note that both the first and second moving bodies may change their motions so as to approach each other's motions.

[0106] Differences from the first embodiment will be described by appropriately interpreting the flowchart shown in Fig. 14. In step S13, the switching unit 14 transfers the motion information A2 of the second moving body (moving body 100B) to the first moving body (moving body 100D). In step S14, the switching unit 14 starts the motion of the first moving body based on the motion information A2. The switching unit 14 controls the first moving body so that the motion information A1 of the first moving body approaches the motion information A2.

[0107] In this way, for example, when the second moving body is moving slower than the first moving body, the first moving body is controlled to decrease its moving speed to match the speed of the second moving body. The speed can be controlled to gradually decrease. Also, when the operator P operates the first moving body, the first moving body can be controlled to move more slowly. In this way, in this modified example, the switching source moving body can perform an operation that matches the operation of the switching destination moving body.

[0108] <Modification 3 of the First Embodiment> Next, a description will be given of Modification 3 of the embodiment 1. In the above-described embodiment 1, an example has been shown in which the switching unit 14 switches the control target from a first moving object to a second moving object, and then causes the display unit 60 to display an image of the second moving object.

[0109] In this modification, the switching unit 14 adjusts the image captured around the second moving body in accordance with the motion information of the first moving body. The display unit 60 displays the adjusted image until the switching of the control target is completed after the second moving body starts to move in accordance with the motion information of the first moving body. After displaying the image, the switching unit 14 switches the control target from the first moving body to the second moving body. By visually checking the adjusted image, the pilot P can reduce the sense of discomfort caused by the switching.

[0110] 14, differences from the first embodiment will be described. After detecting in step S12 that the first moving object (moving object 100D) has approached any of the icons 70a to 70c, the switching unit 14 starts the switching process. Here, the switching unit 14 adjusts the video captured around the second moving object (moving object 100B) according to the motion information of the first moving object.

[0111] Here, the timing when the switching process starts is T0, and the timing when the switching process is completed and the display unit 60 displays the image of the second moving body is T1. The switching unit 14 acquires and adjusts the image captured in the second moving body before T0, and acquires the adjusted image.

[0112] The switching unit 14 adjusts the image according to the motion information A1 of the first moving object. The switching unit 14 adjusts the image captured by the second moving object so as to reduce a sense of incongruity when switching from the image captured by the first moving object to the image captured by the second moving object.

[0113] For example, if a first moving object is moving faster than a second moving object, the surrounding scenery changes slower in the video captured by the second moving object than in the video captured by the first moving object. Therefore, simply switching between the images creates a sense of incongruity.

[0114] The switching unit 14 adjusts the image to display the image captured by the second moving object faster so that the images are smoothly connected at T1. The switching unit 14 may adjust the image so that the speed increases from T0 to T1. The switching unit 14 obtains an adjusted image that is displayed faster than the actual speed of the image captured by the second moving object. The switching unit 14 calculates backwards the timing to display the adjusted image so that the images are connected at T1, and displays it on the display unit 60. As a result, the display unit 60 displays the adjusted image from T0 to T1.

[0115] The switching unit 14 switches the control target from the first moving object to the second moving object after displaying the adjustment image on the display unit 60. The processes from step S13 onwards are the same as those in the first embodiment. Note that the order of the processes relating to the adjustment image described above may be interchanged with the processes of steps S13 and S14.

[0116] As the motion information A1, speed information obtained from sensors provided on the first moving body may be used, or speed information obtained by analyzing a video captured by the first moving body may be used. Furthermore, information other than speed may be used as the motion information A1.

[0117] In this way, according to this modification, by adjusting the playback speed of the video captured by the second moving object, it is possible to reduce the sense of discomfort felt when switching between the objects to be controlled.

[0118] <Embodiment 2> (Configuration of the control device 10a) Next, the second embodiment will be described with reference to FIG. 16. FIG. 16 is a block diagram showing the configuration of a control device 10a according to the present embodiment. The control device 10a differs from the first embodiment in that it includes a switching image generating unit 15 in addition to the configuration of the control device 10 described above. Below, differences from the first embodiment will be mainly described, and overlapping contents will be omitted as appropriate. Note that, as in the first embodiment, an example will be described in which the switching source moving body is moving body 100D and the switching destination moving body is moving body 100B.

[0119] In the first embodiment, a case has been described in which the moving object 100B, which is the switching destination, can quickly start moving in response to control from the control device 10. However, when the control device 10 performs the switching process, there are cases in which the moving object 100B cannot quickly start moving. For example, this is the case when a special operation needs to be performed on the moving object 100B.

[0120] For example, suppose that the moving object 100B is connected to a charging station for charging and is being charged. In this case, in order for the moving object 100B to start operating, it is necessary to remove the moving object 100B from the charging station before performing the switching process. Therefore, even if the control device 10 performs the switching process, the moving object 100B cannot start operating promptly. Therefore, the pilot P cannot continuously switch the control target, which creates a sense of discomfort. This embodiment can address such a problem.

[0121] In the control device 10a, the switching unit 14 determines whether or not switching from the source moving body 100D to the destination moving body 100B is possible based on the operation information A2 of the destination moving body 100B. The switching unit 14 determines whether or not switching from the moving body 100D to the moving body 100B is possible based on the operation information A2 of the moving body 100B.

[0122] For example, the switching unit 14 determines whether switching is possible by determining whether the moving object 100B can operate within a predetermined time based on the operation information A2. For example, when the moving object 100B is connected to the charging station as described above, the switching unit 14 determines that the operation of disconnecting the moving object 100B will take a predetermined time or more. In this case, the switching unit 14 determines that switching to the moving object 100B is not possible.

[0123] When the switching unit 14 determines that switching to the moving body 100B is not possible, the switching unit 14 displays in the image display area 61c the switching image generated by the switching image generating unit 15. After displaying the switching image in the image display area 61c, the switching unit 14 switches the control target from the moving body 100D to the moving body 100B at a predetermined timing.

[0124] The switching unit 14 may determine the predetermined timing in the same manner as in the first embodiment. The switching unit 14 may determine the predetermined timing using a predetermined time interval. Alternatively, the switching unit 14 may determine the predetermined timing in consideration of the operation status of the moving body 100B. For example, the switching unit 14 may display the switching image until the moving body 100B has completely left the charging station, and then switch the target to be controlled.

[0125] When the switching unit 14 determines that switching of the control target is not possible, the switching image generating unit 15 generates a switching image using the image acquired at the moving body 100D and the image acquired at the moving body 100B. Hereinafter, the image acquired at the moving body 100D is referred to as image V1, the image acquired at the moving body 100B as image V2, and the image generated by the switching image generating unit 15 as switching image V3.

[0126] The switching image generating unit 15 generates a switching image V3 that reduces the sense of discomfort felt by the pilot P when the display in the image display area 61c switches from the image V1 to the image V2. The images V1 and V2 may be images stored in the storage units 114 of the moving bodies 100D and 100B, respectively.

[0127] The switching image generating unit 15 generates the switching image V3 using the images V1 and V2 acquired within a predetermined time based on, for example, the shooting date and time information of the images. The switching image generating unit 15 may generate the switching image V3 from the images V1 and V2 using a well-known technique. For example, the switching image generating unit 15 generates the switching image V3 such that a zoom-in operation is performed from the image V1 to the image V2. However, the switching image generating unit 15 may generate the switching image V3 that connects the images V1 and V2 using various methods.

[0128] (Processing performed by the control device 10a) Next, the switching process performed by the control device 10a according to this embodiment will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the process performed by the control device 10a. As in the above description, the control device 10a performs the switching process to switch the control target from the moving body 100D to the moving body 100B. It is also assumed that the control device 10a acquires operation information from each moving body 100 at any time.

[0129] First, the display control unit 13 causes the display unit 60 to display the moving objects 100 that are candidates for switching (S21). The display control unit 13 causes, for example, the icons 70a to 70c corresponding to the moving objects 100A to 100C, respectively, to be displayed in the image display area 61c. As in the first embodiment, the icons 70a to 70c are configured to change shape in response to the movement of the moving object 100D and to be selectable when a predetermined condition is satisfied.

[0130] The switching unit 14 estimates the icon to be selected based on the motion information A1 of the moving object 100D. The switching unit 14 detects that the moving object 100D has approached any of the icons 70a to 70c on the screen (S22). Here, the switching unit 14 detects that the moving object 100D has approached the icon 70b.

[0131] The switching unit 14 determines whether or not switching from the moving body 100D to the moving body 100B is possible based on the motion information A2 of the moving body 100B (S23). If it is determined that switching is possible (YES in S23), the process proceeds to step S26.

[0132] If it is determined that switching is not possible (NO in S23), the switching image generator 15 generates a switching image V3 using the image V1 acquired by the moving body 100D and the image V2 acquired by the moving body 100B (S24). For example, the switching image V3 may be an image that performs a zoom-in operation from the image V1 to the image V2.

[0133] The switching unit 14 displays the switching video V3 in the video display area 61c (S25), which allows the pilot P to view the switching video V3.

[0134] Next, the switching unit 14 starts the operation of the moving body 100B (S26). The switching unit 14 operates the moving body 100B so that switching to the moving body 100B becomes possible. For example, the switching unit 14 controls the moving body 100B so as to separate the moving body 100B from the charging station. In this way, the control device 10a can transition the moving body 100B to a switchable state while the pilot P is watching the switching image V3.

[0135] The switching unit 14 judges whether or not the icon that was estimated to be selected has actually been selected (S27). If it has not been selected (NO in S27), the process returns to step S22.

[0136] When the icon is selected (YES in S27), the switching unit 14 switches the control target from the moving body 100D to the moving body 100B (S28). The switching unit 14 may postpone the switching until a predetermined timing. For example, the switching unit 14 determines whether the moving body 100B has transitioned to a switchable state. The switching unit 14 may switch the control target when it is determined that the moving body 100B has transitioned to a switchable state.

[0137] The display control unit 13 acquires an image captured by the moving body 100B of the switching destination, and displays the image in the image display area 61c (S29). This allows the operator P to operate the moving body 100B using the input unit 11.

[0138] In step S26, the switching unit 14 may further start the motion of the moving object 100B based on the motion information A1 of the moving object 100D, as in the first embodiment. In this case, the switching unit 14 transfers the motion information A1 to the moving object 100B. The motion information A1 is, for example, the speed and moving direction of the moving object 100D. The switching unit 14 controls the moving object 100B so that the motion information A2 of the moving object 100B approaches the motion information A1.

[0139] In this way, while the control device 10a is displaying the switching image V3, the control device 10a can transition the moving body 100B to a switchable state and can make the movement of the moving body 100B closer to the movement of the moving body 100D.

[0140] As described above, in the control system 1 of this embodiment, in the control device 10a that controls multiple moving bodies 100, the switching unit 14 determines whether or not switching from the first moving body, which is the source of switching, to the second moving body is possible based on the operation information of the second moving body, which is the destination of switching.

[0141] If switching is not possible, the switching image generating unit 15 generates a switching image using an image acquired in the first moving body and an image acquired in the second moving body. The switching unit 14 switches the controlled object from the first moving body to the second moving body after displaying the switching image on the screen.

[0142] In this manner, even if the second moving object cannot be switched immediately, the control device 10a can switch the controlled object while reducing the sense of discomfort felt by the control device 10a by displaying the switching image.

[0143] Furthermore, the control device 10a can transfer the motion information of the first moving object to the second moving object while the switching image is being displayed, and therefore can achieve the same effects as the first embodiment.

[0144] <Hardware configuration example> Each functional component of the pilot control device 10 and the moving body 100 may be realized by hardware that realizes each functional component (e.g., hardwired electronic circuits, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). Below, a case where each functional component of the pilot control device 10, etc. is realized by a combination of hardware and software will be described.

[0145] 18 is a block diagram illustrating an example of the hardware configuration of a computer 900 that realizes the pilot control device 10, etc. The computer 900 may be a dedicated computer designed to realize the pilot control device 10, etc., or may be a general-purpose computer. The computer 900 may be a portable computer such as a smartphone or a tablet terminal.

[0146] For example, by installing a specific application on computer 900, each function of the control device 10 and the like is realized by computer 900. The application is constituted by a program for realizing the functional components of the control device 10 and the like.

[0147] The computer 900 includes a bus 902, a processor 904, a memory 906, a storage device 908, an input / output interface 910, and a network interface 912. The bus 902 is a data transmission path for the processor 904, the memory 906, the storage device 908, the input / output interface 910, and the network interface 912 to transmit and receive data to and from each other. However, the method of connecting the processor 904 and other components to each other is not limited to a bus connection.

[0148] The processor 904 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 906 is a main storage device realized using a random access memory (RAM) or the like. The storage device 908 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0149] The input / output interface 910 is an interface for connecting the computer 900 to an input / output device. For example, the input / output interface 910 is connected to an input device such as a keyboard and an output device such as a display device.

[0150] The network interface 912 is an interface for connecting the computer 900 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0151] The storage device 908 stores programs (programs that realize the applications described above) that realize the various functional components of the control device 10, etc. The processor 904 reads these programs into the memory 906 and executes them to realize the various functional components of the control device 10, etc.

[0152] Each of the processors executes one or more programs including instructions for causing a computer to perform an algorithm. The programs include instructions (or software code) for causing a computer to perform one or more functions described in the embodiments when loaded into the computer. The programs may be stored on various types of non-transitory computer readable medium or tangible storage medium. By way of example and not limitation, computer readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may be transmitted on various types of transitory computer readable medium or communication medium. By way of example and not limitation, transitory computer readable medium or communication medium includes electrical, optical, acoustic, or other form of propagated signal.

[0153] The present disclosure is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present disclosure. For example, the control device 10 may transfer environmental information related to the environment of the switching source mobile body 100 to the switching destination mobile body 100 in the same manner as the operation information. The environmental information may include, for example, air pressure, water pressure, wind pressure, air temperature, water temperature, etc. detected in the switching source mobile body 100.

[0154] Furthermore, the above-described embodiments can be implemented in any combination. For example, the first embodiment, the first to third modifications of the first embodiment, and part or all of the second embodiment can be implemented in any suitable combination. [Explanation of symbols]

[0155] 1. Steering System 10, 10a Control device 11 Input section 12 Control device 13 Display control section 14 Switching section 15 Switching image generation unit 19 Memory section 20 Headset 20a Speaker 20b Microphone 30 Controller 30a Cross key 30b Joystick 30c Button 40 Mouse 50 Photography Department 60 Display section 61, 62 display screen 61a Control target display area 61b Pilot display area 61c Image display area 70a~70c Icon 80a~80c Switching destination video 100, 100A~100D Mobile 101 Main body 110 Control device 111 Motion control section 112 Operation information acquisition unit 113 Transfer Department 114 Storage section 120 Head 121 Photography Department 122 Display section 130 Arm section 131 Hands 132 Arm 140 Mobile Unit 900 Computers 902 Bus 904 Processor 906 Memory 908 Storage Devices 910 Input / Output Interface 912 Network Interface

Claims

1. A control system for controlling a moving object to be controlled selected from a plurality of moving objects, comprising: a display unit that displays display information including an image of the surroundings of the moving object; A switching unit that switches the object to be controlled from a first moving body that is a switching source to a second moving body that is a switching destination, using motion information indicating a motion of the moving body; The switching unit is After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and After the display unit displays the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, The switching unit is Whether or not switching from the first moving body to the second moving body is possible is determined based on the operation information of the second moving body, and if it is determined that the switching is not possible, a switching image generated using an image captured around the first moving body and an image captured around the second moving body is displayed on the display unit, and then the control target is switched from the first moving body to the second moving body. Pilot system.

2. the display unit displays an icon for receiving a request to switch the object to be controlled from the operator in association with the moving object that is a candidate for the switching destination; the icon is provided to be selectable in response to an action of the first moving object, The switching unit switches the control target to a moving object associated with the selected icon.

2. The flight control system of claim 1.

3. The switching unit estimates the icon to be selected based on operation information of the first moving object, and switches the control target in accordance with an estimation result.

3. A flight control system as claimed in claim 2.

4. A control system for controlling a target moving body selected from a plurality of moving bodies, comprising: a display unit that displays display information including an image of the surroundings of the moving object; A switching unit that switches the object to be controlled from a first moving body that is a switching source to a second moving body that is a switching destination, using motion information indicating a motion of the moving body; The switching unit is After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and After the display unit displays the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, the display unit displays an icon for receiving a request to switch the object to be controlled from the operator in association with the moving object that is a candidate for the switching destination; the icon is provided to be selectable in response to an action of the first moving object, The switching unit switches the control target to a moving object associated with the selected icon, The switching unit estimates the icon to be selected based on operation information of the first moving object, and switches the control target in accordance with an estimation result. Pilot system.

5. A computer of a control system that controls a target moving object selected from a plurality of moving objects, a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the control target from a first moving body that is a switching source to a second moving body that is a switching destination, using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, In the switching step, Whether or not switching from the first moving body to the second moving body is possible is determined based on the operation information of the second moving body, and if it is determined that the switching is not possible, a switching image generated using an image captured around the first moving body and an image captured around the second moving body is displayed, and then the controlled object is switched from the first moving body to the second moving body. How to operate.

6. A computer of a control system for controlling a target moving object selected from a plurality of moving objects, a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the control target from a first moving body that is a switching source to a second moving body that is a switching destination, using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, In the display step, an icon for receiving a request to switch the object to be controlled from the operator is displayed in association with the moving object that is a candidate for the switching destination; the icon is provided to be selectable in response to an action of the first moving object, In the switching step, the control object is switched to a moving object associated with the selected icon; In the switching step, the icon to be selected is estimated based on the motion information of the first moving object, and the control target is switched in accordance with the estimation result. How to operate.

7. A program to be executed by a computer of a control system for controlling a target moving body selected from a plurality of moving bodies, a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the operation target from a first moving body that is a switching source to a second moving body that is a switching destination, using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, In the switching step, Whether or not switching from the first moving body to the second moving body is possible is determined based on the operation information of the second moving body, and if it is determined that the switching is not possible, a switching image generated using an image captured around the first moving body and an image captured around the second moving body is displayed, and then the controlled object is switched from the first moving body to the second moving body. program.

8. A program to be executed by a computer of a control system for controlling a target moving body selected from a plurality of moving bodies, comprising: a display step of displaying display information including an image of the surroundings of the moving object; A switching step of switching the operation target from a first moving body that is a switching source to a second moving body that is a switching destination, using operation information indicating an operation of the moving body; In the switching step, After the second moving body starts to move based on the motion information of the first moving body, After the first moving body starts to move based on the motion information of the second moving body, and In the display step, after displaying the image adjusted according to the motion information of the first moving object, In at least one of the above cases, the control object is switched from the first moving body to the second moving body, In the display step, an icon for receiving a request to switch the object to be controlled from the operator is displayed in association with the moving object that is a candidate for the switching destination; the icon is provided to be selectable in response to an action of the first moving object, In the switching step, the control object is switched to a moving object associated with the selected icon; In the switching step, the icon to be selected is estimated based on the motion information of the first moving object, and the control target is switched in accordance with the estimation result. program.

Images