Manipulation system, manipulation method, and storage medium

Abstract

The present disclosure relates to a manipulation system, a manipulation method, and a storage medium. The manipulation system manipulates a mobile body that is a manipulation target selected from among a plurality of mobile bodies. The manipulation system includes a display unit that displays display information including an image captured of a periphery of the mobile body, and a switching unit that switches the manipulation target from a first mobile body that is a switching source to a second mobile body that is a switching target, using action information indicating an action of the mobile body. The switching unit switches the manipulation target from the first mobile body to the second mobile body in at least any one of a case where the second mobile body starts an action based on the action information of the first mobile body, a case where the first mobile body starts an action based on the action information of the second mobile body, and a case where the display unit displays the image adjusted based on the action information of the first mobile body.

Description

Technical Field

[0001] This disclosure relates to an operating system, an operating method, and a storage medium. Background Technology

[0002] A technique is known in which an operator remotely controls a mobile body, such as a robot, that moves autonomously via a control terminal. As a related technique, for example, Japanese Patent No. 6,614,607 discloses a remote control system capable of centralized remote control of a moving mobile body while simultaneously appropriately confirming road conditions at a distance. Summary of the Invention

[0003] The system described in Japanese Patent No. 6,614,607 allows an operator to remotely control multiple robots via a control terminal. The control terminal displays images captured by a camera on the robot currently being operated by the operator. Furthermore, the control terminal displays multiple robots in operation on the screen and allows the operator to select the robot to be operated.

[0004] The display screen can be in the form of a list showing all the robots, or a map showing the robots' current positions overlaid on a map. The operator selects the robot to be operated from among several operating robots and issues a switch request. The control terminal accepts the switch request and executes the switch of the target robot.

[0005] In this scenario, the control terminal sends an instruction signal to the robot at the switching source to stop manipulating the currently controlled remote robot. Furthermore, the control terminal needs to temporarily pause the image of the robot at the switching source displayed on the screen and switch the screen to a robot selection screen, allowing the operator to select another robot. Here, when there is a divergence in movement or speed between the robot at the switching source and the robot at the switching target, there is a possibility that the switching of the controlled object may cause discomfort to the remote operator. In such cases, the problem arises that whenever the controlled robot is switched, the sensation of moving using the operating robot is interrupted, thus preventing the operator from experiencing continuous movement.

[0006] In view of the above-mentioned problems, the object of this disclosure is to provide a control system, control method and storage medium that can reduce the discomfort of the operator when switching the movement of the controlled object.

[0007] The control system disclosed herein is a control system for controlling a movable object selected from a plurality of movable bodies. It includes the following components: a display unit that displays display information, the display information including images captured from the vicinity of the movable body; and a switching unit that uses motion information representing the movement of the movable body to switch the controlled object from a first movable body as a switching source to a second movable body as a switching target. The switching unit switches the controlled object from the first movable body to the second movable body in at least one of the following situations: after the second movable body begins to move based on the motion information of the first movable body; after the first movable body begins to move based on the motion information of the second movable body; and after the display unit displays an image adjusted according to the motion information of the first movable body.

[0008] The manipulation method disclosed herein includes: a display step in which a computer of a manipulation system that manipulates a movable object selected from a plurality of movable bodies displays display information, the display information including images captured around the movable body; and a switching step in which the computer switches the manipulated object from a first movable body as a switching source to a second movable body as a switching target using motion information representing the motion of the movable body, wherein, in the switching step, the manipulated object is switched from the first movable body to the second movable body in at least one of the following situations: after the second movable body begins to move based on the motion information of the first movable body; after the first movable body begins to move based on the motion information of the second movable body; and after the display step displays an image adjusted according to the motion information of the first movable body.

[0009] The storage medium disclosed herein stores a program for a computer of a control system that controls a selectable control object from a plurality of mobile bodies to perform the following steps: a display step, displaying display information including images captured by photographs of the area around the mobile body; and a switching step, using motion information representing the motion of the mobile body to switch the control object from a first mobile body as a switching source to a second mobile body as a switching target, wherein, in the switching step, the control object is switched from the first mobile body to the second mobile body in at least one of the following situations: after the second mobile body begins to move based on the motion information of the first mobile body; after the first mobile body begins to move based on the motion information of the second mobile body; and after the display step displays an image adjusted according to the motion information of the first mobile body.

[0010] The control system, control method, and storage medium disclosed herein can reduce the discomfort experienced by the operator when switching the movement of the controlled object. Attached Figure Description

[0011] The features, advantages, technical and industrial significance of representative embodiments of the present invention will be depicted in the following drawings for reference, wherein the same symbols denote the same elements.

[0012] Figure 1 This is a block diagram illustrating the structure of the control system according to Embodiment 1.

[0013] Figure 2 A diagram illustrating an example of the use of the control system according to Embodiment 1.

[0014] Figure 3 This is a block diagram illustrating the structure of the movable body according to Embodiment 1.

[0015] Figure 4 This is a diagram illustrating an example of the appearance of the mobile body according to Embodiment 1.

[0016] Figure 5 This is a block diagram illustrating the structure of the control device according to Embodiment 1.

[0017] Figure 6 This is a diagram illustrating an example of the appearance of the control device according to Embodiment 1.

[0018] Figure 7 This diagram illustrates an example of a display screen shown by the display unit according to Embodiment 1.

[0019] Figure 8This diagram illustrates an example of another display screen shown by the display unit according to Embodiment 1.

[0020] Figure 9 This diagram illustrates an example of a display screen showing icons according to Embodiment 1.

[0021] Figure 10 This diagram illustrates how the display of icons changes as the moving body according to Embodiment 1 moves.

[0022] Figure 11 This diagram illustrates how the display of icons changes as the moving body according to Embodiment 1 moves.

[0023] Figure 12 This diagram illustrates how the display of icons changes as the moving body according to Embodiment 1 moves.

[0024] Figure 13 This is a diagram showing the display screen after the switching process according to Embodiment 1.

[0025] Figure 14 A flowchart illustrating the processes performed by the control device according to Embodiment 1.

[0026] Figure 15 This is a diagram illustrating an example of a display screen involved in a modified embodiment 1.

[0027] Figure 16 This is a block diagram illustrating the structure of the control device according to Embodiment 2.

[0028] Figure 17 A flowchart illustrating the processing performed by the control device according to Embodiment 2.

[0029] Figure 18 A block diagram illustrating the hardware structure of a computer that implements control devices, etc. Detailed Implementation

[0030] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding elements are labeled with the same symbols. Repeated descriptions will be omitted as needed to clarify the explanation.

[0031] Implementation Method 1

[0032] Structure of Control System 1

[0033] Reference Figure 1 The implementation method 1 will now be described. Figure 1This is a block diagram illustrating the structure of the control system 1 according to this embodiment. (See diagram below.) Figure 1 As shown, the control system 1 includes a plurality of movable bodies 100 and a control device 10 for manipulating the plurality of movable bodies 100. The control device 10 may also be referred to as a control system for manipulating the plurality of movable bodies 100. Although three movable bodies 100 are shown in the figures, the number of movable bodies 100 is not limited thereto.

[0034] The control system 1 according to this embodiment is an information processing system that enables an operator to remotely control multiple mobile bodies 100 using a control device 10. The control device 10 controls the multiple mobile bodies 100. Furthermore, the control device 10 receives a selection from the multiple mobile bodies 100 by the operator as the target of control.

[0035] Multiple mobile bodies 100 and control devices 10 are connected together via a network N. The network N is, for example, a wired or wireless communication line. Although an example of one control device 10 connected to multiple mobile bodies 100 is shown here, multiple control devices 10 may also be provided.

[0036] Figure 2 This diagram illustrates an example of the use of the control system 1 according to this embodiment. The mobile body 100 is, for example, a mobile robot capable of autonomous movement within an environment. The environment simply needs to be a space or the like that allowing the mobile body 100 to move. The environment may include, for example, a residence, public facility, office, shop, or hospital. The environment can be either indoors or outdoors.

[0037] The mobile body 100 is a component having a mobile unit for moving within an environment. The mobile unit may include, for example, a drive mechanism for moving on a ground surface, in the air, on water, or in water. The mobile body 100 is not limited to the mobile robot described above, but may also be, for example, an autonomous vehicle or an unmanned aerial vehicle.

[0038] like Figure 2 As shown, the operator P and the mobile body 100 are located in different locations. For example, the operator P may be located in their own home, while the mobile body 100 may be located in a hospital. In the example in the accompanying drawings, a situation is shown where a user U in a wheelchair receives assistance from the mobile body 100 in a hospital. The operator P operates the control device 10 from their own home or other locations to control the mobile body 100. The locations of the operator P, the user U, and the mobile body 100 are not limited to this, but can be any location.

[0039] Structure of the moving body 100

[0040] Next, refer to Figure 3 as well as Figure 4 The structure of the movable body 100 involved in this embodiment will be described. Figure 3 A block diagram representing the structure of the movable body 100. Figure 4 This is a diagram illustrating an example of the appearance of the moving body 100. (See diagram below.) Figure 3 as well as Figure 4 As shown, the mobile body 100 includes a main body 101, a camera unit 121, a display unit 122, a head 120, an arm 130, a moving part 140, and a control device 110.

[0041] The main body 101 is the torso of the mobile body 100. The main body 101 is connected to the head 120, arm 130, and moving part 140. The main body 101 has a built-in control device 110. The main body 101 may also be equipped with a telescopic mechanism for extending and retracting in the vertical direction. In addition, the main body 101 may also be equipped with a rotating mechanism for rotating about the axis of the main body 101.

[0042] The camera unit 121 is a photographic device that captures images of the area around the moving body 100. The camera unit 121 captures images of the environment from a predetermined direction and outputs the captured images to the control device 110. The predetermined direction is, for example, the direction in which the head 120 of the camera unit 121 is facing, but is not limited to this.

[0043] The camera unit 121 may be a two-dimensional camera, for example. The camera unit 121 may also include a three-dimensional camera. The camera unit 121 may be, for example, a camera (RGB-D camera, stereo camera), a laser rangefinder, an ultrasonic sensor, a three-dimensional distance sensor, etc.

[0044] like Figure 4 As shown, the camera unit 121 is provided on the head 120 or the like of the moving body 100, but it is not limited to this. In addition, multiple camera units 121 may be provided on the moving body 100.

[0045] Display unit 122 is a display device for displaying images. For example, display unit 122 displays images acquired from operating device 10. These images may be images acquired by the camera unit 50 of operating device 10, etc. Figure 4 In the example shown, display unit 122 displays an image of the operator P's face. Display unit 122 is not limited to displaying images; it can also display still images and text. Furthermore, display unit 122 can display images captured by camera unit 121, as well as information stored in storage unit 114.

[0046] The display unit 122 may include, for example, a liquid crystal display (LCD) or an organic EL display. The display unit 122 may also be a touch panel that accepts input by means of touch by a user using a finger or the like and has the function of an input unit.

[0047] The head 120 is disposed on the upper part of the main body 101 of the movable body 100. The head 120 includes a camera unit 121 and a display unit 122. The head 120 is provided with a telescopic mechanism (not shown) so that it can be extended and retracted in the vertical direction. As a result, the camera unit 121 can take pictures at different heights. In addition, the head 120 may also be provided with a telescopic mechanism for extending and retracting in the vertical direction. Furthermore, the head 120 may also be provided with a rotating mechanism for rotating about the axis of the head 120. As a result, the camera unit 121 can take pictures in a wider shooting range.

[0048] The arm 130 includes an arm 132 supported on the main body 101 and a gripper 131 disposed at the top of the arm 132. The arm 132 is configured as a multi-joint arm, for example. The arm 132 is disposed on the front surface of the main body 101, but is not limited thereto; it may also be disposed on the side of the main body 101 or elsewhere. The arm 132 changes its position and orientation by means of a drive mechanism (not shown) equipped with a motor or the like, which operates according to a control signal from the control device 110. The arm 132 may also be equipped with a telescopic mechanism and configured to change the length of the arm 132.

[0049] The gripper 131 grips an object or the like according to a control signal from the control device 110. The gripper 131 changes its position and orientation by actuating a drive mechanism (not shown) equipped with a motor or the like. The gripper 131 can be, for example, like... Figure 4 As shown, it is a two-finger gripper with a pair of gripping parts, similar to two human fingers. It can also be a three-finger gripper or a multi-finger gripper with more fingers. It is not limited to these structures and various gripper actuators can be used as gripper 131.

[0050] The moving unit 140 is a moving device that moves the mobile body 100 within the environment in which the mobile body 100 exists. The moving unit 140 moves the mobile body 100 according to a control signal from the control device 110. Furthermore, the moving unit 140 moves the mobile body 100 according to a switching control signal from the operating device 10. The switching control signal may contain motion information of other mobile bodies 100 that serve as the switching source.

[0051] Here, motion information refers to information related to the movement of the moving body 100. Motion information may include, for example, information related to the movement of the moving body 100. Motion information may include parameters such as the velocity, acceleration, jerk, direction of movement, or vibration of the moving body 100. Depending on the structure or mode of movement of the moving body 100, the motion information may also include the pitch angle, roll angle, or yaw angle of the moving body 100. By equipping itself with sensor-like components capable of detecting this motion information, the moving body 100 can acquire various types of motion information as described above.

[0052] The motion information can also be information related to the motion of each functional unit of the moving body 100. For example, the motion information can also be the position or posture of the camera unit 121. The posture of the camera unit 121 is, for example, the tilt of the camera relative to the horizontal plane.

[0053] The moving part 140 includes, for example, a pair of opposing drive wheels and a pair of driven wheels capable of rotating in the left-right direction. Thus, the moving body 100 is configured to move in all directions. By rotating each drive wheel, the moving part 140 can perform any movement of the moving body 100, such as forward and backward movement, turning left and right, acceleration and deceleration, and stopping. The moving part 140 moves the moving body 100, for example, by operating a drive mechanism (not shown) constructed using a motor or the like, based on a control signal from the control device 110.

[0054] Furthermore, although the mobile body 100 in this embodiment uses a mobile robot that travels on the ground, if the mobile body 100 moves by other means, it is sufficient to provide a drive mechanism that is appropriate for the means of movement.

[0055] The control device 110 includes an action control unit 111, an action information acquisition unit 112, a forwarding unit 113, and a storage unit 114.

[0056] The motion control unit 111 controls the movements of each of the aforementioned functional units. That is, the motion control unit 111 controls the movements of the main body 101, the camera unit 121, the display unit 122, the head 120, the arm unit 130, and the movement unit 140.

[0057] The motion information acquisition unit 112 acquires motion information related to the motion of the mobile body 100. As described above, motion information may include, for example, the speed, acceleration, jerk, direction of movement, vibration, position of the camera unit 121, or attitude of the camera unit 121 of the mobile body 100. The motion information acquisition unit 112 acquires motion information by obtaining detection values ​​from sensors capable of detecting this motion information. Alternatively, the motion information acquisition unit 112 may acquire motion information from components other than sensors installed on the mobile body 100. For example, the motion information acquisition unit 112 may also acquire motion information from images captured by high-precision infrastructure cameras (surveillance cameras, etc.), images captured by artificial satellites, or sensor information from portable terminals brought into the mobile body 100. The motion information acquisition unit 112 may use well-known image recognition technologies to acquire motion information. The motion information acquisition unit 112 is not limited to being installed inside the mobile body 100, and may also be installed outside the mobile body 100.

[0058] The forwarding unit 113 forwards the images captured by the camera unit 121 and the motion information acquired by the motion information acquisition unit 112 to the control device 10. The forwarding unit 113 can also forward images and motion information stored in the storage unit 114. The forwarding unit 113 can, for example, be configured to have a communication interface with a network N. The forwarding unit 113 forwards the images and motion information to the control device 10 at a predetermined time interval. The predetermined time interval can also be a predetermined time period. Furthermore, the predetermined time interval can also be when a forwarding request occurs from the control device 10.

[0059] The storage unit 114 is a storage device that stores programs used to implement the various functions of the mobile body 100. In addition, the storage unit 114 can also store images captured by the camera unit 121 and motion information acquired by the motion information acquisition unit 112.

[0060] Structure of the control device 10

[0061] Next, refer to Figure 5 as well as Figure 6 The structure of the control device 10 involved in this embodiment will be described.

[0062] Figure 5 This is a block diagram illustrating the structure of the control device 10. Figure 6 A diagram showing an example of the appearance of the control device 10.

[0063] The operating device 10 is a computer used to perform the switching process involved in this embodiment. The operating device 10 may be, for example, a PC (Personal Computer), a tablet computer, a smartphone, or a mobile phone terminal. The operating device 10 includes an input unit 11, a camera unit 50, a display unit 60, and a control unit 12.

[0064] Input unit 11 is an input device that receives input from operator P. Figure 5 as well as Figure 6 In the example shown is an input unit 11, including a headset 20, a controller 30, and a mouse 40. These components are examples of the input unit 11, so the input unit 11 may also include a portion of these components, or may include other input devices.

[0065] The headset 20 includes a speaker 20a and a microphone 20b. The speaker 20a outputs speech. The microphone 20b picks up the speech of the operator P. By wearing the headset 20 on his head, the operator P uses the speaker 20a and microphone 20b to input and output speech to the control device 10. Thus, the operator P can converse with a user located near the mobile object 100 being controlled. The headset 20 can also be used for voice input to the mobile object 100. For example, the control device 10 can also use well-known speech recognition technology to recognize the speech of the operator P.

[0066] The controller 30 receives operations from the operator P and performs input operations to the control device 10. The controller 30 may be, for example, a game controller used in video games.

[0067] The controller 30 includes, for example, a crosshair 30a, a joystick 30b, and multiple buttons 30c. The operator P uses the controller 30 to manipulate the moving body 100. The mouse 40 receives operations from the operator P and performs input operations on the control device 10. The operator P can use the input section 11 to select icons displayed on the display section 60.

[0068] The camera unit 50 is a photographic device that captures images of the area around the control device 10 from a predetermined position. The predetermined position can be a location where the operator P can be photographed. In the example shown in the accompanying drawing, the camera unit 50 is positioned above the display unit 60. In this example, the camera unit 50 captures an area including the operator P's face. Like the camera unit 121, the camera unit 50 can be a two-dimensional camera or the like. The camera unit 50 outputs the images acquired through photography to the control device 10.

[0069] The display unit 60 displays display information including images captured of the surroundings of the mobile body 100. These images can be captured by the camera unit 121 or by other photographic devices. Other photographic devices could be cameras included in portable terminals brought into the mobile body 100, or cameras included in wearable terminals worn by passengers of the mobile body 100. It is not limited to these methods; images supplied from various image sources can be displayed.

[0070] Display unit 60 can be a display device for displaying images, etc. Similar to display unit 122, display unit 60 can be, for example, a liquid crystal display (LCD), an organic EL display, etc. Furthermore, display unit 60 can also be a touch panel with an input function. Display unit 60 displays images, etc., under the control of display control unit 13. Display unit 60 displays a screen including images relayed from mobile body 100. Not limited to these methods, display unit 60 can also use stereoscopic projection (hologram) technology to display the surrounding environment of the mobile body around the operator P in stereoscopic form. Furthermore, display unit 60 can also use projection mapping technology to project images onto the floor or walls around the operator P.

[0071] Reference Figure 7 Here is an example of a display screen shown on the display unit 60. Figure 7 This diagram illustrates an example of the display screen 61 shown by the display unit 60. The display screen 61 uses icons to show the location of the operator P and the location of each of the multiple mobile bodies 100A to 100D on a map. The operator P can operate the control device 10 while located in his own home or the like to control the mobile body 100 located at a remote location.

[0072] Furthermore, although four mobile bodies 100A to 100D are used to describe this embodiment as multiple mobile bodies 100 in the following description, the number of mobile bodies 100 is not limited to four. Moreover, the following description primarily uses the case where the operator P manipulates mobile body 100D (the first mobile body). In this case, the object of manipulation is mobile body 100D. Furthermore, in this situation, mobile bodies 100A to 100C are candidate mobile bodies to become switching targets.

[0073] For example, suppose operator P wants to switch the manipulated object from mobile body 100D to mobile body 100B (the second mobile body). Operator P requests the switching of the manipulated object by clicking the icon of mobile body 100B displayed on the display screen 61. Alternatively, instead of using an icon, the display screen 61 can display a list of mobile body names from 100A to 100C. Operator P requests the switching by clicking the name of mobile body 100B from the list.

[0074] Here, refer to Figure 8 This will explain the other display screens shown on the display unit 60. Figure 8 This diagram illustrates an example of another display screen 61 displayed by the display unit 60. As shown, the display screen 61 has an object display area 61a, an operator display area 61b, and an image display area 61c.

[0075] The manipulation object display area 61a is an area for displaying information about the moving body 100 being manipulated. The manipulation object display area 61a displays identification information used to identify the moving body 100 currently being manipulated by the operator P. The identification information may be, for example, the identification name or identification number of each of the multiple moving bodies 100. In the example in the accompanying drawing, it is displayed as "100D being manipulated" in the manipulation object display area 61a. Thus, the manipulation object display area 61a shows that the operator P is currently manipulating the moving body 100D.

[0076] The operator display area 61b is an area for displaying information related to the operator P. For example, the operator display area 61b displays images captured by the camera unit 50.

[0077] Image display area 61c is an area for displaying images forwarded from the moving body 100 of the manipulated object. Figure 8 In the example, the image display area 61c displays the image forwarded from the mobile body 100D. However, it is not limited to the above; the display screen 61 may also have various display areas related to the operation of the mobile body 100. For example, the display screen 61 may also have buttons for selecting the start, stop, switching of the operated object, presence or absence of voice output, or presence or absence of image output, etc. Alternatively, these buttons may be configured in areas other than the image display area 61c.

[0078] The operator P manipulates the mobile body 100D by using a headset 20, controller 30, mouse 40, etc., while observing the display screen 61. The manipulation operation is an operation used to control the movement of the mobile body 100D. The manipulation operation may include, for example, operations related to the movement of the mobile body 100. Movement-related operations include operations that cause the mobile body 100 to move forward, backward, accelerate, decelerate, or rotate. In the case where the mobile body 100 is a flyable unmanned aerial vehicle, the manipulation of the mobile body 100 may include raising or lowering the mobile body 100. Furthermore, the manipulation operation may include the extension and retraction of the head 120, the extension and retraction of the arms 130, image capture, or voice output.

[0079] Furthermore, since the above-described example of the control device 10 is just one example, the structure of the control device 10 can be appropriately modified. For example, if the display unit 60 is a touch panel or the like with input functionality, the control device 10 may also be structured without the controller 30 and mouse 40. Additionally, a keyboard or the like may be used as the input unit 11. Furthermore, if the image from the operator's P side is not transmitted to the moving body 100, the camera unit 50 may not be provided.

[0080] In addition, such as Figure 7 As shown in one example, the display unit 60 displays icons for accepting switching requests from the operator P in the image display area 61c in such a way that they correspond to each of the plurality of moving bodies 100.

[0081] When operator P is manipulating moving body 100D, display unit 60 displays in image display area 61c an icon for accepting switching requests from operator P, corresponding to each of the candidate moving bodies 100A to C that are the switching targets. This icon can also be... Figure 7 The icon shown is a replica of the appearance of the mobile unit 100. (See attached image.) Figure 9 This is to explain the icons that use a different display method.

[0082] Figure 9 This diagram illustrates an example of a display screen 61 that shows icons modeled after the shape of a door. As shown, the image display area 61c includes icons 70a to 70c that are associated with the moving bodies 100A to 100C. The image display area 61c uses augmented reality technology to overlay icons 70a to 70c onto an image captured by the moving body 100D.

[0083] like Figure 9As shown, for example, icons 70a to 70c may include an entrance image area representing a hypothetical entrance that corresponds to the moving body 100A to 100C of the switching target. The entrance image area can be formed, for example, using an image of a door, as shown in icons 70a to 70c. The entrance image area can also be, for example, a door, a window, or a cave. It is not limited to this; icons of various shapes can be used as entrance image areas.

[0084] Icons 70a to 70c are configured such that the object to be manipulated can be selected from mobile bodies 100A to 100C based on the movement of the mobile body 100D, which is the switching source of the object to be manipulated. The display mode of icons 70a to 70c can be changed according to the control of the display control unit 13.

[0085] For example, let's say the manipulated object is switched from moving body 100D to moving body 100B. For example, when moving body 100D moves forward according to the operator P's control, the icons 70a to 70c will change their display mode as moving body 100D moves. Specifically, the display unit 60 displays the image display area 61c in such a way that the icons 70a to 70c are enlarged on the screen as moving body 100D moves forward.

[0086] Figures 10 to 12 This diagram illustrates how the display of icons 70a to 70c changes as the moving body 100D moves. It shows how the moving body 100D moves forward under the control of the operator P, thus displaying icons according to… Figure 10 , Figure 11 , Figure 12 The order in which icons 70a to 70c are displayed changes the way they are magnified. Furthermore, when the moving body 100D moves backward, the display unit 60 displays the image display area 61c in a way that shrinks the display of icons 70a to 70c.

[0087] By adopting this method, the display unit 60 is able to show the situation where the moving body 100D approaches the icons 70a to 70c, and the situation where the moving body 100D moves away from the icons 70a to 70c.

[0088] The manipulator P manipulates the moving body 100D by gradually approaching icons 70a to 70c. Furthermore, the manipulator P further manipulates the moving body 100D from... Figure 12 The state begins to advance, so that the moving body 100D passes through the door shown in icon 70b.

[0089] The control device 10 calculates the distance between the moving body 100D and the icon 70b in the displayed imaginary space. If the distance between the moving body 100D and the icon 70b is less than a predetermined threshold, the display unit 60 switches the display screen from the image of the moving body 100D to the image of the moving body 100B. Thus, the display unit 60 can display on the screen the situation where the moving body 100D passes through the door of the icon 70b.

[0090] Thus, when the moving body 100D moves by passing through one of the icons 70a to 70c on the screen, the display unit 60 displays the image captured by the moving body 100 corresponding to the icon passed through in the image display area 61c, thereby switching the image. By employing this method, the operator P can obtain the feeling that they have switched from the moving body of the switching source to the moving body of the switching target.

[0091] Figure 13 This diagram illustrates the display screen 61 after the switch. The image display area 61c shows the image captured by the camera unit 121 of the moving body 100B, which is the target of the switch. Furthermore, the manipulation object display area 61a shows the current manipulation object as the moving body 100B.

[0092] Return to Figure 5 Continuing with the explanation, the control device 12 controls the various functional units of the operating device 10. That is, the control device 12 controls the headset 20, controller 30, mouse 40, camera unit 50, and display unit 60. Furthermore, the control device 12 includes a display control unit 13, a switching unit 14, and a storage unit 19.

[0093] The display control unit 13 generates the display screen on the display unit 60 and controls the display on the display unit 60. For example, the display control unit 13 controls the display of the icons. The display control unit 13 changes the display mode of the icons according to the movement of the moving object 100. For example, the display control unit 13 changes the display mode of icons 70a to 70c according to the movement information of the moving object 100D.

[0094] The switching unit 14 uses the motion information forwarded from the mobile body 100 to switch the manipulation object from the first mobile body, which is the switching source, to the second mobile body, which is the switching target.

[0095] For example, the switching unit 14 performs a switching process to switch the target mobile body 100 based on a switching request from the operator P. The switching request is a request to switch the target mobile body 100 to another mobile body 100. The switching request may include information about the target mobile body 100. This information may include identification information for identifying the mobile body 100. As a result, the switching unit 14 identifies the target mobile body 100.

[0096] The operator P can implement the switching request using various methods. For example, as described above, the operator P can move the manipulated object 100D forward, thereby bringing it closer to one of the icons 70a to 70c. The operator P can implement the switching request to the moving object 100B, which is associated with icon 70b, by moving the moving object 100D through the door shown in icon 70b.

[0097] Not limited to this, the operator P can also select icon 70b by clicking icon 70b using mouse 40. Thus, the operator P can implement the switching request to switch the manipulated object to the moving body 100B that has been associated with icon 70b.

[0098] Furthermore, the operator P can also issue a switching request using voice recognition technology by uttering a voice command such as "Move to mobile body 100B". Alternatively, the operator P can use a head-mounted display or similar device to issue the switching request. For example, the icon can be selected by linking the orientation of the operator P's face with the orientation of the head 120 of the mobile body 100D being manipulated.

[0099] Here, the switching process performed by the switching unit 14 will be described. The switching unit 14 infers the selected icon based on the motion information A1 of the moving body 100D, and begins the switching process according to the inference result. The switching unit 14 detects on the screen that the moving body 100D has approached one of the icons 70a to 70c. For example, if the distance between the moving body 100D and icon 70b on the screen becomes less than a predetermined threshold, the switching unit 14 detects that the moving body 100D has approached icon 70b. The predetermined threshold can be fixed in advance or set in a way that allows for appropriate changes.

[0100] Assuming the switching unit 14 detects that the mobile body 100D has approached the icon 70b, the switching unit 14 infers that the operator P has initiated a switching request from the mobile body 100D to the mobile body 100B.

[0101] The switching unit 14 initiates the operation of the mobile body 100B, which is the switching target. Specifically, the switching unit 14 forwards motion information A1, which indicates the operation of the mobile body 100D, the switching source, to the mobile body 100B, the switching target, and causes the mobile body 100B to begin performing the operation based on the motion information A1. The motion information A1 may include, for example, the speed of the mobile body 100D.

[0102] For example, the switching unit 14 controls the moving body 100B to move at the same speed as the moving body 100D. Furthermore, during this stage, the manipulated object remains the moving body 100D. That is, the switching unit 14 maintains the state where the operator P is manipulating the moving body 100D, and initiates the movement of the moving body 100B based on the motion information A1 of the moving body 100D.

[0103] The moving body 100B performs an action to reduce the difference between action information A2 and action information A1 representing the action of the moving body 100B. For example, suppose that the speed of the moving body 100B is smaller than the speed of the moving body 100D. In order to reduce the difference between the speed of the moving body 100B and the speed of the moving body 100D, the moving body 100B accelerates.

[0104] As described above, after the mobile body 100B begins to move based on the motion information A1, the switching unit 14 switches the manipulated object from the mobile body 100D to the mobile body 100B at a predetermined timing. The switching unit 14 may also use a predetermined time interval to determine the predetermined timing. For example, the switching unit 14 switches the manipulated object to the mobile body 100B after a predetermined time has elapsed since the movement of the mobile body 100B began.

[0105] Furthermore, the switching unit 14 can also determine a predetermined timing considering the operating status of the moving body 100B. For example, the switching unit 14 acquires motion information A1 and motion information A2, and compares the motion information A1 and motion information A2. For example, if the difference between motion information A1 and motion information A2 becomes less than a predetermined threshold, the switching unit 14 switches the manipulated object to the moving body 100B.

[0106] By employing this method, the motion information A2 of the mobile body 100D can be made close to the motion information A1 before the switch to the mobile body 100B is completed. This allows the operator P to switch the manipulated object without discomfort.

[0107] As described above, since the switching unit 14 begins the switching process by switching the manipulated object to the moving body 100B that is presumed to correspond to the selected icon 70b, the operation can begin at an earlier timing. However, it is not limited to this; the switching unit 14 can also begin the switching process when the icon is actually selected.

[0108] Storage unit 19 is a storage device that stores programs used to implement the various functions of the control device 10. Storage unit 19 may also store information acquired from the mobile bodies 100A to 100D. For example, storage unit 19 may also store image or motion information acquired from each of the mobile bodies 100A to 100D.

[0109] The processing performed by the control device 10

[0110] Next, refer to Figure 14 The switching process implemented by the control device 10 according to this embodiment will be described. Figure 14 This is a flowchart illustrating the processing performed by the control device 10. Similar to the description above, here the control device 10 performs a switching process to switch the controlled object from the moving body 100D to the moving body 100B. Furthermore, it is assumed that the moving body 100B is in a switchable state. Furthermore, it is assumed that the control device 10 continuously obtains motion information from each moving body 100.

[0111] First, the display control unit 13 causes the display unit 60 to display the moving body 100 that is a candidate for switching (S11). For example, the display control unit 13 displays icons 70a to 70c corresponding to each of the moving bodies 100A to 100C in the image display area 61c. (Using...) Figures 9 to 12 As explained, icons 70a to 70c change shape according to the movement of the moving body 100D and can be selectively configured by satisfying predetermined conditions.

[0112] The switching unit 14 infers the selected icon based on the motion information A1 of the moving body 100D. Specifically, the switching unit 14 detects on the screen that the moving body 100D has approached one of the icons 70a to 70c (S12). For example, the switching unit 14 detects an approach when the distance between the moving body 100D and one of the icons 70a to 70c becomes less than a predetermined distance. Here, the switching unit 14 detects that the moving body 100D has approached the icon 70b.

[0113] Therefore, the switching unit 14 infers that icon 70b has been selected and begins switching processing for switching the manipulated object. However, although the switching unit 14 infers the selected icon to begin switching processing, it can also begin switching processing based on whether an icon has been selected and the switching requirement has been fulfilled.

[0114] The switching unit 14 forwards the motion information A1 of the mobile body 100D, which is the switching source, to the mobile body 100B, which is the switching target (S13). The motion information A1 includes, for example, the speed and direction of movement of the mobile body 100D. Although an example of the switching unit 14 forwarding motion information A1 is shown here, it is not limited to this. Devices for forwarding motion information to other mobile bodies and devices for switching the manipulated object can also be provided separately. Therefore, the motion information A1 can also be forwarded to the mobile body 100B by a device other than the manipulation device 10.

[0115] The switching unit 14 initiates the movement of the mobile body 100B based on the motion information A1 (S14). The switching unit 14 controls the mobile body 100B so that the motion information A2 of the mobile body 100B is close to the motion information A1.

[0116] The switching unit 14 determines whether the icon that is inferred to be selected has actually been selected (S15). If it has not been selected (NO in S15), the process returns to step S12.

[0117] When an icon is selected (S15: YES), the switching unit 14 switches the manipulated object from mobile body 100D to mobile body 100B (S16). The switching unit 14 may also postpone the switching until a predetermined time is reached. For example, if the motion information A1 and A2 contain speed-related information, the switching unit 14 may switch the manipulated object after the difference between the speed of mobile body 100B and the speed of mobile body 100D becomes less than a predetermined value.

[0118] The display control unit 13 acquires the image captured by the moving body 100B of the switching target and displays the image in the image display area 61c (S17). As a result, the operator P can use the input unit 11 to operate the moving body 100B.

[0119] As explained above, in the control system 1 according to this embodiment, the control device 10, which controls multiple mobile bodies 100, forwards motion information indicating the action of a first mobile body, which is the switching source, to a second mobile body, which is the switching target. After the second mobile body starts to perform an action based on the motion information of the first mobile body, the control device 10 performs a switching process to switch the control target from the first mobile body to the second mobile body.

[0120] By employing this method, the control device 10 can replicate the movement of the switching source and make the switching target move along with it. Furthermore, since the control device 10 uses icons on the display screen to infer the selected moving body and begin the switching process, the switching can be performed more smoothly.

[0121] Thus, the operator can obtain the feeling of moving to another location while remaining in the same mobile body. Therefore, when the operator remotely manipulates a mobile body such as a robot in the imaginary space and moves freely within the imaginary space, the control system 1 can switch the manipulated object without interrupting the feeling of moving within the imaginary space.

[0122] Thus, the control system 1 according to this embodiment can reduce the discomfort when switching the moving body of the controlled object and provide the operator with a continuous movement experience in the imaginary space.

[0123] Furthermore, the structure of the control system 1 described above is merely an example, and the structure of the control system 1 can be appropriately modified. For example, each of the control device 10 and the moving body 100 can also be constructed using a device that integrates multiple structures. Moreover, some or all of the functions of each of the control device 10 and the moving body 100 can be integrated into a single device. For example, although in Figure 1 The control device 10 is shown on the outside of the mobile body 100, but it can also be installed inside the mobile body 100. In that case, the operator P can operate the control device 10 directly by riding on the mobile body 100. Alternatively, the operator P can operate the control device 10 using an information terminal or the like that capable of communicating with it. Furthermore, the functional units of each of the control device 10 and the mobile body 100 can be distributed and processed using multiple devices.

[0124] Example 1 of the modification of implementation method 1

[0125] Reference Figure 15 Here, we will explain a modification example 1 of implementation method 1. Figure 15This diagram illustrates an example of the display screen 62 involved in this modified example. This modified example differs from Embodiment 1 in that the icon further includes an image obtained by a moving body that is a candidate for switching. The structures of the control system 1 and the control device 10 are the same as in Embodiment 1, therefore descriptions are omitted.

[0126] like Figure 15 As shown, on display screen 62, icons 70a to 70c respectively contain switching target images 80a to 80c. Each of the switching target images 80a to 80c is an image acquired by the moving bodies 100A to 100C. Alternatively, the switching target images 80a to 80c may also be still images.

[0127] Thus, by displaying the entrance area image and the switching target image in an overlapping manner in the image display area 61c, the operator P can visually confirm the image of the moving body 100 of the switching target based on the door portion of each icon. Therefore, the operator P can observe the space visible from the moving body 100 other than the controlled object in an imaginary space, across the icons. By employing this method, the operator P can easily select the controlled object for the switching target.

[0128] Example 2 of the modification of implementation method 1

[0129] Next, a modification example 2 of Embodiment 1 will be described. In Embodiment 1 described above, an example was shown where the switching unit 14 switched the object of manipulation from the first moving body to the second moving body after the second moving body started to perform an operation based on the motion information of the first moving body. Thus, in Embodiment 1, the object of manipulation can be switched while the second moving body has taken over the operation of the first moving body.

[0130] In this modified example, after the first moving body begins to perform an action based on the motion information of the second moving body, the switching unit 14 switches the manipulated object from the first moving body to the second moving body. That is, in this modified method, the first moving body coordinates its action with that of the second moving body. Alternatively, both the first and second moving bodies can change their actions in a manner that closely approximates each other's actions.

[0131] Appropriate replacement Figure 14 The flowchart shown illustrates the differences from Embodiment 1. In step S13, the switching unit 14 forwards the motion information A2 of the second mobile body (mobile body 100B) to the first mobile body (mobile body 100D). In step S14, the switching unit 14 initiates the motion of the first mobile body based on the motion information A2. The switching unit 14 controls the first mobile body to make the motion information A1 of the first mobile body approximate the motion information A2.

[0132] By employing this method, when, for example, the second moving body moves at a slower speed compared to the first moving body, the first moving body can be controlled to reduce its speed in accordance with the speed of the second moving body. This speed can be controlled to decrease gradually. Furthermore, when the operator P manipulates the first moving body, it can be controlled to move even slower. Thus, in this modified example, the moving body of the switching source can perform actions that coordinate with the actions of the moving body of the switching target.

[0133] Example 3 of the modification of implementation method 1

[0134] Next, a modified example 3 of Embodiment 1 will be described. In Embodiment 1 described above, an example was shown in which the display unit 60 displays an image of the second moving body after the switching unit 14 switches the manipulated object from the first moving body to the second moving body.

[0135] In this modified example, the switching unit 14 adjusts the image captured around the second mobile body based on the motion information of the first mobile body. After the second mobile body begins to move based on the motion information of the first mobile body, the display unit 60 displays the adjusted image until the switching of the controlled object is completed. After displaying the image, the switching unit 14 switches the controlled object from the first mobile body to the second mobile body. By visually confirming the adjusted image, the operator P can further reduce discomfort caused by the switching.

[0136] exist Figure 14 The flowchart shown illustrates the differences from Embodiment 1. The switching unit 14 initiates a switching process after detecting in step S12 that the first moving body (moving body 100D) has approached one of the icons 70a to 70c. Here, the switching unit 14 adjusts the image captured around the second moving body (moving body 100B) based on the motion information of the first moving body.

[0137] Here, the timing for starting the switching process is set to T0, and the timing for completing the switching process and displaying the image of the second moving body by the display unit 60 is set to T1. The switching unit 14 acquires and adjusts the image captured before T0 in the second moving body, thereby acquiring an adjusted image.

[0138] The switching unit 14 adjusts the image based on the motion information A1 of the first moving body. The switching unit 14 adjusts the image captured by the second moving body to reduce discomfort when switching from the image captured by the first moving body to the image captured by the second moving body.

[0139] For example, suppose the first moving body is moving at a higher speed than the second moving body. In this case, in the image captured by the second moving body, the surrounding scenery and other elements will change at a slower speed compared to the image captured by the first moving body. Therefore, the discomfort is greater when simply switching images.

[0140] The switching unit 14 adjusts the images to display them more quickly, ensuring a smooth transition between the images at point T1. The switching unit 14 can also adjust the images by increasing the speed from T0 to T1. The switching unit 14 obtains an adjusted image that is displayed at a higher speed than the actual speed of the images captured by the second moving body. The switching unit 14 reverses the timing of displaying the adjusted image to ensure the images are connected at point T1 and displayed on the display unit 60. Thus, the display unit 60 displays the adjusted image from T0 to T1.

[0141] After the display unit 60 displays the adjustment image, the switching unit 14 switches the manipulated object from the first moving body to the second moving body. The processing after step S13 is the same as in Embodiment 1. In addition, the processing related to the adjustment image described above can also be reversed in order with the processing of steps S13 and S14.

[0142] Furthermore, the motion information A1 can be obtained from speed information acquired by a sensor-like component mounted on the first moving body, or it can be obtained by analyzing images captured by the first moving body. Alternatively, information other than speed can be used as motion information A1.

[0143] Thus, according to this modified example, by adjusting the playback speed of the images captured by the second moving body, the discomfort of switching the manipulated object can be reduced.

[0144] Implementation Method 2

[0145] Structure of the control device 10a

[0146] Next, refer to Figure 16 The second implementation method will now be described. Figure 16This is a block diagram illustrating the structure of the control device 10a according to this embodiment. The control device 10a differs from Embodiment 1 in that it includes a switching image generation unit 15 in addition to the structure of the control device 10 described above. In the following description, the differences from Embodiment 1 will be mainly explained, and repeated content will be omitted as appropriate. Furthermore, similar to Embodiment 1, an example will be used where the moving body of the switching source is moving body 100D and the moving body of the switching target is moving body 100B for explanation.

[0147] In Embodiment 1, the case where the moving body 100B targeting the switching action can quickly begin to operate under the control of the control device 10 was described. However, there are cases where the moving body 100B cannot quickly begin to operate when the control device 10 performs the switching process. For example, there may be cases where a special action needs to be performed for the moving body 100B.

[0148] For example, suppose the mobile body 100B is connected to a charging station for charging and is charging. In this case, in order for the mobile body 100B to start operating, it needs to be disconnected from the charging station before the switching process is performed. Therefore, even if the control device 10 performs the switching process, the mobile body 100B cannot start operating quickly. As a result, the operator P cannot continuously switch the controlled object, which causes discomfort. This embodiment can solve this problem.

[0149] In the control device 10a, the switching unit 14 determines whether a switch from the switching source mobile body 100D to the switching target mobile body 100B can be performed based on the motion information A2 of the mobile body 100B. The switching unit 14 determines whether a switch from the mobile body 100D to the mobile body 100B can be performed based on the motion information A2 of the mobile body 100B.

[0150] For example, the switching unit 14 determines whether a switch can be performed by judging whether the mobile body 100B can perform an action within a predetermined time based on the action information A2. For example, if the mobile body 100B is connected to a charging station as described above, the switching unit 14 determines that the disconnection action would take more than the predetermined time. In this case, the switching unit 14 determines that a switch to the mobile body 100B cannot be performed.

[0151] If the switching unit 14 determines that switching to the mobile body 100B cannot be performed, it displays the switching image generated by the switching image generation unit 15 in the image display area 61c. After displaying the switching image in the image display area 61c, the switching unit 14 switches the target of manipulation from the mobile body 100D to the mobile body 100B at a predetermined time.

[0152] The switching unit 14 can determine the predetermined timing in the same manner as in Embodiment 1. The switching unit 14 can also use a predetermined time interval to determine the predetermined timing. Alternatively, the switching unit 14 can determine the predetermined timing taking into account the operating conditions of the mobile body 100B. For example, the switching unit 14 can display the switching image until the disengagement from the charging station is complete, and then switch the target of operation.

[0153] If the switching unit 14 determines that the switching of the manipulated object cannot be performed, the switching image generation unit 15 generates a switching image using the image acquired in the mobile body 100D and the image acquired in the mobile body 100B. Hereinafter, the image acquired in the mobile body 100D will be referred to as image V1, the image acquired in the mobile body 100B will be referred to as image V2, and the image generated by the switching image generation unit 15 will be referred to as switching image V3.

[0154] The switching image generation unit 15 generates a switching image V3 that reduces the discomfort felt by the operator P when switching from image V1 to image V2 in the image display area 61c. Images V1 and V2 can be images stored in their respective storage units 114 in the mobile bodies 100D and 100B.

[0155] The switching image generation unit 15 generates a switching image V3 using images V1 and V2 acquired within a predetermined time, for example, based on the date and time information of the images being captured. The switching image generation unit 15 can use well-known techniques to generate the switching image V3 based on images V1 and V2. For example, the switching image generation unit 15 generates a switching image V3 that performs a magnification operation from image V1 to image V2. Not limited to this, the switching image generation unit 15 can use various methods to generate a switching image V3 that connects images V1 and V2.

[0156] The processing performed by the control device 10a

[0157] Next, refer to Figure 17 The switching process implemented by the control device 10a according to this embodiment will be explained. Figure 17 This is a flowchart illustrating the processing performed by the control device 10a. Similar to the description above, here the control device 10a performs a switching process to switch the controlled object from the moving body 100D to the moving body 100B. Furthermore, it is assumed that the control device 10a continuously obtains motion information from each moving body 100.

[0158] First, the display control unit 13 causes the display unit 60 to display the mobile body 100 that has become a candidate for switching (S21). For example, the display control unit 13 displays icons 70a to 70c corresponding to each of the mobile bodies 100A to 100C in the image display area 61c. Similar to Embodiment 1, the icons 70a to 70c change shape according to the movement of the mobile body 100D and are configured in a way that allows selection by satisfying predetermined conditions.

[0159] The switching unit 14 infers the selected icon based on the motion information A1 of the moving body 100D. The switching unit 14 detects on the screen that the moving body 100D has approached one of the icons 70a to 70c (S22). Here, the switching unit 14 detects that the moving body 100D has approached the icon 70b.

[0160] The switching unit 14 determines whether a switch from the mobile body 100D to the mobile body 100B can be performed based on the motion information A2 of the mobile body 100B (S23). If it is determined that a switch can be performed (yes in S23), the process proceeds to step S26.

[0161] If it is determined that switching cannot be performed (No in S23), the switching image generation unit 15 generates a switching image V3 using the image V1 acquired by the moving body 100D and the image V2 acquired by the moving body 100B (S24). For example, the switching image V3 can be an image that performs a magnification operation from image V1 to image V2.

[0162] The switching unit 14 displays the switching image V3 in the image display area 61c (S25). As a result, the operator P can visually confirm the switching image V3.

[0163] Next, the switching unit 14 initiates the operation of the mobile body 100B (S26). The switching unit 14 causes the mobile body 100B to operate, enabling switching to the mobile body 100B. For example, the switching unit 14 controls the mobile body 100B to detach from the charging station. By employing this method, the operating device 10a can switch the mobile body 100B to a switchable state while the operator P is observing the switching image V3.

[0164] The switching unit 14 determines whether the icon that is inferred to be selected has actually been selected (S27). If it has not been selected (No in S27), the process returns to step S22.

[0165] If an icon is selected (as in S27), the switching unit 14 switches the manipulated object from mobile body 100D to mobile body 100B (S28). The switching unit 14 may also postpone the switching until a predetermined time interval. For example, the switching unit 14 determines whether mobile body 100B has changed to a switchable state. The switching unit 14 may also be configured to switch the manipulated object if it determines that mobile body 100B has changed to a switchable state.

[0166] The display control unit 13 acquires the image captured by the moving body 100B of the switching target and displays the image in the image display area 61c (S29). As a result, the operator P can use the input unit 11 to operate the moving body 100B.

[0167] Furthermore, in step S26, the switching unit 14 may, similarly to Embodiment 1, further initiate the movement of the mobile body 100B based on the motion information A1 of the mobile body 100D. In this case, the switching unit 14 forwards the motion information A1 to the mobile body 100B. The motion information A1 may include, for example, the speed and direction of movement of the mobile body 100D. The switching unit 14 controls the mobile body 100B to make the motion information A2 of the mobile body 100B closely approximate the motion information A1.

[0168] By employing this method, the operating device 10a is able to switch the mobile body 100B to a switchable state during the period when the switching image V3 is displayed, and make the movement of the mobile body 100B approximate the movement of the mobile body 100D.

[0169] As explained above, in the control system 1 of this embodiment, in the control device 10a that controls multiple moving bodies 100, the switching unit 14 determines whether a switch from the first moving body, which is the switching source, can be performed based on the motion information of the second moving body, which is the switching target.

[0170] If switching is not possible, the switching image generation unit 15 generates a switching image using the image acquired in the first moving body and the image acquired in the second moving body. After displaying the switching image on the screen, the switching unit 14 switches the target of manipulation from the first moving body to the second moving body.

[0171] By employing this method, the control device 10a can switch the controlled object while reducing the discomfort experienced by the operator, even when the second moving body cannot be switched immediately, by displaying a switching image.

[0172] Furthermore, since the control device 10a can transfer the motion information of the first moving body to the second moving body during the period when the switching image is displayed, it can achieve the same effect as embodiment 1.

[0173] Hardware structure example

[0174] The functional components of the control device 10 and the moving body 100 can be implemented by hardware (e.g., hard-connected circuits) or by a combination of hardware and software (e.g., a combination of circuits and programs that control them). The following will explain the case where the functional components of the control device 10, etc., are implemented by a combination of hardware and software.

[0175] Figure 18 This is a block diagram illustrating the hardware structure of a computer 900 that implements the control device 10, etc. The computer 900 can be a dedicated computer designed to implement the control device 10, etc., or it can be a general-purpose computer. The computer 900 can also be a portable computer such as a smartphone or tablet computer.

[0176] For example, by installing a predetermined application program on the computer 900, the various functions of the operating device 10 and the like can be implemented in the computer 900. The aforementioned application program is composed of programs for implementing the functional structure of the operating device 10 and the like.

[0177] Computer 900 includes a bus 902, a processor 904, a memory 906, a storage device 908, an input / output interface 910, and a network interface 912. Bus 902 serves as a data transmission path for the processor 904, memory 906, storage device 908, input / output interface 910, and network interface 912 to send and receive data. However, the method of interconnecting the processor 904 and other components is not limited to bus connection.

[0178] Processor 904 can be any processor such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). Memory 906 is the main storage device implemented using RAM (Random Access Memory). Storage device 908 is an auxiliary storage device implemented using hard disks, SSDs (Solid State Drives), memory cards, or ROM (Read Only Memory).

[0179] The input / output interface 910 is an interface used to connect the computer 900 and input / output devices. For example, input devices such as a keyboard and output devices such as a display device are connected to the input / output interface 910.

[0180] Network interface 912 is used to connect computer 900 to a network. This network can be either a LAN (Local Area Network) or a WAN (Wide Area Network).

[0181] The storage device 908 stores the programs (programs that implement the application programs described above) that implement the various functional components of the control device 10, etc. The processor 904 implements the various functional components of the control device 10, etc. by reading the programs into the memory 906 and executing them.

[0182] Each processor executes one or more programs containing a set of commands for causing the computer to implement an algorithm. The program includes a set of commands (or software code) for causing the computer to perform one or more functions described in the implementation when read by the computer. The program can also be stored in a wide variety of non-transitory computer-readable media or tangible storage media. As non-limiting examples, computer-readable media or tangible storage media include RAM (random-access memory), ROM (read-only memory), flash memory, SSD (solid-state drive) or other memory technologies, CD-ROM, DVD (digital versatile disc), Blu-ray disc or other optical disc storage, cassette tape, magnetic tape, disk storage, or other magnetic storage devices. The program can also be transmitted on a wide variety of transitory computer-readable media or communication media. As a non-limiting example, temporary computer-readable media or communication media include electrical, optical, acoustic, or other forms of transmitted signals.

[0183] Furthermore, this disclosure is not limited to the above-described embodiments, and appropriate modifications can be made without departing from the spirit of the matter. For example, the control device 10 may also use the same method as the motion information to transfer environmental information related to the environment of the switching source mobile body 100 to the switching target mobile body 100. The environmental information may include, for example, air pressure, water pressure, wind pressure, air temperature, or water temperature detected in the switching source mobile body 100.

[0184] Furthermore, the above-described embodiments can be arbitrarily combined for implementation. For example, Embodiment 1, variations 1 to 3 of Embodiment 1, and part or all of Embodiment 2 can be appropriately combined for implementation.

Claims

1. A manipulation system for manipulating a moving body selected from a plurality of moving bodies as the object of manipulation, comprising: The display unit displays information including images captured from the surroundings of the moving body. as well as The switching unit uses motion information representing the movement of the moving body to switch the manipulated object from a first moving body, which is the switching source, to a second moving body, which is the switching target. The switching unit switches the manipulated object from the first moving body to the second moving body in at least one of the following situations: after the second moving body starts moving based on the motion information of the first moving body; after the first moving body starts moving based on the motion information of the second moving body; and after the display unit displays an image adjusted according to the motion information of the first moving body. The switching unit determines whether a switch from the first mobile body to the second mobile body can be performed based on the motion information of the second mobile body. If the switch is determined to be impossible, the unit displays a switching image and then switches the manipulated object from the first mobile body to the second mobile body. The switching image is generated using images captured around the first mobile body and images captured around the second mobile body.

2. The operating system as claimed in claim 1, wherein, The display unit displays icons that are used to receive switching requests from the operator, and establishes a correspondence between these icons and candidate moving bodies that become the switching targets. The icon is set in a way that allows selection based on the action of the first moving body, and The switching unit switches the manipulated object to a moving body that has been established with a corresponding relationship to the selected icon.

3. The operating system as described in claim 2, wherein, The switching unit infers the icon to be selected based on the action information of the first moving body, and switches the manipulated object according to the inference result.

4. A manipulation system for manipulating a moving body selected from a plurality of moving bodies as the object of manipulation, comprising: The display unit displays information including images captured from the surroundings of the moving body. as well as The switching unit uses motion information representing the movement of the moving body to switch the manipulated object from a first moving body, which is the switching source, to a second moving body, which is the switching target. The switching unit switches the manipulated object from the first moving body to the second moving body in at least one of the following situations: after the second moving body starts moving based on the motion information of the first moving body; after the first moving body starts moving based on the motion information of the second moving body; and after the display unit displays an image adjusted according to the motion information of the first moving body. The display unit displays icons that are used to receive switching requests from the operator, and establishes a correspondence between these icons and candidate moving bodies that become the switching targets. The icon is set in a way that allows selection based on the action of the first moving body, and The switching unit switches the manipulated object to a moving body that has been established with a corresponding relationship to the selected icon. The switching unit infers the icon to be selected based on the action information of the first moving body, and switches the manipulated object according to the inference result.

5. A method of manipulation, comprising: The display step involves a computer of a control system that manipulates a moving object selected from a plurality of moving bodies to display display information, which includes images captured from the surroundings of the moving body. as well as In the switching step, the computer uses motion information representing the movement of the moving body to switch the manipulated object from a first moving body, which is the switching source, to a second moving body, which is the switching target. In the switching step, In at least one of the following cases: after the second moving body begins to move based on the motion information of the first moving body; after the first moving body begins to move based on the motion information of the second moving body; and after the image adjusted according to the motion information of the first moving body is displayed in the display step, the manipulated object is switched from the first moving body to the second moving body. In the switching step, based on the motion information of the second mobile body, it is determined whether a switch from the first mobile body to the second mobile body can be performed. If it is determined that the switch cannot be performed, the manipulation object is switched from the first mobile body to the second mobile body after the display unit displays the switching image. The switching image is generated using images taken around the first mobile body and images taken around the second mobile body.

6. A method of manipulation, comprising: The display step involves a computer of a control system that manipulates a moving object selected from a plurality of moving bodies to display display information, which includes images captured from the surroundings of the moving body. as well as In the switching step, the computer uses motion information representing the movement of the moving body to switch the manipulated object from a first moving body, which is the switching source, to a second moving body, which is the switching target. In the switching step, In at least one of the following cases: after the second moving body begins to move based on the motion information of the first moving body; after the first moving body begins to move based on the motion information of the second moving body; and after the image adjusted according to the motion information of the first moving body is displayed in the display step, the manipulated object is switched from the first moving body to the second moving body. In the display step, icons used to receive switching requests from the operator are displayed in a way that establishes a correspondence between them and candidate moving bodies that become the switching targets. The icon is set in a way that allows selection based on the action of the first moving body, and In the switching step, the manipulated object is switched to a moving body that has been established as a correspondence with the selected icon. In the switching step, the icon to be selected is inferred based on the action information of the first moving body, and the manipulated object is switched according to the inference result.

7. A storage medium storing a program that causes a computer of an operating system, which manipulates a movable object selected from a plurality of movable bodies, to execute the following steps, wherein... The steps include: The display step involves displaying display information, which includes images captured from the surroundings of the moving object; and The switching step involves using motion information representing the movement of the moving body to switch the manipulated object from a first moving body (the switching source) to a second moving body (the switching target). In the switching step, In at least one of the following cases: after the second moving body begins to move based on the motion information of the first moving body; after the first moving body begins to move based on the motion information of the second moving body; and after the image adjusted according to the motion information of the first moving body is displayed in the display step, the manipulated object is switched from the first moving body to the second moving body. In the switching step, based on the motion information of the second mobile body, it is determined whether a switch from the first mobile body to the second mobile body can be performed. If it is determined that the switch cannot be performed, the manipulation object is switched from the first mobile body to the second mobile body after the display unit displays the switching image. The switching image is generated using images taken around the first mobile body and images taken around the second mobile body.

8. A storage medium storing a program for a computer of an operating system that manipulates a movable object selected from a plurality of movable bodies to execute the following steps, wherein... The steps include: The display step involves displaying display information, which includes images captured from the surroundings of the moving object; and The switching step involves using motion information representing the movement of the moving body to switch the manipulated object from a first moving body (the switching source) to a second moving body (the switching target). In the switching step, In at least one of the following cases: after the second moving body begins to move based on the motion information of the first moving body; after the first moving body begins to move based on the motion information of the second moving body; and after the image adjusted according to the motion information of the first moving body is displayed in the display step, the manipulated object is switched from the first moving body to the second moving body. In the display step, icons used to receive switching requests from the operator are displayed in a way that establishes a correspondence between them and candidate moving bodies that become the switching targets. The icon is set in a way that allows selection based on the action of the first moving body, and In the switching step, the manipulated object is switched to a moving body that has been established as a correspondence with the selected icon. In the switching step, the icon to be selected is inferred based on the action information of the first moving body, and the manipulated object is switched according to the inference result.

Images