Mobile body remote operation system, mobile body remote operation device, and mobile body control device

Abstract

The purpose of the present disclosure is to provide a mobile body remote operation system, a mobile body remote operation device, and a mobile body control device that make it possible to operate a plurality of mobile bodies moving by different motion models. A mobile body remote operation system according to the present disclosure comprises: an operation amount acquisition unit that acquires an operation amount from an operation device controlled by an operator; a setting unit that sets an object to be operated among a plurality of mobile bodies moving by different motion models; and a control amount calculation unit that uses the operation amount acquired by the operation amount acquisition unit as a basis to calculate a control amount for controlling the object to be operated set by the setting unit.

Description

Mobile body remote control system, mobile body remote control device, and mobile body control device

[0001] The present disclosure relates to a mobile object remote control system, a mobile object remote control device, and a mobile object control device for controlling multiple mobile objects located in remote locations.

[0002] In recent years, with the advancement of communication technologies such as the Internet, there has been progress in the development of mobile object remote control systems that enable operators to control mobile objects located in remote locations. In particular, mobile object remote control systems that aim to reduce labor by allowing a single operator to monitor and control multiple mobile objects have been developed (see, for example, Patent Document 1).

[0003] Patent No. 6319507

[0004] When controlling a moving object in various situations, it is possible to increase the number of movement mechanisms that each moving object has and switch between them depending on the situation, but this increases the cost of each moving object. Therefore, it is desirable to arrange and control multiple moving objects that operate using different motion models depending on the situation.

[0005] In Patent Document 1, multiple automobiles that operate according to the same motion model are the targets of control, and it is not possible to control multiple moving bodies that operate according to different motion models.

[0006] The present disclosure has been made to solve such problems, and aims to provide a mobile object remote control system, a mobile object remote control device, and a mobile object control device that are capable of controlling multiple mobile objects that operate using different motion models.

[0007] In order to solve the above problems, the mobile object remote control system according to the present disclosure includes a control amount acquisition unit that acquires a control amount from a control device operated by an operator, a setting unit that sets a target to be controlled from among multiple mobile objects that operate using different motion models, and a control amount calculation unit that calculates a control amount for controlling the target to be controlled that has been set by the setting unit, based on the control amount acquired by the control amount acquisition unit.

[0008] According to the present disclosure, it becomes possible to control multiple moving objects that operate using different motion models.

[0009] The objects, features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings.

[0010] 1 is a block diagram showing an example of the configuration of a mobile object remote control system according to embodiment 1. FIG. 2 is a diagram for explaining a mobile object according to embodiment 1. FIG. 3 is a diagram for explaining a mobile object according to embodiment 1. FIG. 4 is a block diagram showing an example of the configuration of a mobile object remote control device according to embodiment 1. FIG. 5 is a block diagram showing an example of the configuration of a mobile object control device and a mobile object according to embodiment 1. FIG. 6 is a diagram showing an example of a combination of an operation mode, a control mechanism, and a mobile object according to embodiment 1. FIG. 7 is a diagram for explaining an automobile mode according to embodiment 1. FIG. 8 is a diagram showing an example of the movement of a differential two-wheel drive in automobile mode according to embodiment 1. FIG. 9 is a diagram for explaining a free mode according to embodiment 1. FIG. 10 is a diagram showing an example of the movement of the automobile in the free mode according to embodiment 1. FIG. 11 is a diagram for explaining a touch panel mode according to embodiment 1. FIG. 12 is a diagram showing an example of the movement of the automobile in the touch panel mode according to embodiment 1. FIG. 13 is a diagram showing an example of a motion model of the automobile according to embodiment 1. FIG. 14 is a diagram showing an example of the configuration of a control amount calculation unit according to embodiment 1. FIG. 15 is a diagram showing an example of the configuration of a control amount calculation unit according to embodiment 1. FIG. 16 is a diagram showing an example of the configuration of a control amount calculation unit according to embodiment 1. FIG. 17 is a flowchart showing an example of the operation of the mobile object remote control device according to embodiment 1. FIG. 18 is a flowchart showing an example of the operation of the mobile object control device according to embodiment 1. FIG. 19 is a flowchart showing an example of the operation of the mobile object control device according to embodiment 1. Fig. 1 is a block diagram showing an example of the configuration of a mobile body remote control device according to embodiment 2. Fig. 2 is a block diagram showing an example of the configuration of a mobile body control device according to embodiment 2. Fig. 3 is a block diagram showing an example of the hardware configuration of a mobile body remote control device according to embodiments 1 and 2. Fig. 4 is a block diagram showing an example of the hardware configuration of a mobile body remote control device according to embodiments 1 and 2.

[0011] <Embodiment 1> <Configuration of a mobile body remote control system> Fig. 1 is a block diagram showing an example of the configuration of a mobile body remote control system according to embodiment 1. The mobile body remote control system is made up of a control system 1, a first mobile body system 2, a second mobile body system 3, a third mobile body system 4, and a network 5. The control system 1 is communicably connected to each of the first mobile body system 2, the second mobile body system 3, and the third mobile body system 4 via the network 5. The first mobile body 22, the second mobile body 32, and the third mobile body 42 each operate using a different motion model.

[0012] Although FIG. 1 illustrates three mobile body systems (first mobile body system 2, second mobile body system 3, and third mobile body system 4), the number of mobile body systems may be two or four or more. Furthermore, mobile bodies other than the first mobile body 22, second mobile body 32, and third mobile body 42 may be the control targets. Although FIG. 1 illustrates three mobile bodies operating under three different motion models, this is not limiting. Three (or two or four or more) mobile bodies operating under at least two different motion models may be the control targets. That is, the first mobile body 22 and the second mobile body 32 operating under the same motion model and the third mobile body 42 operating under a different motion model from the first mobile body 22 and the second mobile body 32 may be the control targets.

[0013] The control system 1 is composed of a mobile object remote control device 11, a control device 12, and an image display device 13. An operator 14 views an image displayed on the image display device 13 and operates the control device 12 to control a mobile object (one of the first mobile object 22, the second mobile object 32, and the third mobile object 42) that is the object to be controlled. The mobile object remote control device 11 transmits the amount of movement (control amount) when the operator 14 operates the control device 12 to control the mobile object to a mobile object control device (one of the mobile object control devices 21, 31, and 41) that controls the operation of the mobile object that is the object to be controlled. The mobile object remote control device 11 also receives information from a camera and a sensor installed on the mobile object from the mobile object control device that controls the operation of the mobile object that is the object to be controlled, and generates an image to be displayed on the image display device 13 based on this information.

[0014] The first mobile body system 2 is composed of a mobile body control device 21 and a first mobile body 22. The mobile body control device 21 controls the operation of the first mobile body 22 based on the operation amount received from the mobile body remote control device 11. The mobile body control device 21 also transmits information from a camera and a sensor provided on the first mobile body 22 to the mobile body remote control device 11. The first mobile body 22 is a mobile body having wheels as a movement mechanism, such as an automobile.

[0015] The second mobile body system 3 is composed of a mobile body control device 31 and a second mobile body 32. The configuration and operation of the mobile body control device 31 are similar to those of the mobile body control device 21. The second mobile body 32 is a mobile body having an infinite mechanism as a movement mechanism, such as a crawler.

[0016] The third mobile body system 4 is composed of a mobile body control device 41 and a third mobile body 42. The configuration and operation of the mobile body control device 41 are similar to those of the mobile body control device 21. The third mobile body 42 is a mobile body having legs as a movement mechanism, such as a legged robot.

[0017] FIG. 2 is a diagram illustrating a moving body according to the first embodiment. A moving body is an object that can change the position and orientation of its own body coordinate system relative to a reference coordinate system by using its own movement mechanism. In FIG. 2, the XY coordinate system is the reference coordinate system, and the XY coordinate system is the body coordinate system. Note that FIG. 2 illustrates a moving body having two differential wheels as the movement mechanism.

[0018] As shown in Fig. 3, the movable area of ​​a moving body varies depending on the moving mechanism that the moving body possesses. The different moving mechanisms of each moving body shown in Fig. 3 result in different motion models (how the body moves and with what operations). For example, in the case of a car, the position and attitude of the car can be controlled by operating the steering wheel, accelerator pedal, and brake pedal. This can be considered the "motion model" of the car.

[0019] The moving object is not limited to an actual moving object, but may be a moving object existing in a virtual space (simulation space). The virtual space may be, for example, a driving simulator. The moving object existing in the virtual space corresponds to a computer (server) incorporating a physical simulator. In this case, an image corresponding to the viewpoint of the moving object in the physical simulator is displayed on the image display device 13. This allows the operator to become familiar with operating each moving object in the virtual space.

[0020] <Configuration of mobile object remote control device> Fig. 4 is a block diagram showing an example of the configuration of the mobile object remote control device 6 according to embodiment 1. The mobile object remote control device 6 includes a control amount acquisition unit 61, a setting unit 62, a transmission unit 63, a reception unit 64, and an image generation unit 65. The mobile object remote control device 6 is also connected to the control device 12 and the image display device 13. The mobile object remote control device 6 corresponds to the mobile object remote control device 11 shown in Fig. 1.

[0021] The control device 12 includes control mechanisms such as a steering wheel, accelerator pedal, and brake pedal, a joystick, a mouse and keyboard, or a tablet. The control device 12 may include one or more of these control mechanisms. When the control device 12 includes multiple control mechanisms, the operator 14 (see FIG. 1 , the same applies below) actually operates one of the control mechanisms.

[0022] The image display device 13 is, for example, a display or a head-mounted display.

[0023] The control amount acquisition unit 61 acquires a control amount from the control device 12 operated by the operator 14. Specifically, the control amount acquisition unit 61 decodes the control amount acquired from the control device 12.

[0024] The setting unit 62 sets the control object and the operation mode based on the setting information specified by the operator 14. The control object is a moving object (moving object 8 shown in FIG. 5 , which will be described later) specified by the operator 14 among a plurality of moving objects that operate according to different motion models. There may be a plurality of control objects. The operation mode is a motion model that causes the control object specified by the operator 14 to operate. Details of the operation mode will be described later. The operator 14 may specify the setting information using the control device 12. Furthermore, the setting unit 62 may automatically select the operation mode when setting the control object specified by the operator 14.

[0025] The transmitter 63 transmits the control amount acquired by the control amount acquisition unit 61 and the operation mode set by the setting unit 62 to a mobile body control device (a mobile body control device 7 shown in FIG. 5 , described later) that controls the control target set by the setting unit 62 via the network 5. The transmitter 63 has a function of switching the mobile body control device that is the destination (communication partner) of the control amount and operation mode based on the control target set by the setting unit 62. Furthermore, when the control target is multiple mobile bodies, the transmitter 63 may transmit the control amount and operation mode simultaneously to the mobile body control devices that control these mobile bodies. In this case, these mobile bodies will move in the same way in response to the control by the operator 14.

[0026] The receiving unit 64 receives image information and sensor information of the periphery of the target (a mobile object 8 shown in FIG. 5 , which will be described later) from a mobile object control device (a mobile object control device 7 shown in FIG. 5 , which will be described later) via the network. The receiving unit 64 has a function of switching the mobile object control device that is the sender (communication partner) of the image information and sensor information based on the target set by the setting unit 62.

[0027] The image generating unit 65 performs image processing based on the image information and sensor information received by the receiving unit 64, and generates a presentation image to be presented to the operator.

[0028] The image generation unit 65 may correct the angle of view and size of the presented image so that the viewpoint is the same as each moving body 8 moves. This can mitigate differences in appearance between each moving body 8. For example, when the operator 14 is operating the object to be controlled by rotating it, the image generation unit 65 corrects the presented image so that the change in viewpoint as the object to be controlled moves appears as if the object to be controlled is rotating on the spot.

[0029] The image generating unit 65 may correct distortion of the presented image due to the roll motion, pitch motion, and up and down motion of the moving body 8 (for example, a legged robot).

[0030] The image generation unit 65 corrects the presented image so that there is no time delay associated with controlling the object to be controlled. Here, the "time delay associated with controlling the moving object" includes not only the delay caused by the inability of the vehicle to rotate in place in response to the rotation operation of the joystick 15 as shown in Figures 9 and 10 described below, but also a communication delay between the moving object remote control device 6 and the moving object control device 7.

[0031] The image generation unit 65 may generate a presentation image including the control amount calculated by the control amount calculation unit 72 (see FIG. 5 described later). In this case, the receiving unit 64 receives the control amount calculated by the control amount calculation unit 72 from the mobile object control device 7 (see FIG. 5 described later). For example, when the operation mode is set to the free mode (see FIG. 6, details will be described later) and the vehicle is rotated on the spot, the angle and time that can be rotated change depending on the presence or absence of obstacles around the vehicle, and such information is included in the presentation image and presented to the operator 14.

[0032] When the motion model corresponding to the control movement of the control device 12 differs from the motion model of the moving body 8 to be controlled, the image generation unit 65 generates a presentation image including an image simulating another control device (a control device other than the control device 12) corresponding to the motion model of the moving body 8 to be controlled. In this case, the control amount acquisition unit 61 converts the control amount acquired from the control device 12 into a control amount for the other control device and outputs it to the image generation unit 65. For example, when the control mechanisms of the control device 12 are a steering wheel, an accelerator pedal, and a brake pedal, and the controlled object is switched to a four-legged robot, the image display device 13 displays a presentation image including an image (an image of a joystick that moves in accordance with the operation of the steering wheel, accelerator pedal, and brake pedal) that makes it appear as if the controlled object is being controlled with a joystick (a control device corresponding to the motion model of the four-legged robot). In this case, the control amount acquisition unit 61 converts the control amounts of the steering wheel, accelerator pedal, and brake pedal into control amounts that would be obtained if the joystick were being controlled. Specifically, for example, the steering amount by rotational operation of the steering wheel is converted into the steering amount by tilting the joystick left and right, and the steering amount by operation of the accelerator pedal and the brake pedal is converted into the steering amount by tilting the joystick forward and backward. At this time, the steering amount acquisition unit 61 acquires the operation mode set by the setting unit 62. Note that the process of converting the steering amount may be executed by the control amount calculation unit 72 of the mobile body control device 7 instead of the steering amount acquisition unit 61.

[0033] <Configuration of mobile body control device and mobile body> Figure 5 is a block diagram showing an example of the configuration of a mobile body control device 7 and a mobile body 8 according to the first embodiment. The mobile body control device 7 is a device that controls the mobile body 8, and is connected to the mobile body 8 so as to be able to communicate with it. The mobile body control device 7 corresponds to any one of the mobile body control devices 21, 31, or 41 shown in Figure 1. The mobile body 8 corresponds to any one of the first mobile body 22, second mobile body 32, and third mobile body 42 shown in Figure 1, and is an object to be controlled designated by the operator 14.

[0034] The mobile object control device 7 includes a receiving unit 71 , a control amount calculation unit 72 , a control amount adjustment unit 73 , an image acquisition unit 74 , a sensor acquisition unit 75 , and a transmission unit 76 .

[0035] The receiving unit 71 receives the operation amount and operation mode transmitted from the mobile object remote control device 6 via the network 5 .

[0036] The control amount calculation unit 72 calculates a control amount for controlling the moving object 8 based on the operation amount and the operation mode received by the receiving unit 71. Specifically, the control amount calculation unit 72 calculates a control amount based on the operation amount in order to make the moving object 8 imitate an operation corresponding to the operation mode designated by the operator 14. Details of the processing by the control amount calculation unit 72 will be described later.

[0037] The control amount adjustment unit 73 adjusts the control amount calculated by the control amount calculation unit 72 in accordance with the scale of the object to be controlled. The control amount adjustment unit 73 outputs the adjusted control amount (adjusted control amount) to the moving object 8. The control amount adjustment unit 73 may be integrated with the control amount calculation unit 72. In this case, the control amount calculation unit 72 has the function of the control amount adjustment unit 73, and the control amount adjustment unit 73 shown in FIG. 5 is omitted. Here, the scale basically refers to a representative length of the moving object, such as the overall length in the case of an automobile. Alternatively, the scale may refer to the mass, moment of inertia, or the like, in addition to the overall length. For example, even if the control amount received by the receiving unit 71 is the same, the control amount adjustment unit 73 adjusts the control amount so that the speed decreases in proportion to the scale. Furthermore, the control amount adjustment unit 73 may appropriately adjust the scale to make it easier for the operator 14 to operate the moving object.

[0038] The image acquisition unit 74 acquires image data captured by a camera 82 provided on the mobile object 8 as image information of the periphery of the mobile object 8. The sensor acquisition unit 75 acquires sensor data detected by a sensor 83 provided on the mobile object 8 as sensor information of the periphery of the mobile object 8. The transmission unit 76 transmits the image information acquired by the image acquisition unit 74 and the sensor information acquired by the sensor acquisition unit 75 to the mobile object remote control device 6 via the network 5.

[0039] The moving object 8 includes a movement control unit 81, a camera 82, and a sensor 83. When the moving object 8 exists in a virtual space, the camera 82, the sensor 83, and the like are numerically calculated in the virtual space.

[0040] The movement control unit 81 controls an actuator (not shown) that operates the moving body 8 based on the adjusted control amount acquired from the moving body control device 7 .

[0041] The camera 82 captures images of the environment surrounding the mobile object 8 and outputs the captured image data to the mobile object control device 7. Examples of the camera 82 include a black-and-white camera, a color camera, an infrared camera, a stereo camera, and a LiDAR (light detection and ranging) camera. The camera 82 may be any camera that outputs image information that enables the operator 14 to recognize the environment surrounding the mobile object 8. Furthermore, if the camera 82 vibrates significantly due to the movement of the mobile object 8 and interferes with the operator 14's control, the camera 82 may be mounted on a vibration suppression device such as a gimbal stabilizer.

[0042] The sensor 83 detects information relating to the movement of the mobile object 8 , such as the acceleration and speed of the mobile object 8 , and outputs the sensor data that is the detection result to the mobile object control device 7 .

[0043] <Operation Mode> The operator 14 can specify the type of motion model that the operator 14 wants the moving object 8 to use. The motion model specified by the operator 14 is called the "operation mode." For example, if the control mechanism of the control device 12 is composed of a steering wheel, an accelerator pedal, and a brake pedal, and the controlled object is switched from an automobile to a crawler, the motion models of the two are different (as shown in FIG. 3, the automobile and the crawler have different movement mechanisms and therefore different motion models), which causes the operator 14 to feel uncomfortable. In this case, the operator 14's sense of discomfort can be alleviated by having the crawler imitate the movement of the automobile. In this way, even if the motion models are different before and after switching the controlled object, the controlled object can be controlled so that the difference is as small as possible.

[0044] 6 is a diagram showing an example of a combination of operation modes, control mechanisms, and moving bodies according to embodiment 1. The operation modes include, but are not limited to, an automobile mode, a flexible mode, and a touch panel mode. The moving bodies include, but are not limited to, an automobile, a crawler, and a legged robot.

[0045] The automobile mode is an operating mode corresponding to the case where the operating mechanism of the control device 12 is a steering wheel, an accelerator pedal, and a brake pedal (see FIG. 7 ). In the automobile mode, when the operating target is a moving body other than an automobile (a crawler or legged robot in the example of FIG. 6 ), the operation of the automobile is simulated for the operating target (automobile simulation control). For example, the operation of an automobile is simulated for a moving body having a differential two-wheeled mechanism as shown in FIG. 8 . This reduces the sense of discomfort felt by the operator 14. Note that when the operating target is an automobile, the operation of the automobile does not need to be simulated. The determination of whether to simulate the operation of the operating target may be made, for example, by the mobile body control device 7 storing an operating mode (automobile mode, flexible mode, or touch panel mode) corresponding to the mobile body 8 and comparing the operating mode with the operating mode (automobile mode, flexible mode, or touch panel mode) received from the mobile body remote control device 6.

[0046] The free mode is an operating mode corresponding to the case where the control mechanism of the control device 12 is a joystick 15 (see FIG. 9). In the free mode, when the control object is a moving body that cannot rotate in place (a car in the example of FIG. 6), a freely rotatable movement of the control object is simulated (free simulation control). For example, a freely rotatable movement of a car as shown in FIG. 10 is simulated. This can reduce the sense of discomfort felt by the operator 14. Note that when the control object is a crawler or legged robot, it is not necessary to simulate a freely rotatable movement.

[0047] The touch panel mode is an operating mode corresponding to the case where the control mechanism of the control device 12 is a touch panel 16 (see FIG. 11). In the touch panel mode, the operator 14 touches and indicates a target movement point on an image displayed on the touch panel 16 (an image captured by a camera attached to the moving object), thereby moving the moving object to the target movement point (target point movement control). For example, a car as shown in FIG. 12 is moved to a target position (target movement point). This reduces the sense of discomfort felt by the operator 14.

[0048] <Processing of control amount calculation unit 72> In order to move a moving body, it is necessary to set a target angular velocity, a target acceleration, and a target speed (these are called "control amounts") with respect to the body coordinate system of the moving body. Therefore, if the control amount calculation unit 72 calculates the control amount based on the operation mode and the operation amount, it is possible to simulate the operation specified by the operator 14 for various moving bodies.

[0049] Typically, a moving body has a motion model that represents the motion of the moving body and a state equation that specifically expresses the motion model in mathematical terms. For example, in the case of an automobile, there is a two-wheel model as a motion model (see FIG. 13), and there is a state equation based on the two-wheel model. In FIG. 13, δ is the steering angle, V is the speed, vx is the speed in the x-axis direction, ψ is the direction, lf is the distance between the center of gravity and the front wheels, lr is the distance between the center of gravity and the rear wheels, ω is the angular velocity, XY is the reference coordinate system, and xy is the body coordinate system. Since the state equation represents the motion of the moving body, the control amount can be calculated from the state equation.

[0050] 14 is a diagram showing an example of the configuration of the control amount calculation unit 72 when the operation mode is the automobile mode. In the automobile mode, the control amount calculation unit 72 executes the functions of the state equation 721 and the output equation 722.

[0051] Specifically, state equation 721 is set to dx / dt = f(x, u). Here, x is the state, and u is the control input. Furthermore, output equation 722 for extracting the control amount from state equation 721 is set to y = h(x, u). In the case of automobile mode, state equation 721 and output equation 722 are, for example, the following equations (1) to (4). State x is calculated by integrating state equation 721. Note that in equation (3), Cf is the cornering stiffness of the front wheels, Cr is the cornering stiffness of the rear wheels, m is the mass, and I is the moment of inertia.

[0052] By using the target angular velocity ω and the target velocity vx obtained from the state equation 721 and the output equation 722 as control variables, the operation of the automobile can be simulated for the object to be controlled.

[0053] The state equation 721 and the output equation 722 may be set in accordance with the moving body to be simulated (based on the operation mode). The moving body to be simulated may be a moving body other than an automobile, or may be a virtually lightened automobile. In this case, the operation of various moving bodies can be simulated.

[0054] A state equation 721 may be constructed to reproduce a phenomenon specific to an automobile (for example, creep phenomenon).

[0055] 15 is a diagram showing an example of the configuration of the control amount calculation unit 72 when the operation mode is the flexible mode. In the flexible mode, the control amount calculation unit 72 executes the function of the flexible control unit 723.

[0056] For example, when steering a car as the steering target in the free mode, the free control unit 723 calculates control variables (target angular velocity ω, target velocity vx) for simulating the car rotating in place.

[0057] 16 is a diagram showing an example of the configuration of the control amount calculation section 72 when the operation mode is the touch panel mode. In the touch panel mode, the control amount calculation section 72 executes the functions of the control section 724.

[0058] For example, when the operator 14 specifies a target position, the control unit 724 calculates the control amount (target angular velocity ω, target velocity vx) for moving the object to be controlled to the target position.

[0059] In this way, the control amount calculation unit 72 has the functions of "state equation 721 and output equation 722 shown in Figure 14," "flexible control unit 723 shown in Figure 15," and "control unit 724 shown in Figure 16," and executes each function depending on the operating mode.

[0060] <Operation> <Operation of mobile body remote control device 6> The operation of the mobile body remote control device 6 is broadly divided into a transmission operation (FIG. 17) for transmitting a control amount and an operation mode to the mobile body control device 7, and a reception operation (FIG. 18) for receiving image information and sensor information from the mobile body control device 7. These operations will be explained in order below.

[0061] FIG. 17 is a flowchart showing an example of the transmission operation of the mobile object remote control device 6 according to the first embodiment.

[0062] In step S11 , the operation amount acquisition unit 61 acquires an operation amount from the control device 12 .

[0063] In step S12 , the setting unit 62 sets the control target and the operation mode based on the setting information specified by the operator 14 .

[0064] In step S13, the transmission unit 63 transmits the steering amount acquired by the steering amount acquisition unit 61 and the operation mode set by the setting unit 62 to the mobile body control device 7 that controls the steering target (mobile body 8) set by the setting unit 62.

[0065] FIG. 18 is a flowchart showing an example of the receiving operation of the mobile object remote control device 6 according to the first embodiment.

[0066] In step S21, the receiving unit 64 receives image information and sensor information of the surroundings of the target (mobile body 8) from the mobile body control device 7.

[0067] In step S22, the image generating unit 65 generates a presentation image to be presented to the operator based on the image information and sensor information received by the receiving unit 64.

[0068] In step S23, the presentation image generated by the image generation unit 65 is displayed on the image display device 13. At this time, the image generation unit 65 may control the image display device 13 to display the presentation image.

[0069] <Operation of mobile object control device 7> The operation of the mobile object control device 7 is roughly divided into a receiving operation (FIG. 19) for receiving an operation amount and an operation mode from the mobile object remote control device 6, and a transmitting operation (FIG. 20) for transmitting image information and sensor information to the mobile object remote control device 6. These operations will be described in order below.

[0070] FIG. 19 is a flowchart showing an example of the receiving operation of the mobile object control device 7 according to the first embodiment.

[0071] In step S31, the receiving unit 71 receives the operation amount and operation mode transmitted from the mobile object remote control device 6.

[0072] In step S32, the control amount calculation unit 72 calculates a control amount for making the moving object 8 imitate an operation corresponding to the operation mode, based on the operation amount and the operation mode received by the receiving unit 71.

[0073] In step S33, the control amount adjuster 73 adjusts the control amount calculated by the control amount calculator 72 in accordance with the scale of the object to be controlled.

[0074] In step S34 , the control amount adjuster 73 outputs the adjusted control amount to the moving object 8 .

[0075] FIG. 20 is a flowchart showing an example of the transmission operation of the mobile object control device 7 according to the first embodiment.

[0076] In step S41, the image acquisition unit 74 acquires image information of the surroundings of the moving object 8 from the camera 82 provided on the moving object 8.

[0077] In step S42 , the sensor acquisition unit 75 acquires sensor information of the surroundings of the mobile object 8 from the sensor 83 provided in the mobile object 8 .

[0078] In step S43 , the transmitting unit 76 transmits the image information acquired by the image acquiring unit 74 and the sensor information acquired by the sensor acquiring unit 75 to the mobile object remote control device 6 .

[0079] <Effects> The mobile body remote control device 6 transmits the control amount used by the operator 14 to control the control device 12 and the operation mode specified by the operator 14 to the mobile body control device 7, which controls the control target specified by the operator 14. The mobile body control device 7 calculates a control amount for controlling the operation of the mobile body 8 based on the control amount and operation mode received from the mobile body remote control device 6, and outputs the control amount to the mobile body 8. This enables the operator 14 to control multiple mobile bodies that operate using different motion models. Furthermore, even if the control mechanism of the control device 12 is not the original control mechanism for controlling the mobile body 8, the mobile body 8 is operated with a control amount corresponding to the operation mode specified by the operator 14, thereby reducing the sense of discomfort felt by the operator 14.

[0080] <Embodiment 2> Fig. 21 is a block diagram showing an example of the configuration of a mobile object remote control device 6 according to embodiment 2. Fig. 22 is a block diagram showing an example of the configuration of a mobile object control device 7 according to embodiment 2.

[0081] As shown in Fig. 21 , the mobile object remote control device 6 according to the second embodiment has a configuration in which the control amount calculation unit 72 and the control amount adjustment unit 73 of the mobile object control device 7 according to the first embodiment (see Fig. 5) are added to the mobile object remote control device 6 according to the first embodiment (see Fig. 4). In the mobile object remote control device 6 according to the second embodiment, the transmission unit 63 transmits the control amount adjusted by the control amount adjustment unit 73 (adjusted control amount) to the mobile object control device 7. Other operations are the same as those of the mobile object remote control device 6 according to the first embodiment.

[0082] As shown in Fig. 22 , the mobile body control device 7 according to the second embodiment has a configuration in which the control amount calculation unit 72 and the control amount adjustment unit 73 are removed from the mobile body control device 7 according to the first embodiment (see Fig. 5 ). In the mobile body control device 7 according to the second embodiment, the receiving unit 71 receives the adjusted control amount transmitted from the mobile body remote control device 6 and outputs the adjusted control amount to the mobile body 8. Other operations are the same as those of the mobile body control device 7 according to the first embodiment.

[0083] As described above, according to the second embodiment, the configuration of the mobile object control device 7 can be simplified.

[0084] <Hardware Configuration> The functions of the control amount acquisition unit 61, setting unit 62, transmission unit 63, reception unit 64, and image generation unit 65 in the mobile object remote control device 6 described in embodiment 1 are realized by processing circuits. That is, the mobile object remote control device 6 includes a processing circuit for acquiring control amounts, setting a control target and an operation mode, transmitting the control target and the operation mode, receiving image information and sensor information about the surroundings of the control target, and generating a presentation image based on the image information and sensor information. The processing circuit may be dedicated hardware, or may be a processor (also referred to as a CPU, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor)) that executes a program stored in memory.

[0085] 23 , the processing circuit 91 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the steering amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65 may be realized individually by the processing circuit 91, or these functions may be realized collectively by a single processing circuit 91.

[0086] When the processing circuit 91 is the processor 92 shown in FIG. 24 , the functions of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 93. The processor 92 realizes each function by reading and executing the program recorded in the memory 93. That is, the mobile object remote control device 6 includes the memory 93 for storing a program that ultimately executes the steps of acquiring a control amount, setting a control target and an operation mode, transmitting the control target and the operation mode, receiving image information and sensor information about the surroundings of the control target, and generating a presentation image based on the image information and sensor information. It can also be said that these programs cause a computer to execute the procedures or methods of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65. Here, the memory may be, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a DVD (Digital Versatile Disc), or any storage medium that will be used in the future.

[0087] It should be noted that with regard to the functions of the steering amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65, some functions may be realized by dedicated hardware, and other functions may be realized by software or firmware.

[0088] Thus, the processing circuitry can implement each of the above-described functions through hardware, software, firmware, or a combination thereof.

[0089] The above describes the hardware configuration of the mobile body remote control device 6 shown in Figure 4, but the hardware configurations of the mobile body control device 7 shown in Figure 5, the mobile body remote control device 6 shown in Figure 21, and the mobile body control device 7 shown in Figure 22 are also similar.

[0090] Within the scope of the present disclosure, the embodiments can be freely combined, modified, or omitted as appropriate.

[0091] Although the present disclosure has been described in detail, the above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned.

[0092] 1 Control system, 2 First mobile body system, 3 Second mobile body system, 4 Third mobile body system, 5 Network, 6 Mobile body remote control device, 7 Mobile body control device, 8 Mobile body, 11 Mobile body remote control device, 12 Control device, 13 Image display device, 14 Operator, 15 Joystick, 16 Touch panel, 21 Mobile body control device, 22 First mobile body, 31 Mobile body control device, 32 Second mobile body, 41 Mobile body control device, 42 Third mobile body, 61 Control amount acquisition unit, 62 Setting unit, 63 Transmitter, 64 Receiver, 65 Image generation unit, 71 Receiver, 72 Control amount calculation unit, 73 Control amount adjustment unit, 74 Image acquisition unit, 75 Sensor acquisition unit, 76 Transmitter, 81 Movement control unit, 82 Camera, 83 Sensor, 91 Processing circuit, 92 Processor, 93 Memory, 721 State equation, 722 Output equation, 723 variable control, 724 control.

Claims

1. a control amount acquisition unit that acquires a control amount from a control device operated by an operator; a setting unit that sets an object to be controlled from among a plurality of moving objects that operate according to different motion models, and sets an operation mode that is a motion model for operating the object to be controlled designated by the operator; a control amount calculation unit that calculates a control amount for controlling the object to be controlled, which is set by the setting unit, based on the operation amount acquired by the operation amount acquisition unit and the operation mode set by the setting unit; A mobile object remote control system comprising:

2. A control amount acquisition unit that acquires a control amount from a control device operated by an operator; A setting unit that sets a target to be controlled from among a plurality of moving objects that operate according to different motion models; a control amount calculation unit that calculates a control amount for controlling the object to be controlled, which is set by the setting unit, based on the control amount acquired by the control amount acquisition unit; a control amount adjustment unit that adjusts the control amount in accordance with a scale of the object to be controlled; A mobile object remote control system comprising:

3. a control amount acquisition unit that acquires a control amount from a control device operated by an operator; a setting unit that sets an object to be controlled from among a plurality of moving objects that operate according to different motion models, and sets an operation mode that is a motion model for operating the object to be controlled designated by the operator; a transmission unit that transmits the steering amount acquired by the steering amount acquisition unit and the operation mode set by the setting unit to a mobile object control device that controls the object to be controlled; A mobile object remote control device comprising:

4. A receiving unit that receives image information and sensor information of the surroundings of the target from the mobile object control device; an image generating unit that generates a presentation image to be presented to the operator based on the image information and the sensor information received by the receiving unit; The mobile remote control device of claim 3 further comprising:

5. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects the presented image so as to eliminate a time delay associated with control of the object to be controlled.

6. The mobile object remote control device according to claim 4, wherein, when the operator is performing a rotation operation, the image generation unit corrects the presented image so that a change in viewpoint accompanying the movement of the object to be controlled makes the object to rotate on the spot.

7. a control amount calculation unit that calculates a control amount for controlling the object to be controlled based on the control amount acquired by the control amount acquisition unit; a control amount adjustment unit that adjusts the control amount in accordance with a scale of the object to be controlled; The mobile remote control device of claim 3 further comprising:

8. The mobile object remote control device according to claim 3 , wherein the setting unit automatically selects the operation mode when the control target designated by the operator is set.

9. The mobile object remote control device according to claim 3 , wherein the mobile object exists in a virtual space.

10. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects an angle of view and a size of the presented image so that a viewpoint accompanying the movement of each of the controlled objects becomes the same.

11. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects distortion of the presented image caused by roll motion, pitch motion, and up-down motion of the controlled object.

12. the receiving unit receives, from the mobile object control device, a control amount for controlling the object to be controlled, the control amount being calculated based on the control amount; The mobile object remote control device according to claim 4 , wherein the image generating unit generates the presentation image including the control amount.

13. The mobile object remote control device of claim 4, wherein when a motion model corresponding to the control movement of the control device is different from a motion model of the object to be controlled, the image generation unit generates the presentation image including an image simulating another control device corresponding to the motion model of the object to be controlled.

14. The mobile object remote control device according to claim 13 , wherein the control amount acquisition unit converts the control amount acquired from the control device into a control amount for the other control device.

15. A control amount calculation unit that calculates a control amount for controlling a target object set from a plurality of moving objects operating with different motion models based on a control amount obtained from a control device operated by an operator; a control amount adjustment unit that adjusts the control amount in accordance with a scale of the object to be controlled; A mobile control device comprising:

16. a receiving unit for receiving the operation amount transmitted from the mobile body remote control device according to claim 3; a control amount calculation unit that calculates a control amount for controlling the object to be controlled based on the control amount received by the receiving unit; A mobile control device comprising:

17. an image acquisition unit that acquires image information of the surroundings of the target from a camera provided on the target; a sensor acquisition unit that acquires sensor information related to a movement of the target object from a sensor provided in the target object; a transmission unit that transmits the image information acquired by the image acquisition unit and the sensor information acquired by the sensor acquisition unit to a mobile object remote control device that outputs the control amount for at least the operator to control the control target; The mobile object control device according to claim 15 or 16, further comprising:

18. A mobile body control device as described in Claim 15, wherein the control quantity calculation unit calculates the control quantity that causes the target to imitate an operation corresponding to an operation mode, which is a motion model for moving the target specified by the operator.

19. The mobile object control device according to claim 16 , further comprising a control amount adjustment unit that adjusts the control amount calculated by the control amount calculation unit in accordance with a scale of the object to be controlled.