Mobile body remote control system, mobile body remote control device, and mobile body control device

Abstract

The purpose of the present disclosure is to provide a mobile body remote operation system, a mobile body remote operation device, and a mobile body control device that make it possible to operate a plurality of mobile bodies moving by different motion models. A mobile body remote operation system according to the present disclosure comprises: an operation amount acquisition unit that acquires an operation amount from an operation device controlled by an operator; a setting unit that sets an object to be operated among a plurality of mobile bodies moving by different motion models; and a control amount calculation unit that uses the operation amount acquired by the operation amount acquisition unit as a basis to calculate a control amount for controlling the object to be operated set by the setting unit.

Description

[Technical Field]

[0001] The present disclosure relates to a mobile object remote control system, a mobile object remote control device, and a mobile object control device for controlling multiple mobile objects located in remote locations. [Background technology]

[0002] In recent years, with the development of communication technologies such as the Internet, mobile object remote control systems that enable operators to control mobile objects located in remote locations have been developed. In particular, mobile object remote control systems that aim to reduce labor by allowing one operator to monitor and control multiple mobile objects have been developed (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6319507 Summary of the Invention [Problem to be solved by the invention]

[0004] When controlling a moving object in various situations, it is possible to increase the number of movement mechanisms that each moving object has and switch between them depending on the situation, but this increases the cost of each moving object. Therefore, it is desirable to arrange and control multiple moving objects that operate using different motion models depending on the situation.

[0005] In Patent Document 1, multiple automobiles that operate according to the same motion model are the targets of control, and multiple moving objects that operate according to different motion models cannot be controlled.

[0006] The present disclosure has been made to solve such problems, and aims to provide a mobile object remote control system, a mobile object remote control device, and a mobile object control device that are capable of controlling multiple mobile objects that operate using different motion models. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the mobile object remote control system according to the present disclosure includes a control amount acquisition unit that acquires a control amount from a control device operated by an operator, and a control amount acquisition unit that sets a target to be controlled from among a plurality of mobile objects that operate with different motion models. and sets the operation mode, which is the motion model that moves the controlled object specified by the operator. a setting unit for setting the steering amount acquired by the steering amount acquisition unit; , and the operation mode set by the setting unit and a control amount calculation unit that calculates a control amount for controlling the object to be controlled set by the setting unit based on the control amount. [Effects of the Invention]

[0008] According to the present disclosure, it becomes possible to control multiple moving objects that operate using different motion models.

[0009] The objects, features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a mobile object remote control system according to a first embodiment. [Figure 2] FIG. 2 is a diagram for explaining a moving body according to the first embodiment. [Figure 3] FIG. 2 is a diagram for explaining a moving body according to the first embodiment. [Figure 4] 1 is a block diagram showing an example of the configuration of a mobile object remote control device according to a first embodiment. [Figure 5] 1 is a block diagram showing an example of the configuration of a mobile object control device and a mobile object according to a first embodiment. [Figure 6] 3 is a diagram showing an example of a combination of an operation mode, a control mechanism, and a moving body according to the first embodiment. FIG. [Figure 7] FIG. 4 is a diagram for explaining an automobile mode according to the first embodiment. [Figure 8]FIG. 4 is a diagram showing an example of the movement of the two differential wheels in the automobile mode according to the first embodiment. [Figure 9] FIG. 3 is a diagram for explaining a flexible mode according to the first embodiment. [Figure 10] 5A and 5B are diagrams illustrating an example of the movement of a car in a free mode according to the first embodiment. [Figure 11] FIG. 2 is a diagram for explaining a touch panel mode according to the first embodiment. [Figure 12] FIG. 4 is a diagram showing an example of the movement of a car in a touch panel mode according to the first embodiment. [Figure 13] FIG. 2 is a diagram showing an example of a motion model of an automobile according to the first embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of the configuration of a control amount calculation unit according to the first embodiment. [Figure 15] FIG. 2 is a diagram illustrating an example of the configuration of a control amount calculation unit according to the first embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of the configuration of a control amount calculation unit according to the first embodiment. [Figure 17] 4 is a flowchart showing an example of the operation of the moving object remote control device according to the first embodiment. [Figure 18] 4 is a flowchart showing an example of the operation of the moving object remote control device according to the first embodiment. [Figure 19] 5 is a flowchart showing an example of the operation of the mobile object control device according to the first embodiment. [Figure 20] 5 is a flowchart showing an example of the operation of the mobile object control device according to the first embodiment. [Figure 21] FIG. 10 is a block diagram showing an example of the configuration of a mobile object remote control device according to a second embodiment. [Figure 22] FIG. 10 is a block diagram showing an example of the configuration of a mobile object control device according to a second embodiment. [Figure 23] 1 is a block diagram showing an example of a hardware configuration of a mobile object remote control device according to first and second embodiments. [Figure 24]1 is a block diagram showing an example of a hardware configuration of a mobile object remote control device according to first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] <First Embodiment> <Configuration of a mobile remote control system> 1 is a block diagram showing an example of the configuration of a mobile object remote control system according to embodiment 1. The mobile object remote control system is made up of a control system 1, a first mobile object system 2, a second mobile object system 3, a third mobile object system 4, and a network 5. The control system 1 is communicably connected to each of the first mobile object system 2, the second mobile object system 3, and the third mobile object system 4 via the network 5. The first mobile object 22, the second mobile object 32, and the third mobile object 42 each operate using a different motion model.

[0012] Although FIG. 1 illustrates three mobile body systems (first mobile body system 2, second mobile body system 3, and third mobile body system 4), the number of mobile body systems may be two or four or more. Furthermore, mobile bodies other than the first mobile body 22, second mobile body 32, and third mobile body 42 may be the control targets. Although the example in FIG. 1 illustrates three mobile bodies that operate using three different motion models, the present invention is not limited to this. Three (or two or four or more) mobile bodies that operate using at least two different motion models may be the control targets. In other words, the first mobile body 22 and the second mobile body 32 that operate using the same motion model and the third mobile body 42 that operates using a different motion model from the first mobile body 22 and the second mobile body 32 may be the control targets.

[0013] The control system 1 is composed of a mobile object remote control device 11, a control device 12, and an image display device 13. An operator 14 views an image displayed on the image display device 13 and operates the control device 12 to control a mobile object to be controlled (any one of the first mobile object 22, the second mobile object 32, and the third mobile object 42). The mobile object remote control device 11 transmits the amount of movement (control amount) when the operator 14 operates the control device 12 to control the mobile object to a mobile object control device (any one of the mobile object control devices 21, 31, and 41) that controls the operation of the mobile object to be controlled. The mobile object remote control device 11 also receives information from a camera and a sensor provided on the mobile object from the mobile object control device that controls the operation of the mobile object to be controlled, and generates an image to be displayed on the image display device 13 based on this information.

[0014] The first mobile body system 2 is composed of a mobile body control device 21 and a first mobile body 22. The mobile body control device 21 controls the operation of the first mobile body 22 based on the operation amount received from the mobile body remote control device 11. The mobile body control device 21 also transmits information from a camera and a sensor provided on the first mobile body 22 to the mobile body remote control device 11. The first mobile body 22 is a mobile body having wheels as a movement mechanism, such as an automobile.

[0015] The second mobile body system 3 is composed of a mobile body control device 31 and a second mobile body 32. The configuration and operation of the mobile body control device 31 are similar to those of the mobile body control device 21. The second mobile body 32 is a mobile body having an infinite mechanism as a movement mechanism, such as a crawler.

[0016] The third moving body system 4 is composed of a moving body control device 41 and a third moving body 42. The configuration and operation of the moving body control device 41 are similar to those of the moving body control device 21. The third moving body 42 is a moving body having legs as a movement mechanism, such as a legged robot.

[0017] FIG. 2 is a diagram illustrating a moving body according to the first embodiment. A moving body is an object that can change the position and posture of its own body coordinate system relative to a reference coordinate system by using its own moving mechanism. In FIG. 2, the XY coordinate system is the reference coordinate system, and the xy coordinate system is the body coordinate system. Note that FIG. 2 illustrates a moving body having two differential wheels as the moving mechanism.

[0018] As shown in FIG. 3, the movable area of ​​a moving body varies depending on the moving mechanism that the moving body possesses. For each moving body shown in FIG. 3, the motion model (how it moves and with what operations) varies depending on the moving mechanism. For example, in the case of a car, the position and attitude of the car can be controlled by operating the steering wheel, accelerator pedal, and brake pedal. This can be considered the "motion model" of the car.

[0019] The moving object is not limited to an actual moving object, but may be a moving object existing in a virtual space (simulation space). The virtual space may be, for example, a driving simulator. The moving object existing in the virtual space corresponds to a computer (server) incorporating a physical simulator. In this case, an image corresponding to the viewpoint of the moving object in the physical simulator is displayed on the image display device 13. This allows the operator to become familiar with operating each moving object in the virtual space.

[0020] <Configuration of a mobile remote control device> 4 is a block diagram showing an example of the configuration of a mobile object remote control device 6 according to embodiment 1. The mobile object remote control device 6 includes a control amount acquisition unit 61, a setting unit 62, a transmission unit 63, a reception unit 64, and an image generation unit 65. The mobile object remote control device 6 is also connected to a control device 12 and an image display device 13. The mobile object remote control device 6 corresponds to the mobile object remote control device 11 shown in FIG. 1.

[0021] The control device 12 is equipped with control mechanisms such as a steering wheel, accelerator pedal, and brake pedal, a joystick, a mouse and keyboard, or a tablet. The control device 12 may be equipped with one or more of these control mechanisms. When the control device 12 is equipped with multiple control mechanisms, the operator 14 (see FIG. 1, the same applies below) actually operates one of the control mechanisms.

[0022] The image display device 13 is, for example, a display or a head-mounted display.

[0023] The control amount acquisition unit 61 acquires the control amount from the control device 12 operated by the operator 14. Specifically, the control amount acquisition unit 61 decodes the control amount acquired from the control device 12.

[0024] The setting unit 62 sets the control target and the operation mode based on the setting information specified by the operator 14. The control target is a moving object (moving object 8 shown in FIG. 5, which will be described later) specified by the operator 14 among a plurality of moving objects that operate according to different motion models. There may be a plurality of control targets. The operation mode is a motion model that causes the control target specified by the operator 14 to operate. Details of the operation mode will be described later. The operator 14 may specify the setting information using the control device 12. Furthermore, the setting unit 62 may automatically select the operation mode when setting the control target specified by the operator 14.

[0025] The transmitter 63 transmits the control amount acquired by the control amount acquisition unit 61 and the operation mode set by the setting unit 62 to a mobile body control device (a mobile body control device 7 shown in FIG. 5 , described later) that controls the control target set by the setting unit 62 via the network 5. The transmitter 63 has a function of switching the mobile body control device that is the destination (communication partner) of the control amount and operation mode based on the control target set by the setting unit 62. If the control target is multiple mobile bodies, the transmitter 63 may transmit the control amount and operation mode simultaneously to the mobile body control devices that control these mobile bodies. In this case, these mobile bodies will move in the same way in response to the control by the operator 14.

[0026] The receiving unit 64 receives image information and sensor information of the surroundings of the control target (a mobile object 8 shown in FIG. 5, which will be described later) from a mobile object control device (a mobile object control device 7 shown in FIG. 5, which will be described later) via the network. The receiving unit 64 has a function of switching the mobile object control device, which is the sender (communication partner) of the image information and sensor information, based on the control target set by the setting unit 62.

[0027] The image generating unit 65 performs image processing based on the image information and sensor information received by the receiving unit 64, and generates a presentation image to be presented to the operator.

[0028] The image generation unit 65 may correct the angle of view and size of the presented image so that the viewpoint is the same as each moving object 8 moves. This can mitigate differences in appearance between the moving objects 8. For example, when the operator 14 is operating the object to be controlled by rotating it, the image generation unit 65 corrects the presented image so that the viewpoint changes as the object to be controlled moves, making it appear as if the object to be controlled is rotating on the spot.

[0029] The image generating unit 65 may correct distortion of the presented image due to the roll motion, pitch motion, and up and down motion of the moving body 8 (for example, a legged robot).

[0030] The image generation unit 65 corrects the presented image so that there is no time delay associated with controlling the object to be controlled. Here, the "time delay associated with controlling the moving object" includes not only the delay caused by the inability of the vehicle to rotate in place in response to the rotation operation of the joystick 15 as shown in Figures 9 and 10 (described later), but also a communication delay between the moving object remote control device 6 and the moving object control device 7.

[0031] The image generation unit 65 may generate a presentation image including the control amount calculated by the control amount calculation unit 72 (see FIG. 5 described later). In this case, the receiving unit 64 receives the control amount calculated by the control amount calculation unit 72 from the mobile object control device 7 (see FIG. 5 described later). For example, when the operation mode is set to the free mode (see FIG. 6, details will be described later) and the automobile is rotated on the spot, the angle and time that can be rotated change depending on the presence or absence of obstacles around the automobile, and such information is included in the presentation image and presented to the operator 14.

[0032] When the motion model corresponding to the control movement of the control device 12 differs from the motion model of the moving object 8 to be controlled, the image generation unit 65 generates a presentation image including an image simulating another control device (a control device other than the control device 12) corresponding to the motion model of the moving object 8 to be controlled. In this case, the control amount acquisition unit 61 converts the control amount acquired from the control device 12 into a control amount for the other control device and outputs it to the image generation unit 65. For example, when the control mechanisms of the control device 12 are a steering wheel, an accelerator pedal, and a brake pedal, and the controlled object is switched to a four-legged robot, the image display device 13 displays a presentation image including an image (an image of a joystick that moves in accordance with the operation of the steering wheel, accelerator pedal, and brake pedal) that makes it appear as if the controlled object is being controlled with a joystick (a control device corresponding to the motion model of a four-legged robot). In this case, the control amount acquisition unit 61 converts the control amounts of the steering wheel, accelerator pedal, and brake pedal into control amounts that would be obtained if a joystick were being controlled. Specifically, for example, the steering amount by rotational operation of the steering wheel is converted into the steering amount by tilting the joystick left and right, and the steering amount by operation of the accelerator pedal and brake pedal is converted into the steering amount by tilting the joystick forward and backward. At this time, the steering amount acquisition unit 61 acquires the operation mode set by the setting unit 62. Note that the process of converting the steering amount may be executed by the control amount calculation unit 72 of the mobile object control device 7 instead of the steering amount acquisition unit 61.

[0033] <Configuration of mobile body control device and mobile body> 5 is a block diagram showing an example of the configuration of a mobile body control device 7 and a mobile body 8 according to the first embodiment. The mobile body control device 7 is a device that controls the mobile body 8, and is connected to the mobile body 8 so as to be able to communicate with the mobile body 8. The mobile body control device 7 corresponds to any one of the mobile body control devices 21, 31, and 41 shown in FIG. 1. The mobile body 8 corresponds to any one of the first mobile body 22, second mobile body 32, and third mobile body 42 shown in FIG. 1, and is an object to be controlled designated by the operator 14.

[0034] The mobile object control device 7 includes a receiving unit 71, a control amount calculation unit 72, a control amount adjustment unit 73, an image acquisition unit 74, a sensor acquisition unit 75, and a transmission unit 76.

[0035] The receiving unit 71 receives the operation amount and operation mode transmitted from the mobile object remote control device 6 via the network 5.

[0036] The control amount calculation unit 72 calculates a control amount for controlling the moving object 8 based on the control amount and operation mode received by the receiving unit 71. Specifically, the control amount calculation unit 72 calculates the control amount based on the control amount in order to make the moving object 8 imitate an operation according to the operation mode designated by the operator 14. Details of the processing by the control amount calculation unit 72 will be described later.

[0037] The control amount adjustment unit 73 adjusts the control amount calculated by the control amount calculation unit 72 in accordance with the scale of the object to be controlled. The control amount adjustment unit 73 outputs the adjusted control amount (adjusted control amount) to the moving object 8. The control amount adjustment unit 73 may be configured integrally with the control amount calculation unit 72. In this case, the control amount calculation unit 72 has the function of the control amount adjustment unit 73, and the control amount adjustment unit 73 shown in FIG. 5 is omitted. Here, the scale basically refers to a representative length of the moving object, such as the overall length in the case of an automobile. In addition to the overall length, the scale may also refer to the mass, moment of inertia, etc. For example, even if the control amount received by the receiving unit 71 is the same, the control amount adjustment unit 73 adjusts the control amount so that the speed decreases in proportion to the scale. Furthermore, the control amount adjustment unit 73 may appropriately adjust the scale to make it easier for the operator 14 to operate the moving object.

[0038] The image acquisition unit 74 acquires image data captured by a camera 82 provided on the mobile object 8 as image information of the periphery of the mobile object 8. The sensor acquisition unit 75 acquires sensor data detected by a sensor 83 provided on the mobile object 8 as sensor information of the periphery of the mobile object 8. The transmission unit 76 transmits the image information acquired by the image acquisition unit 74 and the sensor information acquired by the sensor acquisition unit 75 to the mobile object remote control device 6 via the network 5.

[0039] The moving object 8 includes a movement control unit 81, a camera 82, and a sensor 83. When the moving object 8 exists in a virtual space, the camera 82 and the sensor 83 are calculated numerically in the virtual space.

[0040] The movement control unit 81 controls an actuator (not shown) that operates the moving body 8 based on the adjusted control amount acquired from the moving body control device 7.

[0041] The camera 82 captures images of the environment surrounding the moving object 8 and outputs the captured image data to the moving object control device 7. Examples of the camera 82 include a black-and-white camera, a color camera, an infrared camera, a stereo camera, and a LiDAR (light detection and ranging) camera. The camera 82 may be any camera that outputs image information that enables the operator 14 to recognize the environment surrounding the moving object 8. Furthermore, if the camera 82 vibrates significantly due to the movement of the moving object 8 and interferes with the operator 14's control, the camera 82 may be mounted on a vibration suppression device such as a gimbal stabilizer.

[0042] The sensor 83 detects information relating to the movement of the moving body 8, such as the acceleration and speed of the moving body 8, and outputs sensor data representing the detection results to the moving body control device .

[0043] <Operation mode> The operator 14 can specify the type of motion model that the operator 14 wants the moving object 8 to operate under. The motion model specified by the operator 14 is called an "operation mode." For example, if the control mechanism of the control device 12 is composed of a steering wheel, an accelerator pedal, and a brake pedal, and the controlled object is switched from an automobile to a crawler, the motion models of the two are different (as shown in FIG. 3, an automobile and a crawler have different movement mechanisms and therefore different motion models), which causes the operator 14 to feel uncomfortable. In this case, the operator 14's sense of discomfort can be alleviated by having the crawler imitate the movement of the automobile. In this way, even if the motion models are different before and after switching the controlled object, the controlled object can be controlled so that the difference is as small as possible.

[0044] 6 is a diagram showing an example of a combination of operation modes, control mechanisms, and moving bodies according to embodiment 1. The operation modes include, but are not limited to, an automobile mode, a flexible mode, and a touch panel mode. The moving bodies include, but are not limited to, an automobile, a crawler, and a legged robot.

[0045] The automobile mode is an operation mode corresponding to the case where the operation mechanism of the control device 12 is a steering wheel, an accelerator pedal, and a brake pedal (see FIG. 7). In the automobile mode, when the control target is a moving body other than an automobile (a crawler or legged robot in the example of FIG. 6), the operation of the automobile is simulated for the controlled object (automobile simulation control). For example, the operation of the automobile is simulated for a moving body having a differential two-wheeled mechanism as shown in FIG. 8. This reduces the sense of discomfort felt by the operator 14. Note that when the control target is an automobile, it is not necessary to simulate the operation of the automobile. The determination of whether to simulate the operation of the controlled object may be made, for example, by the moving body control device 7 storing the operation mode (automobile mode, flexible mode, or touch panel mode) corresponding to the moving body 8 and comparing the operation mode with the operation mode (automobile mode, flexible mode, or touch panel mode) received from the moving body remote control device 6.

[0046] The free mode is an operating mode corresponding to the case where the control mechanism of the control device 12 is a joystick 15 (see FIG. 9). In the free mode, when the control object is a moving body that cannot rotate in place (a car in the example of FIG. 6), a freely rotatable movement of the control object is simulated (free simulation control). For example, a freely rotatable movement of a car as shown in FIG. 10 is simulated. This can reduce the sense of discomfort felt by the operator 14. Note that when the control object is a crawler or leg robot, it is not necessary to simulate a freely rotatable movement.

[0047] The touch panel mode is an operation mode corresponding to the case where the control mechanism of the control device 12 is a touch panel 16 (see FIG. 11). In the touch panel mode, the operator 14 touches and indicates a target movement point on an image (an image captured by a camera attached to the moving object) displayed on the touch panel 16, thereby moving the moving object to the target movement point (target point movement control). For example, a car as shown in FIG. 12 is moved to a target position (target movement point). This can reduce the sense of discomfort felt by the operator 14.

[0048] <Processing of the control amount calculation unit 72> In order to move a moving object, it is necessary to set a target angular velocity, a target acceleration, and a target velocity (these are called "control variables") relative to the body coordinate system of the moving object. Therefore, if the control variable calculation unit 72 calculates the control variable based on the operation mode and the operation variable, it is possible to simulate the operation specified by the operator 14 for various moving objects.

[0049] Typically, a moving body has a motion model that represents the motion of the moving body and a state equation that specifically expresses the motion model in mathematical form. For example, in the case of an automobile, there is a two-wheel model as a motion model (see FIG. 13), and there is a state equation based on the two-wheel model. In FIG. 13, δ is the steering angle, V is the speed, vx is the speed in the x-axis direction, ψ is the direction, lf is the distance between the center of gravity and the front wheels, lr is the distance between the center of gravity and the rear wheels, ω is the angular velocity, XY is the reference coordinate system, and xy is the body coordinate system. Since the state equation represents the motion of the moving body, the control amount can be calculated from the state equation.

[0050] 14 is a diagram showing an example of the configuration of the control amount calculation unit 72 when the operation mode is the automobile mode. In the automobile mode, the control amount calculation unit 72 executes the functions of the state equation 721 and the output equation 722.

[0051] Specifically, state equation 721 is set to dx / dt=f(x,u). Here, x is the state, and u is the control input. Furthermore, output equation 722 for extracting the control amount from state equation 721 is set to y=h(x,u). In the case of automobile mode, state equation 721 and output equation 722 are, for example, the following equations (1) to (4). State x is calculated by integrating state equation 721. Note that in equation (3), Cf is the cornering stiffness of the front wheels, Cr is the cornering stiffness of the rear wheels, m is the mass, and I is the moment of inertia.

number

number

number

number

[0052] By using the target angular velocity ω and the target velocity vx obtained from the state equation 721 and the output equation 722 as control variables, the operation of the automobile can be simulated for the controlled object.

[0053] The state equation 721 and the output equation 722 may be set in accordance with the moving body to be simulated (based on the operation mode). The moving body to be simulated may be a moving body other than an automobile, or may be a virtually lightened automobile. In this case, the operation of various moving bodies can be simulated.

[0054] A state equation 721 may be constructed to reproduce a phenomenon specific to an automobile (for example, creep phenomenon).

[0055] 15 is a diagram showing an example of the configuration of the control amount calculation unit 72 when the operation mode is the flexible mode. In the flexible mode, the control amount calculation unit 72 executes the function of the flexible control unit 723.

[0056] For example, when steering a car that is the steering object in the free mode, the free control unit 723 calculates the control amount (target angular velocity ω, target velocity vx) for simulating the operation of the car turning on the spot.

[0057] 16 is a diagram showing an example of the configuration of the control amount calculation section 72 when the operation mode is the touch panel mode. In the touch panel mode, the control amount calculation section 72 executes the functions of the control section 724.

[0058] For example, when the operator 14 specifies a target position, the control unit 724 calculates the control amount (target angular velocity ω, target velocity vx) for moving the object to be controlled to the target position.

[0059] In this way, the control amount calculation unit 72 has the functions of "state equation 721 and output equation 722 shown in Figure 14," "flexible control unit 723 shown in Figure 15," and "control unit 724 shown in Figure 16," and executes each function depending on the operating mode.

[0060] <Operation> <Operation of the mobile object remote control device 6> The operation of the mobile object remote control device 6 is roughly divided into a transmission operation (FIG. 17) for transmitting the operation amount and operation mode to the mobile object control device 7, and a reception operation (FIG. 18) for receiving image information and sensor information from the mobile object control device 7. These operations will be explained below in order.

[0061] FIG. 17 is a flowchart showing an example of the transmission operation of the mobile object remote control device 6 according to the first embodiment.

[0062] In step S11 , the control amount acquisition unit 61 acquires a control amount from the control device 12 .

[0063] In step S12, the setting unit 62 sets the controlled object and the operation mode based on the setting information specified by the operator 14.

[0064] In step S13, the transmission unit 63 transmits the control amount acquired by the control amount acquisition unit 61 and the operation mode set by the setting unit 62 to the mobile body control device 7 that controls the control target (mobile body 8) set by the setting unit 62.

[0065] FIG. 18 is a flowchart showing an example of the receiving operation of the mobile object remote control device 6 according to the first embodiment.

[0066] In step S21, the receiving unit 64 receives image information and sensor information of the surroundings of the control target (moving object 8) from the moving object control device 7.

[0067] In step S22, the image generating unit 65 generates a presentation image to be presented to the operator based on the image information and sensor information received by the receiving unit 64.

[0068] In step S23, the presentation image generated by the image generation unit 65 is displayed on the image display device 13. At this time, the image generation unit 65 may control the image display device 13 so as to display the presentation image.

[0069] <Operation of the mobile object control device 7> The operation of the mobile object control device 7 is roughly divided into a receiving operation (FIG. 19) for receiving the operation amount and operation mode from the mobile object remote control device 6, and a transmitting operation (FIG. 20) for transmitting image information and sensor information to the mobile object remote control device 6. These operations will be explained below in order.

[0070] FIG. 19 is a flowchart showing an example of the receiving operation of the mobile object control device 7 according to the first embodiment.

[0071] In step S31, the receiving unit 71 receives the operation amount and operation mode transmitted from the mobile object remote control device 6.

[0072] In step S32, the control amount calculation unit 72 calculates a control amount for making the moving object 8 imitate an operation according to the operation mode, based on the operation amount and the operation mode received by the receiving unit 71.

[0073] In step S33, the control amount adjuster 73 adjusts the control amount calculated by the control amount calculator 72 in accordance with the scale of the object to be controlled.

[0074] In step S34, the control amount adjuster 73 outputs the adjusted control amount to the moving object 8.

[0075] FIG. 20 is a flowchart showing an example of the transmission operation of the mobile object control device 7 according to the first embodiment.

[0076] In step S41, the image acquisition unit 74 acquires image information of the surroundings of the moving object 8 from the camera 82 provided on the moving object 8.

[0077] In step S42, the sensor acquisition unit 75 acquires sensor information of the periphery of the moving object 8 from the sensor 83 provided in the moving object 8.

[0078] In step S43, the transmission unit 76 transmits the image information acquired by the image acquisition unit 74 and the sensor information acquired by the sensor acquisition unit 75 to the mobile object remote control device 6.

[0079] <Effects> The mobile object remote control device 6 transmits the control amount by which the operator 14 controls the control device 12 and the operation mode specified by the operator 14 to the mobile object control device 7, which controls the control target specified by the operator 14. The mobile object control device 7 calculates a control amount for controlling the operation of the mobile object 8 based on the control amount and operation mode received from the mobile object remote control device 6, and outputs the control amount to the mobile object 8. This enables the operator 14 to control multiple mobile objects that operate with different motion models. Furthermore, even if the control mechanism of the control device 12 is not the original control mechanism for controlling the mobile object 8, the mobile object 8 is operated with a control amount corresponding to the operation mode specified by the operator 14, thereby reducing the sense of discomfort felt by the operator 14.

[0080] <Embodiment 2> Fig. 21 is a block diagram showing an example of the configuration of a mobile object remote control device 6 according to Embodiment 2. Fig. 22 is a block diagram showing an example of the configuration of a mobile object control device 7 according to Embodiment 2.

[0081] As shown in Fig. 21, the mobile object remote control device 6 according to the second embodiment has a configuration in which the control amount calculation unit 72 and the control amount adjustment unit 73 of the mobile object control device 7 according to the first embodiment (see Fig. 5) are added to the mobile object remote control device 6 according to the first embodiment (see Fig. 4). In the mobile object remote control device 6 according to the second embodiment, the transmission unit 63 transmits the control amount adjusted by the control amount adjustment unit 73 (adjusted control amount) to the mobile object control device 7. Other operations are the same as those of the mobile object remote control device 6 according to the first embodiment.

[0082] As shown in Fig. 22, the mobile body control device 7 according to the second embodiment has a configuration in which the control amount calculation unit 72 and the control amount adjustment unit 73 are removed from the mobile body control device 7 according to the first embodiment (see Fig. 5). In the mobile body control device 7 according to the second embodiment, the receiving unit 71 receives the adjusted control amount transmitted from the mobile body remote control device 6 and outputs the adjusted control amount to the mobile body 8. Other operations are the same as those of the mobile body control device 7 according to the first embodiment.

[0083] As described above, according to the second embodiment, the configuration of the moving body control device 7 can be simplified.

[0084] <Hardware configuration> The functions of the control amount acquisition unit 61, setting unit 62, transmission unit 63, reception unit 64, and image generation unit 65 in the mobile object remote control device 6 described in the first embodiment are realized by a processing circuit. That is, the mobile object remote control device 6 includes a processing circuit for acquiring control amounts, setting a control target and an operation mode, transmitting the control target and the operation mode, receiving image information and sensor information about the surroundings of the control target, and generating a presentation image based on the image information and sensor information. The processing circuit may be dedicated hardware, or may be a processor (also referred to as a CPU, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor)) that executes a program stored in a memory.

[0085] When the processing circuit is dedicated hardware, the processing circuit 91 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof, as shown in Fig. 23. The functions of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65 may be realized individually by the processing circuit 91, or these functions may be realized collectively by a single processing circuit 91.

[0086] When the processing circuit 91 is the processor 92 shown in FIG. 24 , the functions of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 93. The processor 92 realizes each function by reading and executing the program recorded in the memory 93. That is, the mobile object remote control device 6 includes the memory 93 for storing a program that ultimately executes the steps of acquiring a control amount, setting a control target and an operation mode, transmitting the control target and the operation mode, receiving image information and sensor information about the surroundings of the control target, and generating a presentation image based on the image information and sensor information. These programs can also be said to cause a computer to execute the procedures or methods of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65. Here, memory may be, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic disk, flexible disk, optical disk, compact disk, DVD (Digital Versatile Disc), or any storage medium that will be used in the future.

[0087] It should be noted that some of the functions of the control amount acquisition unit 61, the setting unit 62, the transmission unit 63, the reception unit 64, and the image generation unit 65 may be realized by dedicated hardware, and other functions may be realized by software or firmware.

[0088] Thus, the processing circuitry can implement each of the above-described functions through hardware, software, firmware, or a combination thereof.

[0089] The above describes the hardware configuration of the mobile object remote control device 6 shown in Figure 4, but the hardware configurations of the mobile object control device 7 shown in Figure 5, the mobile object remote control device 6 shown in Figure 21, and the mobile object control device 7 shown in Figure 22 are also similar.

[0090] Within the scope of the present disclosure, the embodiments can be freely combined, modified, or omitted as appropriate.

[0091] Although the present disclosure has been described in detail, the above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned. [Explanation of symbols]

[0092] 1 Control system, 2 First mobile body system, 3 Second mobile body system, 4 Third mobile body system, 5 Network, 6 Mobile body remote control device, 7 Mobile body control device, 8 Mobile body, 11 Mobile body remote control device, 12 Control device, 13 Image display device, 14 Operator, 15 Joystick, 16 Touch panel, 21 Mobile body control device, 22 First mobile body, 31 Mobile body control device, 32 Second mobile body, 41 Mobile body control device, 42 Third mobile body, 61 Control amount acquisition unit, 62 Setting unit, 63 Transmitter, 64 Receiver, 65 Image generation unit, 71 Receiver, 72 Control amount calculation unit, 73 Control amount adjustment unit, 74 Image acquisition unit, 75 Sensor acquisition unit, 76 Transmitter, 81 Movement control unit, 82 Camera, 83 Sensor, 91 Processing circuit, 92 Processor, 93 Memory, 721 State equation, 722 Output equation, 723 Flexible control, 724 Control.

Claims

1. a control amount acquisition unit that acquires a control amount from a control device operated by an operator; a setting unit that sets an object to be controlled from among a plurality of moving objects that operate according to different motion models, and sets an operation mode that is a motion model that causes the object to operate as specified by the operator; a control amount calculation unit that calculates a control amount for controlling the object to be controlled, which is set by the setting unit, based on the control amount acquired by the control amount acquisition unit and the operation mode set by the setting unit; A mobile remote control system comprising:

2. a control amount acquisition unit that acquires a control amount from a control device operated by an operator; a setting unit that sets a target to be controlled from among a plurality of moving objects that operate according to different motion models; a control amount calculation unit that calculates a control amount for controlling the object to be controlled, which is set by the setting unit, based on the control amount acquired by the control amount acquisition unit; a control amount adjustment unit that adjusts the control amount in accordance with the scale of the object to be controlled; A mobile remote control system comprising:

3. a control amount acquisition unit that acquires a control amount from a control device operated by an operator; a setting unit that sets an object to be controlled from among a plurality of moving objects that operate according to different motion models, and sets an operation mode that is a motion model that causes the object to operate as specified by the operator; a transmitter that transmits the operation amount acquired by the operation amount acquisition unit and the operation mode set by the setting unit to a mobile object control device that controls the object to be controlled; A mobile object remote control device comprising:

4. a receiving unit that receives image information and sensor information of the surroundings of the target object from the mobile object control device; an image generation unit that generates a presentation image to be presented to the operator based on the image information and the sensor information received by the receiving unit; The mobile remote control device of claim 3 further comprising:

5. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects the presented image so as to eliminate a time delay associated with control of the object to be controlled.

6. 5. The mobile object remote control device according to claim 4, wherein, when the operator is performing a rotation operation, the image generation unit corrects the presented image so that a change in viewpoint accompanying the movement of the object to be controlled causes the object to rotate in place.

7. a control amount calculation unit that calculates a control amount for controlling the object to be controlled based on the control amount acquired by the control amount acquisition unit; a control amount adjustment unit that adjusts the control amount in accordance with the scale of the object to be controlled; The mobile remote control device of claim 3 further comprising:

8. The mobile object remote control device according to claim 3 , wherein the setting unit automatically selects the operation mode when the operator sets the object to be controlled.

9. The mobile object remote control device according to claim 3 , wherein the mobile object exists in a virtual space.

10. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects the angle of view and size of the presented image so that the viewpoints of the objects to be controlled are the same as each other as the objects to be controlled move.

11. The mobile object remote control device according to claim 4 , wherein the image generation unit corrects distortion of the presented image caused by roll motion, pitch motion, and up-down motion of the controlled object.

12. the receiving unit receives, from the mobile object control device, a control amount for controlling the object to be controlled, the control amount being calculated based on the control amount; The mobile object remote control device according to claim 4 , wherein the image generation unit generates the presentation image including the control amount.

13. 5. The mobile object remote control device according to claim 4, wherein when a motion model corresponding to the control movement of the control device is different from a motion model of the object to be controlled, the image generation unit generates the presentation image including an image simulating another control device corresponding to the motion model of the object to be controlled.

14. The mobile body remote control device according to claim 13 , wherein the control amount acquisition unit converts the control amount acquired from the control device into a control amount for the other control device.

15. a control amount calculation unit that calculates a control amount for controlling a target object to be controlled, which is set from a plurality of moving objects operating according to different motion models, based on a control amount acquired from a control device controlled by an operator; a control amount adjustment unit that adjusts the control amount in accordance with the scale of the object to be controlled; A mobile object control device comprising:

16. a receiving unit that receives the operation amount and the operation mode transmitted from the mobile body remote control device according to claim 3; a control amount calculation unit that calculates a control amount for controlling the object to be controlled based on the control amount and the operation mode received by the receiving unit; A mobile object control device comprising:

17. an image acquisition unit that acquires image information of the surroundings of the object to be controlled from a camera provided on the object to be controlled; a sensor acquisition unit that acquires sensor information related to a movement of the object to be controlled from a sensor provided in the object to be controlled; a transmitter that transmits the image information acquired by the image acquisition unit and the sensor information acquired by the sensor acquisition unit to a mobile object remote control device that outputs the operation amount for at least the operator to operate the controlled object; The mobile object control device according to claim 15 or 16, further comprising:

18. The mobile body control device according to claim 15 , wherein the control amount calculation unit calculates the control amount that causes the object to imitate an operation corresponding to an operation mode that is a motion model for moving the object specified by the operator.

19. The mobile body control device according to claim 16 , further comprising a control amount adjustment unit that adjusts the control amount calculated by the control amount calculation unit in accordance with a scale of the object to be controlled.

Images