Remote operation system, and remote operation program

Abstract

To provide a remote control system for realizing remote control without requiring position information of a moving object.SOLUTION: A remote control system comprises: a moving object 300; and an operation terminal 100 for remotely controlling the moving object 300 by wireless communication. The moving object 300 has an imaging unit that captures an image around the moving object 300 in an omnidirectional area and the operation terminal 100 has a control unit 102 which causes a display unit to display a first image corresponding to a viewing direction input from an input unit in at least a part of the omnidirectional area imaged by the imaging unit and causes the display unit to display a direction of the first image in the omnidirectional area, a first mark for indicating a traveling direction of the moving object 300, and a second mark for indicating a movement target direction of the moving object 300 while superimposing on the first image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote operation system and a remote operation program.

Background Art

[0002] In a remote operation system, a user (remote operator) remotely operates an unmanned moving body. In a remote operation system, for example, the user can view an image of a remote location by remotely operating an unmanned moving body equipped with a camera. In particular, in a remote operation system in which the camera can be freely moved for imaging, the user can view the situation at the remote location as if they were on-site. For this reason, such a remote operation system is used, for example, for telepresence.

[0003] In a remote operation system, the user operates the moving body relying only on the image captured by the camera mounted on the moving body. For this reason, in a remote operation system, unlike the case where the user is actually on-site and directly observes the movement of the moving body while operating it, it is difficult for the user to move the moving body to the intended location.

[0004] For example, when the user performs an operation to turn the moving body to the left while viewing the image captured by the camera, in the remote operation system, due to a slight delay in the image, it may be difficult to know how much turning operation should be performed. Also, when the user performs an operation to move the moving body straight ahead, in the remote operation system, if the direction of the image and the direction of the moving body do not match, the moving body may move in a diagonal direction. Thus, in a remote operation system, due to the relationship between the direction of the image the user is viewing and the direction of the moving body, or due to delays in the image or operation, etc., the user cannot feel the movement of the remote moving body in real time.

[0005] Conventionally, various technological developments have been made to facilitate remote operation. For example, the technology of Patent Document 1 moves a moving object by designating the moving position of the moving object on a screen. This technology determines the moving position by converting from an image coordinate system to a world coordinate system in order to match the position information on the screen and the position information of the moving object.

[0006] Also, the technology of Patent Document 2 displays a robot to be moved on an overhead display and moves the moving object by designating the moving position with a point-and-click device.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] The technology of Patent Document 1 needs to acquire the position information of the moving object in the world coordinate system. Therefore, this technology has a problem that if the acquired position accuracy is not high, the moving object cannot be accurately moved to the intended position.

[0009] The technology of Patent Document 2 needs to associate the coordinates on the screen with the actual coordinates where the robot exists. Therefore, also in this technology, it is necessary to acquire the position information of the robot. Thus, this technology, like Patent Document 1, has a problem that if the acquired position accuracy is not high, the robot cannot be accurately moved to the intended position.

[0010] Therefore, an object of the present invention is to provide a remote operation system and a remote operation program for realizing remote operation of a moving object without requiring the position information of the moving object.

Means for Solving the Problem

[0011] The above object of the present invention is achieved by the following means.

[0012] (1) A mobile body, An operation terminal for remotely operating the mobile body by wireless communication, A remote operation system having: The mobile body has an imaging unit that images the surroundings of the mobile body over an omnidirectional area, The operation terminal is An input unit, A display unit, A control unit that causes the display unit to display at least a part of the omnidirectional area imaged by the imaging unit, which is a first image corresponding to the line-of-sight direction input from the input unit, and Superimposes and displays on the display unit a direction of the first image in the omnidirectional area, a first mark indicating a traveling direction of the mobile body, and a second mark indicating a moving target direction of the mobile body on the first image, A remote operation system having:

[0013] (2) The operation terminal has sensors for detecting acceleration and angular velocity, The control unit controls the movement of the mobile body based on the output from the sensors, according to the remote operation system described in (1) above.

[0014] (3) The control unit controls the movement of the mobile body based on an input for moving the mobile body from the input unit and the acceleration and angular velocity detected by the sensors, according to the remote operation system described in (2) above.

[0015] (4) The control unit moves the line-of-sight direction based on a first input input from the input unit and the acceleration and angular velocity detected by the sensors after the first input, according to the remote operation system described in (3) above.

[0016] (5) The control unit moves the second mark to the center of the screen of the display unit based on the second input input from the input unit, for the remote operation system according to the above (3) or (4).

[0017] (6) The control unit controls the moving speed of the moving body based on the third input input from the input unit, for the remote operation system according to any one of the above (3) to (5).

[0018] (7) The control unit reverses the moving direction of the moving body based on the fourth input input from the input unit, for the remote operation system according to any one of the above (3) to (6).

[0019] (8) The control unit zooms in or out the first image based on the fifth input input from the input unit, for the remote operation system according to any one of the above (3) to (7).

[0020] (9) The first image is cut out on the moving body side and transmitted from the moving body to the operation terminal, for the remote operation system according to any one of the above (1) to (8).

[0021] (10) The control unit displays, in addition to the first image, a second image including an area in the moving target direction of the moving body and an area different from the first image on the display unit, for the remote operation system according to any one of the above (1) to (9).

[0022] (11) The control unit detects a person and / or an object in the area shown in the second image, for the remote operation system according to the above (10).

[0023] (12) The second image is cut out on the moving body side and transmitted to the operation terminal, for the remote operation system according to the above (10) or (11).

[0024] (13) A mobile body having an imaging unit that images over an omnidirectional area, and a remote control program for remotely controlling the mobile body from an operation terminal connected by wireless communication, displaying a first image corresponding to the input line-of-sight direction in at least a part of the omnidirectional area imaged by the imaging unit (step (a)); superimposing and displaying on the first image a direction of the first image in the omnidirectional area, a first mark indicating a traveling direction of the mobile body, and a second mark indicating a movement target direction of the mobile body (step (b)); A remote control program for causing a computer to execute.

[0025] (14) The operation terminal has a sensor that detects acceleration and angular velocity, The remote control program according to (13) above, further having a step (c) of controlling the movement of the mobile body based on an output from the sensor.

[0026] (15) The step (c) is the remote control program according to (14) above, which controls the movement of the mobile body based on an input for moving the mobile body and the acceleration and angular velocity detected by the sensor.

[0027] (16) The step (c) is the remote control program according to (15) above, which moves the line-of-sight direction based on a first input and the acceleration and angular velocity detected by the sensor after the first input.

[0028] (17) The step (c) is the remote control program according to any one of (14) to (16) above, which moves the second mark to the center of the screen where the first image is displayed based on a second input.

[0029] (18) The step (c) is the remote control program according to any one of (14) to (17) above, which controls the moving speed of the mobile body based on a third input.

[0030] (19) The step (c) is the remote operation program according to any one of (14) to (18) above, which reverses the moving direction of the moving body based on the fourth input.

[0031] (20) The step (c) is the remote operation program according to any one of (14) to (19) above, which zooms in or out the first image based on the fifth input.

[0032] (21) The remote operation program according to any one of (13) to (20) above, which has a step (d) of receiving the first image cut out on the moving body side from the moving body.

[0033] (22) The remote operation program according to any one of (13) to (19) above, which has a step (e) of displaying a second image including an area in the moving target direction of the moving body and an area different from the first image.

[0034] (23) The remote operation program according to (22) above, which further has a step (f) of detecting a person and / or an object in the area shown in the second image.

[0035] (24) The remote operation program according to (22) or (23) above, which has a step (g) of receiving the second image cut out on the moving body side from the moving body.

Effect of the Invention

[0036] The moving body of the remote operation system has an imaging unit that images in all directions. Thereby, when the remote operator moves the moving body, the remote operator can freely look at and operate in the direction he / she wants to see. For this reason, the remote operator can operate the moving body while looking at the direction he / she wants to move. Therefore, remote operation of the moving body can be realized without the need for the position information of the moving body.

Brief Description of the Drawings

[0037]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Embodiments for Carrying Out the Invention

[0038] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0039] FIG. 1 is a schematic diagram for explaining a remote control system according to an embodiment.

[0040] As shown in FIG. 1, the remote control system 1 according to Embodiment 1 includes an operation terminal 100 and a moving body 300. The operation terminal 100 and the moving body 300 are connected by a network 200. The operation terminal 100 is connected to the moving body 300 by wireless communication via the network 200.

[0041] The details of each part will be described below.

[0042] Figure 2 is a block diagram for explaining the configuration of the operation terminal 100.

[0043] As shown in Figure 2, the operation terminal 100 includes a communication unit 101, a control unit 102, a storage unit 103, a sensor 104, and an operation display unit 105.

[0044] The communication unit 101 performs wireless communication with the moving body 300 via the network 200. The communication unit 101 has an interface according to a wireless communication standard such as 5G or IEEE802.11 (for example, WiFi) corresponding to a wireless base station or an access point (both not shown) of the network 200. The communication unit 101 may have a plurality of interfaces corresponding to a plurality of wireless communication standards. In addition, the communication unit 101 also has an antenna (not shown) corresponding to a wireless communication station or an access point that may be connected. Note that the wireless communication standard used is not limited to 5G or IEEE802.11 as long as it can communicate with the communication unit 101 of the operation terminal 100.

[0045] The communication unit 101 receives image data from the moving body 300, the current moving direction of the moving body 300, and the current moving speed, and outputs them to the control unit 102. In addition, the communication unit 101 transmits a control signal for controlling the moving body 300 output from the control unit 102 to the moving body 300. The control signal has information for moving the moving body 300.

[0046] The control unit 102 is a computer. The control unit 102 controls the movement of the moving body 300 by executing a program for remote operation described later.

[0047] Similar to a well-known computer, the control unit 102 has a CPU (Central Processing Unit) serving as an arithmetic element, a ROM (Read Only Memory) and a RAM (Random Access Memory) serving as a storage unit 103, and the like.

[0048] In addition to the ROM and the RAM, the control unit 102 also has a storage unit 103. The storage unit 103 is a non-volatile storage device such as an eMMC (embedded MultiMediaCard), an SSD (Solid State Drive), or an HDD (Hard Disk Drive). Further, the storage unit 103 may be a portable storage medium such as a memory card.

[0049] The sensor 104 is, for example, an acceleration sensor and an angular velocity sensor (gyro sensor). The sensor 104 outputs the detected values of acceleration and angular velocity to the control unit 102. The control unit 102 obtains the rotation direction and the rotation amount (rotation angle) of the operation terminal 100 itself from the detected values of acceleration and angular velocity.

[0050] FIG. 3 is a schematic diagram for explaining the rotation of the operation terminal 100 itself when the operation terminal 100 is used in the horizontal position. FIG. 4 is a schematic diagram for explaining the rotation of the operation terminal 100 itself when the operation terminal 100 is used in the vertical position. Here, the case where a smartphone is used as the operation terminal 100 will be described as an example.

[0051] When the smartphone (operation terminal 100) is used in the horizontal position, the rotation of the operation terminal 100 itself is a rotation about the rotation axis AX passing through the center of the long side of the smartphone, as shown in FIG. 3. On the other hand, when the smartphone (operation terminal 100) is used in the vertical position, the rotation of the operation terminal 100 itself is a rotation about the rotation axis AX passing through the center of the short side of the smartphone, as shown in FIG. 4. The rotation axis AX is a perpendicular line (a line perpendicular to the horizontal line) in both cases.

[0052] The operation display unit 105 is an input unit and a display unit. The operation display unit 105 is, for example, a touch panel type display, which displays various information and accepts various inputs from the user. An image captured by the omnidirectional camera 320 mounted on the moving body 300 is displayed on the operation display unit 105. The displayed image is an image of a partial area in the omnidirectional image. Further, the operation display unit 105 accepts an input of an instruction for controlling the moving body 300 when a remote operator (user) touches the operation display unit 105.

[0053] The operation terminal 100 having the functions of such each part is, for example, a smartphone, a tablet computer, or the like. The operation terminal 100 may be other personal computers (PCs), dedicated controllers, or the like.

[0054] The network 200 is a network 200 using a 5G communication line, a wireless local area network (LAN), or the like. The network 200 has a wireless base station and an access point for the operation terminal 100 and the moving body 300 to be connected by wireless communication. These networks 200 may be further connected to the Internet.

[0055] In this embodiment, the operation terminal 100 and the moving body 300 are connected by wireless communication via the network 200, but the operation terminal 100 and the moving body 300 may be directly connected by wireless communication.

[0056] The moving body 300 has a moving body main body 310, an omnidirectional camera 320, and a movement control terminal 330. The moving body 300 is, for example, a robot car that travels on the ground by remote control (also referred to as wireless steering). Further, the moving body 300 may be, for example, an aerial drone, an underwater or surface drone, or the like. In this embodiment, a robot car is described as an example of the moving body 300.

[0057] In this embodiment, the mobile body main body 310 has a drive mechanism including a motor, wheels, etc. and a steering mechanism. The mobile body main body 310 is also provided with an acceleration sensor and a gyro sensor (both not shown). The acceleration sensor and the gyro sensor detect the moving direction and moving speed of the mobile body 300. Note that the acceleration sensor and the gyro sensor may be provided in the movement control terminal 330 or the omnidirectional camera 320.

[0058] The omnidirectional camera 320 is an imaging unit. The omnidirectional camera 320 is also referred to as a 360-degree camera. The omnidirectional camera 320 captures images globally. The images captured by the omnidirectional camera 320 are moving images. In a robot car or other mobile body 300 that travels on the ground, an omnidirectional camera 320 that captures only the upper hemispherical portion from the mobile body main body 310 may be used.

[0059] Also, the movement control terminal 330 receives a control signal from the operation terminal 100 and controls the mobile body 300 according to the instructions included in the control signal. The movement control terminal 330 is a computer for movement control. The control signal has information such as the line-of-sight direction input to the operation terminal 100, the direction for moving the mobile body 300, and the indicated speed. The movement control terminal 330 controls the omnidirectional camera 320 based on the control signal. The movement control terminal 330 controls the drive mechanism and the steering mechanism of the mobile body main body 310 based on the control signal to move the mobile body 300 in the direction and at the speed input from the operation terminal 100.

[0060] [[ID=II]] Further, the mobile control terminal 330 wirelessly communicates with the operation terminal 100 via the network 200. The mobile control terminal 330 has an interface according to a wireless communication standard such as 5G or the IEEE 802.11 standard (e.g., WiFi) corresponding to a wireless base station or an access point (both not shown) of the network 200. It may have a plurality of interfaces corresponding to a plurality of wireless communication standards. Also, the mobile control terminal 330 has an antenna (not shown) corresponding to a wireless communication station or an access point that may be connected. Note that the wireless communication standard used is not limited to 5G or IEEE 802.11 as long as it can communicate with the communication unit 101 of the operation terminal 100.

[0061] The mobile control terminal 330 cuts out a part of the omnidirectional area imaged by the omnidirectional camera 320. The mobile control terminal 330 transmits the image of the cut-out area as image data to the operation terminal 100. The mobile control terminal 330 transmits the moving direction and moving speed of the mobile body 300 to the operation terminal 100.

[0062] Next, an operation example will be described. Here, an example using a smartphone as the operation terminal 100 will be described.

[0063] An image captured by the omnidirectional camera 320 is transmitted from the mobile body 300 to the smartphone in real time. The remote operator can view the image captured by the omnidirectional camera 320 mounted on the remotely located mobile body 300 through the screen of the smartphone. Then, the remote operator can control the movement of the mobile body 300 by operating the smartphone.

[0064] The mobile body 300 is a robot car. The moving direction of the robot car is only the horizontal direction of front, rear, left, and right. Also, in the robot car, the operation of changing the traveling direction varies depending on the configuration of the drive system such as wheels. As a configuration for changing the traveling direction, for example, a configuration in which the front wheels change their angles to gradually turn left (or right) like a normal vehicle can be adopted. Also, as a configuration for changing the traveling direction, a configuration in which it rotates in place and then moves forward after turning the vehicle body in the traveling direction like a mecanum wheel can be adopted. Also, as a configuration for changing the traveling direction, a configuration in which it moves straight in an oblique direction like a mecanum wheel can be adopted.

[0065] Here, unless otherwise specified, the mobile body 300 that changes its traveling direction in the same way as a normal vehicle will be described as an example.

[0066] FIG. 5 is an explanatory diagram for explaining Example 1 of the display image displayed on the operation display unit 105.

[0067] As shown in FIG. 5, on the operation display unit 105, an image of a partial area within the image of the omnidirectional area captured by the omnidirectional camera 320 is displayed. This image is the first image. The image of the omnidirectional area is an image of a spherical surface viewed from the inside of a sphere (described later). Therefore, a partial image within the image of the omnidirectional area is also curved. Thus, in the operation terminal 100, the curved image of the selected partial area is corrected to be planar. The corrected image is displayed on the operation display unit 105 of the operation terminal 100.

[0068] On the operation display unit 105, a front mark, a reticle, a direction bar, and a speedometer are displayed together with an image of a partial area of the image from the omnidirectional camera 320.

[0069] In Example 1 of the display screen shown in FIG. 5, the center of the screen is the line-of-sight direction.

[0070] The front mark is the first mark indicating the current direction (front direction) of the moving body 300. In the display screen example 1, the front mark is displayed at the center of the screen. The front mark moves on the direction bar. The front mark indicates in which direction and to what extent the moving body 300 is oriented with respect to the line-of-sight direction (center of the screen).

[0071] The reticle is the second mark indicating the moving target direction. In the display screen example 1, the reticle is displayed at the center of the screen, and the front mark is also displayed indicating the center of the screen direction. The moving body 300 moves forward in the direction pointed to by the reticle, that is, the same direction as the line-of-sight direction. The method of operating the forward movement, acceleration, and deceleration of the moving body 300 will be described later.

[0072] The direction bar is a guide for easily displaying the direction.

[0073] The speedometer indicates the current speed of the moving body 300. Although the speedometer is displayed using a graphical figure, instead of or in addition to this, the actual speed may be displayed numerically.

[0074] FIG. 6 is an explanatory diagram for explaining the display image example 2 displayed on the operation display unit 105.

[0075] The display image example 2 shown in FIG. 6 shows the case where the line-of-sight direction of the moving body 300 is changed. When the line-of-sight direction is changed, the position of the reticle moves within the screen. For example, when the line of sight is shifted slightly upward and to the right, the image displayed on the screen moves in that direction. On the other hand, as shown in FIG. 6, the reticle moves slightly downward and to the left from the center of the screen, which is the line-of-sight direction. Also, as shown in FIG. 6, since the front mark is on the right side of the reticle, it can be seen that the moving body 300 is currently facing the right side with respect to the indicated moving direction. Then, by an instruction input for forward movement, the moving body 300 will move forward while turning left in the direction indicated by the reticle.

[0076] The image displayed on the operation display unit 105 is a part of the image of the omnidirectional area (360 degrees). When the front mark is located at the central part of the direction bar (in the case of FIG. 5), the moving body 300 is moving in the same direction as the line-of-sight direction. Also, when the front mark is located on the right side of the direction bar (in the case of FIG. 6), the moving body 300 is moving in a direction to the right of the line-of-sight direction.

[0077] FIG. 7 is an explanatory diagram for explaining a display image example 3 displayed on the operation display unit 105.

[0078] The display image example 3 shown in FIG. 7 shows a case where the movement target does not fit on the screen of the operation display unit 105. Such a screen occurs, for example, when trying to change the moving direction beyond the range of the image displayed on the screen. In such a case, as shown in FIG. 7, the reticle is displayed only halfway at the edge of the screen. Note that the display method when trying to change the moving direction beyond the range of the image displayed on the screen is not limited to the method shown in FIG. 7. For example, it may be shown that the moving direction is changed beyond the range of the image displayed on the screen by changing the color of the reticle or by canceling the display of the reticle. Further, a warning display or the like indicating that an attempt is being made to change the moving direction beyond the range of the image displayed on the screen may be performed.

[0079] Also, there may be a case where the front mark does not fit on the screen of the operation display unit 105. In such a case, the front mark is displayed at the edge of the direction bar. Also, in such a case, the front mark may be displayed in a changed color. Also, in such a case, in addition to the display of the front mark, a warning may be displayed.

[0080] The vertical position of the reticle within the screen will be described. FIGS. 8 and 9 are explanatory diagrams for explaining the vertical position of the reticle within the screen.

[0081] When the moving body 300 is traveling on flat ground and the true front (traveling direction) of the moving body 300 is the same as the line-of-sight direction L (described later), the reticle is located at the vertical center within the screen.

[0082] On the one hand, for example, when the moving body 300 is going uphill and the line of sight direction L is the same as the front (travel direction) of the moving body 300, as shown in FIG. 8, the line of sight direction L is upward. The vertical direction indicated by the reticle is the same as the horizontal line (a line parallel to the flat ground). Therefore, the reticle is located below the center in the vertical direction of the screen.

[0083] Conversely, when the moving body 300 is going downhill and the line of sight direction L is the same as the front (travel direction) of the moving body 300, as shown in FIG. 9, the line of sight direction L is downward. The vertical direction indicated by the reticle is the same as the horizontal line (a line parallel to the flat ground). Therefore, the reticle is located above the center in the vertical direction of the screen.

[0084] However, for an aerial drone or an underwater drone that moves three-dimensionally, the restriction of fixing the reticle in a direction parallel to the flat ground (horizontal line) may be removed. And for an aerial drone or an underwater drone that moves three-dimensionally, if the position of the reticle within the screen is above the center in the vertical direction, it may be processed as an instruction to ascend, and if it is below the center, it may be processed as an instruction to descend, etc.

[0085] Next, the image captured by the omnidirectional camera 320 will be described.

[0086] FIG. 10 is an explanatory diagram for explaining Example 1 of the captured image captured by the omnidirectional camera 320. FIG. 11 is an explanatory diagram for explaining Example 2 of the captured image captured by the omnidirectional camera 320.

[0087] The imaging image examples 1 and 2 shown in FIGS. 10 and 11 are shown as virtual spherical surfaces viewed from the inside (the center point of the sphere) of the images captured by the omnidirectional camera 320. As shown in FIGS. 10 and 11, when viewed from the center point of the sphere, the direction L (Line Of Sight) indicates the line-of-sight direction that the remote operator is looking at, the direction F (Forward) indicates the traveling direction (front) of the moving body 300, and the direction R (Reticle) indicates the direction in which the moving body 300 is desired to move, that is, the moving target direction pointed to by the reticle. Hereinafter, each direction is referred to as the line-of-sight direction L, the traveling direction F, and the reticle direction R. The information on these directions is stored as internal parameters in the storage unit 103 in the control unit 102.

[0088] In the imaging image example 1 shown in FIG. 10, the region A1 is the region that is displayed as the display screen example 1 shown in FIG. 5. The region A1 is cut out on the moving body 300 side within a predetermined range and transmitted to the operation terminal 100.

[0089] In FIG. 10, the line-of-sight direction L, the traveling direction F, and the reticle direction R are all in the same direction. In the present embodiment, the state in which the line-of-sight direction L, the traveling direction F, and the reticle direction R are all in the same direction is set as the initial state.

[0090] On the other hand, in the imaging image example 2 shown in FIG. 11, the region A2 is the region that is displayed as the display screen example 2 shown in FIG. 6. This region A2, like the region A1, is cut out on the moving body 300 side within a predetermined range and transmitted to the operation terminal 100.

[0091] In FIG. 11, the line-of-sight direction L indicates the direction changed by the remote operator, the traveling direction F indicates the direction in which the moving body 300 is traveling at that time, and the reticle direction R indicates the direction of the moving target at that time.

[0092] Note that within the images of the omnidirectional region (FIGS. 10 to 12), the position in the vertical direction of the reticle direction R is always parallel to the flat ground, and thus is on the plane H.

[0093] As shown in FIGS. 10 and 11, the image of the omnidirectional region is entirely spherical. Therefore, if left as it is, the image displayed on the operation display unit 105 will be a curved image. Thus, in the present embodiment, when displaying an image on the operation display unit 105, regions A1 and A2 within a predetermined range from the center O are flattened and displayed with the point of intersection of the straight line in the line-of-sight direction L and the spherical surface as the center O. The flattening is executed by the control unit 102 of the operation terminal 100.

[0094] The ranges of regions A1 and A2 (the predetermined range from the center O) are the same size and differ only in their directions. The ranges of regions A1 and A2 are determined in advance according to the screen size, shape, resolution, and allowable level of image distortion correction of the operation display unit 105.

[0095] The straight line in the traveling direction F is always on a plane H parallel to the flat ground.

[0096] The direction bar displayed on the operation display unit 105 indicates the range of region A1 in the meridian direction of the spherical surface of the captured image example 1.

[0097] The forward mark displayed on the operation display unit 105 is displayed at the position where the point of intersection of the straight line in the traveling direction F and the spherical surface is moved along the meridian and intersects the direction bar within region A1.

[0098] The straight line in the reticle direction R is always on a plane H parallel to the flat ground. The point P where the straight line in the reticle direction R intersects the spherical surface indicates the position of the reticle, that is, the direction of the movement target. The display of these forward marks and reticles is the same in region A2.

[0099] Here, a case where the forward mark and the reticle do not fit within the image in the line-of-sight direction displayed on the operation display unit 105 will be described.

[0100] FIG. 12 is an explanatory diagram for explaining an example of a captured image 3 captured by the omnidirectional camera 320. The captured image example 3 shows a case where the front mark and the reticle do not fit within the screen (image) in the line-of-sight direction. Note that also in the captured image example 3, the definitions of the line-of-sight direction L, the traveling direction F, and the reticle direction R are the same as those in the captured image examples 1 and 2.

[0101] When the traveling direction F of the moving body 300 is outside the range of the region A3 cut out as an image in the line-of-sight direction, if the point where the straight line of the traveling direction F intersects the spherical surface is moved along the meridian, it does not intersect the direction bar. In such a state, the front mark displayed on the operation display unit 105 is displayed at the end of the direction bar within the region A3. Therefore, when the point where the straight line of the traveling direction F intersects the spherical surface and is moved along the meridian is outside the region A3 on the right side, the front mark is displayed at the right end of the direction bar, and when the point is outside the region A3 on the left side, the front mark is displayed at the left end of the direction bar.

[0102] The reticle may also be outside the range of the image, similar to the front mark. When the point P of the movement target is outside the range of the region A3, the reticle is displayed at the position where the left end of the region A3 intersects the plane H, as shown in FIG. 12.

[0103] As a method of specifying the line-of-sight direction L, the traveling direction F, and the reticle direction R, vector coordinates are used. For example, by fixing the coordinate system to the omnidirectional camera 320 and fixing the direction of the front of the moving body 300 to the coordinate axis x, the traveling direction F can be represented only by the inclination of the moving body 300 with respect to the flat ground, and the calculation can be omitted. The line-of-sight direction L and the reticle direction R are calculated by measuring the rotation amount of the moving body 300 at any time and reflecting the measured rotation amount with respect to the vector coordinates of the fixed traveling direction F.

[0104] As a method for measuring the amount of rotation, the change in the orientation of the moving body 300 is directly measured by a gyro sensor, an acceleration sensor, etc. mounted on the moving body 300. Also, as a method for measuring the amount of rotation, object detection by a TOF (Time Of Flight) sensor 104 used in LiDAR (Light Detection And Ranging) or the like, or object detection by an image recognition AI may be performed, and it may be calculated from the detection information.

[0105] As described above, the positions of the front mark and the reticle on the image of a partial area cut out from the image of the omnidirectional area captured by the omnidirectional camera 320 are associated with the line-of-sight direction L, the traveling direction F, and the reticle direction R, which are physical directions related to the image captured by the omnidirectional camera 320.

[0106] Next, the operation for changing the line-of-sight direction L will be described.

[0107] FIG. 13 is an explanatory diagram for explaining the operation for changing the line-of-sight direction L.

[0108] As described above, in the initial state, the line-of-sight direction L and the traveling direction F coincide. That is, in the initial state, the line-of-sight direction L faces the same direction as the traveling direction F of the moving body 300 on a plane H parallel to the flat ground.

[0109] The change in the line-of-sight direction L is performed by rotating the operation terminal 100 while making a first input from this initial state. The first input is performed, for example, by touching the screen of the operation display unit 105 (the screen of a smartphone, hereinafter simply referred to as the "screen") with one finger as shown in FIG. 13.

[0110] First, as the first input, when the screen is touched with one finger, the control unit 102 executes screen locking. Here, screen locking is a state in which the line-of-sight direction L can be changed in accordance with the movement of the operation terminal 100 (smartphone). While in this state, the operation terminal 100 is rotated. The rotation has already been described as above.

[0111] The operation terminal 100 is equipped with an acceleration sensor and a gyro sensor. From the acceleration sensor and the gyro sensor, the rotation direction and the amount of rotation of the operation terminal 100 are output to the control unit 102. The control unit 102 measures the amount of rotation from the moment when the screen is touched with one finger. Then, the control unit 102 transmits the measured amount of rotation to the moving body 300 as the amount of rotation in the line-of-sight direction L. The moving body 300 moves the center O on the spherical surface, which is the omnidirectional image shown in FIGS. 10 and 11. In conjunction with the movement of the center O, for example, the region A1 moves to the region A2 (accurately, the cut-out position of the image changes). The control unit 102 receives the region A2 updated by this movement from the moving body 300. The control unit 102 flattens the image of the received region A2 and displays it on the operation display unit 105 in real time.

[0112] Returning to FIG. 13 for explanation. When the smartphone is slid (rotated) to the right from the state where the screen is touched with one finger, the image on the right side with respect to the traveling direction of the moving body 300 is displayed. As a result, the image displayed on the operation display unit 105 moves from the state before the slide (rotation) to the image after the slide (rotation). At this time, the traveling direction F and the reticle direction R do not change. For this reason, in the image on the operation display unit 105, the forward mark and the reticle move to the left side.

[0113] Note that portions not included in the image captured by the omnidirectional camera 320 are displayed as blanks on the operation display unit 105. Specifically, for example, in the case of a robot car or the like, the lower part of the moving body main body 310 is not imaged. Therefore, when the operation terminal 100 is tilted downward, the center O moves in the direction of viewing the portion that has not been imaged as an omnidirectional image. In such a case, since no image has been acquired in the range below the moving body main body 310 in the first place, the image displayed on the operation display unit 105 is displayed as partially blank.

[0114] Next, an operation for changing the reticle direction R will be described.

[0115] FIG. 14 is an explanatory diagram for explaining an operation of changing the reticle direction R.

[0116] As described above, in the initial state, the reticle direction R and the traveling direction F coincide. That is, in the initial state, the reticle direction R faces the same direction as the traveling direction F of the moving body 300.

[0117] The change of the reticle direction R is performed by rotating the operation terminal 100 while performing the second input from this initial state. The second input is performed, for example, by touching the screen with two fingers as shown in FIG. 14.

[0118] First, as the second input, when the screen is touched with two fingers, the control unit 102 executes screen locking. Here, the screen locking is a state in which the reticle direction R can be changed in accordance with the movement of the operation terminal 100 itself. At this time, the control unit 102 sets and aligns the horizontal direction of the reticle direction R with the line-of-sight direction L simultaneously with the screen locking. In this state, the control unit 102 enables the reticle direction R to move while always pointing in the same horizontal direction as the line-of-sight direction L. The vertical direction of the reticle direction R is always parallel to the flat ground.

[0119] When the operation terminal 100 is rotated in this state, the line-of-sight direction L changes in the same manner as the operation of the line-of-sight direction L described above, and the range of the region A1 changes. On the other hand, the horizontal direction of the reticle direction R always points to the center and relatively moves with respect to the image of the region A1. The vertical direction of the reticle direction R points in a direction parallel to the flat ground. At this time, the front mark moves according to the rotation amount of the operation terminal 100.

[0120] For example, as shown in FIG. 14, when the operation terminal 100 is slid (rotated) to the right while the screen is touched by two fingers, the image displayed on the operation display unit 105 moves to the right. At this time, the reticle always remains at the center in the horizontal direction of the screen. That is, the image that has moved to the operation display unit 105 moves (the image flows) as the operation terminal 100 slides (rotates), but the reticle does not move from the center of the screen. The vertical position of the reticle is at the same position as the line-of-sight direction L and depends on the position of the image in the area being viewed at that time. As a result, the reticle will move in the direction of the moved image. At this time, the forward mark is displayed on the left side of the screen.

[0121] Then, when the moving body 300 moves forward, it moves in the direction indicated by the reticle direction R, that is, it curves to the right. Along with this movement, the forward mark moves toward the center on the screen so as to approach the reticle.

[0122] Such an operation of changing the reticle direction R can also be performed while the moving body 300 is moving. Therefore, the remote operator can move the moving body 300 forward while curving it left and right by simply changing the orientation of the operation terminal 100 in the same way as operating the steering wheel of a vehicle. Also, by this operation, the remote operator can easily perform fine operations on the traveling direction by simply moving the operation terminal 100 so as to capture the direction in which they want to proceed at the center of the screen.

[0123] The control unit 102 releases the linkage between the screen lock and the reticle at the moment when a finger is lifted from the screen of the operation display unit 105. When the linkage between the screen lock and the reticle is released, the image from the camera changes as the moving body 300 moves, but the line-of-sight direction L and the reticle direction R do not change. Then, when the screen is next touched with two fingers, the control unit 102 displays the positions of the line-of-sight direction L and the reticle direction R at the moment when the finger was lifted last time, with the image of the orientation of the operation terminal 100 at the moment of touching, and sets it to the screen lock state.

[0124] Next, the operation of accelerating and decelerating the moving body 300 will be described.

[0125] Acceleration and deceleration of the moving body 300 are performed by the third input.

[0126] FIG. 15 is an explanatory diagram for explaining the acceleration and deceleration operations of the moving body 300.

[0127] The third input is performed, for example, by an operation (drag) of sliding while touching the screen with one finger.

[0128] As shown in FIG. 15, when dragged upward on the screen, the control unit 102 transmits an instruction (control signal) for accelerating the moving body 300. When the moving body 300 is in a stopped state and is dragged upward on the screen, the moving body 300 starts (moves forward).

[0129] Conversely, when dragged downward on the screen, the control unit 102 transmits an instruction (control signal) for decelerating the moving body 300. When the moving body 300 is moving at a low speed and is dragged downward on the screen, the moving body 300 stops.

[0130] The acceleration and deceleration operations are executed by various methods in addition to using one finger as described above. Other acceleration and deceleration operations are executed using two fingers as the third input, for example.

[0131] FIG. 16 is an explanatory diagram for explaining other acceleration and deceleration operations of the moving body 300.

[0132] As shown in FIG. 16, other acceleration and deceleration operations are performed by moving one finger upward or downward on the screen (dragging) while the screen is touched with two fingers. When dragged upward on the screen, the control unit 102 sends an instruction (control signal) to accelerate the moving body 300. Note that when the moving body 300 is stopped and dragged, the moving body 300 starts moving (advancing). Conversely, when dragged downward on the screen, the control unit 102 sends an instruction (control signal) to decelerate the moving body 300. Note that when the moving body 300 is moving at a low speed and dragged downward on the screen, the moving body 300 stops.

[0133] Acceleration and deceleration operations can also be performed in various other ways. Although not shown, for example, the speed may be specified by the amount of dragging (the amount of movement from the position where the finger first touched the screen). Also, for example, when holding the smartphone, tilting it away from the remote operator in the upward direction performs a forward and acceleration operation, and tilting it toward the remote operator in the upward direction performs a deceleration and stop operation. Additionally, for example, icons such as forward, stop, backward, acceleration, and deceleration may be displayed on the screen, and corresponding operations may be performed when they are touched.

[0134] Next, an operation for reversing the moving direction of the moving body 300 will be described.

[0135] The reversal of the moving direction of the moving body 300 is performed by a fourth input.

[0136] FIG. 17 is an explanatory diagram for explaining an operation for reversing the moving direction of the moving body 300. FIG. 18 is an explanatory diagram for explaining an omnidirectional image when the moving direction of the moving body 300 is reversed.

[0137] The fourth input is performed, for example, as shown in FIG. 17, by a motion (flick) in which the finger moves rapidly upward or downward on the screen and then leaves the screen while the screen is touched with one finger.

[0138] When flicked, the control unit 102 transmits a control signal for reversing the traveling direction of the moving body 300 by 180 degrees. At this time, the displayed image is changed as follows. For the all-round image after reversal, as shown in FIG. 18, when the spherical center point C is included and the direction obtained by reversing the surface S that intersects perpendicularly to the traveling direction F by 180 degrees is taken as the traveling direction F', this traveling direction F' becomes the new front of the moving body 300. That is, when there is a distinction between the front and rear of the moving body 300, after reversal, the rear of the moving body 300 becomes the front, and it will move backward.

[0139] Then, the control unit 102 flattens the region A' that is mirror-symmetric to the region A with respect to the surface S and displays it on the operation display unit 105. At this time, the center O moves to the center O', and the line-of-sight direction L will face the direction of the line-of-sight direction L'. Also, the displayed reticle direction R does not change until a movement operation of the reticle direction R is performed by the remote operator. The control unit 102 recalculates the positional relationship between the reticle direction R and the region A' and displays the reticle at a new position.

[0140] Also, when the moving body 300 is moving and is flicked, the control unit 102 decelerates and stops the moving body 300. When the reticle is within the region A and is flicked, the position of the reticle will be outside the range of the new region A'. In FIG. 18, for example, when the reticle is in the lower left within the region A, after the operation of reversing by flicking is executed, the reticle moves to the right end (lower side) of the region A'. When moving forward in this state, the moving body 300 moves in a U-turn to the right. Usually, after being flicked, the moving body 300 is advanced after resetting the position of the reticle (that is, the direction of the moving target).

[0141] Next, the zoom operation will be described.

[0142] The zoom operation is performed by the fifth input.

[0143] FIG. 19 is an explanatory diagram for explaining the zoom operation.

[0144] The fifth input is performed, for example, as shown in FIG. 19, by an operation of widening the space between the fingers while touching the screen (pinch out), or an operation of narrowing the space between the fingers while touching the screen (pinch in).

[0145] When pinch-out is performed, the control unit 102 determines the position where pinch-out is performed in the line-of-sight direction L, and makes the horizontal direction of the reticle direction R coincide with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the region A. Then, the control unit 102 enlarges (zooms in) the image displayed on the screen of the operation display unit 105 according to the amount by which the fingers are widened by the pinch-out operation. Further, the control unit 102 causes the storage unit 103 to store the enlarged image at the moment when the fingers for pinch-out are released. Thereafter, the control unit 102 returns the image displayed on the screen of the operation display unit 105 to the image at the magnification before pinch-out. Subsequently, the control unit 102 advances the moving body 300 until the image received from the moving body 300 has the same size as the enlarged image at the time of pinch-out stored earlier.

[0146] On the other hand, when pinch-in is performed, the control unit 102 determines the position where pinch-in is performed in the line-of-sight direction L, and makes the horizontal direction of the reticle direction R coincide with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the region A. Then, the control unit 102 reduces (zooms out) the image displayed on the screen of the operation display unit 105 according to the amount by which the fingers are narrowed by the pinch-in operation. Further, the control unit 102 causes the storage unit 103 to store the reduced image at the moment when the fingers for pinch-in are released. Thereafter, the control unit 102 returns the image displayed on the screen of the operation display unit 105 to the image at the magnification before pinch-in. Subsequently, the control unit 102 retreats the moving body 300 until the image received from the moving body 300 has the same size as the enlarged image at the time of pinch-in stored earlier.

[0147] As described above, the zoom operation is performed in various ways in addition to the pinch operation. Other zoom operations are performed, for example, using one finger as the fifth input.

[0148] FIG. 20 is an explanatory diagram for explaining other zoom operations.

[0149] The fifth input of the other zoom operation is, for example, a long press on the screen with one finger as shown in FIG. 20.

[0150] When long-pressed, the control unit 102 determines the long-pressed position in the line-of-sight direction L and aligns the horizontal direction of the reticle direction R with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the region A. Then, while the long press is being performed, the control unit 102 advances the moving body 300, and when the finger performing the long press is released from the screen, the control unit 102 stops the moving body 300. Therefore, the moving body 300 advances in that direction while being long-pressed. For this reason, to the remote operator, the object shown in the image appears to be enlarged.

[0151] Among the zoom operations as described above, operations such as pinch-out or long press are used, for example, when the object is far from the moving body 300 and difficult to see. For example, when the characters written on a distant wall are small and difficult to read, the remote operator can perform a zoom-in operation, not only enlarging the image but also bringing the moving body 300 closer to the wall. This makes it easier for the remote operator to read characters and the like.

[0152] On the other hand, the zoom-out operation is used, for example, when there is an object near the moving body 300 and the whole cannot be seen. For example, when there is an object that is too close to the moving body 300 and the whole cannot be seen, the remote operator can perform a zoom-out operation, not only reducing the image but also moving the moving body 300 away from the object. This enables the remote operator to see the whole object.

[0153] During such a zoom operation, for example, the color of the reticle may be changed. For example, it may be changed to green in the case of zooming in and to red in the case of zooming out. Of course, any color may be used. Also, characters may be displayed, such as during zooming in or during zooming out.

[0154] Note that during the movement by the zoom operation, if no operation is performed, the reticle direction R does not change. However, as the moving body 300 moves, the direction of viewing the object captured by the omnidirectional camera 320 also changes. For this reason, the enlarged image at the time of zooming in or the reduced image at the time of zooming out does not exactly match the image after the movement. Also, the amount of movement (the amount of forward or backward movement) during zooming in or zooming out is an approximate value calculated by triangulation.

[0155] Next, collision detection will be described.

[0156] FIG. 21 is an explanatory diagram for explaining collision detection.

[0157] As shown in FIG. 21, collision detection is performed by displaying, on the operation display unit 105, in addition to the display of the region A, an image of a region B centered on the reticle direction R in which the moving body 300 is about to move. The region B is a predetermined range centered on the reticle direction R. The region B includes a range different from that of the region A. This region B is displayed on the operation display unit 105 as a second image.

[0158] The moving body 300 transmits a region B obtained by cutting out a predetermined range centered on the reticle direction R to the operation terminal 100. The control unit 102 flattens the received region B and displays it on the operation display unit 105.

[0159] The remote operator can view the image in the direction in which the moving body 300 is about to travel. For this reason, the remote operator can perform the operation while checking for the presence of a person and / or an object in the direction in which the moving body 300 is about to travel, or whether the person and / or the object does not move.

[0160] Alternatively, an AI model for detecting people and objects may be pre-installed in the control unit 102 to detect people and / or objects from the image of area B. Thereby, when a person or an object is detected in the reticle direction R by the AI model, the control unit 102 stops the moving body 300.

[0161] Also, a TOF sensor used in LiDAR or the like may be installed, and information on the reticle direction R may be notified to such a sensor to detect an object in that direction. Thereby, when a person or an object is detected in the reticle direction R by the sensor, the control unit 102 stops the moving body 300.

[0162] In this way, by setting and displaying in the operation display unit 105 not only area A but also area B, which is a narrower range than area A in the direction of movement, the remote operator can easily detect people and objects at the destination of movement. Also, by setting such an area B, when detecting people and objects by an AI, a TOF sensor, etc., it is possible to detect people and objects from a small amount of data with a narrowed detection range. For this reason, even with a small processing capacity, highly accurate detection and a collision avoidance operation based on it can be performed at high speed.

[0163] Note that although the above-described area B has a smaller range than area A, it is not limited to this, and area B may have the same size as area A or may be larger than area A. Also, the display (second image) of this area B may not be provided.

[0164] Next, the details of the movement control will be described.

[0165] FIG. 22 is a vector diagram related to the movement control. FIG. 22 shows a three-dimensional coordinate system.

[0166] The vectors related to movement control are, as already explained, the line-of-sight direction L, the traveling direction F, and the reticle direction R. Each of these directions can be represented by a vector in the polar coordinate system, as shown in Fig. 22. Here, only the direction of the vector is necessary. Therefore, each direction can be represented by a vector P(Θ,φ) that is a function of the declination angles Θ and φ, with the length r of the vector being a constant.

[0167] In the initial state, the line-of-sight direction L, the traveling direction F, and the reticle direction R coincide. Setting Θ = C0 and φ = 0, the line-of-sight direction L, the traveling direction F, and the reticle direction R are all the vector P0(C0,0). Here, C0 is the initial value of the inclination of the moving body 300 with respect to the flat ground. The inclination of the moving body 300 can be obtained by the acceleration sensor and gyro sensor mounted on the moving body 300. This coordinate system is fixed to the omnidirectional camera 320 mounted on the moving body 300, with the front face of the moving body 300 fixed to the x-axis of the coordinate system. The traveling direction F is always at φ = 0, and Θ is the inclination of the moving body 300 with respect to the flat ground.

[0168] Fig. 23 is a vector diagram for explaining the case when the moving body 300 is inclined.

[0169] For example, when the moving body 300 is on a slope with an inclination angle α and the deflection angle Θ of the line-of-sight direction L is ΘL, the line-of-sight direction L is looking in the direction of the elevation angle 90° - ΘL + α with respect to the flat ground (G (horizontal plane) in FIG. 23). In this case, even if the moving body 300 moves on the slope and the inclination angle α changes, the deflection angle Θ will not change unless the line-of-sight direction L or the reticle direction R is moved. Therefore, even if the inclination angle α changes, the coordinate system on the spherical surface of the image captured by the omnidirectional camera 320 does not change, and the image shown therein will change. That is, as can be seen by comparing FIGS. 22 and 23, it means that the vector P(Θ, φ) in the coordinate system of the moving body 300 does not change even if the moving body 300 tilts. However, the reticle direction R always points in a direction horizontal to the flat ground G. For this reason, the reticle direction R is corrected by the deflection angle Θ corresponding to the vertical component of the inclination due to the movement of the moving body 300. The correction of the reticle direction R is performed by the movement control terminal 330.

[0170] On the other hand, the deflection angle φ is updated one by one in accordance with the rotation (change in azimuth angle) of the moving body 300. FIGS. 24 and 25 are vector diagrams for explaining the case when the direction of the moving body 300 changes. FIGS. 24 and 25 show the x-y two-dimensional coordinates among the three-dimensional coordinate systems shown in FIG. 22.

[0171] For example, if the moving body 300 rotates by the azimuth displacement ω = β as shown in FIG. 25 from the state of the deflection angle φ1 as shown in FIG. 24, the deflection angle φ2 at that time is represented by the following equation (1).

[0172] φ2 = φ1 - β (1) Here, the azimuth displacement ω is obtained by the acceleration sensor and gyro sensor mounted on the moving body 300.

[0173] Therefore, when the line-of-sight direction L and the reticle direction R are not being changed, for the vertical movement of the moving body 300, the image moves up and down, but horizontally, an image in a substantially constant direction is displayed.

[0174] Next, the rotation control of the mobile body 300 will be described.

[0175] The vector coordinates of the line-of-sight direction L are represented as L(ΘL, φL), and the vector coordinates of the reticle direction R are represented as R(ΘR, φR). Also, let the displacement of the azimuth angle of the mobile body 300 be ω. These values are managed by the movement control terminal 330 (computer). In the initial state, ΘL = 0, φL = 0, ΘR = 0, and φR = 0. ω is a value obtained from the acceleration sensor and gyro sensor of the mobile body 300 (displacement from the previous acquisition) and is updated at a fixed period. Also, φL and φR are updated simultaneously with this update. Therefore, the updated φL and φR are expressed by the following equations (2) and (3).

[0176] φL = φL - ω (2) φR = φR - ω (3) Also, when φR is not 0, the mobile body 300 is bent (or rotated) so that φR approaches 0. This is an operation to bend the mobile body 300 in the reticle direction R. Regarding how fast and how φR approaches 0, it depends on the performance of the mechanical control of the mobile body 300.

[0177] Figure 26 is a flowchart showing the rotation operation procedure. The control unit 102 executes a program created according to this rotation control procedure.

[0178] First, the control unit 102 sets ΘL = 0, φL = 0, ΘR = 0, φR = 0, and resets ω (S101).

[0179] Subsequently, the control unit 102 acquires ω and calculates φL = φL - ω and φR = φR - ω (S102).

[0180] Subsequently, the control unit 102 determines whether φR > 0 (S103). If φR > 0 (S103: YES), then subsequently, the control unit 102 transmits a control signal for bending the mobile body 300 to the left to the mobile body 300 (S104).

[0181] The control signal transmitted from the operation terminal 100 is received by the movement control terminal 330, directly interpreted there, and the moving body 300 is controlled. At this time, the control signal may include instructions other than rotation. For example, the control signal includes instructions such as acceleration and deceleration. The movement control terminal 330 executes those instructions (the same applies in the following processes).

[0182] After that, the control unit 102 determines whether the process has ended (S107). If the process has not ended (S107: NO), the control unit 102 returns to S101 and continues the process. Thereby, the operation of the moving body 300 is continuously executed. On the other hand, if the process has ended (S107: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

[0183] In the step of S103, if φR>0 is not satisfied (S103: NO), it is determined whether φR<0 (S105). If φR<0 (S105: YES), subsequently, the control unit 102 transmits a control signal for turning the moving body 300 to the right to the moving body 300 (S106).

[0184] After that, the control unit 102 determines whether the process has ended (S107). Also, in the step of S105, if φR<0 is not satisfied (S105: YES), the control unit 102 also determines whether the process has ended (S107). If the process has not ended (S107: NO), the control unit 102 returns to S101 and continues the process. Thereby, the operation of the moving body 300 is continuously executed. On the other hand, if the process has ended (S107: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

[0185] Next, the movement control in the line-of-sight direction L will be described.

[0186] As already described, when the orientation of the operation terminal 100 (smartphone) is changed, the acceleration sensor and gyro sensor within the operation terminal 100 detect the direction and amount of rotation of the operation terminal 100. Since this amount can be represented by the declination angles Θ and φ, this amount is directly reflected as the amount of rotation of the viewing direction of the omnidirectional camera 320.

[0187] Specifically, the displacement angles Θd and φd from the moment of being touched by one finger are transmitted from the operation terminal 100 to the moving body 300. After a certain period of time, the displacement angles Θd and φd from the previous transmission time are transmitted to the moving body 300. Thereafter, this operation is repeated at short intervals until one finger is released.

[0188] The movement control terminal 330 receives the displacement angles Θd and φd, and calculates and updates the coordinates L(ΘL, φL) of the line-of-sight direction L according to the following equations (4) and (5).

[0189] ΘL = ΘL + Θd (4) φL = φL + φd (5) Then, the movement control terminal 330 cuts out a region A within a predetermined range centered on the coordinates L(ΘL, φL) (i.e., the line-of-sight direction L) and transmits it to the operation terminal 100. The control unit 102 flattens and displays the region A.

[0190] FIG. 27 is a flowchart showing the movement control procedure of the line-of-sight direction L. The control unit 102 executes a program created according to this movement control procedure of the line-of-sight direction L.

[0191] First, the control unit 102 determines whether the operation display unit 105 has been touched with one finger (S201). If the operation display unit 105 has been touched with one finger (S201: YES), then subsequently, the control unit 102 recognizes that the operation display unit 105 has been touched with one finger and resets the declination angles Θ and φ (S202).

[0192] Subsequently, the control unit 102 determines whether the operation display unit 105 is being touched with one finger (S203). If the operation display unit 105 is being touched with one finger (S203: YES), subsequently, the control unit 102 acquires Θd and φd and transmits them to the moving body 300 (S204).

[0193] On the side of the moving body 300 that has received Θd and φd, ΘL = ΘL + Θd and φL = φL + φd are set, and a new area A is transmitted to the operation terminal 100 (S205). This step of S205 is a process in the movement control terminal 330.

[0194] The control unit 102 that has received the new area A flattens the received area A and displays it on the operation display unit 105 (S206).

[0195] After that, the control unit 102 determines whether the process is finished (S207). Also, in the step of S201, if the operation display unit 105 is not being touched with one finger (S201: NO), and in the step of S203, if the finger has left the operation display unit 105 (S203: NO), the control unit 102 also determines whether the process is finished (S207). If the process is not finished (S207: NO), the control unit 102 returns to S201 and continues the process. Thereby, the operation of the moving body 300 is continuously executed. On the other hand, if the process is finished (S207: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

[0196] Note that the rotation control of the moving body 300 and the movement control in the line-of-sight direction L described above are performed simultaneously. Even during the operation of changing the line-of-sight direction L, the moving body 300 rotates. Therefore, the image captured by the omnidirectional camera 320 may change. Also, there is a delay time until the image captured by the omnidirectional camera 320 is displayed on the operation display unit 105. Considering these, it is preferable to perform the coordinate update process at a small frequency so as not to give a sense of discomfort to the remote operator.

[0197] Next, the movement control in the reticle direction R will be described.

[0198] The reticle direction R is represented by declination angles Θ and φ, similar to the case of the line-of-sight direction L, and the rotation amount of the operation terminal 100 is reflected in the rotation amount of the reticle direction R.

[0199] Specifically, first, when the operation display unit 105 of the operation terminal 100 is touched with two fingers, the operation terminal 100 immediately transmits the fact that it has been touched to the moving body 300. The movement control terminal 330 aligns the coordinates of the line-of-sight direction L and the reticle direction R. That is, the movement control terminal 330 sets each coordinate as shown in the following equations (6) and (7).

[0200] ΘR = ΘL (6) φR = φL (7) Thereafter, the control unit 102 continuously calculates the displacement angles Θd and φd as the terminal device rotates and transmits them to the moving body 300. When the two fingers are released from the operation display unit 105, the operation terminal 100 transmits this to the moving body 300. The movement control terminal 330 calculates the coordinates L(ΘL, φL) and the coordinates R(ΘR, φR) as shown in the following equations (8) to (11) and updates them simultaneously.

[0201] ΘL = ΘL + Θd (8) ΘR = C (9) φL = φL + φd (10) φR = φL (11) In the equations, C is the rotation amount necessary to align the reticle direction R with the plane H parallel to the flat ground. This rotation amount C indicates how much and in which direction the moving body 300 is tilted from the plane parallel to the flat ground. How much and in which direction the moving body 300 is tilted from the plane parallel to the flat ground is measured, for example, by a gyro sensor of the moving body 300.

[0202] As a result, the direction in which the line-of-sight direction L is projected onto the plane H parallel to the flat ground becomes the reticle direction R. If the moving body 300 is parallel to the flat ground, then C = 90°.

[0203] Then, the movement control terminal 330 cuts out a region A within a predetermined range centered on the coordinates L(ΘL, φL) (i.e., the line-of-sight direction L) and transmits it to the operation terminal 100. The control unit 102 flattens and displays the region A. Also, the control unit 102 displays the reticle at the center of the screen.

[0204] FIG. 28 is a flowchart showing the movement control procedure in the reticle direction R. The control unit 102 executes a program created according to this movement control procedure in the reticle direction R.

[0205] First, the control unit 102 determines whether the operation display unit 105 is touched with two fingers (S301). If the operation display unit 105 is touched with two fingers (S201: YES), then subsequently, the control unit 102 recognizes that the operation display unit 105 is touched with two fingers and transmits ΘR = ΘL and φR = φL to the moving body 300 (S302).

[0206] Subsequently, the control unit 102 determines whether the operation display unit 105 remains touched with two fingers (S303). If the operation display unit 105 remains touched with two fingers (S303: YES), then subsequently, the control unit 102 acquires Θd and φd and transmits them to the moving body 300 (S304).

[0207] The moving body 300 that has received Θd and φd calculates each value as ΘL = ΘL + Θd, φL = φL + φd, ΘR = C, φR = φL, and transmits a new region A to the operation terminal 100 (S305). This step of S305 is a process executed in the movement control terminal 330.

[0208] The control unit 102 that has received the new region A flattens the received region A and displays it on the operation display unit 105, and also displays the reticle at the center of the screen (S306).

[0209] Thereafter, the control unit 102 determines whether or not the processing has ended (S307). Furthermore, if the operation display unit 105 is not touched with two fingers in step S301 (S301: NO), or if the fingers are removed from the operation display unit 105 in step S303 (S303: NO), the control unit 102 also determines whether or not the processing has ended (S307). If the processing has not ended (S307: NO), the control unit 102 returns to S301 and continues the processing. This allows the operation of the moving object 300 to continue. On the other hand, if the processing has ended (S307: YES), the control unit 102 ends the processing. The end of the processing is, for example, an end instruction from the user.

[0210] According to the present embodiment described above, the following effects are achieved.

[0211] In the remote control system of this embodiment, the moving body 300 has an omnidirectional camera 320 that captures images in all directions. This allows the remote operator to freely look in the direction he or she wants to see when moving the moving body 300 and operate it. Therefore, in this embodiment, position information of the moving body 300 is not required. This allows the remote operator to operate the moving body 300 while looking in the direction he or she wants to move it. Furthermore, in this embodiment, when the remote operator wants to look in the direction he or she wants to see, there is no need to perform an operation to physically change the imaging direction, such as changing the direction of the moving body 300 or changing the imaging direction of the camera. Therefore, in this embodiment, no operational delay occurs due to changing the imaging direction.

[0212] Furthermore, in this embodiment, even when moving along a complex route, it is not necessary to use a point-and-click device to specify multiple points as a route.

[0213] Furthermore, in this embodiment, the moving body 300 extracts an area to be displayed on the operation terminal 100 from an image captured in all directions and transmits the extracted area to the operation terminal 100. This reduces the amount of data to be transmitted, thereby reducing data delays in wireless communication.

[0214] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments.

[0215] In the above-described embodiment, it is assumed that a smartphone is used as the operation terminal 100, but an operation terminal 100 in which the display is separated from an operation unit (operation controller) equipped with an acceleration sensor and a gyro sensor may also be used.

[0216] In the above-described embodiment, operations are realized only by touch operations on the screen of the operation display unit 105 and by rotating the operation terminal 100 itself. However, the operation terminal 100 may be provided with an operation unit such as a game controller and receive multiple button operations to achieve more precise operations. The operation terminal 100 may also be realized by separating an operation unit capable of performing operations equivalent to touch operations on a smartphone and XR glasses equipped with an acceleration sensor and a gyro sensor. In other words, instead of the remote operator moving the smartphone with his / her hand, the remote operator wearing the XR glasses may be able to specify the line of sight direction and reticle direction by moving his / her head. The XR glasses are high-performance glasses that realize virtual reality (VR), augmented reality (AR), mixed reality (MR), and the like.

[0217] Alternatively, instead of the omnidirectional camera 320, the imaging unit may be configured to attach wide-angle cameras to the front, rear, left and right of the moving body 300 to capture images in all directions, and combine these images to obtain an omnidirectional image.

[0218] Furthermore, the input operations such as touching, dragging, and flicking in the above-described embodiments may be gesture operations that do not involve touching the screen. Conversely, these input operations may be performed by touching buttons or icons that correspond to the input operations and are displayed on the screen.

[0219] Furthermore, while the movement of the vehicle 300 has been described as changing its direction of travel (turning) by changing the angle of the front wheels, as in a conventional vehicle, the present invention can also be applied to a form in which the direction of travel is changed by a Mecanum wheel. In a case where the vehicle 300 rotates on the spot to face the direction of travel and then moves forward, as in the case of a Mecanum wheel, when an operation to change the reticle direction R is performed, the vehicle 300 rotates on the spot and can immediately align the direction of travel F with the reticle direction R. Furthermore, in a configuration in which the vehicle 300 moves diagonally in a straight line, as in the case of a Mecanum wheel, the vehicle 300 can move in any direction, so the direction of travel F can be considered to always coincide with the reticle direction R. Furthermore, in this case, the vehicle 300 can also move sideways, so a touch input operation on a smartphone indicating sideways movement may be added. For example, the vehicle 300 may be configured to move horizontally left (right) by dragging a finger left (right) on the smartphone.

[0220] In the above-described embodiment, the moving object 300 cuts out an image of a partial area from the omnidirectional area and transmits the image data to the operation terminal 100. However, the present invention is not limited to this. For example, the moving object 300 may transmit image data of the entire omnidirectional area to the operation terminal 100, and the operation terminal 100 may then cut out a partial area. In this case, the moving object 300 does not need to cut out an image. In such a case, for example, the moving object 300 may transmit the moving speed and moving direction to the operation terminal 100, and the omnidirectional camera 320 may directly transmit image data of the omnidirectional area to the operation terminal 100.

[0221] In addition, the program according to the present invention can also be realized by a dedicated hardware circuit. Further, this program can be provided by a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM (Read Only Memory), or can be provided online via a network 200 such as the Internet without relying on a recording medium. When provided online, this program is recorded on a recording medium such as a magnetic disk in a computer connected to the network 200.

[0222] In addition, the present invention can be variously modified based on the configurations described in the claims, and these are also within the scope of the present invention.

Explanation of Reference Numerals

[0223] 1 Remote operation system, 100 Operation terminal, 101 Communication unit, 102 Control unit, 103 Storage unit, 104 Sensor, 105 Operation display unit, 200 Network, 300 Mobile body, 310 Mobile body main body, 320 Omnidirectional camera, 330 Mobile body control terminal.

Claims

1. A mobile body, An operation terminal for remotely operating the mobile body by wireless communication, A remote operation system having, The mobile body has an imaging unit that images the surroundings of the mobile body over an omnidirectional area, The operation terminal is, An input unit, A display unit, At least a part of the omnidirectional area imaged by the imaging unit, and a first image corresponding to the line-of-sight direction input from the input unit is displayed on the display unit. At the same time, a first mark indicating the traveling direction of the mobile body and a second mark indicating the moving target direction of the mobile body are superimposed on the first image and displayed on the display unit. A control unit, Having, The mobile body has a detection unit that calculates acceleration and angular velocity, The control unit obtains a deviation angle with respect to the traveling direction from the output from the detection unit and the position of the moving target direction on the first image, and controls the movement of the mobile body. Remote operation system.

2. The imaging unit is an omnidirectional camera and is fixed to the mobile body, The control unit obtains the deviation angle between the traveling direction and the moving target direction of the mobile body by using the correspondence between the position of the moving target direction on the first image and the incident angle of the video from the moving target direction with respect to the lens of the omnidirectional camera. The remote operation system according to claim 1.

3. The operation terminal has sensors for detecting acceleration and angular velocity, The control unit further controls the movement of the mobile body based on the output from the sensors. The remote operation system according to claim 1 or claim 2.

4. The control unit moves the line-of-sight direction based on a first input input from the input unit and the acceleration and angular velocity detected by the sensors after the first input. The remote operation system according to claim 3.

5. The control unit moves the second mark to the center of the screen of the display unit based on a second input input from the input unit. The remote operation system according to claim 3 or 4.

6. The control unit controls the moving speed of the mobile body based on a third input input from the input unit. The remote operation system according to any one of claims 3 to 5.

7. The control unit reverses the moving direction of the mobile body based on a fourth input input from the input unit. The remote operation system according to any one of claims 3 to 6.

8. The remote operation system according to any one of claims 3 to 7, wherein the control unit zooms in or out the first image based on a fifth input input from the input unit.

9. The remote operation system according to any one of claims 1 to 8, wherein the first image is cut out on the moving body side and transmitted from the moving body to the operation terminal.

10. The remote operation system according to any one of claims 1 to 9, wherein the control unit displays, on the display unit, a second image corresponding to a moving target direction of the moving body in addition to the first image.

11. The remote operation system according to claim 10, wherein the control unit detects a person and / or an object in a region shown in the second image.

12. The remote operation system according to claim 10 or 11, wherein the second image is cut out on the moving body side and transmitted to the operation terminal.

13. A remote operation program for remotely operating a moving body from an operation terminal connected by wireless communication, the moving body having an imaging unit that images over an omnidirectional region, a step (a) of displaying a first image corresponding to an input line-of-sight direction, which is at least a part of the omnidirectional region imaged by the imaging unit; a step (b) of superimposing and displaying a first mark indicating a traveling direction of the moving body and a second mark indicating a moving target direction of the moving body on the first image; wherein the moving body has a detection unit that calculates an acceleration and an angular velocity, and further has a step (c) of obtaining a deviation angle with respect to the traveling direction from an output from the detection unit and a position on the first image of the moving target direction, and controlling the movement of the moving body. A remote operation program for causing a computer to execute.

14. The imaging unit is an omnidirectional camera and is fixed to the moving body. In the step (c), the deviation angle between the traveling direction and the moving target direction of the moving body is obtained using a correspondence relationship between a position of the moving target direction on the first image and an incident angle of an image from the moving target direction with respect to a lens of the omnidirectional camera. The remote operation program according to claim 13.

15. The operation terminal has sensors for detecting an acceleration and an angular velocity. In the step (c), the movement of the moving body is further controlled based on an output from the sensors. The remote operation program according to claim 13 or claim 14.

16. The remote operation program according to claim 15, wherein the step (c) moves the line-of-sight direction based on the first input and the acceleration and angular velocity detected by the sensor after the first input.

17. The remote operation program according to any one of claims 14 to 16, wherein the step (c) moves the second mark to the center of the screen on which the first image is displayed based on a second input.

18. The remote operation program according to any one of claims 14 to 17, wherein the step (c) controls the moving speed of the moving body based on a third input.

19. The remote operation program according to any one of claims 14 to 18, wherein the step (c) reverses the moving direction of the moving body based on a fourth input.

20. The remote operation program according to any one of claims 14 to 19, wherein the step (c) zooms in or out the first image based on a fifth input.

21. The remote operation program according to any one of claims 13 to 20, further comprising a step (d) of receiving the first image cut out on the moving body side from the moving body.

22. The remote operation program according to any one of claims 13 to 19, further comprising a step (e) of displaying a second image corresponding to the moving target direction of the moving body in addition to the first image.

23. The remote operation program according to claim 22, further comprising a step (f) of detecting a person and / or an object in the area shown in the second image.

24. The remote operation program according to claim 22 or 23, further comprising a step (g) of receiving the second image cut out on the moving body side from the moving body.

Images