Mobile device, its control device and control method, and program

Abstract

This improves convenience for users who operate mobile devices remotely. [Solution] The control device for a mobile body having a display device comprises a display control unit that displays an avatar on the display device, and a drive unit that drives the mobile body in response to a request from an external device, wherein the display control unit changes the orientation of the avatar's face in response to a request obtained from an external device.

Description

Technical Field

[0001] The present invention relates to a mobile object, a control device and a control method therefor, and a program.

Background Art

[0002] Various techniques for remotely controlling a mobile object have been proposed. Patent Document 1 describes a technique of transmitting an image captured by a camera of a mobile object to a remote smartphone, and allowing a remote user to operate the mobile object by viewing the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0004] It is useful for the user if people around the mobile object can recognize the direction that the user operating the mobile object from a distant location is viewing. One aspect of the present invention aims to improve the convenience of a user who operates a mobile object from a distant location.

Means for Solving the Problem

[0005] According to some embodiments, there is provided a control device for a mobile object having a display device, the control device comprising: display control means for displaying an avatar on the display device; and drive means for driving the mobile object in response to a request from an external device, wherein the display control means changes the direction of the face of the avatar in response to the request acquired from the external device.

Effects of the Invention

[0006] According to some embodiments, the convenience of a user who operates a mobile object from a distant location is improved.

Brief Description of the Drawings

[0007] [Figure 1] A block diagram illustrating an example configuration of a mobile object control system according to one embodiment. [Figure 2] A block diagram illustrating an example configuration of a mobile body according to one embodiment. [Figure 3] A block diagram illustrating an example configuration of an HMD according to one embodiment. [Figure 4A] A schematic diagram illustrating examples of changes to the display range and avatar according to certain embodiments. [Figure 4B] A schematic diagram illustrating examples of changes to the display range and avatar according to certain embodiments. [Figure 5] A schematic diagram illustrating examples of changes to the display range and avatar according to certain embodiments. [Figure 6] A flowchart illustrating an example of the operation of a mobile body according to some embodiments. [Figure 7] A flowchart illustrating an example of the operation of an HMD according to one embodiment. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined in any way. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0009] <Example of a mobile control system configuration> Referring to Figure 1, an example configuration of a mobile control system according to some embodiments will be described. The mobile control system in Figure 1 comprises a mobile body 100, a head-mounted display (HMD) 110, and a controller 111. The mobile body 100 is controlled based on operations performed by a user 120 on the HMD 110 and the controller 111. The mobile body 100 and the HMD 110 can communicate with each other via a network 130. The HMD 110 is an external device to the mobile body 100. The network 130 may be a personal area network (PAN), local area network (LAN), cellular network, private network, internet, other network, or any combination thereof. The HMD 110 and the controller 111 may communicate with each other via the network 130, or they may communicate directly via Bluetooth® or the like. The user 120 may be located far away from the mobile body 100.

[0010] The mobile body 100 comprises a body 101, wheels 102, a display device 103, and a camera 104. In the example shown in Figure 1, the mobile body 100 is a vehicle that moves on the ground using wheels 102. Alternatively, the mobile body 100 may be an aircraft that moves through the air (e.g., a drone), a robot that moves on the ground using two or more legs, or another form of mobile body. The following description will focus on the case where the mobile body 100 is a vehicle, but the following description applies similarly to other forms of mobile bodies.

[0011] The display device 103 displays the user 120's avatar 105. The avatar 105 includes at least a face. The avatar 105 may further include part or all of the torso. In the example in Figure 1, the portion of the avatar 105 above the chest is displayed. The mobile device 100 displays the face of the avatar 105 so as to match the direction of the user 120's face. For example, if the user 120's face is facing forward, the mobile device 100 displays the avatar 105 so as to face the front of the mobile device 100. If the user 120's face is facing left, the mobile device 100 displays the avatar 105 so as to face left of the mobile device 100. This allows people around the mobile device 100 to recognize which direction the user 120's face is facing. Furthermore, the mobile device 100 may display the avatar 105's line of sight so as to match the direction of the user 120's gaze.

[0012] The display device 103 may have a display surface facing only one direction, or it may have display surfaces facing multiple directions. The display surface facing multiple directions may face 180 degrees, 360 degrees, or other angles. The display surface facing multiple directions may consist of a single curved display surface, or it may consist of multiple planar or curved display surfaces.

[0013] If the display device 103 has a display surface facing only one direction, the direction of the avatar 105's face may be represented by the depiction of the avatar 105's face. If the display device 103 has display surfaces facing multiple directions, the direction of the avatar 105's face may be represented by the position in which the avatar 105's face is displayed. For example, by displaying a depiction of the avatar 105 facing forward on the left side of the display device 103, it may be represented that the avatar 105's face is facing the left side of the mobile body 100. Furthermore, the direction of the avatar 105's face may be represented by combining the display position of the avatar 105 and the depiction of the avatar 105. The same applies to representing the direction of the avatar 105's torso. In the following description, when the control device 201 of the mobile body 100 displays the avatar 105 on the display device 103, it may be said that the mobile body 100 displays the avatar 105.

[0014] Camera 104 captures the environment surrounding the mobile object 100. In the following description, the environment surrounding the mobile object 100 is referred to as the mobile object environment. Images of the mobile object environment are transmitted to the HMD 110. Camera 104 is capable of capturing images with a field of view larger than the field of view of the images displayed on the HMD 110. For example, camera 104 may be an omnidirectional camera. In this case, camera 104 captures omnidirectional images. Alternatively, camera 104 may be equipped with a fisheye lens and capture wide-angle images.

[0015] The images of the mobile environment may be still images or moving images. For example, the mobile device 100 may sequentially transmit each frame of a moving image captured by the camera 104 at a predetermined frame rate to the HMD 110.

[0016] The HMD110 is worn on the head of user 120. The HMD110 displays a portion of the image of the moving environment. User 120 changes the orientation of the HMD110 by moving their own head. In response to the change in orientation, the HMD110 changes the portion of the image of the moving environment that is displayed. This allows user 120 to view a desired portion of the moving environment.

[0017] The controller 111 acquires an instruction from the user 120. The instruction from the user 120 includes an instruction related to movement of the mobile object 100. The instruction related to movement of the mobile object 100 may include an instruction to rotate the mobile object 100 in place (hereinafter referred to as a rotation instruction) and an instruction to cause the mobile object 100 to travel straight (hereinafter referred to as a straight travel instruction). Further, the instruction related to movement of the mobile object 100 may include an instruction to move the mobile object 100 backward and an instruction to move the mobile object 100 along a curved line.

[0018] <Configuration Example of Mobile Object> A configuration example of the mobile object 100 will be described with reference to Figure 2. The mobile object 100 includes the components shown in Figure 2. The mobile object 100 may include components not shown in Figure 2, or may not include some of the components shown in Figure 2.

[0019] The control device 201 controls the operation of the mobile object 100. The control device 201 includes a processor 202 and a memory 203. The control device 201 including the processor 202 and the memory 203 may be regarded as a computer. The control device 201 may be configured by an ECU (Electronic Control Unit). The processor 202 may be configured by a CPU (Central Processing Unit). The memory 203 may be configured by a combination of a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 203 stores programs executed by the processor 202, data used by the processor 202, and the like. The control device 201 may include only one processor 202, or may include a plurality of processors 202 that can communicate with each other. The same applies to the memory 203.

[0020] The processor 202 implements multiple functions, including a driving control unit 204, an avatar control unit 205, and an HMD communication unit 206, by executing a program stored in the memory 203. The driving control unit 204 controls the movement of the mobile body 100. For example, the driving control unit 204 rotates the mobile body 100 in accordance with a rotation request from the HMD 110 and moves the mobile body 100 in a straight line in accordance with a straight line request from the HMD 110. The avatar control unit 205 controls the display of the avatar 105 on the display device 103. The HMD communication unit 206 communicates with the HMD 110. For example, the HMD communication unit 206 transmits images captured by the camera 104 to the HMD 110 and receives requests for the mobile body 100 from the HMD 110.

[0021] The driving unit 210 drives the mobile body 100 to move. The movement of the mobile body 100 may include moving in a straight line and turning in place. The movement of the mobile body 100 may also include moving backward and moving along curves. The driving unit 210 consists of wheels 102, a drive source (e.g., a motor), etc. The driving unit 210 may be a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, or other form.

[0022] The battery 211 supplies operating power to each component of the mobile unit 100. The battery 211 is a secondary battery, such as a lithium-ion battery.

[0023] Speaker 212 outputs sound towards the surroundings of the mobile device 100. For example, the control device 201 may output the voice of user 120 received from HMD 110 through speaker 212. Microphone 213 acquires sound from the surroundings of the mobile device 100. For example, the control device 201 may transmit the sound acquired by microphone 213 to HMD 110. People around the mobile device 100 can converse with user 120 through speaker 212 and microphone 213.

[0024] The communication device 214 is a device for communicating with other devices (e.g., the HMD 110) via the network 130. The communication device 214 may perform wired communication, wireless communication, or both. When performing wired communication, the communication device 214 may include a connector for connecting a cable and a signal processing circuit. When performing wireless communication, the communication device 214 may include an antenna and a signal processing circuit.

[0025] <Exemplary Configuration of HMD> An exemplary configuration of the HMD 110 and the controller 111 will be described with reference to Figure 3. The HMD 110 and the controller 111 include the components shown in Figure 3. The HMD 110 and the controller 111 may include components not shown in Figure 3, or may not include some of the components shown in Figure 3.

[0026] The control device 301 controls the operation of the HMD 110. The control device 301 has a processor 302 and a memory 303. The control device 301 having the processor 302 and the memory 303 may be regarded as a computer. The processor 302 may be configured by a CPU. The memory 303 may be configured by a combination of ROM and RAM. The memory 303 stores programs executed by the processor 302, data used by the processor 302, and the like. The control device 301 may have only one processor 302, or may have a plurality of processors 302 that can communicate with each other. The same applies to the memory 303.

[0027] The processor 302 implements multiple functions, including the display acquisition unit 304, the input acquisition unit 305, and the mobile device coordination unit 306, by executing a program stored in the memory 303. The display acquisition unit 304 controls the display on the HMD 110. For example, the display acquisition unit 304 displays a display range, which is a part of the image received from the mobile device 100, on the display device 311. The display acquisition unit 304 also changes the display range in the image, as will be described later. The input acquisition unit 305 acquires inputs made by the user 120 to the HMD 110 and the controller 111. Inputs by the user 120 include changes in the orientation of the HMD 110. The mobile device coordination unit 306 coordinates with the mobile device 100. For example, the mobile device coordination unit 306 receives data (for example, images captured by the camera 104) from the mobile device 100 and sends requests to the mobile device 100.

[0028] The battery 310 supplies operating power to each component of the HMD110. The battery 310 is a rechargeable battery, such as a lithium-ion battery.

[0029] The display device 311 displays an image to the user 120 wearing the HMD 110. The display device 311 may be a display type, a projection type, or any other type. In the following description, when the control device 301 of the HMD 110 displays an image on the display device 311, it may be said that the HMD 110 displays an image.

[0030] Speaker 312 outputs sound towards the surrounding area of ​​the HMD 110 (in particular, towards the user 120). For example, the control device 301 may output the audio received from the mobile device 100 through speaker 312. Microphone 313 acquires sounds from the surrounding area of ​​the HMD 110 (in particular, the voice of the user 120). For example, the control device 301 may transmit the sound acquired by microphone 313 to the mobile device 100. The user 120 can converse with people around the HMD 110 through speaker 312 and microphone 313.

[0031] The communication device 314 is a device for communicating with other devices (e.g., HMD 110) via the network 130. The communication device 314 also communicates with the controller 111. The communication device 314 may perform wired communication, wireless communication, or both. When performing wired communication, the communication device 314 may include a connector for connecting a cable and a signal processing circuit. When performing wireless communication, the communication device 314 may include an antenna and a signal processing circuit.

[0032] The attitude sensor 315 is a sensor for determining the attitude of the HMD 110. The attitude sensor 315 may consist of, for example, an accelerometer, a gyroscope, a geomagnetic sensor, etc. In lieu of or in addition to the measurements taken by the attitude sensor 315, the control device 301 may determine the attitude of the HMD 110 based on changes in the surrounding image captured by the camera 316. The attitude of each component fixed to the HMD 110 changes in the same way as the attitude of the HMD 110.

[0033] Camera 316 captures images of the area around the HMD 110. For example, camera 316 may capture images in front of the HMD 110. The control device 301 may obtain user 120's instructions based on the user 120's hand gestures captured by camera 316.

[0034] Next, the components of the controller 111 will be described. The communication device 320 is a device for communicating with the HMD 110. The battery 321 supplies operating power to each component of the controller 111.

[0035] The input device 322 is a device for obtaining input from the user 120. The input device 322 may consist of, for example, hardware buttons or a joystick. The controller 111 transmits the input from the user 120 to the input device 322 to the HMD 110.

[0036] The attitude sensor 323 is a sensor for determining the attitude of the HMD 110. The attitude sensor 323 may consist of, for example, an accelerometer, a gyroscope, or a geomagnetic sensor. The controller 111 transmits the attitude of the attitude sensor 323 to the HMD 110. Based on the attitude of the controller 111, the control device 301 may identify the position that the user 120 is pointing to in the image displayed on the display device 311.

[0037] HMD110 is an example of a display system having a control device 301 and a display device 311 controlled by the control device 301. In HMD110, the control device 301 and the display device 311 are mounted in the same device. Alternatively, the control device 301 and the display device 311 may be mounted in separate devices. For example, the processing of the control device 301 described below may be performed by a control device included in the controller 111, or by another device different from HMD110 and the controller 111 (e.g., an external server).

[0038] <Examples of changing the display range and avatar> Referring to Figures 4A to 5, examples of changes to the display range of the HMD 110 and changes to the avatar 105 based on operations by user 120 will be explained. Situations 400, 410, 420, and 430 in Figures 4A and 4B show the mobile body 100 and HMD 110 viewed from above and the avatar 105 viewed from the front of the mobile body 100, respectively, at a specific point in time. Arrow 401 represents the direction of user 120's face relative to the ground. Arrow 402 represents the direction of the front of the mobile body 100 relative to the ground. Arrow 403 represents the direction of avatar 105's face relative to the ground.

[0039] Image 404 represents an image of the mobile environment captured by the camera 104 of the mobile body 100. As described above, Image 404 is transmitted from the mobile body 100 to the HMD 110. In the examples of Figures 4A and 4B, Image 404 is an omnidirectional image. Figures 4A and 4B show the intersection (circle) of Image 404 (omnidirectional image) with the horizontal plane. When the HMD 110 receives a moving image from the mobile body 100, Image 404 represents the latest frame at each point in time. The display range 405 represents the portion of Image 404 that is displayed on the display device 311. The display range 405 is a part of Image 404. The display range 405 is a two-dimensional region. Figures 4A and 4B show the intersection (arc) of the display range 405 with the horizontal plane.

[0040] In situation 400, user 120 is facing forward relative to the torso. In the following description, user 120 operates the HMD 110 and controller 111 without moving the torso (i.e., without changing the direction of the torso). HMD 110 stores the orientation of HMD 110 when user 120 is facing forward as the initial orientation. HMD 110 determines the direction that user 120's face is facing based on the change from the initial orientation of HMD 110. HMD 110 requests mobile body 100 to orient the face and torso of avatar 105 toward the front of mobile body 100 when user 120 is facing forward. Mobile body 100 displays avatar 105 in accordance with this request, such that the face and torso of avatar 105 are facing toward mobile body 100.

[0041] The HMD 110 determines the position of the image display range 405 so that when the user 120 is facing forward, the portion of the moving environment that is in front of the moving object 100 is displayed on the display device 311. In situation 400, assume that object A is in front of the moving object 100 (i.e., in the direction of arrow 402). In this case, object A is displayed in the center of the display range 405. Also, in situation 400, assume that object B is located 45 degrees to the left of the moving object 100. Object B is displayed on the left side within the display range 405.

[0042] Referring to Figure 4A, an example of how user 120 changes the portion of the mobile environment displayed on the display device 311 by changing the orientation of the HMD 110 will be described. In the following description, the portion of the mobile environment displayed on the display device 311 will be referred to as the display portion of the mobile environment. Assume that user 120 rotates their face 45 degrees to the left (counterclockwise when viewed from above) in order to view object B from the front. Situation 410 shows the mobile body 100 and HMD 110 as viewed from above after user 120 has rotated their face 45 degrees to the left. HMD 110 detects the change in the orientation of user 120's face based on the change in the orientation of HMD 110 measured by the orientation sensor 315. In response to the change in the orientation of user 120's face, HMD 110 moves the display range 405 in image 404 to the left. As a result, object A moves to the right side within the display range 405, and object B moves to the center of the display range 405.

[0043] Based on the fact that user 120 has rotated their face 45 degrees to the left from the front, HMD 110 requests mobile device 100 to rotate the face direction of avatar 105 by 45 degrees to the left. In accordance with this request, mobile device 100 changes the display of avatar 105 so that the face of avatar 105 is facing 45 degrees to the left from the front of mobile device 100.

[0044] In the operation of situation 410, the user 120's torso remains facing forward. Therefore, when the HMD 110 requests a change in the direction of the avatar 105's face, it does not request a change in the direction of the avatar 105's torso. Consequently, the mobile unit 100 maintains the direction of the avatar 105's torso facing forward.

[0045] In the example in Figure 4A, user 120 changes the direction of their face from facing forward to the left. Even if user 120 changes the direction of their face to another direction (for example, to the right, down, or up), the HMD 110 moves the display range 405 in accordance with the change in the direction of user 120's face, and the mobile body 100 changes the direction of the avatar 105's face in accordance with the change in the direction of user 120's face. As shown in situation 410, when user 120 changes the direction of their face, the direction of the mobile body 100 does not change.

[0046] Referring to Figure 4B, an example of how the user 120 can change the display portion of the mobile environment in a manner different from changing the posture of the HMD 110 will be described. For example, the user 120 may use the input device 322 (e.g., joystick) of the controller 111 to instruct the HMD 110 to change the display portion of the mobile environment. In addition to or instead of this, if the HMD 110 is capable of recognizing hand gestures, the user 120 may use hand gestures to instruct the HMD 110 to change the display portion of the mobile environment. In this method of instruction, the user 120 does not need to change the posture of the HMD 110.

[0047] Suppose user 120 instructs HMD 110 to rotate the display portion of the moving object environment 45 degrees to the left so that object B is displayed in the center of the display device 311. As shown in situation 420, HMD 110 moves the display range 405 of image 404 45 degrees to the left in response to user 120's instruction. As a result, the display device 311 displays the portion of the moving object environment that was rotated 45 degrees to the left from the portion displayed in situation 400. In the image displayed on the display device 311, object A is located on the right and object B is located in the center.

[0048] Since user 120's face is facing object B, HMD 110 requests mobile unit 100 to orient avatar 105's face toward object B. Also, since user 120's face remains facing the front of the torso, HMD 110 requests mobile unit 100 to orient avatar 105's torso toward object B, along with its face. In response to this request, mobile unit 100 changes the display of avatar 105 so that its face and torso face object B (i.e., rotates avatar 105's face and torso 45 degrees to the left from the state of situation 400). As a result, the object located in front of user 120 on HMD 110 (object B in situation 420) and the object located in front of avatar 105's face (object B in situation 420) coincide.

[0049] Through the above operation, user 120 can change the displayed portion of the mobile environment without changing the posture of the HMD 110. As a result, user 120 can easily see parts of the display that are difficult to see by changing the direction of the face (for example, the part behind the mobile object 100). Furthermore, since the face of avatar 105 is facing the part of the mobile environment that user 120 is looking at, people around the mobile object 100 can correctly recognize the direction that user 120 is looking.

[0050] In situation 420, the direction of the user's torso and the front of the mobile body 100 do not coincide. Therefore, if the user 120 moves the mobile body 100 in a straight line in this state, the user 120 will perceive the mobile body 100 as moving diagonally. Specifically, even though object B is visible to the user 120, the mobile body 100 moves in a straight line towards object A. In this case, the HMD 110 may request the mobile body 100 to face the portion of the mobile body environment displayed in the center of the display range 405.

[0051] As shown in situation 430, the mobile body 100 rotates to face object B in response to a request from the HMD 110 (i.e., rotates 45 degrees to the left). As the mobile body 100 rotates, the camera 104 mounted on the mobile body 100 also rotates, and the image 404 captured by the camera 104 also rotates. Therefore, the HMD 110 moves the display range 405 of the image so that object B remains displayed in the center (specifically, rotates 45 degrees to the right).

[0052] Furthermore, when the mobile body 100 rotates, the display device 103 mounted on the mobile body 100 also rotates, and the avatar 105 displayed on the display device 103 also rotates. Therefore, the mobile body 100 changes the display of the avatar 105 so that the face and torso of the avatar 105 continue to face object B (specifically, it rotates the direction of the face and torso of the avatar 105 by 45 degrees to the right).

[0053] As a result of the above operation, even if the mobile object 100 rotates, the display portion of the mobile object environment shown on the HMD 110 does not change, thus reducing the sense of unease experienced by the user 120. In addition, even if the mobile object 100 rotates, the orientation of the avatar 105's face and torso does not change, thus reducing the sense of unease experienced by people around the mobile object 100.

[0054] Referring to Figure 5, the horizontal axes of graphs 501 to 505, which illustrate the change in the position of the display range 405 and the change in the direction of the avatar 105's face from situation 400 to situation 430 in Figure 4B, all represent time. The vertical axis of graph 501 represents the direction of the front of the moving object 100 relative to the ground. The vertical axis of graph 502 represents the position of the display range 405 in image 404. The vertical axis of graph 503 represents the direction of the moving object environment displayed in the center of the display device 103 of the HMD 110. The vertical axis of graph 504 represents the direction of the avatar 105's face relative to the moving object 100. The vertical axis of graph 505 represents the direction of the avatar 105's face relative to the ground. In all graphs 501 to 505, the vertical axis represents the amount of rotation, with the position at time t0 being 0 and leftward rotation (counterclockwise when viewed from above) being positive.

[0055] Suppose that at time t0, the HMD 110 receives an instruction from user 120 to rotate the display portion of the mobile environment to the left by a target amount TGT. In response to this instruction, the HMD 110 moves the display range 405 in image 404 to the left by a target amount TGT. Also in response to this instruction, the HMD 110 requests the mobile device 100 to rotate the direction of the avatar 105's face and torso to the left by a target amount TGT. In response to this request, the mobile device 100 changes the display of the avatar 105 so that the direction of the avatar 105's face and torso is rotated to the left by a target amount TGT. Throughout time t0 to t1, the mobile device 100 ensures that the direction of the avatar 105's face and torso faces the portion of the mobile environment that is displayed in the center of the display range 405.

[0056] Assume that at time t1, the display range 405 has finished moving by the target amount TGT. Simultaneously, the direction of the avatar 105's face and torso also finishes moving by the target amount TGT. The movement speed of the display range 405 may be a pre-set speed or a speed instructed by the user 120. The movement speed of the display range 405 may be greater than the rotational speed of the moving object 100.

[0057] At time t2, the HMD 110 requests the moving object 100 to rotate to the left by a target amount TGT. The HMD 110 may make this request in response to instructions from the user 120. Alternatively, the HMD 110 may make this request after a predetermined time has elapsed since the end of movement within the display range 405. The moving object 100 begins to rotate in response to this request.

[0058] While the mobile body 100 is rotating, the HMD 110 moves to the right within the display range 405 at the same speed as the rotation speed of the mobile body 100. This maintains the orientation of the mobile body environment displayed in the center of the HMD 110's display device 103, as shown in Graph 503. Also, while the mobile body 100 is rotating, the mobile body 100 rotates the orientation of the avatar 105's face and torso to the right at the same speed as the rotation speed of the mobile body 100. This maintains the orientation of the avatar 105's face and torso relative to the ground, as shown in Graph 505.

[0059] Assume that at time t3, the moving object 100 has finished rotating by the target amount TGT. Accordingly, the HMD 110 finishes moving within the display range 405 and finishes changing the orientation of the avatar 105's face and torso.

[0060] In the example shown in Figure 5, the rotation amounts of the moving object 100, the rotation amounts of the display range 405, and the rotation amounts of the avatar 105's face are changed linearly. Alternatively, these rotation amounts may be changed curvely.

[0061] In the example shown in Figure 5, the moving body 100 begins to rotate after the display range 405 has finished moving by the target amount TGT. Alternatively, the moving body 100 may begin to rotate before the display range 405 has finished moving by the target amount TGT. For example, the moving body 100 may begin to rotate after the display range 405 has started to move, in response to instructions from the user 120, or after a predetermined amount of time has elapsed. Alternatively, the moving body 100 may begin to rotate simultaneously with the start of movement of the display range 405. If the moving body 100 begins to rotate while the display range 405 is moving toward the target value TGT, the direction of the moving body environment displayed in the center of the HMD 110's display device 103 will rotate by the amount of rotation of the moving body 100. Therefore, the HMD 110 may end its leftward rotation in response to the direction of the moving body environment displayed in the center of the HMD 110's display device 103 rotating by the target value TGT, and begin rotating to the right so that the direction of the moving body environment becomes constant. The same applies to the rotation of the avatar's face and torso.

[0062] <Operation Example of HMD110> With reference to FIG. 6, an example of a method by which the control device 301 controls the HMD 110 to operate the HMD 110 as described above will be described. Each step in FIG. 6 may be executed by the processor 302 executing a program stored in the memory 303. Alternatively, at least some steps of the method in FIG. 6 may be executed by a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit). The method in FIG. 6 is started, for example, in response to the start of cooperation between the mobile object 100 and the HMD 110. The method in FIG. 6 ends, for example, in response to the end of cooperation between the mobile object 100 and the HMD 110. It is assumed that the user 120 is wearing the HMD 110 on the head at the start point of the method in FIG. 6.

[0063] In S601, the control device 301 starts acquiring (receiving) the image 404 of the mobile object environment captured by the camera 104 of the mobile object 100. When the camera 104 captures a moving image, the control device 301 receives the frame from the HMD 110 every time a new frame is generated. The control device 301 displays a display range 405 which is a part of the acquired image 404 on the display device 311. At this point, the control device 301 determines the display range 405 of the image so that the portion of the mobile object environment that is in front of the mobile object 100 is displayed on the display device 311. The operation of S601 corresponds to the operation of the situation 400 in FIG. 4A and FIG. 4B.

[0064] In S602, the control device 301 determines whether the posture of the HMD 110 has been changed. When it is determined that the posture of the HMD 110 has been changed ("YES" in S602), the control device 301 shifts the process to S603, and otherwise ("NO" in S602), shifts the process to S604. The change in the posture of the HMD 110 may be detected by a measurement value from the posture sensor 315.

[0065] In S603, the control device 301 moves the display range 405 of the image 404 in response to a change in the posture of the HMD 110. Specifically, the control device 301 moves the display range 405 of the image 404 in accordance with the direction and amount of change (e.g., change angle) of the change in the posture of the HMD 110. The direction of change in the posture of the HMD 110 may be upward, downward, left, right, or in any direction between these. Also, when the control device 301 moves the display range 405 of the image 404 in response to a change in the posture of the HMD 110, it requests the mobile body 100 to change the direction of the avatar 105's face. The control device 301 may include the direction and amount of change (change angle) of the avatar 105's face in this request. In this step, the control device 301 does not request a change in the direction of the avatar 105's torso. The operation in S603 corresponds to the operation in situation 410 in Figure 4A.

[0066] In S604, the control device 301 determines whether an instruction has been received from the user 120 to change the portion of the mobile environment that is to be displayed on the display device 311, in a manner different from changing the orientation of the display device. An instruction to change the portion of the mobile environment that is to be displayed on the display device 311 is referred to as a change instruction. The portion of the mobile environment that is designated to be displayed on the display device 311 is referred to as a designated portion. If the control device 301 determines that a change instruction has been received in a manner different from changing the orientation of the display device (YES in S604), it proceeds to S605; otherwise (NO in S604), it proceeds to S608.

[0067] A change instruction may include the direction and amount of change (e.g., change angle) for displaying a specified portion of the mobile environment on the display device 311. The change instruction is an instruction to display a specified portion on the display device 311 that is different from the currently displayed portion of the display device 311. The change instruction may be performed by operations on the input device 322 of the controller 111 (e.g., the tilt direction and amount or duration of the joystick tilt), hand gestures (e.g., the direction and amount or duration of hand movement), changes in the posture of the controller 111, or specifying a position within the display range 405 of the image 404 (the specified position becomes the center of the specified portion). The direction specified in the change instruction may be the direction in which the mobile body 100 can rotate. For example, if the mobile body 100 can rotate to the left and to the right, the direction specified in the change instruction may be either to the left or to the right.

[0068] In S605, the control device 301, upon receiving a change instruction, moves the display range 405 of the image 404 by a specified amount in the direction specified by the change instruction so that the specified portion of the mobile environment is displayed on the display device 311.

[0069] Furthermore, when moving within the display range 405 of the image 404, the control device 301 requests the moving body 100 to change the orientation of the avatar 105's face and torso. The control device 301 may also include in this request the direction and amount (angle of change) of the avatar 105's face and torso. The operation in S605 corresponds to the operation in situation 420 of Figure 4B.

[0070] In S606, the control device 301 requests the mobile body 100 to face a designated portion of the mobile environment. The mobile body 100 rotates in response to this request to face the designated portion of the mobile environment. This request may be made after the designated portion of the mobile environment is displayed on the display device 311. Alternatively, this request may be made before the designated portion of the mobile environment is displayed on the display device 311 (for example, while the display range 405 is moving, or at the start of the movement of the display range 405).

[0071] In S607, while the mobile body 100 is rotating to face the designated portion, the control device 301 moves the display range 405 of image 404 back to its position before the movement in S605. Specifically, in this movement, the display range 405 is moved in the opposite direction to the direction specified by the change instruction. The control device 301 may also move the display range 405 of image 404 so that the designated portion of the mobile body continues to be displayed on the display device 311 after the designated portion of the mobile body environment is displayed on the display device 311, while the mobile body 100 is rotating to face the designated portion. The operation in S607 corresponds to the operation in situation 430 in Figure 4B.

[0072] In S608, the control device 301 determines whether an instruction to move the mobile body 100 has been received from the user 120. An instruction to move the mobile body 100 is referred to as a move instruction. If the control device 301 determines that a move instruction has been received (YES in S608), it proceeds to S609; otherwise (NO in S608), it proceeds to S602.

[0073] The movement instruction may include the direction and amount of movement of the mobile body 100. The movement instruction may be performed by operations on the input device 322 of the controller 111 (e.g., the tilt direction and amount or duration of the joystick tilt), hand gestures (e.g., the direction and amount or duration of hand movement), changes in the orientation of the controller 111, or specifying a position within the display range 405 of the image 404 (the mobile body 100 moves to the specified position). The direction specified in the movement instruction may be any direction in which the mobile body 100 can move. For example, if the mobile body 100 can move along the ground, the direction specified in the movement instruction may be any direction in the horizontal plane.

[0074] In S609, the control device 301 requests the mobile body 100 to move to a specified position. In this step, the control device 301 does not move the display range 405 of the image 404. Furthermore, the control device 301 does not request the mobile body 100 to change the orientation of the avatar 105's face and torso.

[0075] <Example of operation of mobile unit 100> Referring to Figure 7, an example of how the control device 201 controls the mobile body 100 to operate it as described above will be explained. Each step in Figure 7 may be performed by the processor 202 executing a program stored in memory 203. Alternatively, at least some steps of the method in Figure 7 may be performed by a dedicated integrated circuit such as an ASIC. The method in Figure 7 is started, for example, when the mobile body 100 and the HMD 110 begin coordinating. The method in Figure 7 is terminated, for example, when the mobile body 100 and the HMD 110 end coordinating. In the method in Figure 7, a change in the orientation of the avatar 105's face or torso refers to a change relative to the mobile body 100.

[0076] At S701, the control device 201 begins transmitting images 404 of the mobile environment of the mobile body 100, captured by the camera 104, to the HMD 110. When the camera 104 captures moving images, the control device 201 transmits each new frame to the HMD 110. The control device 201 also begins displaying the avatar 105. At this point, the control device 201 displays the avatar 105 so that its face and torso are facing the front of the mobile body 100. The operation at S601 corresponds to the operation in situation 400 in Figures 4A and 4B.

[0077] In S702, the control device 201 determines whether it has received a request from the HMD 110 to change the avatar 105. A request to change the avatar 105 is referred to as an avatar change request. If the control device 201 determines that it has received an avatar change request (YES in S702), it proceeds to S703; otherwise (NO in S702), it proceeds to S704. The avatar change request may also be a request to change the orientation of the avatar 105's face (the request in S603 in Figure 6). This request does not request to change the orientation of the avatar 105's torso. Alternatively, the avatar change request may be a request to change the orientation of both the avatar 105's face and torso (the request in S605 in Figure 6).

[0078] In S703, the control device 201 changes the avatar 105 displayed on the display device 103. Specifically, the control device 201 rotates the direction of the avatar 105's face in the direction specified in the avatar change request so that the avatar 105's face faces a different part of the mobile environment it is currently facing. If the avatar change request also requests a change in the direction of the avatar 105's torso, the control device 201 rotates the avatar 105's torso so that it faces the same direction as the avatar 105's face. This operation corresponds to the operation in situation 420 of Figure 4B. If the avatar change request does not request a change in the direction of the avatar 105's torso, the control device 201 maintains the direction of the avatar 105's torso facing the front of the mobile 100. This operation corresponds to the operation in situation 410 of Figure 4A.

[0079] In S704, the control device 201 determines whether it has received a request from the HMD 110 to rotate the mobile body 100. The request to rotate the mobile body 100 is referred to as a rotation request. If the control device 201 determines that it has received a rotation request (YES in S704), it proceeds to S705; otherwise (NO in S704), it proceeds to S706.

[0080] In S705, the control device 201 rotates the mobile body 100 in response to the rotation request. The rotation request may also be the request in S603 of Figure 6. That is, the rotation request may be a request to rotate the mobile body 100 so that it faces the part of the mobile body environment specified in the avatar change request. In this case, the control device 201 rotates the mobile body 100 so that it faces the specified part of the mobile body environment.

[0081] If, at the start of S705, the control device 201 is not facing the front of the mobile body 100, it may rotate the direction of the face and torso of the avatar 105 so that the face and torso of the avatar 105 are facing the front of the mobile body 100. In this case, while the mobile body 100 is rotating to face the designated part, the control device 201 rotates the face and torso of the avatar 105 in the opposite direction to the rotation direction of S703. Alternatively, after the face and torso of the avatar 105 are facing the designated part, the control device 201 may rotate the face and torso of the avatar 105 so that the face and torso of the avatar 105 continue to face the designated part while the mobile body 100 is rotating to face the designated part. Alternatively, after the face of the avatar 105 is facing the designated part, the control device 201 may rotate the mobile body 100 to face the designated part. The operation in S705 corresponds to the operation in situation 430 in Figure 4B.

[0082] In S706, the control device 201 determines whether it has received a request from the HMD 110 to move the mobile body 100. The request to move the mobile body 100 is referred to as a movement request. If the control device 201 determines that it has received a movement request (YES in S706), it proceeds to S707; otherwise (NO in S706), it proceeds to S702. The movement request may include the direction and distance of movement of the mobile body 100.

[0083] In S707, the control device 201 moves the mobile body 100 in response to a movement request. If the direction of movement specified in the movement request is in front of the mobile body 100, the control device 201 moves the mobile body 100 in a straight line for the specified distance. If the direction of movement specified in the movement request is not in front of the mobile body 100, the control device 201 rotates the mobile body 100 in the direction of movement and then moves the mobile body 100 in a straight line for the specified distance. In this step, the control device 201 does not change (maintain) the orientation of the avatar 105's face and torso.

[0084] In the above embodiment, an image 404 of the moving environment captured by the camera 104 is displayed on the display device 311 of the HMD 110. Instead of the HMD 110, a device not intended to be worn on the user's head (for example, a smartphone) may be used. The orientation of such a device may or may not be detectable. If the orientation of the device is not detectable, the operation utilizing orientation detection in the above embodiment is omitted.

[0085] <Summary of Embodiments> (Item 1) A control device (201) for a mobile body (100) having a display device (103), The display control means for displaying an avatar (105) on the display device, The system includes a drive means for driving the mobile body in response to a request from an external device, The display control means is a control device that changes the orientation (403) of the avatar's face in response to a request obtained from the external device. According to this section, people around the moving object can perceive the orientation of the user's torso in relation to the external device. This improves user convenience. (Item 2) The display control means is While the avatar's face is facing a first portion of the environment surrounding the mobile body, in response to a request received from the external device, the avatar's face is rotated in the first direction so that it faces a second portion of the environment. The control device according to item 1, which rotates the avatar's face in a second direction opposite to the first direction while the moving body is rotating so that it faces the second part. According to this section, it is possible to mitigate the unintended changes in the orientation of the avatar's face caused by the rotation of a moving object. (Item 3) The control device according to item 2, wherein the display control means rotates the avatar's face so that it continues to face the second part of the environment after the avatar's face has turned toward the second part of the environment, while the moving body rotates to face the second part. According to this section, it is possible to prevent the orientation of the avatar's face from unintentionally changing due to the rotation of the moving object. (Item 4) The control device according to item 2 or 3, wherein the driving means rotates the moving body so that it faces the second part after the face of the avatar faces the second part of the environment. According to this section, it is possible to suppress excessive rotation of the moving object. (Item 5) The control device according to any one of items 1 to 4, wherein the display control means changes the orientation of the avatar's torso in response to a request obtained from the external device. According to this item, people around the moving object can perceive the orientation of the torso of the user of the external device. (Item 6) The control device according to any one of items 1 to 5, wherein the display control means rotates the torso of the avatar so that it faces the same direction as the face of the avatar in response to a request from the external device. According to this item, people around the moving object can recognize when the user of the external device changes the direction they are looking while remaining facing forward. (Item 7) The control device according to any one of items 1 to 6, wherein the display control means maintains the orientation of the avatar's torso facing the front (402) of the moving body when it is not requested to change the orientation of the avatar's torso. According to this section, people around the moving object can recognize that the user of the external device is only moving their face. (Item 8) The aforementioned moving body is A camera (104) that photographs the environment around the aforementioned moving object, A control device according to any one of items 1 to 7, further comprising: a transmission means (214) for transmitting an image of the environment to an external device. According to this item, the user of the external device can perceive the environment surrounding the moving object. (Item 9) A mobile body equipped with a control device as described in any one of items 1 through 8. According to this section, a mobile device that achieves the above advantages is provided. (Item 10) A program for causing a computer to function as one of the control devices described in any one of items 1 through 8. According to this section, a program is provided that realizes the above advantages. (Item 11) A method for controlling a mobile body (100) having a display device (103), A display control step for displaying an avatar on the aforementioned display device, The system includes a drive step for driving the moving body in response to a request from an external device, The display control step is a control method that includes changing the orientation (403) of the avatar's face in response to a request obtained from the external device. According to this section, people around the moving object can perceive the orientation of the user's torso in relation to the external device. This improves user convenience.

[0086] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of Symbols]

[0087] 100 mobile devices, 105 avatars, 110 head-mounted displays, 111 controllers

Claims

1. A control device for a mobile body having a display device, The display control means for displaying an avatar on the aforementioned display device, The system includes a drive means for driving the mobile body in response to a request from an external device, The display control means is a control device that changes the orientation of the avatar's face in response to a request obtained from the external device.

2. The display control means is While the avatar's face is facing a first portion of the environment surrounding the mobile body, in response to a request received from the external device, the avatar's face is rotated in the first direction so that it faces a second portion of the environment. The control device according to claim 1, wherein while the moving body is rotating to face the second part, the face of the avatar is rotated in a second direction opposite to the first direction.

3. The control device according to claim 2, wherein the display control means rotates the avatar's face so that it continues to face the second part of the environment after the avatar's face has turned toward the second part of the environment, while the moving body is rotating to face the second part.

4. The control device according to claim 2, wherein the driving means rotates the moving body so that it faces the second part after the face of the avatar faces the second part of the environment.

5. The control device according to claim 1, wherein the display control means changes the orientation of the avatar's torso in response to a request obtained from the external device.

6. The control device according to claim 1, wherein the display control means rotates the torso of the avatar so that it faces the same direction as the face of the avatar in response to a request from the external device.

7. The control device according to claim 1, wherein the display control means maintains the orientation of the avatar's torso facing the front of the moving body when it is not requested to change the orientation of the avatar's torso.

8. The aforementioned moving body is A camera that photographs the environment around the aforementioned moving object, The control device according to claim 1, further comprising a transmission means for transmitting an image of the environment to an external device.

9. A mobile body comprising the control device according to any one of claims 1 to 8.

10. A program for causing a computer to function as one of the means of the control device described in any one of claims 1 to 8.

11. A method for controlling a mobile body having a display device, A display control step for displaying an avatar on the aforementioned display device, The system includes a drive step for driving the moving body in response to a request from an external device, The display control step is a control method that includes changing the orientation of the avatar's face in response to a request obtained from the external device.

Images