Image generating device and image generating method

The image generation device addresses the challenge of users outside vehicles grasping vehicle movements by generating and switching between viewpoint images based on vehicle behavior, providing an immersive experience.

JP7581093B2Active Publication Date: 2024-11-12NISSAN MOTOR CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021042758
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-16
Publication Date
2024-11-12
Estimated Expiration
2041-03-16

AI Technical Summary

Technical Problem

Existing virtual space display systems for vehicles struggle to allow users not in the vehicle to visually grasp the movement of the vehicle effectively.

Method used

An image generation device that acquires vehicle information, including behavior data, generates a first viewpoint image from a virtual perspective, and switches to a second viewpoint image based on vehicle behavior changes, displaying these images on a remote display device to reflect vehicle movements.

Benefits of technology

Enables users outside the vehicle to intuitively understand vehicle movements through visually immersive second viewpoint images, enhancing their perception of vehicle behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581093000001
    Figure 0007581093000001
  • Figure 0007581093000002
    Figure 0007581093000002
  • Figure 0007581093000003
    Figure 0007581093000003
Patent Text Reader

Abstract

To provide an image generation apparatus configured to allow a user who is not on a vehicle to visually identify the motion of the vehicle easily, and an image generation method.SOLUTION: An image generation apparatus includes: a vehicle information acquisition unit which acquires vehicle information including information on behavior of a vehicle from a device disposed on the vehicle; an image generation unit which generates an image of a view in a predetermined direction seen from a virtual viewpoint in a vehicle interior space of the vehicle, as a first viewpoint image; and a display control unit which outputs a control signal for displaying the image generated by the image generation unit on a display device arranged in a space different from the vehicle interior space. The image generation unit generates a second viewpoint image by changing a relative positional relationship between the virtual viewpoint and a position of a subject in the vehicle interior space included in the first viewpoint image, on the basis of the behavior of the vehicle.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image generating device and an image generating method. [Background technology]

[0002] A virtual space display system is known that displays a virtual space in which objects existing in real space are replaced with objects that do not exist in real space to the vehicle occupants as the vehicle moves, thereby preventing the occupants from feeling uncomfortable about the vehicle's movement (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-146963 A Summary of the Invention [Problem to be solved by the invention]

[0004] The virtual space display system described in Patent Document 1 has a problem in that, because it displays objects that do not exist in the real space, it is difficult for a user who is not in the vehicle to visually grasp the movement of the vehicle.

[0005] An object of the present invention is to provide an image generating device and an image generating method that allow a user who is not in a vehicle to easily grasp the movement of the vehicle through vision. [Means for solving the problem]

[0006] The present invention obtains vehicle information including information on the behavior of the vehicle from a device provided in the vehicle, and displays the image on a display device provided in a space different from the vehicle cabin. The image viewed in a predetermined direction from a virtual viewpoint in the vehicle cabin is generated as a first viewpoint image, and the relative positional relationship between the position of a subject in the vehicle cabin captured in the first viewpoint image and the virtual viewpoint is changed based on the behavior of the vehicle to generate a second viewpoint image.The system can switch between a first mode in which a first viewpoint image and a second viewpoint image are generated, and a second mode in which a first viewpoint image is generated but a second viewpoint image is not generated. When the behavior of the vehicle is constantly changing, the system can switch from the first mode to the second mode. This solves the above problem. Effect of the Invention

[0007] According to the present invention, a user who is not in a vehicle can easily visually grasp the movement of the vehicle by viewing the second viewpoint image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram of an image display system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram for explaining a first viewpoint image. [Diagram 3] 13 is an explanatory diagram showing a change in the display mode of an interior image when the size of the interior image is enlarged or reduced. FIG. [Figure 4] FIG. 11 is an explanatory diagram showing a change in the display mode of an interior image when the display position of the interior image is changed in the vertical direction. [Diagram 5] FIG. 11 is an explanatory diagram showing a change in the display mode of the interior image when the interior image is tilted left or right. [Figure 6] 5 is a flowchart showing image display control executed by the image generating device according to the present embodiment. [Figure 7] 10 is a flowchart showing an image display control executed by an image generating device according to a modified example. [Figure 8] 13 is an example of a virtual space when the display device is incorporated in a head mounted display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, an image generating device and an image generating method according to the present invention will be described using an example in which the image generating device and the image generating method are applied to an image display system.

[0010] 1 is a block diagram of an image display system 100 according to this embodiment. The image display system 100 according to this embodiment is used in a scene where a virtual space that reproduces a space where a user B is present is displayed to a user A who is in a space different from the user B. In this embodiment, a user who is in a remote space 30 that is a place away from the vehicle 1 is taken as an example of the user A, and a user who is in the vehicle 1 and in the in-vehicle space 20 is taken as an example of the user B. In the following description, for convenience, the user A who is in the remote space 30 is referred to as the remote user A, and the user B who is in the in-vehicle space 20 is referred to as the in-vehicle user B.

[0011] In this embodiment, the states of the remote user A and the in-vehicle user B are not particularly limited. For example, in FIG. 1, the state of the remote user A is shown as a seated state, but the remote user A may be standing. Also, for example, in FIG. 1, the state of the in-vehicle user B is shown as a state in which the steering wheel of the vehicle 1 is being operated, but the in-vehicle user B may not be the driver, but may be a passenger sitting in the passenger seat or the back seat.

[0012] In this embodiment, the image display system 100 is used in a system that uses virtual reality technology. As shown in FIG. 1, the remote user A and the in-car user B are in different spaces, so the remote user A cannot confirm what the in-car user B is viewing in the in-car space 20. However, in the virtual reality technology that allows a user to perceive a virtual space, which is a world created by a computer, as reality, the computer can create a virtual space for the remote user A as if he or she were in the in-car space 20. In the virtual space, the remote user A perceives as if he or she were in the in-car space 20 through at least the sense of sight and hearing. Furthermore, the image generating device and the image generating method according to the present invention change the virtual space according to the behavior of the vehicle 1, so that the remote user A can visually grasp the behavior of the vehicle 1 even while in the remote space 30. Hereinafter, each configuration of the image display system 100 will be described with reference to FIG. 1.

[0013] 1, the image display system 100 includes an image generating device 10, an in-vehicle space 20, and a remote space 30. The remote space 30 is a space separated from the in-vehicle space 20. The remote space 30 may be, for example, an interior of a vehicle other than the vehicle 1 or a room in a building, but the remote space 30 is not particularly limited as long as it is a space separate from the in-vehicle space 20.

[0014] The vehicle interior space 20 is a space showing the interior of the vehicle 1. The vehicle 1 is provided with an on-board camera 21, an on-board sensor 22, and an on-board communication device 23. In FIG. 1, these devices are shown as being provided in the vehicle interior space 20, but the on-board camera 21, the on-board sensor 22, and the on-board communication device 23 are assumed to be mounted on the vehicle 1. The devices provided in the vehicle 1 are connected to each other by, for example, a Controller Area Network (CAN) or other on-board network (such as Ethernet (registered trademark)) in order to transmit and receive information to and from each other.

[0015] The on-board camera 21 is an imaging device that captures the surroundings of the vehicle 1. The on-board camera 21 is preferably provided so as to capture the range visible to the occupants of the vehicle 1, and may be configured with multiple cameras, such as a front camera that captures the front of the vehicle, a side camera that captures the side of the vehicle, and a rear camera that captures the rear of the vehicle. The captured images captured by the on-board camera 21 are output to the on-board communication device 23 as captured image data of the surroundings of the vehicle 1. The on-board camera 21 may be a device or equipment that captures the range visible to the occupants of the vehicle 1, and its form, number, characteristics, installation location, etc. are not particularly limited. For example, the on-board camera 21 may be configured with multiple cameras or multiple types of cameras.

[0016] The on-board sensor 22 is a sensor that detects the current state of the vehicle 1. The current state of the vehicle 1 is the current behavior of the vehicle 1. The current behavior of the vehicle 1 includes the current attitude of the vehicle 1. The type and number of the on-board sensors 22 are not particularly limited. In this embodiment, the acceleration and angular acceleration of the vehicle 1 are described as examples of parameters indicating the behavior of the vehicle 1, but the parameters indicating the behavior of the vehicle 1 may include at least one of the acceleration and angular velocity of the vehicle 1. As shown in FIG. 1, the on-board sensor 22 is composed of an acceleration sensor 22a and an angular acceleration sensor 22b.

[0017] The acceleration sensor 22a is a so-called three-axis acceleration sensor, which is a sensor that measures the acceleration on three axes of the vehicle 1. The three axes of the vehicle 1 are an axis (x-axis) along the longitudinal direction of the vehicle 1, an axis (y-axis) along the lateral direction of the vehicle 1, and an axis (z-axis) along the up-down direction of the vehicle 1. An example of the acceleration sensor 22a is a MEMS type acceleration sensor. Note that the method of measuring the acceleration in the acceleration sensor 22a is not particularly limited. The acceleration on each axis of the vehicle 1 measured by the acceleration sensor 22a is output to the in-vehicle communication device 23 as acceleration data of the vehicle 1. The acceleration sensor 22a may be a device or equipment that measures the acceleration on three axes of the vehicle 1, and the form, number, characteristics, installation location, etc. of the acceleration sensor 22a are not particularly limited. For example, the acceleration sensor 22a may be configured with a plurality of acceleration sensors or a plurality of types of acceleration sensors.

[0018] The angular acceleration sensor 22b is a sensor that measures the rotational motion of the vehicle 1. The rotational motion of the vehicle 1 includes the rotational motion of the vehicle 1 in the roll direction, the pitch direction, and the yaw direction. The angular acceleration sensor 22b is composed of, for example, a roll angular acceleration sensor that measures the angular acceleration of the vehicle 1 in the roll direction, a pitch angular acceleration sensor that measures the angular acceleration of the vehicle 1 in the pitch direction, and a yaw angular acceleration sensor that measures the angular acceleration of the vehicle 1 in the yaw direction. The method of measuring the angular acceleration in the angular acceleration sensor 22b is not particularly limited. The angular acceleration in each direction of the vehicle 1 measured by the angular acceleration sensor 22b is output to the in-vehicle communication device 23 as angular acceleration data of the vehicle 1. The angular acceleration sensor 22b may be a device or equipment that measures the angular acceleration in three directions of the vehicle 1, and its form, number, characteristics, installation location, and the like are not particularly limited. For example, the angular acceleration sensor 22b may be composed of multiple angular acceleration sensors or multiple types of angular acceleration sensors.

[0019] The in-vehicle communication device 23 is a device having a wireless communication function for wirelessly transmitting and receiving data to and from the outside of the vehicle 1. An example of the in-vehicle communication device 23 is a telematics control unit (TCU). The in-vehicle communication device 23 is connected to the Internet using a communication standard such as 4G / LTE or Wifi (registered trademark) by the wireless communication function, and transmits and receives various data to and from a server or a system provided outside the vehicle 1. The in-vehicle communication device 23 receives peripheral captured image data of the vehicle 1 from the in-vehicle camera 21. The in-vehicle communication device 23 also receives acceleration data and angular acceleration data of the vehicle 1 from the in-vehicle sensor 22. The in-vehicle communication device 23 transmits the peripheral captured image data, acceleration data, and angular acceleration data of the vehicle 1 to the image generating device 10 at predetermined intervals.

[0020] The remote space 30 will now be described. As shown in Fig. 1, the remote space 30 is provided with a remote communication device 31 and a display device 32. The devices provided in the remote space 30 are connected to each other by, for example, Ethernet (registered trademark) in order to transmit and receive information to each other.

[0021] The remote communication device 31 is a device equipped with a wireless communication function for wirelessly transmitting and receiving data to and from the outside of the remote space 30. An example of the remote communication device 31 is a router. The wireless communication function of the remote communication device 31 is similar to the wireless communication function of the in-vehicle communication device 23, so the description of the wireless communication function of the in-vehicle communication device 23 will be used here. In this embodiment, the remote communication device 31 transmits and receives data to and from the image generation device 10. Image data is input to the remote communication device 31 from the image generation device 10, and the remote communication device 31 outputs the image data to the display device 32.

[0022] The display device 32 displays the image generated by the image generation device 10. The image data generated by the image generation device 10 is input to the display device 32 via the remote communication device 31. In this embodiment, the display device 32 has a display such as a liquid crystal panel. For example, the display device 32 may be a television or a monitor.

[0023] The image generating device 10 will be described. In this embodiment, the image generating device 10 is provided in a server (not shown) located outside the vehicle interior space 20 and the remote space 30. The image generating device 10 wirelessly transmits and receives data between the in-vehicle communication device 23 and the remote communication device 31 via a communication device (not shown) in the server. The installation location of the image generating device 10 is not particularly limited, and for example, the image generating device 10 may be mounted on the vehicle 1. Furthermore, for example, the image generating device 10 may be provided in the remote space 30.

[0024] The image generating device 10 is a computer including a ROM (Read Only Memory) storing a program for generating and processing an image to be viewed by a remote user A, a CPU (Central Processing Unit) that executes the program stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. The image generating device 10 includes a vehicle information acquiring unit 11, a storage unit 12, an image generating unit 13, and a display control unit 14. The image generating device 10 can execute the functions of the vehicle information acquiring unit 11, the storage unit 12, the image generating unit 13, and the display control unit 14 by executing the program stored in the ROM. Each function of the image generating device 10 will be described.

[0025] The vehicle information acquisition unit 11 acquires vehicle information including information on the behavior of the vehicle 1 from a device provided in the vehicle 1. Specifically, the vehicle information acquisition unit 11 acquires, via the in-vehicle communication device 23, an image of the surroundings of the vehicle 1 captured by the in-vehicle camera 21, and the acceleration and angular acceleration of the vehicle 1 detected by the in-vehicle sensor 22. For example, the vehicle information acquisition unit 11 acquires vehicle information from the vehicle 1 at predetermined intervals.

[0026] The storage unit 12 stores an interior image showing the interior of the vehicle 1. The interior image is used by the image generation unit 13 to generate a virtual space that reproduces the interior space 20 of the vehicle 1. The storage unit 12 stores a plurality of types of interior images, and stores an interior image for each model and grade of the vehicle 1, for example. Examples of the interior image include an image of an instrument panel including a steering wheel, a shift lever, a navigation operation unit, an air conditioner operation unit, a dashboard, a speedometer, and the like. Note that the interior image is not limited to an image of an instrument panel, and may be, for example, an image of a driver's seat, an image of a passenger seat, an image of a back seat, an image showing the inside of a door, an image of a ceiling or a floor, and the like.

[0027] The image generator 13 generates a virtual space that reproduces the vehicle interior space 20, and generates an image viewed from a predetermined virtual viewpoint. In this embodiment, an image viewed in a predetermined direction from a virtual viewpoint in the vehicle interior of the vehicle 1 is referred to as a first viewpoint image, and an image obtained by changing the first viewpoint image according to the behavior of the vehicle 1 is referred to as a second viewpoint image.

[0028] In order to generate the first viewpoint image, the image generating unit 13 sets a virtual viewpoint at a predetermined position in the cabin of the vehicle 1. There is no particular limitation on where in the cabin the virtual viewpoint is set, but it is preferable that the virtual viewpoint is set at a position where, when an image is displayed on the display device 32, the remote user A viewing the image can feel that he or she is riding in the vehicle 1. For example, the image generating unit 13 sets the virtual viewpoint at a position corresponding to the head of an occupant of the vehicle 1. Note that position information of the head of the occupant of the vehicle 1 is stored in advance in the storage unit 12.

[0029] Next, an example of a method for generating a first viewpoint image will be described. When the image generating unit 13 sets a virtual viewpoint, it generates an image showing the surroundings of the vehicle 1 based on the captured image of the surroundings of the vehicle 1. Then, the image generating device 10 reads out an interior image of the vehicle 1 seen from the virtual viewpoint from the storage unit 12, and generates a virtual space that reproduces the interior space 20 of the vehicle 1 by superimposing the interior image on the image showing the surroundings of the vehicle 1. Then, the image generating unit 13 generates a first viewpoint image by cutting out an image of a part of the virtual space that is included in the range of the virtual field of view based on the virtual viewpoint, based on the range of the virtual field of view set in advance.

[0030] Fig. 2 is an explanatory diagram for explaining the first viewpoint image. Fig. 2(A) is an example of a virtual space showing the interior of the vehicle 1 generated by the image generation unit 13. In Fig. 2(A), A0 is an interior image of the vehicle 1 including a display a and a steering wheel b of a car navigation system, and B is a captured image of the surroundings of the vehicle 1. The image generation unit 13 generates a virtual space showing the interior of the vehicle 1 by superimposing and displaying the interior image A0 on the captured image of the surroundings B. Also, in Fig. 2(A), C0 indicates a portion included in the range of the virtual field of view when the front of the vehicle 1 is viewed from the virtual viewpoint.

[0031] Fig. 2(B) shows the first viewpoint image C0 displayed on the display device 32. The display device 32 displays the first viewpoint image C0 as an image of the front of the vehicle 1 viewed from a virtual viewpoint. In the example of Fig. 2, the first viewpoint image C0 includes an interior image A0 showing the state of the interior of the vehicle 1 as viewed from the virtual viewpoint, and a surrounding captured image B of the vehicle 1 showing the surrounding state of the vehicle 1 as viewed from the virtual viewpoint.

[0032] Next, the second viewpoint image will be described. The image generating unit 13 generates the second viewpoint image by changing the relative positional relationship between the position of the subject in the vehicle interior captured in the first viewpoint image and the virtual viewpoint based on the behavior of the vehicle 1. The second viewpoint image is a viewpoint image that reflects the change in the posture of the occupant of the vehicle 1 and the change in the posture of the vehicle 1 due to the behavior of the vehicle 1, with respect to the virtual viewpoint set when generating the first viewpoint image. Therefore, unlike the first viewpoint image, the second viewpoint image changes into various images depending on the behavior of the vehicle 1. Note that the subject in the vehicle interior captured in the first viewpoint image is an object captured in the interior image. Examples of objects captured in the interior image include the display a and the steering wheel b shown in FIG. 2(A).

[0033] The image generating unit 13 calculates the displacement direction and displacement amount of the virtual viewpoint corresponding to the change in the behavior of the vehicle 1 based on the behavior of the vehicle 1. In this embodiment, the image generating unit 13 calculates the displacement direction and displacement amount of the virtual viewpoint corresponding to the change in the behavior of the vehicle 1 based on the acceleration and angular acceleration of the vehicle 1 among the vehicle information of the vehicle 1 acquired by the vehicle information acquiring unit 11. Then, the image generating unit 13 changes the display mode of the interior image included in the first viewpoint image based on the displacement direction and displacement amount of the virtual viewpoint. The change in the display mode of the interior image includes at least one of a change in the size of the interior image, a change in the display position of the interior image, and a change in the inclination angle of the interior image. In this embodiment, the image generating unit 13 sets the rate at which the display mode of the interior image is changed to the same rate as the displacement amount of the virtual viewpoint corresponding to the behavior of the vehicle 1. In other words, the image generating unit 13 changes the display mode of the interior image in direct proportion to the displacement amount of the virtual viewpoint.

[0034] Next, an example of the second viewpoint image will be described with reference to Figs. 3 to 5. Figs. 3 to 5 are divided into (A) to (C), and each of (A) to (C) shows two diagrams, one on the left and one on the right. In each of (A) and (C), the left diagram shows a virtual space showing the interior of the vehicle 1, and the right diagram shows the second viewpoint image displayed on the display device 32. In each of (B), the left diagram shows a virtual space showing the interior of the vehicle 1, and the right diagram shows the first viewpoint image displayed on the display device 32. That is, the left diagrams of Figs. 3(B), 4(B), and 5(B) correspond to Fig. 2(A), and the right diagrams of Figs. 3(B), 4(B), and 5(B) correspond to Fig. 2(B).

[0035] An example of reflecting a change in behavior of the vehicle 1 in the front-rear direction of the vehicle 1 to a virtual viewpoint by enlarging or reducing the size of an interior image in a virtual space will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram of a case where the size of an interior image is enlarged or reduced as a change in the display mode of the interior image.

[0036] For example, when the vehicle 1 decelerates and a backward acceleration occurs in the vehicle 1, a forward acceleration occurs in the occupant of the vehicle 1. In this case, the image generating unit 13 generates a virtual space in which the virtual viewpoint is displaced forward of the vehicle 1 by enlarging the interior image. For example, as shown in the left diagram of FIG. 3(A), the image generating unit 13 generates an interior image A1 by enlarging the interior image A0 shown in FIG. 3(B) at a predetermined magnification. This generates a virtual space including the interior image A1 obtained by enlarging the interior image A0. Then, the image generating unit 13 generates a second viewpoint image C1 as an image expressing a state in which the distance between the virtual viewpoint and the display a shown in the interior image is shortened by cutting out the range of C1 from the generated virtual space. Since the second viewpoint image C1 as shown in the right diagram of FIG. 3(A) is displayed on the display device 32, the remote user A has the illusion that his / her viewpoint has been moved forward of the vehicle 1, and it becomes easier for him / her to grasp that the vehicle 1 is decelerating.

[0037] On the other hand, when the vehicle 1 accelerates and a forward acceleration occurs in the vehicle 1, the occupant of the vehicle 1 accelerates backward. In this case, the image generating unit 13 reduces the interior image to generate a virtual space in which the virtual viewpoint is displaced backward of the vehicle 1. For example, as shown in the left diagram of FIG. 3(C), the image generating unit 13 reduces the interior image A0 shown in FIG. 3(B) by a predetermined magnification to generate an interior image A2. This generates a virtual space including the interior image A2 in which the interior image A0 is reduced. Then, the image generating unit 13 cuts out the range of C2 from the generated virtual space to generate a second viewpoint image C2 as an image expressing a state in which the distance between the virtual viewpoint and the display a shown in the interior image has increased. Since the display device 32 displays the second viewpoint image C2 as shown in the right diagram of FIG. 3(C), the remote user A has the illusion that his / her viewpoint has been moved backward of the vehicle 1, making it easier for him / her to grasp that the vehicle 1 is accelerating. In addition, in the left drawings of Fig. 3(A) to Fig. 3(C), C0 to C2 have the same shape and size. Also, a method for setting the enlargement ratio or reduction ratio of the interior image by the image generating unit 13 will be described later with reference to the flowchart shown in Fig. 6.

[0038] Next, an example of reflecting a change in behavior of the vehicle 1 in the pitch direction of the vehicle 1 in the virtual viewpoint by changing the display position of the interior image in the virtual space in the vertical direction will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram of a case where the display position of the interior image is changed in the vertical direction as a change in the display mode of the interior image.

[0039] For example, when the vehicle 1 accelerates and the vehicle 1 sinks backward, the viewpoint of the passenger of the vehicle 1 is moved upward. In this case, the image generating unit 13 generates a virtual space in which the virtual viewpoint is displaced upward of the vehicle 1 by moving the display position of the interior image downward. For example, as shown in the left diagram of FIG. 4(A), the image generating unit 13 generates an interior image A3 by moving the interior image A0 shown in FIG. 4(B) downward by a predetermined distance. This generates a virtual space including the interior image A3 in which the display position of the interior image A0 has been moved downward. Then, the image generating unit 13 generates a second viewpoint image C3 by cutting out the range of C3 from the generated virtual space as an image expressing the state in which the display a shown in the interior image seen from the virtual viewpoint has moved downward. Since the display device 32 displays a second viewpoint image C3 as shown in the right diagram of Figure 4(A), the remote user A has the illusion that his / her viewpoint has been moved upward above the vehicle 1, making it easier for him / her to understand that the vehicle 1 is tilting backward.

[0040] On the other hand, for example, when the attitude of the vehicle 1 sinks forward due to deceleration of the vehicle 1, the viewpoint of the occupant of the vehicle 1 is moved downward. In this case, the image generating unit 13 generates a virtual space in which the virtual viewpoint is displaced downward of the vehicle 1 by moving the display position of the interior image upward. For example, as shown in the left diagram of FIG. 4(C), the image generating unit 13 generates an interior image A4 by moving the interior image A0 shown in FIG. 4(B) upward by a predetermined distance. This generates a virtual space including the interior image A4 in which the display position of the interior image A0 has been moved upward. Then, the image generating unit 13 generates a second viewpoint image C4 as an image expressing the state in which the display a shown in the interior image seen from the virtual viewpoint has been moved upward by cutting out the range of C4 from the generated virtual space. Since the display device 32 displays the second viewpoint image C4 as shown in the right diagram of Fig. 4(C), the remote user A has the illusion that his / her viewpoint has been moved downward from the vehicle 1, and can easily understand that the vehicle 1 is tilting forward. Note that, in the left diagrams of Figs. 4(A) to 4(C), C0, C3, and C4 have the same shape and size. Also, a method of determining the movement direction of the interior image by the image generating unit 13 and a method of setting the movement distance will be described later with reference to the flowchart shown in Fig. 6.

[0041] Next, an example of reflecting the behavior of the vehicle 1 in the roll direction of the vehicle 1 in the virtual viewpoint by tilting the interior image in the virtual space in the left / right direction will be described with reference to Fig. 5. Fig. 5 is an explanatory diagram of a case where the interior image is tilted in the left / right direction as a change in the display mode of the interior image.

[0042] For example, when the vehicle 1 is tilted to the left while traveling around a curve to the left, the posture of the occupant of the vehicle 1 is also tilted to the left. In this case, the image generating unit 13 tilts the interior image to the left to generate a virtual space in which the posture of the vehicle 1 is tilted to the left. For example, as shown in the left diagram of FIG. 5(A), the image generating unit 13 generates an interior image A5 by tilting the interior image A0 shown in FIG. 5(B) to the left (counterclockwise direction) by a predetermined angle. This generates a virtual space including the interior image A5 in which the interior image A0 is tilted to the left. Then, the image generating unit 13 generates a second viewpoint image C5 as an image expressing the state in which the display a shown in the interior image seen from the virtual viewpoint is tilted to the left by cutting out the range of C5 from the generated virtual space. Since the display device 32 displays a second viewpoint image C5 as shown in the right diagram of Figure 5 (A), the remote user A is given the illusion that his / her own posture is leaning to the left relative to the direction of travel of the vehicle 1, making it easier for him / her to understand that the vehicle 1 is leaning to the left.

[0043] On the other hand, for example, when the vehicle 1 is tilted to the right while traveling around a right-hand curve, the posture of the occupant of the vehicle 1 is also tilted to the right. In this case, the image generating unit 13 tilts the interior image to the right to generate a virtual space in which the posture of the vehicle 1 is tilted to the right. For example, as shown in the left diagram of FIG. 5(C), the image generating unit 13 generates an interior image A6 by tilting the interior image A0 shown in FIG. 5(B) to the right (clockwise direction) by a predetermined angle. This generates a virtual space including the interior image A6 in which the interior image A0 is tilted to the right. Then, the image generating unit 13 generates a second viewpoint image C6 by cutting out the range of C6 from the generated virtual space as an image expressing the state in which the display a shown in the interior image seen from the virtual viewpoint is tilted to the right. Since the display device 32 displays the second viewpoint image C6 as shown in the right diagram of Fig. 5(C), the remote user A has the illusion that his / her posture is tilted to the right with respect to the traveling direction of the vehicle 1, and it becomes easier for him / her to understand that the vehicle 1 is tilted to the right. Note that, in the left diagrams of Figs. 5(A) to 5(C), C0, C5, and C6 have the same shape and size. Also, a method of determining the tilt direction of the interior image by the image generating unit 13 and a method of setting the tilt angle will be described later with reference to the flowchart shown in Fig. 6.

[0044] Returning to Fig. 1 again, the functions of the image generating device 10 will be described. The display control unit 14 outputs a control signal for displaying the first viewpoint image and the second viewpoint image generated by the image generating unit 13 on the display device 32 provided in the remote space 30. For example, the display control unit 14 converts image data of the first viewpoint image and the second viewpoint image into a control signal that can be displayed by the display device 32. The display control unit 14 transmits the converted control signal to the remote communication device 31 via a communication device (not shown) provided in the server.

[0045] Next, an image display control flow executed by the image generating device 10 will be described with reference to the flowchart shown in Fig. 6. Fig. 6 is a flowchart showing the image display control executed by the image generating device 10. The image generating device 10 executes the process of the flowchart shown in Fig. 6 at predetermined intervals.

[0046] In step S101, the image generating device 10 acquires vehicle information from a device provided in the vehicle 1. The vehicle information includes a captured image of the surroundings of the vehicle 1 captured by the on-board camera 21, acceleration in three axes of the vehicle 1 measured by the acceleration sensor 22a, and angular acceleration in three directions of the vehicle 1 measured by the angular acceleration sensor 22b. In step S101, the image generating device 10 acquires the vehicle information at predetermined intervals, and ends the process of step S101 after a predetermined time has elapsed.

[0047] In step S102, the image generating device 10 generates a first viewpoint image based on the vehicle information acquired in step S101. For example, the image generating device 10 sets a virtual viewpoint at a position corresponding to the head of an occupant of the vehicle 1, and reads out an interior image of the vehicle 1 seen from the set virtual viewpoint. Next, the image generating device 10 generates a virtual space that reproduces the interior space of the vehicle 1 by superimposing the interior image on the peripheral captured image of the vehicle 1 (see FIG. 2(A)). Then, the image generating device 10 generates a first viewpoint image, which is an image seen in a predetermined direction from the virtual viewpoint in the cabin of the vehicle 1, by cutting out an image of a part included in a virtual field of view based on the virtual viewpoint from the virtual viewpoint out of the virtual space (see FIG. 2(B)).

[0048] In step S103, the image generating device 110 calculates a change amount per predetermined time (a time difference in acceleration) for the acceleration of the vehicle 1 among the vehicle information acquired in step S101. For example, the image generating device 10 calculates a difference value between the acceleration at a specific time and the acceleration after a predetermined time has elapsed from the time for the acceleration of the vehicle 1 acquired at predetermined intervals in step S101.

[0049] In step S104, the image generating device 10 sets the enlargement or reduction ratio of the interior image based on the calculation result of step 103. For example, if the acceleration difference value in step S103 is negative, the image generating device 10 determines that the vehicle 1 is decelerating, and sets the enlargement ratio of the interior image to a magnification proportional to the absolute value of the difference value (in terms of magnification, a value greater than 1). On the other hand, for example, if the acceleration difference value in step S103 is positive, the image generating device 10 determines that the vehicle 1 is accelerating, and sets the reduction ratio of the interior image to a magnification proportional to the absolute value of the difference value (in terms of magnification, a value smaller than 1).

[0050] In step S105, the image generating device 10 calculates a time integral value obtained by integrating the angular acceleration in the pitch direction of the vehicle 1 from the vehicle information acquired in step S101 over a predetermined period of time, thereby calculating the angular velocity in the pitch direction of the vehicle 1.

[0051] In step S106, the image generating device 10 sets the movement direction and movement distance of the interior image based on the calculation result of step S105. For example, when the image generating device 10 obtains an angular velocity of the vehicle 1 in the backward pitch direction as the calculation result of step S105, the image generating device 10 determines that the attitude of the vehicle 1 has sunk backward, and sets the movement direction of the interior image to the upward direction. Furthermore, the image generating device 10 sets a distance proportional to the absolute position of the angular velocity in the pitch direction as the movement distance of the interior image. On the other hand, for example, when the image generating device 10 obtains an angular velocity of the vehicle 1 in the forward pitch direction as the calculation result of step S105, the image generating device 10 determines that the attitude of the vehicle 1 has sunk forward, and sets the movement direction of the interior image to the downward direction. Furthermore, the image generating device 10 sets a distance proportional to the absolute position of the angular velocity in the pitch direction as the movement distance of the interior image.

[0052] In step S107, the image generating device 10 calculates a time integral value obtained by integrating, over a predetermined period, the angular acceleration in the roll direction of the vehicle 1 from the vehicle information acquired in step S101, thereby calculating the angular velocity in the roll direction of the vehicle 1.

[0053] In step S108, the tilt direction and tilt angle of the interior image are set based on the calculation result of step S107. For example, when an angular velocity in the roll direction toward the left with respect to the traveling direction of the vehicle 1 is obtained as the calculation result of step S107, the image generating device 10 determines that the attitude of the vehicle 1 is tilted to the left, and sets the tilt direction of the interior image to the left. Furthermore, the image generating device 10 sets an angle proportional to the absolute position of the angular velocity in the roll direction as the tilt angle of the interior image. On the other hand, for example, when an angular velocity in the roll direction toward the right with respect to the traveling direction of the vehicle 1 is obtained as the calculation result of step S107, the image generating device 10 determines that the attitude of the vehicle 1 is tilted to the right, and sets the tilt direction of the interior image to the right. Furthermore, the image generating device 10 sets an angle proportional to the absolute position of the angular velocity in the roll direction as the tilt angle of the interior image.

[0054] In step S109, the image generating device 10 generates a second viewpoint image based on the first viewpoint image generated in step S102, based on the processing results of steps S104, S106, and S108. Specifically, the image generating device 10 enlarges or reduces the interior image used when generating the first viewpoint image according to the enlargement ratio or reduction ratio set in step S104 (see FIG. 3). The image generating device 10 also moves the interior image used when generating the first viewpoint image according to the movement direction and movement distance set in step S016 (see FIG. 4). The image generating device 10 also tilts the interior image used when generating the first viewpoint image according to the tilt direction and tilt angle set in step S108 (see FIG. 5). As a result, a second viewpoint image is generated in which the change in the speed of the vehicle 1 and the change in the attitude of the vehicle 1 in the pitch direction and roll direction are reflected in the virtual viewpoint.

[0055] In step S110, the image generating device 10 outputs the second viewpoint image generated in step S109 to the display device 32 via the communication device provided in the server and the remote communication device 31. When the process in step S110 ends, the image generating device 10 ends the process of the flowchart shown in FIG.

[0056] As described above, the image generating device 10 according to the present embodiment includes the vehicle information acquiring unit 11 that acquires vehicle information including information on the behavior of the vehicle 1 from the on-board camera 21 and the on-board sensor 22 provided in the vehicle 1, the image generating unit 13 that generates an image viewed in a predetermined direction from a virtual viewpoint in the cabin of the vehicle 1 as a first viewpoint image, and the display control unit 14 that outputs a control signal for displaying the image generated by the image generating unit 13 on the display device 32 provided in the remote space 30. In the present embodiment, the image generating unit 13 generates the second viewpoint image by changing the relative positional relationship between the position of the subject in the cabin captured in the first viewpoint image and the virtual viewpoint based on the behavior of the vehicle 1. This allows the remote user A who is not in the vehicle 1 to visually grasp the behavior of the vehicle 1 by viewing the second viewpoint image in which the behavior of the vehicle 1 is reflected.

[0057] Moreover, in this embodiment, the first viewpoint image includes an interior image showing the interior of the vehicle 1 as seen from a virtual viewpoint, and the second viewpoint image is an image in which the display mode of the interior image has changed according to the behavior of the vehicle 1. This allows the remote user A to intuitively grasp how the behavior of the vehicle 1 has changed according to the change in the display mode of the interior image.

[0058] Furthermore, in this embodiment, the image generator 13 calculates a displacement direction and a displacement amount of the virtual viewpoint corresponding to the change in the behavior of the vehicle 1 based on the behavior of the vehicle 1, and changes the display mode of the interior image based on the displacement direction and the displacement amount of the virtual viewpoint. Since the change in the behavior of the vehicle 1 can be expressed as a change in the display mode of the interior image, the remote user A has the illusion that the virtual viewpoint has been displaced in response to the change in the behavior of the vehicle 1, and can intuitively grasp the behavior of the vehicle 1.

[0059] Additionally, in this embodiment, the information on the behavior of the vehicle 1 includes the acceleration of the vehicle 1, and the image generator 13 changes the size of the interior image based on the acceleration of the vehicle 1. The second viewpoint image is an image that creates the illusion that the subject shown in the interior image of the vehicle 1 is approaching or receding depending on the direction of the acceleration of the vehicle 1, so that the remote user A can easily understand that the vehicle 1 is accelerating or decelerating.

[0060] Furthermore, in this embodiment, the information on the behavior of the vehicle 1 includes the angular acceleration in the pitch direction of the vehicle 1, and the image generator 13 changes the display position of the interior image based on the angular acceleration in the pitch direction of the vehicle 1. The second viewpoint image is an image that creates the illusion that the subject shown in the interior image of the vehicle 1 moves upward or downward depending on the direction of the angular acceleration in the pitch direction of the vehicle 1, so that the remote user A can easily recognize that the vehicle 1 is tilting forward or backward.

[0061] Furthermore, in this embodiment, the information on the behavior of the vehicle 1 includes the angular acceleration in the roll direction of the vehicle 1, and the image generator 13 tilts the interior image based on the angular acceleration in the roll direction of the vehicle 1. The second viewpoint image is an image that creates the illusion that the subject shown in the interior image of the vehicle 1 is tilting to the left or right depending on the direction of the angular acceleration in the roll direction of the vehicle 1, so that the remote user A can easily recognize that the vehicle 1 is tilting to the left or right.

[0062] Additionally, in this embodiment, the image generator 13 sets a virtual viewpoint at a position corresponding to the head of an occupant of the vehicle 1. As a result, the second viewpoint image becomes an image seen from the viewpoint of the occupant of the vehicle 1, so that the remote user A can obtain a sense of immersion as if he or she were riding in the vehicle 1 through his or her vision.

[0063] In this embodiment, the image generator 13 sets the rate at which the display mode of the interior image is changed to the same rate as the displacement amount of the virtual viewpoint. This allows the display mode of the interior image to change in accordance with the change in the behavior of the vehicle 1, so that the remote user A can intuitively grasp through his / her eyes how much the behavior of the vehicle 1 has changed.

[0064] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

[0065] For example, in the above-described embodiment, in the processing in the flowchart of FIG. 6, a configuration was given in which the time difference of acceleration, the time integral of angular acceleration in the pitch direction, and the time integral of angular acceleration in the roll direction are calculated, and then a change in the display mode of the interior image is set. However, the display mode of the interior image may be set by directly using the measurement results of the in-vehicle sensor 22 without performing any calculations.

[0066] An image display control flow executed by the image generating device 10 according to the modified example will be described with reference to Fig. 7. Fig. 7 is a flowchart showing image display control executed by the image generating device 10 according to the modified example. In the flowchart of Fig. 7, steps S103 to S109 are replaced with steps S201 to S204, as compared with the flowchart of Fig. 6. Therefore, for the steps described using the flowchart of Fig. 6, the above-mentioned explanation will be used and steps S201 to S204 will be described.

[0067] In step S201, the image generating device 10 determines whether to enlarge or reduce the interior image according to the direction of the acceleration of the vehicle 1 among the vehicle information acquired in step S101, and sets the magnification for enlarging or reducing the interior image according to the magnitude of the absolute value. For example, when the image generating device 10 acquires the backward acceleration of the vehicle 1 in the longitudinal direction of the vehicle 1, that is, when a negative acceleration is acquired on the x-axis of the vehicle 1, the image generating device 10 determines that the vehicle 1 is decelerating. In this case, the image generating unit 13 sets the magnification of the interior image to a magnification proportional to the magnitude of the absolute value of the acceleration. On the other hand, for example, when the image generating device 10 acquires the forward acceleration of the vehicle 1 in the longitudinal direction of the vehicle 1, that is, when a positive acceleration is acquired on the x-axis of the vehicle 1, the image generating device 10 determines that the vehicle 1 is accelerating. In this case, the image generating device 10 sets the reduction ratio of the interior image to a magnification proportional to the magnitude of the absolute value of the acceleration.

[0068] In step S202, the image generating device 10 sets the movement direction of the interior image to the upward or downward direction according to the angular acceleration in the pitch direction of the vehicle 1 among the vehicle information acquired in step S101, and sets the movement distance according to the magnitude of the absolute value of the angular acceleration. For example, when the image generating device 10 acquires the backward angular acceleration of the vehicle 1 in the pitch direction of the vehicle 1, it determines that the attitude of the vehicle 1 has sunk backward. In this case, the image generating device 10 sets the movement direction of the interior image to the upward direction, and sets the movement distance of the interior image to a distance proportional to the magnitude of the angular acceleration. On the other hand, for example, when the image generating device 10 acquires the forward angular acceleration of the vehicle 1 in the pitch direction of the vehicle 1, it determines that the attitude of the vehicle 1 has sunk forward. In this case, the image generating device 10 sets the movement direction of the interior image to the downward direction, and sets the movement distance of the interior image to a distance proportional to the magnitude of the angular acceleration.

[0069] In step S203, the image generating device 10 sets the tilt direction of the interior image to the left or right depending on the angular acceleration in the roll direction of the vehicle 1 among the vehicle information acquired in step S101, and sets the tilt angle depending on the magnitude of the absolute value of the angular acceleration. For example, when the image generating device 10 acquires angular acceleration to the left with respect to the traveling direction of the vehicle 1 in the roll direction of the vehicle 1, it determines that the attitude of the vehicle 1 has tilted to the left. In this case, the image generating unit 13 sets the left side as the tilt direction of the interior image, and sets the tilt angle of the interior image to an angle proportional to the magnitude of the angular acceleration. On the other hand, for example, when the image generating device 10 acquires angular acceleration to the right with respect to the traveling direction of the vehicle 1 in the roll direction of the vehicle 1, it determines that the attitude of the vehicle 1 has tilted to the right. In this case, the image generating device 10 sets the right side as the tilt direction of the interior image, and sets the tilt angle of the interior image to an angle proportional to the magnitude of the angular acceleration.

[0070] In step S204, the image generating device 10 generates a second viewpoint image based on the first viewpoint image generated in step S102, based on the processing results of steps S201 to S203. The method of generating the second viewpoint image is similar to the description of step S109 in the above embodiment.

[0071] As described above, according to the image generating device 10 according to the modified example, the second viewpoint image can be generated using the measurement results of the vehicle-mounted sensor 22 as they are, and therefore the calculation load can be reduced.

[0072] In the above embodiment, the ratio of changing the display mode of the interior image is set to the same ratio as the displacement amount of the virtual viewpoint in response to the change in the behavior of the vehicle 1, but the ratio of changing the display mode of the interior image is not limited to this. For example, the image generating unit 13 may set the ratio of changing the display mode of the interior image to a ratio larger than the displacement amount of the virtual viewpoint in response to the change in the behavior of the vehicle 1. This causes the display mode of the interior image to change more significantly than the actual change in the behavior of the vehicle 1, so that a second viewpoint image in which the change in the behavior of the vehicle 1 is exaggerated can be generated. Even if the change in the behavior of the vehicle 1 is minor, the remote user A can intuitively grasp it through his / her eyes.

[0073] In the above embodiment, the configuration for generating the first viewpoint image and the second viewpoint image has been described as an example, but the generation of the viewpoint image is not limited to this, and the first viewpoint image may be generated without generating the second viewpoint image. For example, the image generating unit 13 may switch between a first mode for generating the first viewpoint image and the second viewpoint image and a second mode for generating the first viewpoint image and not generating the second viewpoint image. For example, in a scene in which the behavior of the vehicle 1 changes constantly, such as when the vehicle 1 runs on an unpaved road, the image generating unit 13 can prevent the second viewpoint image from constantly changing by switching from the first mode to the second mode, and can reduce the possibility that the remote user A becomes sick due to a change in the display mode of the interior image.

[0074] In the above embodiment, the display device 32 has a display such as a television or a monitor. However, the present invention is not limited to this. For example, the display device 32 may be a screen display portion of a head-mounted display worn by the remote user A. In this case, the virtual viewpoint is not a preset viewpoint, but the viewpoint of the remote user A, and the viewpoint moves according to the movement of the head of the remote user A. In this case, as shown in the example of FIG. 8, the virtual space has a spherical shape representing the interior of the vehicle 1. The image generating device 10 generates an interior image according to the movement of the head of the remote user A in addition to the behavior of the vehicle 1, so that the remote user A can easily grasp the behavior of the vehicle 1 even on a device where the remote user A can arrive. FIG. 8 shows an example of a virtual space in the case where the display device 32 is incorporated in a head-mounted display.

[0075] In the above embodiment, an image viewed from a virtual viewpoint toward the front of the vehicle 1 has been described as a virtual image, but the direction viewed from the virtual viewpoint is not limited to the front of the vehicle 1, and for example, an image viewed from a virtual viewpoint toward the left or right side of the vehicle 1 may be used as a virtual image. In this case, the image generator 13 changes the size of the interior image used when generating the first viewpoint image based on the lateral acceleration of the vehicle 1, moves the display position of the interior image in the vertical direction based on the angular acceleration in the roll direction of the vehicle 1, and further tilts the interior image in the horizontal direction based on the angular acceleration in the pitch direction of the vehicle 1, thereby generating a second viewpoint image.

[0076] In the above embodiment, the angular velocity in the pitch direction is calculated from the angular acceleration in the pitch direction, and the angular velocity in the roll direction is calculated from the angular acceleration in the roll direction. However, the angular velocities in the pitch direction and the roll direction may be values ​​measured by a gyro sensor. In this case, the image generating device 10 acquires the angular velocities in the pitch direction and the roll direction from the gyro sensor mounted on the vehicle 1.

[0077] In addition, in the above-described embodiment, an example was given of a configuration in which an interior image of vehicle 1 is stored in memory unit 12, but the interior image of vehicle 1 may also be an image captured by a camera mounted on vehicle 1 that captures images of the interior of vehicle 1. [Explanation of symbols]

[0078] 100...Image display system 1. Vehicle 21…In-vehicle camera 22…In-vehicle sensor 22a...Acceleration sensor 22b...Angular acceleration sensor 23...In-vehicle communication device 10...Image generating device 11…Vehicle information acquisition unit 12...Storage section 13...Image generation unit 14...Display control unit 31...Remote communication equipment 32...Display device

Claims

1. A vehicle information acquisition unit that acquires vehicle information including information on a behavior of the vehicle from a device provided in the vehicle; an image generating unit that generates an image viewed in a predetermined direction from a virtual viewpoint in a cabin of the vehicle as a first viewpoint image; A display control unit that outputs a control signal for displaying the image generated by the image generation unit on a display device provided in a space different from the vehicle interior, The image generating unit includes: generating a second viewpoint image by changing a relative positional relationship between a position of a subject in the vehicle interior captured in the first viewpoint image and the virtual viewpoint based on the behavior; a first mode in which the first viewpoint image and the second viewpoint image are generated and a second mode in which the first viewpoint image is generated but the second viewpoint image is not generated can be switched; If the behavior is constantly changing, the image generating device switches from the first mode to the second mode.

2. The first viewpoint image includes an interior image showing the interior of the vehicle as seen from the virtual viewpoint, The image generating device according to claim 1 , wherein the second viewpoint image is an image in which a display state of the interior image is changed in accordance with the behavior.

3. The image generating unit, Calculating a displacement direction and a displacement amount of the virtual viewpoint according to the change in the behavior based on the behavior; The image generating device according to claim 2 , wherein a display mode of the interior image is changed based on a displacement direction and a displacement amount of the virtual viewpoint.

4. The behavior information includes an acceleration of the vehicle, The image generating device according to claim 2 , wherein the image generating unit changes a size of the interior image based on an acceleration of the vehicle.

5. The behavior information includes angular acceleration of the vehicle in a pitch direction, The image generating device according to any one of claims 2 to 4, wherein the image generating unit changes a display position of the interior image based on the angular acceleration in the pitch direction.

6. The behavior information includes an angular acceleration of the vehicle in a roll direction, The image generating device according to any one of claims 2 to 5, wherein the image generating unit tilts the interior image based on angular acceleration in the roll direction.

7. An image generating device as described in any one of claims 1 to 6, wherein the image generating unit sets the virtual viewpoint at a position corresponding to the head of an occupant of the vehicle.

8. An image generating device as described in any one of claims 1 to 7, wherein the image generation unit switches from the first mode to the second mode when the vehicle is traveling on an unpaved road.

9. The image generating device of claim 3, wherein the image generating unit sets the rate at which the display mode of the interior image is changed to the same rate as the amount of displacement of the virtual viewpoint.

10. An image generating device as described in Claim 3, wherein the image generating unit sets a rate at which the display mode of the interior image is changed to a rate greater than the amount of displacement of the virtual viewpoint.

11. 1. An image generating method executed by a processor to generate an image to be displayed on a display device provided in a space other than a vehicle interior, comprising: Acquiring vehicle information including information on a behavior of the vehicle from a device provided in the vehicle; generating an image seen in a predetermined direction from a virtual viewpoint in the interior of the vehicle as a first viewpoint image; generating a second viewpoint image by changing a relative positional relationship between a position of the subject captured in the first viewpoint image and the virtual viewpoint based on the behavior; a first mode in which the first viewpoint image and the second viewpoint image are generated and a second mode in which the first viewpoint image is generated but the second viewpoint image is not generated can be switched; If the behavior is constantly changing, the method switches from the first mode to the second mode.

Citation Information

Patent Citations

  • Vehicle remote control system, remote controller, vehicle controller, and remote control method for vehicle

    JP2004206218A

  • Mobile body simulation device and mobile body simulation program

    JP2006031229A

  • Image display system in train

    JP2009093076A

  • Virtual space display system

    JP2017146963A

  • Image processing apparatus, image processing system, image processing method, and program

    JP2020145612A