Display control method and display control apparatus

Abstract

To enhance the entertainment of an occupant on a vehicle who is watching indoor and outdoor images, and provide a comfortable vehicle space.SOLUTION: A display control apparatus 100 controls display devices 30, 40 mounted on a vehicle 1. The display control apparatus 100 includes: a display mode switching unit 143 which selects a first display mode that superimposes a virtual object configured so that front and rear can be specified, on a first captured image obtained by imaging the outside of the vehicle 1, or a second display mode that superimposes the virtual object in a display mode different from the first display mode, on a second captured image obtained by imaging the inside of the vehicle 1; and a display control unit 146 which displays, in the first display mode, the virtual object so that the front part of the virtual object faces an advancing direction of the vehicle 1, and displays, in the second display mode, the virtual object so that the front part of the virtual object faces a direction of the front of a seat inside the vehicle 1.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a display control method and a display control device for controlling a display device. [Background technology]

[0002] In recent years, there have been devices that superimpose characters on captured images and display them on a display device. For example, a technology has been proposed in which an enemy character corresponding to a target marker included in a captured image appears on a game screen displayed on a touch panel of a user terminal mounted on a gun-shaped controller (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-176635 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described conventional technology, an enemy character can appear in a captured image in the user's traveling direction. A vehicle has both an interior and exterior space. Therefore, it is conceivable to use an imaging device installed in the vehicle to capture images of subjects inside and outside the vehicle and display at least one of the captured images from the interior and exterior of the vehicle on a display device. It is also conceivable to superimpose a virtual object on the captured image. However, because a vehicle has multiple spaces, such as the interior and exterior of the vehicle, there is room for improvement in display control to improve the user experience in the vehicle by utilizing these spaces.

[0005] The present invention aims to improve the enjoyment of passengers viewing captured images of the interior and exterior of the vehicle while riding in the vehicle, and to provide a comfortable vehicle space. [Means for solving the problem]

[0006] One aspect of the present invention is a display control method for controlling a display device mounted on a vehicle. The display control method includes a switching process for switching between a first display mode in which a virtual object having a form that allows identification of front and rear is superimposed on a first captured image of the exterior of the vehicle and a second display mode in which a virtual object having a display form different from that in the first display mode is superimposed on a second captured image of the interior of the vehicle, and a display control process for displaying the virtual object in the first display mode so that a front portion of the virtual object faces in the direction in which the vehicle is moving forward, and for displaying the virtual object in the second display mode so that a front portion of the virtual object faces the direction of the front of the seat inside the vehicle. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the enjoyment of passengers viewing captured images of the interior and exterior of the vehicle while riding in the vehicle, and to provide a comfortable vehicle space. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the external configuration of an imaging device and a display device. [Figure 2] FIG. 2 is a diagram showing an example of the external configuration of the imaging device and the display device. [Figure 3] FIG. 3 is a diagram showing an example of the system configuration of a vehicle management system that manages images acquired in a vehicle. [Figure 4] FIG. 4 is a block diagram showing an example of the functional configuration of a display control device mounted on a vehicle. [Figure 5] FIG. 5 is a diagram showing a captured image generated by an imaging device and a display example in which a virtual object is superimposed on the captured image. [Figure 6] FIG. 6 is a diagram showing a captured image generated by an imaging device and a display example in which a virtual object is superimposed on the captured image. [Figure 7]FIG. 7 shows an example of image transition when switching the display mode of an image displayed on a display device, and a display example when changing the display mode of a character image and a vehicle image based on a vehicle control signal. [Figure 8] FIG. 8 is a flowchart showing an example of a display control process in the display control device. [Figure 9] FIG. 9 is a flowchart showing an example of a display control process in the display control device. [Figure 10] FIG. 10 is a diagram showing an example of transition of an image when the display mode of an image displayed on a display device is automatically switched. [Figure 11] FIG. 11 is a diagram showing an example of a detection method for detecting a specific object existing around a vehicle. [Figure 12] FIG. 12 is a flowchart showing an example of a display mode switching process in the display control device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Example of vehicle internal configuration] 1 and 2 are diagrams showing an example of the external configuration of imaging devices 10, 20 and display devices 30, 40. Note that Figs. 1 and 2 show a simplified view of the interior of vehicle 1, and omit illustrations of only a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, a car navigation device 6, a driver's seat 7, and a passenger seat 8. Fig. 1 shows an example of the external configuration when the area in front of the driver's seat 7 and passenger seat 8 is viewed from the rear of vehicle 1, and Fig. 2 shows an example of the external configuration when the area in front of rear seats 11, 12 (see Fig. 6) is viewed from the rear of vehicle 1.

[0011] The imaging device 10 is provided on the ceiling inside the vehicle 1 and captures an image of a subject inside the vehicle 1 to generate an image (image data). For example, the imaging device 10 can be provided above the windshield 4, i.e., above the rearview mirror 5. The imaging device 20 is provided at the left end of the dashboard 2 and captures an image of a subject outside the vehicle 1 to generate an image (image data). The imaging devices 10 and 20 are configured, for example, with one or more camera devices or image sensors capable of capturing an image of a subject. While this example illustrates an example in which at least two imaging devices 10 and 20 are provided, it is also possible to provide three or more imaging devices and use some of the images from these imaging devices. The installation locations of the imaging devices are not limited to the examples shown in FIGS. 1 and 2 and can be changed as appropriate. One or more devices, for example, a 360-degree camera, capable of capturing subjects in all directions around the vehicle 1 and subjects inside the vehicle 1 may also be used.

[0012] The display devices 30 and 40 are display devices that display various images under the control of the controller 140 (see FIG. 4). The display devices 30 and 40 can be, for example, touch panels that can accept user operations by touching the screen. The display device 30 can be called an assistant monitor, and the display device 40 can be called a rear monitor. Instead of the display device 30, various images may be displayed on the car navigation device 6.

[0013] Specifically, the display device 30 is provided on the passenger seat 8 side of the dashboard 2 so that it can be seen by a passenger seated in the passenger seat 8, and displays various images on the display unit 31. The display device 40 is provided near the center of the ceiling inside the vehicle 1 so that it can be seen by a passenger seated in the rear seat, and displays various images on the display unit 41. For example, the display device 40 may be a flip-down display device attached to the ceiling using a support member 42, i.e., a foldable display device. Note that while this example illustrates an example in which at least two display devices 30 and 40 are provided, one or three or more display devices may be provided, and display control may be performed for only some of these display devices. Furthermore, the installation locations of each display device are not limited to the example shown in FIGS. 1 and 2 and can be changed as appropriate. Furthermore, display examples of the display devices 30 and 40 are shown in FIGS. 5 to 7 and 10.

[0014] [Example of vehicle management system configuration] FIG. 3 is a diagram showing an example of the system configuration of a vehicle management system 50 that manages images acquired in the vehicle 1. As shown in FIG.

[0015] The vehicle management system 50 includes, for example, a network 60 and a management server 70, and is configured to be able to communicate with the display control device 100 of the vehicle 1, the electronic device 80, etc. The electronic device 80 is an electronic device carried by a passenger in the vehicle 1, and is a wireless device that can connect to the network 60 using wireless communication. The electronic device 80 is, for example, an information processing device such as a smartphone, a tablet device, or a portable personal computer.

[0016] The network 60 is a network such as a public line network or the Internet. The devices constituting the vehicle management system 50 are connected to the network 60 by either a communication method using wireless communication or a communication method using wired communication, or by both methods.

[0017] The management server 70 includes a communication unit 71, a control unit 72, and a storage unit 73. For example, the management server 70 can be a server that manages image content transmitted from the display control device 100 of the vehicle 1 and provides the image content in response to a request from the electronic device 80. Furthermore, for example, the management server 70 can manage position information transmitted from a plurality of vehicles and provide each piece of position information in response to a request from each vehicle.

[0018] The communication unit 71 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 72. That is, the communication unit 71 functions as an image receiving unit and an image transmitting unit.

[0019] The control unit 72 controls each unit based on various programs stored in the storage unit 73. The control unit 72 is realized by a processing device such as a CPU (Central Processing Unit), for example.

[0020] The storage unit 73 is a storage medium that stores various types of information. For example, the storage unit 73 stores various types of information (for example, control programs and image content) required for the control unit 72 to perform various processes. The storage unit 73 also stores various types of information acquired via the communication unit 71. The storage unit 73 can be, for example, a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0021] The electronic device 80 can display the image content transmitted from the management server 70 on the display unit. Although Fig. 3 shows an example in which information is exchanged between the electronic device 80 and the display control device 100 via the network 60, information may also be exchanged directly between the electronic device 80 and the display control device 100. In this case, for example, short-range wireless communication (for example, wireless LAN such as Wi-Fi, Bluetooth (registered trademark)) can be used.

[0022] [Configuration example of display control device 100] 4 is a block diagram showing an example of the functional configuration of a display control device 100 mounted on a vehicle 1. The display control device 100 causes, for example, the display devices 30 and 40 to display an image in which a virtual object is superimposed on a captured image generated by the imaging devices 10 and 20. Note that the virtual object refers to various types of virtual objects that are displayed superimposed on real space, and can take various forms such as an icon or a character. Display examples of this virtual object are shown in FIGS. 5 to 7 and 10.

[0023] The display control device 100 includes a vehicle system 110, a UI unit 120, an image acquisition unit 130, a controller 140, a storage unit 150, and a communication unit 160. The controller 140 also includes an occupant recognition unit 141, a display mode determination unit 142, a display mode switching unit 143, a display object determination unit 144, an imaging control unit 145, and a display control unit 146.

[0024] The vehicle system 110 is a control system installed in the vehicle 1 and includes, for example, various sensors. The vehicle system 110 outputs detection information acquired by each sensor, control signals for the vehicle 1, and the like to the display mode determination unit 142 and the display object determination unit 144. The sensors include, for example, millimeter-wave radar, LIDAR (Light Detection and Ranging), sonar, a stroke sensor, an acceleration sensor, a wheel measurement sensor, a lateral G sensor, a longitudinal G sensor, a vehicle speed sensor, a sound acquisition unit, a seating sensor, and a position information acquisition unit. The sound acquisition unit is provided inside or outside the vehicle 1 and acquires sounds around the vehicle 1 (sounds outside the vehicle) and sounds inside the vehicle 1 based on the control of the controller 140. The sound acquisition unit can be, for example, one or more microphones or sound acquisition sensors. The vehicle system 110 can acquire information about the traveling of the vehicle 1, information about the surroundings of the vehicle 1, and the like based on the detection information acquired by each sensor. Furthermore, the vehicle system 110 generates a control signal for the vehicle 1 based on the driving operation by the driver, detection information acquired by each sensor, etc. This control signal is a signal used when executing various controls for driving the vehicle 1. Then, based on the control signal for the vehicle 1, the vehicle system 110 executes various controls related to the driving of the vehicle 1, for example, forward / reverse movement and steering.

[0025] The UI unit 120 functions as an interface that accepts operations from the occupants of the vehicle 1 and provides various types of information to the occupants of the vehicle 1, and is equipped with display devices 30 and 40. The UI unit 120 outputs the contents of the accepted operations to the controller 140 and provides various types of information under the control of the controller 140.

[0026] The image acquisition unit 130 acquires various images (still images, moving images, etc.) related to the vehicle 1, and includes the imaging devices 10 and 20. The imaging devices 10 and 20 output captured images (image data) generated by imaging processing to the controller 140.

[0027] The controller 140 controls each unit based on various programs stored in the storage unit 150. The controller 140 is realized by, for example, a processing device such as a CPU. Note that the vehicle ECU (Electronic Control Unit) of the vehicle 1 may also be used as the controller 140, or a processing device different from the vehicle ECU may be provided as the controller 140.

[0028] The occupant recognition unit 141 recognizes occupants seated in each seat of the vehicle 1 based on the captured image generated by the imaging device 10 and outputs the recognition result to the display object determination unit 144. That is, the occupant recognition unit 141 recognizes occupants seated in each seat of the vehicle 1, thereby determining whether or not a person is present in each seat of the vehicle 1, and based on the determination result, determines whether or not a seat in the vehicle 1 is vacant. In this way, the occupant recognition unit 141 detects vacant seats in the vehicle 1 to search for the display position of the virtual object. For example, by performing a seat detection process for detecting each seat and a person detection process for recognizing people, it is possible to determine whether or not a person is present in each seat position. Note that the seat detection process and the person detection process can use known object detection technology or person detection technology. Note that if the imaging device 10 is installed in a fixed position, the seat detection process can be omitted and person detection process for detecting people in each seat position set in advance can be performed. Note that the occupant recognition unit 141 may determine whether or not a seat in the vehicle 1 is vacant based on internal information acquired from the vehicle system 110. The occupant recognition unit 141 can determine whether or not a seat in the vehicle 1 is vacant by using, for example, a seating sensor installed in each seat of the vehicle 1.

[0029] The display mode determination unit 142 determines the display mode of the display devices 30 and 40 based on the information output from the vehicle system 110 and the operation information output from the display devices 30 and 40. Then, the display mode determination unit 142 outputs the determination result to the display mode switching unit 143 and the display object determination unit 144. For example, when the exterior display mode switching button 302 (see FIGS. 7(A) and (B)) is pressed, it is determined that the display mode has been switched to the exterior display mode. On the other hand, when the interior display mode switching button 303 (see FIGS. 7(A) and (B)) is pressed, it is determined that the display mode has been switched to the interior display mode. Furthermore, the display mode determination unit 142 determines automatic switching of the display mode of the display devices 30 and 40 based on the information output from the vehicle system 110. An example of this automatic switching determination will be described in detail with reference to FIG. 12.

[0030] The display mode switching unit 143 switches between the images generated by the image capture devices 10 and 20 based on the display mode determination result output from the display mode determination unit 142, and outputs the switched image to the display control unit 146. For example, if it is determined that the display mode has been switched to the in-vehicle display mode, the display mode switching unit 143 outputs the captured image generated by the image capture device 10 to the display control unit 146. On the other hand, if it is determined that the display mode has been switched to the outside-vehicle display mode, the display mode switching unit 143 outputs the captured image generated by the image capture device 20 to the display control unit 146.

[0031] The display object determination unit 144 determines a virtual object to be displayed on the image capture devices 10 and 20 based on the information output from the vehicle system 110, the occupant recognition unit 141, the display mode determination unit 142, and the storage unit 150. For example, the display object determination unit 144 can generate a virtual object to be displayed based on virtual object information acquired from the storage unit 150. In this case, the display object determination unit 144 determines the orientation, position in the captured image, and movement of the virtual object to be displayed based on the information acquired from the vehicle system 110, the occupant recognition unit 141, and the display mode determination unit 142. For example, the display object determination unit 144 determines the orientation, position, and movement of a character image as a virtual object to be displayed, and whether or not to display a vehicle image. The virtual object information is information used when displaying virtual objects such as character images and vehicle images on the display devices 30 and 40. The display object determination unit 144 then outputs information for displaying the determined virtual object to the display control unit 146.

[0032] The photographing control unit 145 executes a storage process for storing the image displayed on the display device 30, 40 in the storage unit 150. Specifically, the photographing control unit 145 determines whether or not a pressing operation has been performed on the shutter operation area 304 (see FIGS. 7A and 7B), and stores the image displayed on the display device 30, 40 when the shutter operation area 304 was pressed as image content in the storage unit 150. Specifically, the photographing control unit 145 acquires the image displayed on the display device 30, 40 when the shutter operation area 304 was pressed from the display control unit 146, and stores the acquired image as image content in the storage unit 150. Note that video content may be stored in addition to the still image content at the timing of the pressing operation on the shutter operation area 304. The photographing control unit 145 also transmits the image content stored in the storage unit 150 to the management server 70 via the communication unit 160. In this way, the image content acquired when the shutter operation area 304 is pressed is stored in the storage unit 150, and is also transmitted to the management server 70, where it can be managed. This allows the image content managed by the management server 70 to be easily viewed using another device, for example, the electronic device 80 (see FIG. 3).

[0033] The display control unit 146 controls the display devices 30, 40 to display various images. Specifically, the display control unit 146 superimposes the virtual object determined by the display object determination unit 144 on the image output from the display mode switching unit 143 and displays the superimposed image on the display devices 30, 40. In this case, the display control unit 146 may display the same image on both the display devices 30, 40, or may display different images on each of the display devices 30, 40. The display control unit 146 also outputs the images displayed on the display devices 30, 40 to the shooting control unit 145. Display examples on the display devices 30, 40 are shown in FIGS. 5 to 7, etc.

[0034] The storage unit 150 is a storage medium that stores various types of information. For example, the storage unit 150 stores various types of information (e.g., control programs, map information, virtual object information, and image content) required for the controller 140 to perform various processes. The storage unit 150 also stores various types of information acquired via the communication unit 160. For example, a ROM, a RAM, an HDD, an SSD, or a combination of these can be used as the storage unit 150. The image content is a file of an image that was acquired based on a photographing operation by an occupant of the vehicle 1 from among the images displayed on the display devices 30 and 40.

[0035] The communication unit 160 exchanges various types of information with other devices using wireless communication under the control of the controller 140. For example, under the control of the imaging control unit 145, the communication unit 160 transmits images displayed on the display devices 30 and 40 by the display control unit 146 as image content to the management server 70 via the network 60. The communication unit 160 also receives information transmitted from the management server 70 or other external devices via the network 60.

[0036] [Example of an image captured by the imaging device] 5 and 6 are diagrams showing captured images generated by the imaging devices 10 and 20, and display examples in which virtual objects are superimposed on the captured images.

[0037] FIG. 5 shows a display example when the vehicle exterior display mode is set. FIG. 5 also shows vehicle images 212, 222, and 232 representing virtual vehicles and character images 211, 221, and 231 representing characters that can ride in the virtual vehicles, as examples of virtual objects. The characters constituting the virtual objects are objects that can ride in the virtual vehicles and are examples of characters that resemble objects with a face and a body. Furthermore, the character images 211, 221, and 231 are the same character in different display modes. Thus, the virtual object shown in this embodiment has a form that allows identification of the front and rear, and is, for example, a character that resembles an object with a face and a body, or a virtual vehicle. The characters may be mechanical characters that resemble robots, or biological characters that resemble animals, plants, or the like.

[0038] Image 200 shown in the upper left of FIG. 5 is a captured image generated by imaging device 20, and is an image including a subject ahead of vehicle 1. FIG. 5 shows image 200 including a plurality of vehicles traveling ahead of vehicle 1, an oncoming vehicle, etc. In this case, as shown in the upper right of FIG. 5, a composite image of a character riding in a virtual vehicle can be superimposed on the lower left of image 200 as vehicle image 212 and character image 211 to form image 210. Furthermore, as shown in the lower left of FIG. 5, vehicle image 222 and character image 221 can be superimposed on the lower right of image 200 to form image 220. Furthermore, as shown in the lower right of FIG. 5, vehicle image 232 and character image 231 can be superimposed on the upper left of image 200 to form image 230.

[0039] The positions at which the character image and the vehicle image are superimposed may be set in advance or may be set based on a user instruction. Furthermore, they may be set appropriately depending on the environment around the vehicle 1 or the environment inside the vehicle 1. For example, an object included in the captured image generated by the imaging device 20, such as a preceding vehicle, may be detected, and a position where no preceding vehicle is present may be set as the position at which the character image and the vehicle image are superimposed. For example, if there is congestion ahead of the vehicle 1, an image 230 may be displayed in which a vehicle image 232 and a character image 231 appear to be flying in the sky, as shown in the lower right of FIG. 5.

[0040] It should be noted that, when the vehicle 1 is moving forward, it is preferable that the display mode of the character image and vehicle image be such that they are moving ahead of the vehicle 1. For example, it is preferable that the character and virtual vehicle are facing the direction of travel of the vehicle 1, as in the vehicle images 212, 222, and 232 and the character images 211, 221, and 231 in the images 210, 220, and 230.

[0041] Furthermore, the display modes of the character image and the vehicle image may be changed as appropriate based on a control signal from the vehicle 1. For example, the display modes can be changed by tilting the character and the virtual vehicle, making them run, or moving their display positions in the captured image. For example, when the vehicle 1 turns right, the character and the virtual vehicle can be made to face right, and when the vehicle 1 turns left, the character and the virtual vehicle can be made to face left. In this case, only a part of the character or only a part of the virtual vehicle may be turned, and an effect may be executed in which the character's face looks ahead of the turn or the character's body leans in the direction of the turn. An example of a display in which the character and the virtual vehicle face left is shown in FIG. 7(D).

[0042] Furthermore, for example, when the vehicle 1 travels uphill, the virtual object consisting of the character and the virtual vehicle can be made to appear to go uphill, and when the vehicle 1 travels downhill, the virtual object can be made to appear to go downhill. That is, the pitch direction of the virtual vehicle can be changed to increase, or the position of the virtual vehicle in the captured image can be changed by moving it vertically.

[0043] Furthermore, for example, when vehicle 1 is traveling on a highway, the character and the virtual vehicle can be made to travel at high speed. That is, the character and the virtual vehicle can be changed to travel at high speed far ahead of vehicle 1.

[0044] Also, for example, when the vehicle 1 moves backward, only the face of the character image can be turned to face backward. An example of this display is shown in Fig. 7(C).

[0045] Furthermore, the display modes of the character image and the vehicle image may be set appropriately depending on the environment around the vehicle 1 and the environment inside the vehicle 1. For example, the display modes of the character image and the vehicle image can be changed depending on the weather when the vehicle 1 is traveling. For example, when the vehicle 1 is traveling in the rain, the character riding in the virtual vehicle can be shown holding an umbrella, a roof can be provided on the virtual vehicle in which the character is riding, or the character riding in the virtual vehicle can be shown soaking wet. Weather information can be acquired via the network 60 from an external device, such as a weather information server. Furthermore, for example, when the volume of the voice of the occupant of the vehicle 1 exceeds a reference value, a display mode can be such that a speech bubble image murmuring "That looks fun" is displayed near the character image, or a character image turning around with a smile can be displayed.

[0046] In this way, actions according to the state of the vehicle 1 can be performed using the character image and the vehicle image.

[0047] As described above, by displaying images 210, 220, and 230 on display devices 30 and 40, in which virtual objects made up of vehicle images 212, 222, and 232 and character images 211, 221, and 231 are superimposed on image 200 that is the same as the image that the occupant of vehicle 1 actually sees, the occupant of vehicle 1 can virtually experience riding in vehicle 1 together with the virtual object characters. Note that while Fig. 5 shows an example in which a composite image in which a character is riding in a virtual vehicle is displayed as a virtual object, an image in which a character is running on a road without riding in a virtual vehicle may also be displayed as a virtual object.

[0048] Fig. 6 shows a display example when the in-car display mode is set. Fig. 6 also shows character images 261, 271, and 281 that represent the same characters as those shown in Fig. 5. Note that character images 261, 271, and 281 are the same characters as those shown in Fig. 5, but in a different display mode, and are the characters shown in Fig. 5 facing forward.

[0049] An image 250 shown in the upper left of FIG. 6 is a captured image generated by the imaging device 10, and is an image including a subject inside the vehicle 1. FIG. 6 shows an image 250 including a driver U1 in the driver's seat 7 of the vehicle 1, a passenger U2 in the rear seat 11, and the like. In this case, as shown in the upper right of FIG. 6, an image 260 can be created in which a character image 261 is superimposed on the face of the driver U1. As shown in the lower left of FIG. 6, an image 270 can be created in which a character image 271 is superimposed on the passenger seat 8. This image 270 can be displayed with a seat belt image 272 superimposed on the character image 271 to make it appear as if the character is fastening a seat belt. As shown in the lower right of FIG. 6, an image 280 can be created in which a character image 281 is superimposed on the lap of the passenger U2.

[0050] The position at which the character image is superimposed may be set in advance or based on a user instruction. It may also be set appropriately depending on the environment around the vehicle 1 or the environment inside the vehicle 1. For example, an occupant included in a captured image generated by the imaging device 10 may be detected, and a character image may be superimposed on the position of an empty seat where no occupant is present. For example, as shown in the lower left of FIG. 6, an image 270 in which a character is sitting in the passenger seat may be generated. Furthermore, for example, a character image may be superimposed on a portion of an occupant included in a captured image generated by the imaging device 10. For example, as shown in the upper right of FIG. 6, an image 260 in which a character is present near the face of the driver U1 may be generated. In this case, for example, by displaying the character image so that it is superimposed over the entire face of the driver U1, the driver U1 does not need to be concerned about his or her own face being displayed on the display device 30, and can concentrate on driving. Furthermore, for example, if the vehicle 1 is a taxi, an image of only the taxi passenger can be displayed together with the character by superimposing a character image on the taxi driver. Furthermore, for example, as shown in the lower right of FIG. 6, an image 280 in which the occupant is holding the character may be generated. This allows the character to appear as if he is sitting with the passenger or riding in the vehicle 1 together.

[0051] It is preferable that the character images and vehicle images are displayed so that the character's face faces forward of the seats inside vehicle 1. For example, it is preferable that the character's face faces forward of the seats, as in character images 261, 271, and 281 in images 260, 270, and 280. It is also possible that in some vehicles, such as buses, the seats face left, right, or rearward rather than forward of the vehicle. In this case, it is preferable that the character's face faces in the direction of the seats facing left, right, or rearward.

[0052] Furthermore, the display manner of the character image and vehicle image may be changed as appropriate based on a control signal from the vehicle 1. For example, when the vehicle 1 turns right, the character may face right, and when the vehicle 1 turns left, the character may face left. For example, when the vehicle 1 travels uphill, the character may look up, and when the vehicle 1 travels downhill, the character may look down. For example, when the vehicle 1 travels in reverse, only the face of the character image may face backward.

[0053] In this way, by displaying images 260, 270, 280 on display devices 30, 40, in which character images 261, 271, 281 are superimposed on image 250 representing the interior space of vehicle 1, occupant U2 of vehicle 1 can virtually experience riding in vehicle 1 together with the virtual object characters.

[0054] The display contents of the display devices 30 and 40 may be displayed in the same display mode, or may be displayed in different display modes.

[0055] [Example of switching display modes] 7A and 7B are diagrams showing an example of image transition when switching the display mode of images displayed on the display devices 30 and 40, and a display example when changing the display manner of a character image and a vehicle image based on a control signal from the vehicle 1. Note that the character image 305 and the vehicle image 306 shown in Fig. 7A and the character image 371 and the vehicle image 372 shown in Fig. 7B correspond to the character image 211 and the vehicle image 212 shown in the upper right of Fig. 5. Furthermore, the character image 321 shown in Fig. 7A and the character image 351 shown in Fig. 7B correspond to the character image 261 shown in the upper right of Fig. 6.

[0056] 7A shows an example of a transition when a switching operation from the vehicle exterior display mode to the vehicle interior display mode is performed. Specifically, when the vehicle exterior display mode is set, an vehicle exterior display mode screen 300 including an image captured by the imaging device 20 is displayed on the display device 30, 40. Note that on the vehicle exterior display mode screen 300, a mode switching area 301 and a shutter operation area 304 are superimposed on the image 210 shown in the upper right of FIG. 5.

[0057] 7(B) shows an example of transition when a switching operation from the in-vehicle display mode to the outside-vehicle display mode is performed. Specifically, when the in-vehicle display mode is set, an in-vehicle display mode screen 350 including a captured image generated by the imaging device 10 is displayed on the display device 30, 40. Note that on the in-vehicle display mode screen 350, a mode switching area 301 and a shutter operation area 304 are superimposed on the image 260 shown in the upper right of FIG. 6.

[0058] The mode switching area 301 is an area used when performing a switching operation to switch the display mode of an image displayed on the display devices 30 and 40. Specifically, the mode switching area 301 displays an exterior display mode switching button 302 and an interior display mode switching button 303.

[0059] 7(A), when the exterior display mode screen 300 is displayed, if the passenger presses the interior display mode switching button 303, the interior display mode is set. In this case, an interior display mode switching screen 310 for notifying the driver that the display mode will be switched to the interior display mode is temporarily displayed, and then an interior display mode screen 320 including an image in which a character image 321 is superimposed on a captured image generated by the imaging device 10 is displayed.

[0060] 7(B), when the inside-vehicle display mode screen 350 is displayed, if the outside-vehicle display mode switching button 302 is pressed by the occupant, the outside-vehicle display mode is set. In this case, after an outside-vehicle display mode switching screen 360 for notifying the user that the display mode will be switched to the outside-vehicle display mode is temporarily displayed, an outside-vehicle display mode screen 370 including an image in which a character image 371 and a vehicle image 372 are superimposed on a captured image generated by the imaging device 20 is displayed.

[0061] The shutter operation area 304 is an area used when recording an image displayed on the display device 30, 40. Specifically, when the shutter operation area 304 is pressed by an occupant, the image displayed on the display device where the pressing operation was performed is stored as image content in the storage unit 150. In this case, together with the image at the time of pressing, images within a predetermined time before and after that time may be stored as still image content or video content. Furthermore, together with the still image, sound acquired by the sound acquisition unit within that predetermined time may be recorded as character information associated with the image. Furthermore, sound information may be recorded together with the video.

[0062] Although an example in which the display mode is switched manually by the occupant is shown here, the display mode may be switched by other methods. For example, the display mode may be switched by a voice uttered by the occupant, or the display mode may be switched automatically. For example, the display mode may be switched when the occupant utters, "Switch the display mode." In this case, the voice uttered by the occupant can be acquired by analyzing sound information acquired from the sound acquisition unit. An example of automatically switching the display mode will be described in detail with reference to FIGS. 10 to 12.

[0063] 7(C) shows an example of an exterior display mode screen 380 that is displayed when the vehicle 1 moves backward in the exterior display mode. As described above, when the vehicle 1 moves backward, only the face of the character image 381 riding in the vehicle image 382 can be displayed facing backward. In other words, the display can be such that only a part of the character and the virtual vehicle, such as the face or arms, faces backward from the vehicle 1, and the other parts face forward from the vehicle 1.

[0064] 7(D) shows an example of an exterior display mode screen 390 that is displayed when the vehicle 1 turns left when the exterior display mode is set. As described above, when the vehicle 1 turns left, the character image 391 and the vehicle image 392 can be displayed facing the direction of the turn, i.e., the left direction.

[0065] [Example of display control device operation] 8 is a flowchart showing an example of display control processing in the display control device 100. This display control processing is executed by the controller 140 based on a program stored in the storage unit 150. This display control processing is constantly executed for each control cycle. This display control processing will be explained with appropriate reference to FIGS. 1 to 7.

[0066] Although Figure 8 shows an example in which the display modes of the display device 30 and the display device 40 are switched simultaneously, the display modes of the display device 30 and the display device 40 may be switched separately for each device in accordance with the switching operation on each device.

[0067] In step S501, the display mode determination unit 142 determines the current display mode. For example, if the display mode has not been switched by the occupant, it is determined that the current display mode is continuing. If the display mode has been switched by the occupant's operation, it is determined that the display mode after the switch is to be continued. For example, if the exterior display mode switching button 302 is pressed in the mode switching area 301 shown in FIGS. 7(A) and 7(B), it is determined that the display mode has been switched to the exterior display mode. On the other hand, if the interior display mode switching button 303 is pressed in the mode switching area 301 shown in FIGS. 7(A) and 7(B), it is determined that the display mode has been switched to the interior display mode. If the current display mode is set to the exterior display mode or if the display mode has been switched from the interior display mode to the exterior display mode, the process proceeds to step S502. On the other hand, if the current display mode is set to the interior display mode or if the display mode has been switched from the exterior display mode to the interior display mode, the process proceeds to step S509.

[0068] In step S502, the display object determination unit 144 acquires related information including a control signal for the vehicle 1, information about the traveling of the vehicle 1, and information about the surroundings of the vehicle 1 from the vehicle system 110. Then, the display object determination unit 144 determines a character image and a vehicle image to be displayed on the display devices 30 and 40 based on the acquired related information.

[0069] In step S503, the display control unit 146 determines whether the previous display mode was the vehicle exterior display mode. If the previous display mode was the vehicle exterior display mode, the process proceeds to step S506. On the other hand, if the previous display mode was the vehicle interior display mode, the process proceeds to step S504.

[0070] In step S504, the display control unit 146 displays a display mode switching screen for notifying the occupant that the display mode will be switched on the display devices 30, 40. For example, as shown in FIG. 7(B), an exterior display mode switching screen 360 is displayed on the display devices 30, 40.

[0071] In step S505, the display control unit 146 causes the display devices 30, 40 to display a character riding on a virtual vehicle moving forward in the traveling direction of the vehicle 1, superimposed on an actual scene of the vehicle 1. Specifically, the display control unit 146 causes the display devices 30, 40 to display an exterior-of-vehicle display mode screen in which the character image and vehicle image determined by the display object determination unit 144 are superimposed on the captured image generated by the imaging device 20. For example, as shown in FIG. 7(B), an exterior-of-vehicle display mode screen 370 is displayed on the display devices 30, 40.

[0072] In step S506, the display control unit 146 changes the virtual vehicle and character to be displayed on the display devices 30, 40 based on related information including a control signal for the vehicle 1, information about the traveling of the vehicle 1, and information about the surroundings of the vehicle 1. Specifically, the display control unit 146 causes the display devices 30, 40 to display an exterior display mode screen in which the character image and vehicle image determined by the display object determination unit 144 are superimposed on the captured image generated by the imaging device 20. Note that if there is no change in the character image and vehicle image, the display of the same character image and vehicle image continues.

[0073] In step S507, the shooting control unit 145 determines whether or not the shooting button has been pressed. For example, it determines whether or not the shutter operation area 304 (see FIGS. 7(A) and (B)) displayed on the display devices 30 and 40 has been pressed. If the shooting button has been pressed, the process proceeds to step S508. On the other hand, if the shooting button has not been pressed, the operation of the display control process ends.

[0074] In step S508, the photographing control unit 145 executes a storage process to store the image displayed on the display device 30, 40 when the photographing button was pressed as image content in the storage unit 150. The photographing control unit 145 also transmits the image content stored in the storage unit 150 to the management server 70 via the communication unit 160.

[0075] In step S509, the display object determination unit 144 acquires interior information related to the interior of the vehicle 1 from the vehicle system 110. Then, the display object determination unit 144 determines a character image and a vehicle image to be displayed on the display devices 30 and 40, based on the acquired interior information.

[0076] In step S510, the display control unit 146 determines whether the previous display mode was the in-vehicle display mode. If the previous display mode was the in-vehicle display mode, the process proceeds to step S515. On the other hand, if the previous display mode was the outside-vehicle display mode, the process proceeds to step S511.

[0077] In step S511, the display control unit 146 displays a display mode switching screen for notifying the occupant that the display mode will be switched on the display devices 30, 40. For example, as shown in FIG. 7(A), an in-vehicle display mode switching screen 310 is displayed on the display devices 30, 40.

[0078] In step S512, the occupant recognition unit 141 determines whether or not there is an empty seat in the vehicle 1 based on the captured image generated by the imaging device 10. If there is an empty seat in the vehicle 1, the process proceeds to step S513. On the other hand, if there is no empty seat in the vehicle 1, the process proceeds to step S514.

[0079] In step S513, the display control unit 146 causes the display devices 30, 40 to display a character image superimposed on an empty seat in the vehicle 1. Specifically, the display object determination unit 144 determines that a character image should be placed on a seat determined to be empty by the occupant recognition unit 141. Then, the display control unit 146 causes the display devices 30, 40 to display the character image determined by the display object determination unit 144 superimposed on the empty seat according to the determination.

[0080] In step S514, the display control unit 146 causes the display devices 30, 40 to display a character superimposed on a position other than the seat inside the vehicle 1. Specifically, the display object determination unit 144 determines to place a character image on a position other than the seat inside the vehicle 1. Then, the display control unit 146 causes the display devices 30, 40 to display the character image determined by the display object determination unit 144 superimposed on a position according to the determination.

[0081] In step S515, the display object determination unit 144 determines whether or not there has been a change inside the vehicle 1, based on the internal information acquired from the vehicle system 110. For example, based on the images of the interior of the vehicle generated by the imaging device 10, if there is a change in the posture of the occupants sitting in the seats of the vehicle 1, or if there is a change in the voices or conversations of the occupants sitting in the seats of the vehicle 1, it can be determined that there has been a change inside the vehicle 1.

[0082] In step S516, the display control unit 146 changes the display mode of the character to be displayed on the display devices 30, 40, based on the change inside the vehicle 1. Specifically, the display control unit 146 causes the display devices 30, 40 to display the image of the character determined by the display object determination unit 144.

[0083] [Example of switching effects when switching display modes] The above describes an example in which the exterior display mode switching screen 360 (see FIG. 7(B)) is temporarily displayed when switching from the interior display mode to the exterior display mode, and the interior display mode switching screen 310 (see FIG. 7(A)) is temporarily displayed when switching from the exterior display mode to the interior display mode. Here, by performing a predetermined effect using a character when switching the display mode, it is thought that it is possible to increase the sense of familiarity of the occupants with the character and further improve the user experience while riding in the car. Therefore, below is a description of an example in which a predetermined switching effect using a character is performed when switching the display mode.

[0084] Fig. 9 is a flowchart showing an example of display control processing in the display control device 100. Note that this display control processing is a modified example of the display control processing shown in Fig. 8, and therefore parts common to Fig. 8 are assigned the same reference numerals, and the explanation will focus on the points that differ from Fig. 8.

[0085] In step S521, the display control unit 146 executes a predetermined switching effect when switching from the in-vehicle display mode to the outside-vehicle display mode.

[0086] In step S522, the display control unit 146 executes a predetermined switching effect when switching from the vehicle exterior display mode to the vehicle interior display mode.

[0087] Specifically, the display object determination unit 144 determines a switching effect to be performed on at least one of the character image and the vehicle image when switching the display mode. Then, the display control unit 146 displays the character image and the vehicle image on the display devices 30 and 40 so that the switching effect determined by the display object determination unit 144 is performed on at least one of the character image and the vehicle image.

[0088] As a switching effect when switching from the interior display mode to the exterior display mode, for example, an effect can be executed in which a character displayed at a predetermined position inside the vehicle walks outside the vehicle and gets into the virtual vehicle. Also, as a switching effect when switching from the exterior display mode to the interior display mode, for example, an effect can be executed in which a character gets out of the virtual vehicle. Also, an effect can be executed in which a character moves to an image of the interior of the vehicle and sits in a seat and fastens a seat belt.

[0089] When performing these switching effects, the switching from the vehicle interior image to the vehicle exterior image may be instantaneous, or may be sequential based on the transition of the character's movement. Also, only the background of the character image may be switched while maintaining the continuity of the character image.

[0090] [Example of automatic display mode switching] The above describes an example in which the display mode is switched by an operation by an occupant of the vehicle 1. However, if a predetermined condition is satisfied, the display mode may be switched automatically. Therefore, the following describes an example in which the display mode is switched automatically.

[0091] FIG. 10 is a diagram showing an example of image transition when the display mode of the images displayed on the display devices 30 and 40 is automatically switched.

[0092] FIG. 10(A) shows an example of transition when the display mode is automatically switched when a landmark is present around the vehicle 1 and the landmark is included in the imaging range of the imaging device 20. A landmark is a building or landform that serves as a landmark or symbol. For example, buildings such as Tokyo Tower and Kyoto Tower, and landforms such as Mount Fuji and Lake Biwa can be called landmarks. FIG. 10(A) shows an example when ABC Tower 423 is detected as a landmark. The location of the landmark can be associated with a location in map information, for example.

[0093] 10(A) shows an example in which an ABC tower 423 is detected to the right front of the vehicle 1. As shown in Fig. 10(A), when the in-vehicle display mode is set and the ABC tower 423 is detected around the vehicle 1, the outside-vehicle display mode is set to notify the occupants of the presence of the ABC tower 423. In this case, an outside-vehicle display mode switching screen 410 for notifying the occupants of the presence of the ABC tower 423 is temporarily displayed, and then an outside-vehicle display mode screen 420 including the captured image generated by the imaging device 20 is displayed.

[0094] Note that, when an ABC tower 423 is detected around the vehicle 1, a predetermined switching effect may be executed on the interior display mode screen 400 before the exterior display mode is set. For example, when an ABC tower 423 is detected around the vehicle 1, a display mode in which the character image 401 on the interior display mode screen 400 notices something, such as a facial expression that notices something or an action of noticing something, may be executed. Fig. 10(A) shows an example in which a light bulb image 402 is displayed near the character image 401 as the action of the character image 401 noticing something.

[0095] On the exterior display mode screen 420, an effect for notifying the occupants of the ABC tower 423 is executed by at least one of the character image 421 and the vehicle image 422. For example, the character image 421 may be displayed in a display mode in which it points at the ABC tower 423 with a finger, and a balloon image 424 may be displayed in which the character image 421 is saying, "It's the ABC tower." Alternatively, for example, a display mode in which a light emitted from the vehicle image 422 points to the ABC tower 423 may be used. In addition, in this switching effect, a marker sign 425 or an arrow sign 426 indicating the position of the ABC tower 423 in the captured image may be displayed.

[0096] 10(B) shows an example of a transition when the display mode is automatically switched when another vehicle equipped with a display control device capable of executing the display control process of this embodiment is present around vehicle 1 and the vehicle is included in the imaging range of imaging device 20. Note that the other vehicle shown here is a vehicle capable of displaying an in-vehicle display mode screen including a character image and an outside-vehicle display mode screen including a character image and a vehicle image on the display device, as shown in FIGS. 5 to 7. FIG. 10(B) also shows an example when vehicle 453 is detected as the other vehicle.

[0097] FIG. 10B also shows an example in which different virtual objects are assigned to each vehicle equipped with a display control device capable of executing the display control process of this embodiment. The virtual objects assigned to each vehicle are managed by an external device, such as a management server 70. Each vehicle can grasp the positions of other vehicles based on vehicle-to-vehicle communication. Each vehicle can grasp the positions of other vehicles by sequentially transmitting its own position information to an external device, such as the management server 70, and acquiring position information of other vehicles from the external device as needed. Each vehicle can download and display information about the virtual objects assigned to other vehicles using vehicle-to-vehicle communication or from an external device.

[0098] 10(B) shows an example in which a vehicle 453 is detected ahead of the vehicle 1. As shown in Fig. 10(B), when the in-vehicle display mode is set and a vehicle 453 is detected around the vehicle 1, the outside-vehicle display mode is set to notify the occupants of the presence of the vehicle 453. In this case, an outside-vehicle display mode switching screen 440 for notifying the occupants of the presence of the vehicle 453 is temporarily displayed, and then an outside-vehicle display mode screen 450 including a captured image generated by the imaging device 20 is displayed.

[0099] 10(A), a predetermined switching effect may be executed on the interior display mode screen 430 before the exterior display mode is set when a vehicle 453 is detected around the vehicle 1. In FIG. 10(B), an example is shown in which the character image 431 notices something by pointing outside with its right hand and issuing a message saying "There's something outside" (balloon image 433), and a light bulb image 432 is also displayed.

[0100] On the exterior display mode screen 450, an effect for notifying the occupants of the vehicle 453 is executed by at least one of the character image 451 and the vehicle image 452. For example, the character image 451 may be displayed in a display mode in which it points at the vehicle 453 with a finger, and a balloon image 454 may be displayed in which the character image 451 says, "There's a friend here." Alternatively, for example, a display mode in which a light emitted from the vehicle image 422 points to the vehicle 453 may be used. In addition, in this switching effect, a marker sign 457 or an arrow sign 455 indicating the position of the vehicle 453 in the image may be displayed.

[0101] Furthermore, on the exterior vehicle display mode screen 450, a virtual object (character image 456) assigned to the vehicle 453 is displayed in association with the vehicle 453. For example, the character image 456 can be displayed as if it is riding on the roof of the vehicle 453.

[0102] In this way, every time another vehicle equipped with the same display control device as vehicle 1 appears around vehicle 1, the occupants of vehicle 1 can view the virtual objects assigned to the other vehicle in association with the other vehicle. This allows the occupants of vehicle 1 to enjoy the virtual objects assigned to the other vehicle as well as the virtual objects assigned to their own vehicle 1. This further increases the enjoyment for the occupants of viewing captured images of the interior and exterior of the vehicle, thereby improving the user experience.

[0103] [Example of detecting a specific object] Fig. 11 is a diagram showing an example of a detection method for detecting a specific object present around the vehicle 1. The specific object is, for example, the above-mentioned landmark or another vehicle equipped with a display control device capable of executing the display control process in this embodiment. Fig. 11 also shows an example of detecting a specific object present around the vehicle 1 traveling in the direction of arrow AW1.

[0104] First, a case where a specific object fixed on the ground is detected will be described. In this case, a specific object existing around the vehicle 1 can be detected based on map information associated with specific object information related to the specific object and the position information of the vehicle 1 acquired by a position information acquisition unit of the vehicle system 110. The position information acquisition unit acquires position information related to the location of the vehicle 1. For example, this can be realized by a GNSS receiver that acquires position information using a GNSS (Global Navigation Satellite System). The position information includes position-related data such as latitude, longitude, and altitude at the time of receiving a GNSS signal. The position information may also be acquired by other methods of acquiring position information. For example, the position information may be derived using information from surrounding access points or base stations. For example, the position information may also be derived using a position estimation technique implemented by a navigation device.

[0105] Furthermore, the map information associated with the specific object information may be stored in the storage unit 150 and used, or may be acquired from an external device, for example, a map server and used.

[0106] The display mode determination unit 142 determines whether or not a specific object exists within a predetermined range PR1 around the vehicle 1, based on the position of the vehicle 1, based on the position of the vehicle 1, based on the position information of the vehicle 1 acquired from the vehicle system 110 and the map information stored in the storage unit 150. Here, the predetermined range PR1 can be set appropriately based on experimental data, etc. For example, the predetermined range PR1 can be a circle with a radius R1 of several tens of meters to several kilometers.

[0107] Next, the display mode determination unit 142 determines whether the detected specific object is included in the imaging range IR1 of the imaging device 20, based on the position information of the vehicle 1 acquired from the vehicle system 110 and the map information stored in the storage unit 150. Here, the imaging range IR1 can be set based on the traveling direction of the vehicle 1 and the imaging range (angle of view) of the imaging device 20 in the horizontal direction.

[0108] Then, if a specific object exists within the predetermined range PR1 and is included in the imaging range IR1 of the imaging device 20, the display mode determination unit 142 determines to switch from the in-vehicle display mode to the outside-vehicle display mode.

[0109] 11, it is assumed that specific objects LM1 to LM5 exist around the vehicle 1. Note that specific objects LM1 to LM3 exist within a predetermined range PR1, but specific objects LM4 and LM5 exist outside the predetermined range PR1. Also, it is assumed that specific objects LM1 and LM5 exist within an imaging range IR1, but specific objects LM2 to LM4 exist outside the imaging range IR1. In this case, since specific object LM1 exists within the predetermined range PR1 and is included in the imaging range IR1 of the imaging device 20, the display mode determination unit 142 determines to switch from the in-vehicle display mode to the exterior display mode.

[0110] In this case, the display object determination unit 144 and the display control unit 146 acquire various information, such as position information and image information, related to the specific object LM1 detected by the display mode determination unit 142 from the storage unit 150. Then, based on the various information related to the specific object LM1, the display object determination unit 144 causes the vehicle image and the object image to perform an effect for notifying the occupant of the specific object LM1 included in the captured image generated by the imaging device 20. For example, as shown in Fig. 10(A), a character image 421 as a display mode pointing to the ABC tower 423 and a balloon image 424 from the character image 421 can be displayed.

[0111] Furthermore, based on various information about the specific object LM1, the display control unit 146 can execute various effects related to the specific object LM1 in the captured image generated by the imaging device 20. For example, as shown in Fig. 10(A), a marker sign 425 and an arrow sign 426 indicating the position of the ABC tower 423 can be displayed.

[0112] In the above example, a determination is made to switch from the in-vehicle display mode to the exterior display mode when a specific object is present around the vehicle 1 within a predetermined range PR1 based on the position of the vehicle 1 and the specific object is included in the imaging range IR1 of the imaging device 20. However, the relationship between the road on which the vehicle 1 is traveling, the traveling direction of the vehicle 1, and the imaging range of the imaging device 20 may be stored in advance in map information, and the determination may be made based on the map information.

[0113] For example, a road A, a traveling direction AW1, a predetermined position B, and a specific object LM1 are stored in association with each other in the map information. When the vehicle 1 is traveling in the direction of the arrow AW1 on the road A, and has passed the predetermined position B on the road A, the display mode determination unit 142 may determine that the specific object LM1 is present within the predetermined range PR1 and that the specific object LM1 is included in the imaging range IR1 of the imaging device 20. In this case, the display mode determination unit 142 determines to switch from the in-vehicle display mode to the outside-vehicle display mode.

[0114] Next, a case where a specific object moving on the ground, for example, another vehicle equipped with the same display control device as vehicle 1, is detected will be described. In this case, other vehicles present around vehicle 1 can be detected based on the position information of the other vehicles and the position information of vehicle 1 acquired by the position information acquisition unit of vehicle system 110. Here, the positions of the other vehicles can be acquired by various position estimation processes. The position estimation processes include, for example, a position estimation process using an image of the other vehicle included in the captured image generated by imaging device 20, a position estimation process using each sensor of vehicle system 110, such as LIDAR or sonar, a position estimation process using position information of the other vehicle acquired from management server 70 via communication unit 160, and a position estimation process using vehicle-to-vehicle communication.

[0115] The display mode determination unit 142 determines whether or not another vehicle is present around the vehicle 1 within a predetermined range PR1 based on the position of the vehicle 1, based on the position estimation process described above.

[0116] Next, the display mode determination unit 142 determines whether the other vehicle is included in the imaging range IR1 of the imaging device 20 based on the position of the other vehicle relative to the position of vehicle 1, the direction of the other vehicle relative to the position of vehicle 1, and the imaging range IR1 of the imaging device 20.

[0117] If another vehicle is present within the predetermined range PR1 and is included in the imaging range IR1 of the imaging device 20, the display mode determination unit 142 determines to switch from the in-vehicle display mode to the outside-vehicle display mode.

[0118] In this case, the display object determination unit 144 and the display control unit 146 acquire various information, such as position information and image information, related to the other vehicle detected by the display mode determination unit 142. Then, based on the various information related to the other vehicle, the display object determination unit 144 causes the vehicle image and the object image to perform an effect for notifying the occupant of the other vehicle included in the captured image generated by the imaging device 20. For example, as shown in Fig. 10(B), a character image 451 as a display mode pointing to a vehicle 453 and a balloon image 454 from the character image 451 can be displayed.

[0119] Furthermore, based on various information about other vehicles, the display control unit 146 can execute various effects related to other vehicles in the captured image generated by the imaging device 20. For example, as shown in Fig. 10(B), a marker sign 457 and an arrow sign 455 indicating the position of the vehicle 453 can be displayed, and a character image 456 assigned to the vehicle 453 can be displayed in association with the vehicle 453.

[0120] The above describes an example in which it is determined to switch from the in-vehicle display mode to the exterior display mode when a specific object is present around the vehicle 1 within a predetermined range PR1 based on the position of the vehicle 1 and the specific object is included in the imaging range IR1 of the image capture device 20. However, even if a specific object is present around the vehicle 1 within the predetermined range PR1 based on the position of the vehicle 1 but is not included in the imaging range IR1 of the image capture device 20, the in-vehicle display mode may be automatically switched to the exterior display mode. In this case, the occupant may be notified by using audio information, a character image, a vehicle image, or the like that a specific object is nearby, even though it is not displayed in the exterior image.

[0121] Here, it is assumed that the imaging device 20 is a 360-degree camera that can capture images in all directions of the vehicle 1. In this case, it is possible to address this issue by switching the imaging range of the vehicle exterior image displayed on the display devices 30 and 40 to the direction of a specific object that exists within the predetermined range PR1. It is also assumed that the imaging device 20 is a movable camera that can move in at least one of the left-right and up-down directions. In this case, it is possible to address this issue by driving the movable camera so that the imaging range of the vehicle exterior image displayed on the display devices 30 and 40 includes the specific object that exists within the predetermined range PR1.

[0122] Note that other object detection technologies may be used as a detection method for detecting specific objects present around the vehicle 1. For example, artificial intelligence (AI) may be used to detect specific objects present around the vehicle 1. For example, the number of occupants in the vehicle 1, the vehicle 1's traveling position on the road, the display mode switching timing, and the captured image generated by the imaging device 20 may be learned in advance, and this learning data may be used to detect specific objects. For example, for objects included in the captured image when the display mode switching operation is performed, the learning data may be used to calculate an evaluation value for determining the presence or absence of a specific object, and the presence or absence of a specific object may be determined based on the evaluation value. Note that the learning data may be set appropriately through experiments, etc.

[0123] [Example of display control device operation] 12 is a flowchart showing an example of a display mode switching process in the display control device 100. This display mode switching process is executed by the controller 140 based on a program stored in the storage unit 150. This display mode switching process is also executed constantly for each control cycle. This display mode switching process will be described with appropriate reference to FIGS. 10 and 11.

[0124] In step S531, the display mode determination unit 142 determines whether or not a display mode switching operation has been performed by an occupant of the vehicle 1. If a display mode switching operation has been performed, the process proceeds to step S532. On the other hand, if a display mode switching operation has not been performed, the process proceeds to step S533.

[0125] In step S532, the display mode determination unit 142 sets the display mode according to the display mode switching operation by the occupant of the vehicle 1.

[0126] In step S533, the display mode determination unit 142 determines whether the in-car display mode is set. If the in-car display mode is set, the process proceeds to step S534. On the other hand, if the in-car display mode is not set, the display mode switching process ends.

[0127] In step S534, the display mode determination unit 142 determines whether or not a landmark is present within a predetermined range PR1 around the vehicle 1, based on the position of the vehicle 1. If a landmark is present within the predetermined range PR1, the process proceeds to step S535. On the other hand, if no landmark is present within the predetermined range PR1, the display mode switching process ends.

[0128] In step S535, the display mode determination unit 142 determines whether or not the landmark present in the predetermined range PR1 is included in the imaging range IR1 of the imaging device 20. If the landmark is included in the imaging range IR1, the process proceeds to step S538. On the other hand, if the landmark is not included in the imaging range IR1, the process proceeds to step S536.

[0129] In step S536, the display mode determination unit 142 determines whether there are other vehicles equipped with the same display control device as vehicle 1 within a predetermined range PR1 around vehicle 1, which is based on the position of vehicle 1. If there are other vehicles within the predetermined range PR1, the process proceeds to step S537. On the other hand, if there are no other vehicles within the predetermined range PR1, the display mode switching process ends.

[0130] In step S537, the display mode determination unit 142 determines whether or not another vehicle present within the predetermined range PR1 is included in the image capture range IR1 of the image capture device 20. If the vehicle is included in the image capture range IR1, the process proceeds to step S538. On the other hand, if the vehicle is not included in the image capture range IR1, the operation of the display mode switching process ends.

[0131] In step S538, the display mode determination unit 142 sets the vehicle exterior display mode.

[0132] The above describes an example of automatically switching from the interior display mode to the exterior display mode when the interior display mode is set. However, when the exterior display mode is set, the display mode may also be automatically switched from the exterior display mode to the interior display mode. For example, the display mode can be automatically switched from the exterior display mode to the interior display mode when a predetermined change occurs in the interior environment of the vehicle. For example, when the volume of the voice of an occupant of the vehicle 1 exceeds a reference value or when laughter from an occupant is detected, it can be determined that a predetermined change has occurred in the interior environment of the vehicle. Note that laughter can be determined using known sound analysis technology. Furthermore, when a predetermined facial expression, such as a smiling face or an angry face, is detected as the facial expression of an occupant of the vehicle 1, it can be determined that a predetermined change has occurred in the interior environment of the vehicle.

[0133] Furthermore, automatic switching of the display mode may be performed based on other conditions. For example, if the display mode has not been switched for a predetermined time or longer, the display mode may be automatically switched to another display mode. This makes it possible to provide an occupant who has been looking at an outside image for a long time with an opportunity to view an inside image. On the other hand, it is possible to provide an occupant who has been looking at an inside image for a long time with an opportunity to view an outside image. This makes it possible to appropriately switch the mood of the occupant and provide a comfortable vehicle space.

[0134] Other determination methods may also be used as a determination method for determining the timing for automatically switching the display mode. For example, artificial intelligence may be used to determine the timing for automatically switching the display mode. For example, the number of passengers in the vehicle 1, the vehicle 1's traveling position on the road, and the timing for switching the display mode may be learned in advance, and this learning data may be used to determine the timing for automatically switching the display mode. The learning data may be set appropriately through experiments, etc.

[0135] In this embodiment, an example is shown in which the display device 30 is provided on the passenger seat 8 side, but other display media may be used. For example, if the windshield 4 functions as a display media for a HUD (Head Up Display) of the vehicle 1, the above-mentioned displays can be displayed on the HUD display section of the windshield 4.

[0136] As described above, in this embodiment, the imaging devices 10 and 20 installed in the vehicle 1 can capture images of subjects inside and outside the vehicle, and at least one of the captured images from the inside and outside of the vehicle can be displayed on the display devices 30 and 40. In this case, a virtual object, such as a character image or a vehicle image, can be superimposed on the captured image. That is, the user experience can be improved by displaying a character or the like in captured images of the vehicle 1 from different directions. In other words, by displaying the same character or the like superimposed on captured images of the inside and outside of the vehicle, an interesting effect can be achieved in which the character moves together with the occupants inside the vehicle. This makes it possible to display an effect that improves the user experience in the vehicle 1, which has multiple spaces, such as the inside and outside of the vehicle, by making use of these spaces. That is, the occupants' riding experience when viewing captured images from the inside and outside of the vehicle can be improved, providing a comfortable vehicle space.

[0137] Furthermore, according to this embodiment, by appropriately setting the orientation of the virtual object to create an effect in which the vehicle 1 in which the occupant is riding and the virtual object are traveling together, and an effect in which the occupant is riding together, it is possible to realize a user experience in which the occupant inside the vehicle and the virtual object are moving together as one. This improves the enjoyment of the occupant's time riding in the vehicle by viewing captured images of the inside and outside of the vehicle, and provides a comfortable vehicle space.

[0138] For example, assume that vehicle 1 is a shared car and a person travels using vehicle 1. In this case, by displaying a character image or a vehicle image on display devices 30 and 40, the passengers in vehicle 1 can have a simulated experience of the character riding with them in vehicle 1. For example, by displaying an image of a character riding in a virtual vehicle, the feeling that the character is traveling together with vehicle 1 can be enhanced. Furthermore, the passengers in vehicle 1 can have a simulated experience of traveling with the character. This allows the passengers to develop an attachment to the character displayed on display devices 30 and 40, which motivates them to want to ride in the shared car again next time.

[0139] In the present embodiment, an example has been shown in which the display control process and the display mode determination process are executed by the display control device 100, but all or part of these processes may be executed by other devices. In this case, the display control system of the vehicle 1 is configured by the devices that execute part of these processes. For example, at least part of the display control process and the display mode determination process may be executed using electronic device 80 (see FIG. 3) used by the occupants inside the vehicle 1, various information processing devices such as servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0140] Furthermore, a part (or all) of the display control system of the vehicle 1 that can execute the functions of the display control device 100 may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0141] Each process described in this embodiment is executed based on a program that causes a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to a display control device can store the program in the storage device of the display control device. This makes it possible to cause the updated display control device to execute each process described in this embodiment.

[0142] [Configuration and Effects of This Embodiment] The display control method for the vehicle 1 according to this embodiment is a display control method for controlling the display devices 30 and 40 mounted on the vehicle 1. This display control method includes a switching process (steps S501, S531 to S538) for switching between an exterior display mode (an example of the first display mode) in which a virtual object having a form that enables identification of front and rear is superimposed on a first captured image capturing the exterior of the vehicle 1, and an interior display mode in which a virtual object having a display form different from that of the exterior display mode (an example of the first display mode) is superimposed on a second captured image capturing the interior of the vehicle 1, and a display control process (steps S502 to S506, S509 to S516, S521, S522) for displaying a virtual object in the exterior display mode so that the front of the virtual object faces in the direction in which the vehicle 1 is moving forward, and for displaying a virtual object in the interior display mode so that the front of the virtual object faces the direction in which the seat inside the vehicle 1 faces.

[0143] According to this configuration, by appropriately setting the orientation of the virtual object, it is possible to realize an effect in which the vehicle 1 in which the occupants are riding and the virtual object are traveling together (exterior vehicle display mode) and an effect in which the occupants are riding together (interior vehicle display mode). This allows for a user experience in which the occupants in the vehicle and the virtual object are moving together as one. Furthermore, it is possible to improve the enjoyment of the occupants' riding experience when viewing captured images of the interior and exterior of the vehicle, and provide a comfortable vehicle space.

[0144] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (steps S505, S506, S513, S514, and S516) displays, as virtual objects, object images representing objects that can be ridden in a virtual vehicle in an interior display mode (an example of a second display mode), and displays, as virtual objects, composite images in which object images are mounted on vehicle images representing a virtual vehicle in an exterior display mode (an example of a first display mode). For example, as shown in FIG. 5, composite images in which character images 211, 221, and 231 are mounted on vehicle images 212, 222, and 232 representing virtual vehicles can be displayed as virtual objects. Furthermore, as shown in FIG. 6, character images 261, 271, and 281 representing objects that can be ridden in a virtual vehicle can be displayed as virtual objects.

[0145] With this configuration, it is possible to realize an effect in which the vehicle 1 with passengers on board is traveling together with the vehicle images 212, 222, 232 and character images 211, 221, 231 (exterior vehicle display mode), and an effect in which the character images 261, 271, 281 are also traveling together (interior vehicle display mode).

[0146] Furthermore, the display control method for vehicle 1 according to this embodiment further includes an acquisition process (step S502) for acquiring a control signal for vehicle 1, and the display control process (step S506) changes the display manner of some or all of the vehicle image and object image in the exterior display mode (an example of a first display mode) based on the control signal from vehicle 1. For example, as shown in Fig. 7(C), when vehicle 1 is moving backward, character image 381 in which a character riding in vehicle image 382 is looking back can be displayed as a virtual object.

[0147] According to this configuration, the display mode of some or all of the vehicle image and object image can be changed according to the movement of the vehicle 1, so that the occupants of the vehicle 1 can enjoy the changes in the vehicle image and object image corresponding to the movement of the vehicle 1.

[0148] Furthermore, in the display control method for vehicle 1 according to this embodiment, the object image is a character image that resembles an object having a face and a body, and the display control process (step S506) changes the display mode of the character image based on a control signal from vehicle 1 in an exterior display mode (an example of a first display mode) so that at least a part of the face and body faces in the traveling direction of vehicle 1. For example, as shown in Fig. 7(C), when vehicle 1 is moving backward, character image 381 can be displayed as a virtual object so that the face of the character riding in vehicle image 382 faces in the traveling direction (rear) of vehicle 1.

[0149] According to this configuration, it is possible to display a character that is thought to be familiar to the occupants of vehicle 1, and it is possible to realize an effect in which the face or body is turned in the direction of travel of vehicle 1 according to the movement of vehicle 1, so that the occupants of vehicle 1 can enjoy the changes in the character.

[0150] In the display control method for the vehicle 1 according to this embodiment, the display control process (step S513) places a virtual object on a seat in the second captured image in the in-car display mode (an example of the second display mode). For example, as in image 270 shown in FIG. 6, a character image 271 can be placed on the vacant passenger seat 8.

[0151] According to this configuration, by displaying the character as if it were sitting in an empty seat in the vehicle 1, it is possible to further enhance the feeling that the passenger is riding along with the character.

[0152] In addition, the display control method for vehicle 1 according to this embodiment further includes a detection process (step S512) for detecting vacant seats among the seats that are not occupied by an occupant, and the display control process (steps S513, S514) is such that, in an in-car display mode (an example of a second display mode), if there is an vacant seat in the second captured image, a virtual object is placed so as to be superimposed on the vacant seat, and if there is no vacant seat in the second captured image, a virtual object is placed so as to be superimposed on an occupant sitting in the seat in the second captured image.

[0153] According to this configuration, the virtual objects can be arranged differently depending on whether or not there are vacant seats in the vehicle 1, which can improve the enjoyment of the passengers viewing the captured images of the interior of the vehicle.

[0154] Furthermore, in the display control method for the vehicle 1 according to this embodiment, in the in-car display mode (an example of the second display mode), the display control process (step S514) arranges a virtual object so as to be superimposed on the face of the driver in the second captured image. For example, as in image 260 shown in FIG. 6, a character image 261 can be displayed superimposed on part of the face of the driver U1.

[0155] According to this configuration, it is possible to create an unexpected effect by placing a virtual object on the driver's face, which can further increase the enjoyment for passengers viewing captured images from inside the vehicle.

[0156] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control processing (steps S521, S522) performs a predetermined switching effect using a virtual object when switching from the exterior display mode (an example of the first display mode) to the interior display mode (an example of the second display mode) or when switching from the interior display mode to the exterior display mode.

[0157] According to this configuration, even when the display mode is switched, it is possible to provide effects using virtual objects, further increasing the enjoyment for occupants viewing captured images of the inside and outside of the vehicle.

[0158] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control processing (steps S521, S522) executes a switching effect in which, when switching from the exterior display mode (an example of a first display mode) to the interior display mode (an example of a second display mode), the display modes of the object image and the vehicle image are sequentially changed so that the object image descends from the vehicle image, and when switching from the interior display mode to the exterior display mode, the display modes of the object image and the vehicle image are sequentially changed so that the object image ascends into the vehicle image.

[0159] With this configuration, when the display mode is switched, a switching effect can be produced using virtual objects according to the switch, further increasing the enjoyment for occupants viewing captured images of the inside and outside of the vehicle.

[0160] Furthermore, in the display control method for vehicle 1 according to this embodiment, when switching from the exterior display mode (an example of the first display mode) to the interior display mode (an example of the second display mode), the display control process (step S522) moves the object image to an empty seat in the second captured image in which no occupant is seated, and executes a switching performance in which the display mode of the object image is sequentially changed so that the object image appears to be fastening a seat belt in that empty seat.

[0161] With this configuration, when the display mode is switched, an unexpected switching effect can be created in which the object image fastens a seat belt, further increasing the enjoyment for occupants viewing captured images of the inside and outside of the vehicle.

[0162] Furthermore, in the display control method for vehicle 1 according to this embodiment, when the in-vehicle display mode (an example of the second display mode) is set, the switching process (steps S531 to S538) switches from the in-vehicle display mode to the outside-vehicle display mode (an example of the first display mode) when a specific object is present around vehicle 1 within a predetermined range PR1 based on the position of vehicle 1.

[0163] This configuration allows occupants looking at captured images from inside the vehicle to recognize the presence of specific objects outside the vehicle, further increasing the enjoyment for occupants looking at captured images from inside and outside the vehicle.

[0164] In the display control method for the vehicle 1 according to this embodiment, the specific object is a landmark on the ground. The switching process (steps S534, S535, and S538) switches from the interior display mode (an example of the second display mode) to the exterior display mode (an example of the first display mode) when it is determined that a landmark exists within the predetermined range PR1 and that the landmark is included in the imaging range IR1 of the first captured image. When the switching process switches from the interior display mode to the exterior display mode due to the presence of a landmark, the display control process (steps S521 and S505) causes the virtual object to execute an effect to notify the occupant of the landmark included in the first captured image. For example, as shown in FIG. 10(A), an effect can be executed in which a character image 421 is displayed in a display mode pointing at ABC Tower 423 with a finger, and a speech bubble image 424 in which the character image 421 says, "It's ABC Tower."

[0165] According to this configuration, landmarks existing around vehicle 1 can be notified using virtual objects, thereby increasing the enjoyment of occupants who are interested in the landmarks around the area where vehicle 1 is traveling.

[0166] In the display control method for the vehicle 1 according to this embodiment, the specific object is another vehicle for which the above-described display control process can be executed. The switching process (steps S536 to S538) switches from the interior display mode (an example of the second display mode) to the exterior display mode (an example of the first display mode) when it is determined that another vehicle exists within the predetermined range PR1 and that the other vehicle is included in the image capture range of the first captured image. When the switching process switches from the interior display mode to the exterior display mode due to the presence of another vehicle, the display control process (steps S521 and S505) causes the virtual object to execute an effect notifying the occupant of the other vehicle included in the first captured image, and also causes the virtual object assigned to the other vehicle to be displayed in association with the other vehicle. For example, as shown in FIG. 10(B), an effect can be executed in which a character image 451 is displayed in a display mode pointing at a vehicle 453 with a finger and a speech bubble image 454 in which the character image 451 says, "There's a friend here." In addition, it is possible to create an effect in which the character image 456 appears to be riding on the roof of the vehicle 453.

[0167] This configuration makes it possible to use virtual objects to notify other vehicles that exist around vehicle 1. In addition, virtual objects assigned to other vehicles can be displayed in association with the other vehicles, which improves the enjoyment of passengers who are interested in the virtual objects of each vehicle.

[0168] The display control device 100 is a display control device that controls the display devices 30 and 40 mounted on the vehicle 1. The display control device 100 includes a display mode switching unit 143 that switches between an exterior display mode (an example of the first display mode) that displays a virtual object having a form that enables identification of front and rear on a first captured image that captures the exterior of the vehicle 1, and an interior display mode that displays a virtual object in a display form different from that in the exterior display mode (an example of the first display mode) that is superimposed on a second captured image that captures the interior of the vehicle 1, and a display control unit 146 that displays the virtual object in the exterior display mode so that the front of the virtual object faces in the direction in which the vehicle 1 is moving forward, and displays the virtual object in the interior display mode (an example of the second display mode) so that the front of the virtual object faces the direction in which the front of the seat inside the vehicle 1 faces.

[0169] According to this configuration, by appropriately setting the orientation of the virtual object, it is possible to realize an effect in which the vehicle 1 in which the occupants are riding and the virtual object are traveling together (exterior vehicle display mode) and an effect in which the occupants are riding together (interior vehicle display mode). This allows for a user experience in which the occupants in the vehicle and the virtual object are moving together as one. Furthermore, it is possible to improve the enjoyment of the occupants' riding experience when viewing captured images of the interior and exterior of the vehicle, and provide a comfortable vehicle space.

[0170] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0171] 1 vehicle, 2 dashboard, 3 steering wheel, 4 windshield, 5 rearview mirror, 6 car navigation device, 7 driver's seat, 8 passenger seat, 10, 20 imaging device, 11, 12 rear seat, 30, 40 display device, 50 vehicle management system, 60 network, 70 management server, 71 communication unit, 72 control unit, 73 memory unit, 80 electronic device, 100 display control device, 110 vehicle system, 120 UI unit, 130 image acquisition unit, 140 controller, 141 occupant recognition unit, 142 display mode determination unit, 143 display mode switching unit, 144 display object determination unit, 145 shooting control unit, 146 display control unit, 150 memory unit, 160 communication unit

Claims

1. A display control method for controlling a display device mounted on a vehicle, comprising: a switching process for switching between a first display mode in which a virtual object having a form that allows identification of front and rear is superimposed on a first captured image of the exterior of the vehicle, and a second display mode in which the virtual object in a display form different from that in the first display mode is superimposed on a second captured image of the interior of the vehicle; a display control process for displaying the virtual object such that a front portion of the virtual object faces a direction in which the vehicle moves forward in the first display mode, and for displaying the virtual object such that a front portion of the virtual object faces a front portion of a seat inside the vehicle in the second display mode, Display control method.

2. 2. The display control method according to claim 1, In the display control process, in the second display mode, an object image representing an object that can be ridden in a virtual vehicle is displayed as the virtual object, and in the first display mode, a composite image in which the object image is ridden in a vehicle image representing the virtual vehicle is displayed as the virtual object. Display control method.

3. 3. The display control method according to claim 2, further comprising an acquisition process for acquiring a control signal for the vehicle; In the first display mode, the display control process changes a display state of the vehicle image and a part or all of the object image based on a control signal of the vehicle. Display control method.

4. The display control method according to claim 3, the object image is a character image that resembles an object having a face and a body, In the first display mode, the display control process changes the display mode of the character image based on a control signal of the vehicle so that at least a part of the face and body faces in a traveling direction of the vehicle. Display control method.

5. 5. A display control method according to claim 1, comprising: the display control process, in the second display mode, places the virtual object on the seat in the second captured image. Display control method.

6. 6. A display control method according to claim 5, a detection process for detecting an empty seat among the seats where no occupant is seated; In the display control process, in the second display mode, if an empty seat exists in the seat in the second captured image, the virtual object is arranged so as to be superimposed on the empty seat, and if an empty seat does not exist in the seat in the second captured image, the virtual object is arranged so as to be superimposed on an occupant sitting in the seat in the second captured image. Display control method.

7. 5. A display control method according to claim 1, comprising: In the second display mode, the display control process arranges the virtual object so as to be superimposed on a face of the driver in the second captured image. Display control method.

8. 8. A display control method according to claim 1, comprising: the display control process executes a predetermined switching effect using the virtual object when switching from the first display mode to the second display mode or when switching from the second display mode to the first display mode. Display control method.

9. The display control method according to claim 8, In the second display mode, an object image representing an object that can be ridden in a virtual vehicle is displayed as the virtual object, and in the first display mode, a composite image in which the object image is ridden in a vehicle image representing the virtual vehicle is displayed as the virtual object, The display control process includes: When switching from the first display mode to the second display mode, a performance is executed as the switching performance in which the display modes of the object image and the vehicle image are sequentially changed so that the object image descends from the vehicle image; When switching from the second display mode to the first display mode, a performance is executed as the switching performance in which the display modes of the object image and the vehicle image are sequentially changed so that the object image gets into the vehicle image. Display control method.

10. The display control method according to claim 8, In the second display mode, an object image representing an object that can be ridden in a virtual vehicle is displayed as the virtual object, and in the first display mode, a composite image in which the object image is ridden in a vehicle image representing the virtual vehicle is displayed as the virtual object, The display control process includes: When switching from the first display mode to the second display mode, the object image is moved to an empty seat in the second captured image where no occupant is seated, and the switching performance is performed by sequentially changing the display mode of the object image so that the object image appears to be fastening a seat belt in the empty seat. Display control method.

11. 11. A display control method according to claim 1, the switching process switches from the second display mode to the first display mode when a specific object is present around the vehicle within a predetermined range based on a position of the vehicle, when the second display mode is set. Display control method.

12. The display control method according to claim 11, the specific object is a landmark on the ground, the switching process includes switching from the second display mode to the first display mode when it is determined that the landmark is present within the predetermined range and that the landmark is included in an imaging range of the first captured image; When the switching process switches from the second display mode to the first display mode due to the presence of the landmark, the display control process causes the virtual object to execute an effect for notifying an occupant of the landmark included in the first captured image. Display control method.

13. The display control method according to claim 11, the specific object is another vehicle capable of executing the display control process, the switching process switches from the second display mode to the first display mode when it is determined that the other vehicle is present within the predetermined range and that the other vehicle is included in an imaging range of the first captured image; When the switching process switches from the second display mode to the first display mode due to the presence of the other vehicle, the display control process causes the virtual object to perform an effect for notifying a passenger of the other vehicle included in the first captured image, and also causes the virtual object assigned to the other vehicle to be displayed in association with the other vehicle. Display control method.

14. A display control device that controls a display device mounted on a vehicle, a display mode switching unit that switches between a first display mode in which a virtual object having a form that allows identification of front and rear is superimposed on a first captured image of the exterior of the vehicle, and a second display mode in which the virtual object in a display form different from that in the first display mode is superimposed on a second captured image of the interior of the vehicle; a display control unit that displays the virtual object such that a front portion of the virtual object faces a direction in which the vehicle moves forward in the first display mode, and that displays the virtual object such that a front portion of the virtual object faces a front portion of a seat inside the vehicle in the second display mode. Display control device.

Images