Display control method and display control device

Abstract

To enhance the entertainment of an occupant on a vehicle who is watching indoor and outdoor images, and provide a comfortable vehicle space.SOLUTION: A display control apparatus 100 controls display devices 30, 40 mounted on a vehicle 1. The display control apparatus 100 includes: a display mode switching unit 143 which selects a first display mode that superimposes a virtual object configured so that front and rear can be specified, on a first captured image obtained by imaging the outside of the vehicle 1, or a second display mode that superimposes the virtual object in a display mode different from the first display mode, on a second captured image obtained by imaging the inside of the vehicle 1; and a display control unit 146 which displays, in the first display mode, the virtual object so that the front part of the virtual object faces an advancing direction of the vehicle 1, and displays, in the second display mode, the virtual object so that the front part of the virtual object faces a direction of the front of a seat inside the vehicle 1.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a display control method and a display control device for controlling a display device.

Background Art

[0002] In recent years, there are devices that superimpose characters on a captured image and display it on a display device. For example, in a game screen displayed on a touch panel of a user terminal mounted on a gun-shaped controller, a technique has been proposed to make an enemy character appear corresponding to a target marker included in the captured image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, an enemy character can be made to appear in a captured image in the advancing direction of the user. Here, in a vehicle, there are spaces inside and outside the vehicle. Therefore, it is conceivable to use an imaging device installed in the vehicle to capture subjects inside and outside the vehicle and display at least one captured image of the inside and outside of the vehicle on a display device. It is also conceivable to superimpose and display a virtual object on the captured image. However, in a vehicle, since there are a plurality of spaces such as inside and outside the vehicle, there is room for improvement in display control for improving the user experience in the vehicle by making use of these spaces.

[0005] An object of the present invention is to improve the amusement of a passenger viewing captured images inside and outside the vehicle during riding and to provide a comfortable vehicle space.

Means for Solving the Problems

[0006] One aspect of the present invention is a display control method for controlling a display device mounted on a vehicle. This display control method includes a switching process that switches between a first display mode in which a virtual object having a form that allows the front and rear to be identified is superimposed on a first image captured of the outside of the vehicle, and a second display mode in which a virtual object having a different display mode from the first display mode is superimposed on a second image captured of the inside of the vehicle, and in the first display mode, the orientation of the virtual object is determined based on first information regarding the orientation of the vehicle, and in the second display mode, ,before Vehicles Seats inside This includes a display control process in which the orientation of a virtual object is determined based on second information regarding its orientation. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the enjoyment of passengers while riding in a vehicle by viewing captured images of the interior and exterior of the vehicle, and to provide a comfortable vehicle space. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the external configuration of an imaging device and a display device. [Figure 2] Figure 2 shows an example of the external configuration of an imaging device and a display device. [Figure 3] Figure 3 shows an example of a system configuration for a vehicle management system that manages images acquired from a vehicle. [Figure 4] Figure 4 is a block diagram showing an example of the functional configuration of a display control device installed in a vehicle. [Figure 5] Figure 5 shows an example of a display when an image generated by an imaging device is superimposed on that image, along with a virtual object. [Figure 6] Figure 6 shows an example of a display when an image generated by an imaging device is superimposed on that image, along with a virtual object. [Figure 7]Figure 7 shows examples of image transitions when switching the display mode of an image displayed on a display device, and examples of displays when changing the display manner of character images and vehicle images based on vehicle control signals. [Figure 8] Figure 8 is a flowchart showing an example of display control processing in a display control device. [Figure 9] Figure 9 is a flowchart showing an example of display control processing in a display control device. [Figure 10] Figure 10 shows an example of image transitions when the display mode of an image displayed on a display device is automatically switched. [Figure 11] Figure 11 shows an example of a detection method for detecting specific objects present around a vehicle. [Figure 12] Figure 12 is a flowchart showing an example of the display mode switching process in a display control device. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the attached drawings.

[0010] [Example of vehicle internal configuration] Figures 1 and 2 show an example of the external configuration of the imaging devices 10 and 20 and the display devices 30 and 40. In Figures 1 and 2, the interior of the vehicle 1 is shown in a simplified manner, and only the dashboard 2, steering wheel 3, windshield 4, rearview mirror 5, car navigation device 6, driver's seat 7, and passenger seat 8 are shown. Furthermore, Figure 1 shows an example of the external configuration when viewed from the rear of the vehicle 1, with the area in front of the driver's seat 7 and passenger seat 8 in the foreground, and Figure 2 shows an example of the external configuration when viewed from the rear of the vehicle 1, with the area in front of the rear seats 11 and 12 (see Figure 6) in the foreground.

[0011] The imaging device 10 is installed on the ceiling inside the vehicle 1 and captures images (image data) of subjects inside the vehicle 1. For example, the imaging device 10 can be installed on the upper part of the windshield 4, i.e., above the rearview mirror 5. The imaging device 20 is installed on the left end of the dashboard 2 and captures images (image data) of subjects outside the vehicle 1. The imaging devices 10 and 20 are composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. In this example, an example with at least two imaging devices 10 and 20 is shown, but it is also possible to have three or more imaging devices and use images from some of these imaging devices. Furthermore, the installation location of each imaging device is not limited to the examples shown in Figures 1 and 2 and can be changed as appropriate. In addition, one or more devices capable of acquiring images of subjects present in all directions of the vehicle 1 and subjects inside the vehicle 1, such as a 360-degree camera, may be used.

[0012] Display devices 30 and 40 are display devices that display various images based on the control of the controller 140 (see Figure 4). Display devices 30 and 40 can be, for example, touch panels that can accept user operation by touch operation by the user. Display device 30 may be called an assistant monitor, and display device 40 may be called a rear monitor. Alternatively, various images may be displayed on the car navigation device 6 instead of display device 30.

[0013] Specifically, the display device 30 is provided on the passenger seat 8 side of the dashboard 2 so that a passenger sitting on the passenger seat 8 can view it, and various images are displayed on the display unit 31. The display device 40 is provided near the center of the ceiling inside the vehicle 1 so that a passenger sitting on the rear seat can view it, and various images are displayed on the display unit 41. As the display device 40, for example, a flip-down type display device, that is, a foldable display device attached to the ceiling using a support member 42 can be used. In this example, an example including at least two display devices 30 and 40 is shown, but one or three or more display devices may be provided, and partial display control of these display devices may be executed. Also, the installation locations of the respective display devices are not limited to the examples shown in FIGS. 1 and 2 and can be changed as appropriate. Also, display examples of the display devices 30 and 40 are shown in FIGS. 5 to 7 and FIG. 10.

[0014] [Configuration Example of Vehicle Management System] FIG. 3 is a diagram showing an example of the system configuration of a vehicle management system 50 that manages images acquired in the vehicle 1.

[0015] The vehicle management system 50 includes, for example, a network 60 and a management server 70, and is configured to be communicable with the display control device 100, the electronic device 80, etc. of the vehicle 1. The electronic device 80 is an electronic device possessed by a passenger riding in the vehicle 1 and is a wireless device that can be connected to the network 60 using wireless communication. The electronic device 80 is, for example, an information processing device such as a smartphone, a tablet device, or a portable personal computer.

[0016] The network 60 is a network such as a public line network or the Internet. Also, each device constituting the vehicle management system 50 is connected to the network 60 by either a communication method using wireless communication or a communication method using wired communication, or both methods.

[0017] The management server 70 comprises a communication unit 71, a control unit 72, and a storage unit 73. For example, the management server 70 can be a server that manages image content transmitted from the display control device 100 of vehicle 1 and provides image content in response to requests from electronic devices 80. Alternatively, for example, the management server 70 can manage location information transmitted from multiple vehicles and provide each location information in response to requests from each vehicle.

[0018] The communication unit 71, based on the control of the control unit 72, exchanges various types of information with other devices using wired or wireless communication. In other words, the communication unit 71 functions as both an image receiving unit and an image transmitting unit.

[0019] The control unit 72 controls each part based on various programs stored in the memory unit 73. The control unit 72 is implemented by a processing unit such as a CPU (Central Processing Unit).

[0020] The memory unit 73 is a storage medium for storing various types of information. For example, the memory unit 73 stores various types of information necessary for the control unit 72 to perform various processes (e.g., control programs, image content). The memory unit 73 also stores various types of information acquired via the communication unit 71. As the memory unit 73, for example, ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0021] The electronic device 80 can display image content transmitted from the management server 70 on its display unit. While Figure 3 shows an example of information exchange between the electronic device 80 and the display control device 100 via the network 60, information may also be exchanged directly between the electronic device 80 and the display control device 100. In this case, for example, short-range wireless communication (e.g., Wi-Fi or other wireless LAN, Bluetooth®) can be used.

[0022] [Example of the display control device 100 configuration] Figure 4 is a block diagram showing an example of the functional configuration of a display control device 100 mounted on vehicle 1. The display control device 100 displays images on display devices 30 and 40, for example, images on which virtual objects are superimposed on captured images generated by imaging devices 10 and 20. Virtual objects refer to various virtual objects that are displayed superimposed on real space, and can take various forms such as icons and characters. Examples of the display of these virtual objects are shown in Figures 5 to 7 and 10.

[0023] The display control device 100 comprises a vehicle system 110, a UI unit 120, an image acquisition unit 130, a controller 140, a storage unit 150, and a communication unit 160. The controller 140 also comprises a occupant recognition unit 141, a display mode determination unit 142, a display mode switching unit 143, a display object determination unit 144, an image capture control unit 145, and a display control unit 146.

[0024] The vehicle system 110 is a control system installed in the vehicle 1 and includes, for example, various sensors. The vehicle system 110 also outputs detection information acquired by each sensor, control signals for the vehicle 1, etc., to the display mode determination unit 142 and the display object determination unit 144. The sensors include, for example, a millimeter-wave radar, LIDAR (Light Detection and Ranging), sonar, stroke sensor, acceleration sensor, wheel measurement sensor, lateral G sensor, longitudinal G sensor, vehicle speed sensor, sound acquisition unit, seating sensor, and position information acquisition unit. The sound acquisition unit is provided inside or outside the vehicle 1 and acquires sounds around the vehicle 1 (sounds outside the vehicle) and sounds inside the vehicle based on the control of the controller 140. For example, one or more microphones or sound acquisition sensors can be used as the sound acquisition unit. The vehicle system 110 can acquire information related to the vehicle 1's movement and information related to the vehicle 1's surroundings based on the detection information acquired by each sensor. Furthermore, the vehicle system 110 generates control signals for vehicle 1 based on driving operations by the driver and detection information acquired by various sensors. These control signals are used when executing various controls to drive vehicle 1. Based on the control signals for vehicle 1, the vehicle system 110 then executes various controls related to the vehicle 1's movement, such as forward and reverse movement and steering.

[0025] The UI unit 120 functions as an interface that receives operations from the occupants of the vehicle 1 and provides various information to the occupants of the vehicle 1, and includes display devices 30 and 40. The UI unit 120 outputs the received operations to the controller 140 and provides various information based on the control of the controller 140.

[0026] The image acquisition unit 130 acquires various images (still images and videos, etc.) related to the vehicle 1 and includes imaging devices 10 and 20. The imaging devices 10 and 20 output the captured images (image data) generated by the imaging process to the controller 140.

[0027] The controller 140 controls each part based on various programs stored in the memory unit 150. The controller 140 is implemented by a processing unit such as a CPU. The vehicle ECU (Electronic Control Unit) of the vehicle 1 may be used as the controller 140, or a processing unit different from the vehicle ECU may be provided as the controller 140.

[0028] The occupant recognition unit 141 recognizes occupants seated in each seat of the vehicle 1 based on the captured images generated by the imaging device 10, and outputs the recognition result to the display object determination unit 144. That is, by recognizing occupants seated in each seat of the vehicle 1, the occupant recognition unit 141 determines whether or not there is a person in each seat of the vehicle 1, and based on this determination result, it determines whether or not there is an empty seat in the vehicle 1. In this way, the occupant recognition unit 141 detects empty seats in the vehicle 1 in order to find the display position of virtual objects. For example, by executing a seat detection process to detect each seat and a person detection process to recognize a person, it is possible to determine whether or not there is a person in each seat's position. Known object detection techniques and person detection techniques can be used for the seat detection process and person detection process. If the imaging device 10 is fixedly installed, the seat detection process can be omitted, and a person detection process that detects a person at a pre-set position in each seat can be executed. The occupant recognition unit 141 may also determine whether or not there is an empty seat in the vehicle 1 based on internal information acquired from the vehicle system 110. The occupant recognition unit 141 can determine, for example, whether or not there is an empty seat in the vehicle 1 by using seating sensors installed in each seat of the vehicle 1.

[0029] The display mode determination unit 142 determines the display mode of the display devices 30 and 40 based on the information output from the vehicle system 110 and the operation information output from the display devices 30 and 40. The display mode determination unit 142 then outputs the determination result to the display mode switching unit 143 and the display object determination unit 144. For example, if the external display mode switching button 302 (see Figures 7(A) and 7(B)) is pressed, it is determined that the display mode has been switched to the external display mode. On the other hand, if the internal display mode switching button 303 (see Figures 7(A) and 7(B)) is pressed, it is determined that the display mode has been switched to the internal display mode. The display mode determination unit 142 also determines the automatic switching of the display modes of the display devices 30 and 40 based on the information output from the vehicle system 110. An example of this automatic switching determination will be explained in detail with reference to Figure 12.

[0030] The display mode switching unit 143 switches the images generated by the imaging devices 10 and 20 based on the display mode determination result output from the display mode determination unit 142, and outputs the switched image to the display control unit 146. For example, if it is determined that the display has been switched to the in-vehicle display mode, the image captured by the imaging device 10 is output to the display control unit 146. On the other hand, if it is determined that the display has been switched to the out-of-vehicle display mode, the image captured by the imaging device 20 is output to the display control unit 146.

[0031] The display object determination unit 144 determines the virtual objects to be displayed on the imaging devices 10 and 20 based on the information output from the vehicle system 110, the occupant recognition unit 141, the display mode determination unit 142, and the storage unit 150. For example, the display object determination unit 144 can generate virtual objects to be displayed based on the virtual object information acquired from the storage unit 150. In this case, the display object determination unit 144 determines the orientation, position in the captured image, and movement of the virtual objects to be displayed based on the information acquired from the vehicle system 110, the occupant recognition unit 141, and the display mode determination unit 142. For example, the display object determination unit 144 determines the orientation, position, and movement of a character image as a virtual object to be displayed, and whether or not to display a vehicle image. The virtual object information is information used when displaying virtual objects such as character images and vehicle images on the display devices 30 and 40. The display object determination unit 144 then outputs the information necessary to display the determined virtual objects to the display control unit 146.

[0032] The shooting control unit 145 performs storage processing to store the images displayed on the display devices 30 and 40 in the storage unit 150. Specifically, the shooting control unit 145 determines whether or not the shutter operation area 304 (see Figure 7(A)(B)) has been pressed, and stores the images displayed on the display devices 30 and 40 as image content in the storage unit 150 when the shutter operation area 304 has been pressed. Specifically, the shooting control unit 145 obtains the images displayed on the display devices 30 and 40 when the shutter operation area 304 has been pressed from the display control unit 146, and stores the obtained images as image content in the storage unit 150. In addition to still image content at the time of the shutter operation area 304 press operation, video content may also be stored. Furthermore, the shooting control unit 145 transmits the image content stored in the storage unit 150 to the management server 70 via the communication unit 160. In this way, the image content acquired when the shutter operation area 304 is pressed is stored in the storage unit 150 and also transmitted to the management server 70, where it can also be managed. This makes it possible to easily view the image content managed by the management server 70 using other devices, such as electronic devices 80 (see Figure 3).

[0033] The display control unit 146 controls the display devices 30 and 40 to display various images. Specifically, the display control unit 146 superimposes virtual objects determined by the display object determination unit 144 onto the image output from the display mode switching unit 143 and displays them on the display devices 30 and 40. In this case, the display control unit 146 may display the same image on both the display devices 30 and 40, or it may display different images on each of the display devices 30 and 40. The display control unit 146 also outputs the images displayed on the display devices 30 and 40 to the shooting control unit 145. Examples of displays on the display devices 30 and 40 are shown in Figures 5 to 7, etc.

[0034] The memory unit 150 is a storage medium for storing various types of information. For example, the memory unit 150 stores various types of information necessary for the controller 140 to perform various processes (e.g., control programs, map information, virtual object information, image content). The memory unit 150 also stores various types of information acquired via the communication unit 160. The memory unit 150 can be, for example, ROM, RAM, HDD, SSD, or a combination thereof. The image content is a file of images acquired based on the photography operation of the occupant of the vehicle 1 from among the images displayed on the display devices 30 and 40.

[0035] The communication unit 160, based on the control of the controller 140, uses wireless communication to exchange various types of information with other devices. For example, based on the control of the image capture control unit 145, the communication unit 160 transmits images displayed on the display devices 30 and 40 by the display control unit 146 as image content to the management server 70 via the network 60. The communication unit 160 also receives information transmitted from the management server 70 and other external devices via the network 60.

[0036] [Example of an image generated by an imaging device] Figures 5 and 6 show the captured images generated by the imaging devices 10 and 20, and examples of how they are displayed when virtual objects are superimposed on those captured images.

[0037] Figure 5 shows an example of the display when the external display mode is set. Also in Figure 5, vehicle images 212, 222, and 232 representing a virtual vehicle and character images 211, 221, and 231 representing a character that can ride in that virtual vehicle are shown as examples of virtual objects. The characters that make up the virtual objects are examples of characters that are modeled after objects that can ride in the virtual vehicle and have a face and a body. Furthermore, character images 211, 221, and 231 are variations in the display mode of the same character. Thus, the virtual objects shown in this embodiment have a form that allows the front and back to be identified, for example, a character modeled after an object with a face and a body, or a virtual vehicle. Note that the characters can be characters modeled after mechanical robots, or characters modeled after biological animals, plants, etc.

[0038] Image 200, shown in the upper left of Figure 5, is an image captured by the imaging device 20 and includes a subject in front of vehicle 1. Figure 5 shows image 200 including multiple vehicles traveling in front of vehicle 1, oncoming vehicles, etc. In this case, as shown in the upper right of Figure 5, a composite image of a character riding in a virtual vehicle can be superimposed on the lower left of image 200 as vehicle image 212 and character image 211 to create image 210. Also, as shown in the lower left of Figure 5, vehicle image 222 and character image 221 can be superimposed on the lower right of image 200 to create image 220. Furthermore, as shown in the lower right of Figure 5, vehicle image 232 and character image 231 can be superimposed on the upper left of image 200 to create image 230.

[0039] The positions for superimposing the character image and vehicle image may be set in advance, or they may be set based on user instructions. They may also be set appropriately according to the environment around vehicle 1 and the environment inside vehicle 1. For example, objects included in the captured image generated by the imaging device 20, such as a preceding vehicle, can be detected, and the position where no preceding vehicle exists can be set as the position for superimposing the character image and vehicle image. For example, if the area in front of vehicle 1 is congested, the vehicle image 232 and character image 231 can be made to appear as if they are flying in the sky, as shown in the lower right of Figure 5, resulting in image 230.

[0040] Furthermore, regarding the display of character images and vehicle images, it is preferable that when vehicle 1 is moving forward, the images are displayed in a manner that they are moving in front of vehicle 1. For example, as in vehicle images 212, 222, 232 and character images 211, 221, 231 in images 210, 220, and 230, it is preferable that the characters and virtual vehicles are displayed in a manner that they are facing the direction of vehicle 1's movement.

[0041] Furthermore, the display manner of the character image and vehicle image may be changed as appropriate based on the control signal of vehicle 1. For example, the display manner can be changed by tilting the character and virtual vehicle, making them move, or moving their display position in the captured image. For example, when vehicle 1 turns to the right, the character and virtual vehicle can be set to face to the right, and when vehicle 1 turns to the left, the character and virtual vehicle can be set to face to the left. In this case, only a part of the character or only a part of the virtual vehicle may be bent, and effects such as the character's face looking in the direction of the turn or the character leaning its body in the direction of the turn may be implemented. An example of a display where the character and virtual vehicle face to the left is shown in Figure 7(D).

[0042] Furthermore, for example, when vehicle 1 is traveling uphill, the virtual object consisting of the character and the virtual vehicle can be configured to move uphill, and when vehicle 1 is traveling downhill, the virtual object can be configured to move downhill. In other words, the pitch direction of the virtual vehicle can be changed to increase, or the position of the virtual vehicle in the captured image can be changed by moving it vertically.

[0043] Furthermore, for example, when vehicle 1 is traveling on a highway, the character and virtual vehicle can be configured to travel at high speed. That is, the character and virtual vehicle can be modified to move at high speed in the distance ahead of vehicle 1.

[0044] Furthermore, for example, when vehicle 1 is moving in reverse, the display can be configured so that only the character's face is facing backward. An example of this display is shown in Figure 7(C).

[0045] Furthermore, the display modes for character images and vehicle images may be set appropriately according to the environment surrounding vehicle 1 and the environment inside vehicle 1. For example, the display modes for character images and vehicle images can be changed according to the weather when vehicle 1 is running. For example, if vehicle 1 is running in the rain, the character riding in the virtual vehicle may be shown holding an umbrella, a roof may be added to the virtual vehicle the character is riding in, or the character riding in the virtual vehicle may be shown getting soaked. Weather information can be obtained via network 60 from external devices, such as a weather information server. Also, for example, if the volume of the voices of the occupants of vehicle 1 exceeds a certain level, a speech bubble image saying "Looks like fun" may be displayed near the character image, or a character image showing the character turning around with a smile may be displayed.

[0046] In this way, actions corresponding to the state of vehicle 1 can be performed using character images and vehicle images.

[0047] As described above, by displaying images 210, 220, and 230 on the display devices 30 and 40, which superimpose virtual objects consisting of vehicle images 212, 222, and 232 and character images 211, 221, and 231 onto image 200, the same image that the occupants of vehicle 1 actually see, the occupants of vehicle 1 can have a simulated experience of riding in vehicle 1 together with the virtual character objects. In Figure 5, an example is shown in which a composite image of a character riding in a virtual vehicle is displayed as a virtual object, but an image of a character running on a road without riding in a virtual vehicle may also be displayed as a virtual object.

[0048] Figure 6 shows an example of the display when the in-vehicle display mode is set. Figure 6 also shows character images 261, 271, and 281, which represent the same characters as those shown in Figure 5. Note that character images 261, 271, and 281 are variations of the characters shown in Figure 5, with the characters facing forward.

[0049] Image 250, shown in the upper left of Figure 6, is an image captured by the imaging device 10 and includes subjects inside the vehicle 1. Figure 6 shows image 250 including the driver U1 sitting in the driver's seat 7 of the vehicle 1, and the passenger U2 sitting in the rear seat 11. In this case, as shown in the upper right of Figure 6, the character image 261 can be superimposed on the face of the driver U1 to create image 260. Also, as shown in the lower left of Figure 6, the character image 271 can be superimposed on the passenger seat 8 to create image 270. In this image 270, to make it appear as if the character is wearing a seat belt, the seat belt image 272 can be superimposed on the character image 271. Also, as shown in the lower right of Figure 6, the character image 281 can be superimposed on the knees of passenger U2 to create image 280.

[0050] The position where the character image is superimposed may be set in advance, or it may be set based on user instructions. It may also be set appropriately according to the environment around the vehicle 1 or the environment inside the vehicle 1. For example, the system can detect occupants included in the captured image generated by the imaging device 10 and superimpose the character image on the position of an empty seat where no occupants exist. For example, as shown in the lower left of Figure 6, an image 270 can be created in which the character appears to be sitting in the passenger seat. Alternatively, for example, the character image can be superimposed on a portion of the occupants included in the captured image generated by the imaging device 10. For example, as shown in the upper right of Figure 6, an image 260 can be created in which the character appears to be near the face of the driver U1. In this case, for example, by displaying the character image superimposed on the entire face of the driver U1, the driver U1 does not need to worry about their own face being displayed on the display device 30 and can concentrate on driving. Also, for example, if the vehicle 1 is a taxi, by superimposing the character image on the taxi driver, an image of only the taxi passenger can be displayed along with the character. Furthermore, for example, as shown in the lower right of Figure 6, an image 280 can be created in which an occupant appears to be holding the character. These features allow for effects such as making it appear as if the character is sitting with the crew or riding in the same vehicle (vehicle 1).

[0051] Furthermore, regarding the display of character images and vehicle images, it is preferable that the character's face faces forward towards the seats inside the vehicle 1. For example, as shown in character images 261, 271, and 281 in images 260, 270, and 280, it is preferable that the character's face faces forward towards the seats. However, it is conceivable that in some vehicles such as buses, the seats may face to the left, right, or rear, rather than forward. In this case, it is preferable that the character's face faces towards the seats that are facing to the left, right, or rear.

[0052] Furthermore, the display manner of the character image and vehicle image may be changed as appropriate based on the control signal of vehicle 1. For example, when vehicle 1 turns to the right, the character may be positioned to face to the right, and when vehicle 1 turns to the left, the character may be positioned to face to the left. Also, for example, when vehicle 1 is traveling uphill, the character may be positioned to look upwards, and when vehicle 1 is traveling downhill, the character may be positioned to look downwards. Also, for example, when vehicle 1 is moving in reverse, only the face of the character image may be positioned to face backwards.

[0053] In this way, by displaying images 260, 270, and 280 on the display devices 30 and 40, which are created by superimposing character images 261, 271, and 281 onto image 250 representing the interior space of vehicle 1, the occupant U2 of vehicle 1 can have a simulated experience of riding in vehicle 1 together with the virtual object characters.

[0054] The display content of the display devices 30 and 40 may be the same display mode, or they may each be a different display mode.

[0055] [Example of switching display modes] Figure 7 shows examples of image transitions when switching the display mode of images displayed on the display devices 30 and 40, and examples of displays when changing the display mode of character images and vehicle images based on control signals from vehicle 1. Character image 305 and vehicle image 306 shown in Figure 7(A) and character image 371 and vehicle image 372 shown in Figure 7(B) correspond to character image 211 and vehicle image 212 shown in the upper right of Figure 5. Also, character image 321 shown in Figure 7(A) and character image 351 shown in Figure 7(B) correspond to character image 261 shown in the upper right of Figure 6.

[0056] Figure 7(A) shows an example of the transition when switching from the external display mode to the internal display mode. Specifically, when the external display mode is set, the external display mode screen 300, which includes the captured image generated by the imaging device 20, is displayed on the display devices 30 and 40. The external display mode screen 300 has the mode switching area 301 and the shutter operation area 304 superimposed on the image 210 shown in the upper right of Figure 5.

[0057] Figure 7(B) shows an example of the transition when switching from in-vehicle display mode to exterior display mode. Specifically, when in-vehicle display mode is set, the in-vehicle display mode screen 350, which includes the captured image generated by the imaging device 10, is displayed on the display devices 30 and 40. The in-vehicle display mode screen 350 has the mode switching area 301 and the shutter operation area 304 superimposed on the image 260 shown in the upper right of Figure 6.

[0058] The mode switching area 301 is an area used when performing a switching operation to switch the display mode of the images displayed on the display devices 30 and 40. Specifically, the mode switching area 301 displays an exterior display mode switching button 302 and an interior display mode switching button 303.

[0059] As shown in Figure 7(A), when the exterior display mode screen 300 is displayed, pressing the interior display mode switch button 303 by an occupant sets the interior display mode. In this case, an interior display mode switch screen 310 is temporarily displayed to notify the user that the system is switching to interior display mode, and then an interior display mode screen 320 is displayed, which includes an image in which a character image 321 is superimposed on an image captured by the imaging device 10.

[0060] As shown in Figure 7(B), when the in-vehicle display mode screen 350 is displayed, pressing the exterior display mode switch button 302 by an occupant sets the exterior display mode. In this case, an exterior display mode switch screen 360 is temporarily displayed to notify the user that the system is switching to exterior display mode, and then an exterior display mode screen 370 is displayed, which includes an image in which a character image 371 and a vehicle image 372 are superimposed on an image captured by the imaging device 20.

[0061] The shutter operation area 304 is an area used to record images displayed on the display devices 30 and 40. Specifically, when the shutter operation area 304 is pressed by a crew member, the image that was displayed on the display device at the time of the press is stored as image content in the storage unit 150. In this case, images from a predetermined time before and after the press may be stored as still image content or video content along with the image at the time of the press. In addition, sound acquired by the sound acquisition unit within that predetermined time may be recorded as text information associated with the image along with the still image. Sound information may also be recorded along with the video.

[0062] Here, we show an example of switching the display mode by manual operation by the crew, but the display mode may be switched by other methods. For example, the display mode may be switched by a voice command from the crew, or it may be switched automatically. For example, the display mode may be switched when the crew says, "Switch the display mode." In this case, it is possible to obtain the voice command from the crew by analyzing the sound information acquired from the sound acquisition unit. An example of automatically switching the display mode will be explained in detail with reference to Figures 10 to 12.

[0063] Figure 7(C) shows an example of the external display mode screen 380 displayed when the vehicle 1 is reversing, when the external display mode is set. As described above, when the vehicle 1 is reversing, the display can be configured so that only the face of the character image 381 riding in the vehicle image 382 is facing backward. In other words, the display can be configured so that only a part of the character and virtual vehicle, such as the face or arms, is facing the rear of the vehicle 1, while the other parts are facing forward.

[0064] Figure 7(D) shows an example of the display of the exterior display mode screen 390 that is shown when the vehicle 1 turns to the left, when the exterior display mode is set. As described above, when the vehicle 1 turns to the left, the character image 391 and the vehicle image 392 can be configured to face the direction of the turn, i.e., to the left.

[0065] [Example of display control device operation] Figure 8 is a flowchart illustrating an example of the display control process in the display control device 100. This display control process is executed by the controller 140 based on a program stored in the memory unit 150. This display control process is executed continuously at each control cycle. This display control process will be explained with reference to Figures 1 to 7 as appropriate.

[0066] Although Figure 8 shows an example of simultaneously switching the display modes of display device 30 and display device 40, the switching of the display modes of display device 30 and display device 40 may be performed individually for each device according to the switching operation in each device.

[0067] In step S501, the display mode determination unit 142 determines the current display mode. For example, if the display mode has not been switched by the occupant, it is determined that the current display mode is continuing. If the display mode has been switched by the occupant, the display mode after the switch is determined. For example, in the mode switching area 301 shown in Figures 7(A) and 7(B), if the external display mode switching button 302 is pressed, it is determined that the display mode has been switched to the external display mode. On the other hand, in the mode switching area 301 shown in Figures 7(A) and 7(B), if the in-vehicle display mode switching button 303 is pressed, it is determined that the display mode has been switched to the in-vehicle display mode. If the current display mode is set to external display mode, or if the display mode has been switched from in-vehicle display mode to external display mode, the process proceeds to step S502. On the other hand, if the current display mode is set to in-vehicle display mode, or if the display mode has been switched from external display mode to in-vehicle display mode, the process proceeds to step S509.

[0068] In step S502, the display object determination unit 144 acquires relevant information from the vehicle system 110, including control signals for vehicle 1, information regarding the vehicle 1's movement, and information regarding the vehicle 1's surroundings. Based on the acquired relevant information, the display object determination unit 144 determines the character image and vehicle image to be displayed on the display devices 30 and 40.

[0069] In step S503, the display control unit 146 determines whether the previous display mode was the external display mode. If the previous display mode was the external display mode, the process proceeds to step S506. On the other hand, if the previous display mode was the internal display mode, the process proceeds to step S504.

[0070] In step S504, the display control unit 146 displays a display mode switching screen on the display devices 30 and 40 to notify the occupants that the display mode is being switched. For example, as shown in Figure 7(B), the external display mode switching screen 360 is displayed on the display devices 30 and 40.

[0071] In step S505, the display control unit 146 overlays a character riding in a virtual vehicle moving forward in the direction of travel of vehicle 1 onto the actual scenery of vehicle 1 and displays it on the display devices 30 and 40. Specifically, the display control unit 146 displays an external display mode screen on the display devices 30 and 40, which is an external display mode screen in which the character image and vehicle image determined by the display object determination unit 144 are overlaid on the captured image generated by the imaging device 20. For example, as shown in Figure 7(B), the external display mode screen 370 is displayed on the display devices 30 and 40.

[0072] In step S506, the display control unit 146 changes the virtual vehicle and character to be displayed on the display devices 30 and 40 based on relevant information including control signals for vehicle 1, information regarding the movement of vehicle 1, and information regarding the surroundings of vehicle 1. Specifically, the display control unit 146 displays an external display mode screen on the display devices 30 and 40, in which the character image and vehicle image determined by the display object determination unit 144 are superimposed on the captured image generated by the imaging device 20. If there are no changes to the character image and vehicle image, the display of the same character image and vehicle image continues.

[0073] In step S507, the shooting control unit 145 determines whether or not the shooting button has been pressed. For example, it determines whether or not the shutter operation area 304 (see Figures 7(A) and 7(B)) displayed on the display devices 30 and 40 has been pressed. If the shooting button has been pressed, the process proceeds to step S508. On the other hand, if the shooting button has not been pressed, the display control process terminates.

[0074] In step S508, the shooting control unit 145 performs a storage process to store the image that was displayed on the display devices 30 and 40 when the shooting button was pressed as image content in the storage unit 150. The shooting control unit 145 also transmits the image content stored in the storage unit 150 to the management server 70 via the communication unit 160.

[0075] In step S509, the display object determination unit 144 acquires internal information about the interior of the vehicle 1 from the vehicle system 110. Based on the acquired internal information, the display object determination unit 144 determines the character image and vehicle image to be displayed on the display devices 30 and 40.

[0076] In step S510, the display control unit 146 determines whether the previous display mode was the in-vehicle display mode. If the previous display mode was the in-vehicle display mode, the process proceeds to step S515. On the other hand, if the previous display mode was the exterior display mode, the process proceeds to step S511.

[0077] In step S511, the display control unit 146 displays a display mode switching screen on the display devices 30 and 40 to notify the occupants that the display mode is being switched. For example, as shown in Figure 7(A), the in-vehicle display mode switching screen 310 is displayed on the display devices 30 and 40.

[0078] In step S512, the occupant recognition unit 141 determines whether or not there are vacant seats in vehicle 1 based on the captured image generated by the imaging device 10. If there are vacant seats in vehicle 1, the process proceeds to step S513. On the other hand, if there are no vacant seats in vehicle 1, the process proceeds to step S514.

[0079] In step S513, the display control unit 146 superimposes character images onto the empty seats in vehicle 1 and displays them on the display devices 30 and 40. Specifically, the display object determination unit 144 decides to place character images on the seats that the occupant recognition unit 141 has determined to be empty. The display control unit 146 then superimposes the character images determined by the display object determination unit 144 onto the corresponding empty seats and displays them on the display devices 30 and 40.

[0080] In step S514, the display control unit 146 superimposes the character onto a location other than the seats inside the vehicle 1 and displays it on the display devices 30 and 40. Specifically, the display object determination unit 144 decides to place the character image on a location other than the seats inside the vehicle 1. Then, the display control unit 146 superimposes the character image determined by the display object determination unit 144 onto the display devices 30 and 40 at the location determined by that decision.

[0081] In step S515, the display object determination unit 144 determines whether or not there has been a change inside the vehicle 1 based on the internal information acquired from the vehicle system 110. For example, based on the interior images generated by the imaging device 10, if the posture of the occupants sitting in each seat of the vehicle 1 changes, or if the voices or conversations of the occupants sitting in each seat of the vehicle 1 change, it can be determined that there has been a change inside the vehicle 1.

[0082] In step S516, the display control unit 146 changes the display mode of the character to be displayed on the display devices 30 and 40 based on changes inside the vehicle 1. Specifically, the display control unit 146 displays the character image determined by the display object determination unit 144 on the display devices 30 and 40.

[0083] [Example of executing a transition animation when switching display modes] The above examples show that when switching from in-vehicle display mode to exterior display mode, the exterior display mode switching screen 360 (see Figure 7(B)) is temporarily displayed, and when switching from exterior display mode to in-vehicle display mode, the in-vehicle display mode switching screen 310 (see Figure 7(A)) is temporarily displayed. Here, it is thought that by performing a predetermined animation using a character when switching display modes, the passenger's sense of familiarity with the character can be improved, and the user experience while riding can be further enhanced. Therefore, the following examples show a predetermined switching animation using a character when switching display modes.

[0084] Figure 9 is a flowchart showing an example of the display control process in the display control device 100. Note that this display control process is a modified version of the display control process shown in Figure 8; therefore, parts common to both Figure 8 and Figure 8 are denoted by the same reference numerals, and the explanation will focus on the differences from Figure 8.

[0085] In step S521, the display control unit 146 performs a predetermined switching animation when switching from the in-vehicle display mode to the exterior display mode.

[0086] In step S522, the display control unit 146 performs a predetermined switching animation when switching from the external display mode to the internal display mode.

[0087] Specifically, the display object determination unit 144 determines a switching animation to be performed on at least one of the character image and vehicle image when switching display modes. The display control unit 146 then displays the character image and vehicle image on the display devices 30 and 40 so that the switching animation determined by the display object determination unit 144 is performed on at least one of the character image and vehicle image.

[0088] As a transition animation when switching from in-car display mode to exterior display mode, for example, a character displayed in a designated position inside the car can be shown walking outside and getting into a virtual vehicle. Similarly, as a transition animation when switching from exterior display mode to in-car mode, for example, a character can be shown getting out of the virtual vehicle. Furthermore, a character that has moved to the interior image can be shown sitting in a seat and fastening a seatbelt.

[0089] When performing these transition effects, the switch from the interior image to the exterior image may be instantaneous, or it may be a sequential switch based on the character's movement. Alternatively, the continuity of the character image may be maintained while only the background of the character image is changed.

[0090] [Example of automatic display mode switching] The above example illustrates how the display mode can be switched by the operation of the occupant of vehicle 1. However, under certain conditions, the display mode may be switched automatically. Therefore, the following example illustrates how the display mode can be switched automatically.

[0091] Figure 10 shows an example of image transitions when the display mode of images displayed on the display devices 30 and 40 is automatically switched.

[0092] Figure 10(A) shows an example of the automatic switching of the display mode when a landmark exists around vehicle 1 and that landmark is included in the imaging range of the imaging device 20. A landmark is a building or landform that serves as a landmark or symbol. For example, buildings such as Tokyo Tower and Kyoto Tower, or landforms such as Mount Fuji and Lake Biwa can be called landmarks. Figure 10(A) shows an example where ABC Tower 423 is detected as a landmark. The location of a landmark can also be associated with, for example, its location in map information.

[0093] Figure 10(A) shows an example where the ABC Tower 423 is detected to the right front of Vehicle 1. As shown in Figure 10(A), when the in-vehicle display mode is set and the ABC Tower 423 is detected around Vehicle 1, the external display mode is set to notify the occupants of the presence of the ABC Tower 423. In this case, the external display mode switching screen 410 for notifying the occupants of the presence of the ABC Tower 423 is temporarily displayed, and then the external display mode screen 420 including the captured image generated by the imaging device 20 is displayed.

[0094] Furthermore, if the ABC Tower 423 is detected around the vehicle 1, a predetermined switching animation may be performed on the in-vehicle display mode screen 400 before the exterior display mode is set. For example, if the ABC Tower 423 is detected around the vehicle 1, the character image 401 on the in-vehicle display mode screen 400 can be displayed in a way that indicates it has noticed something, such as an expression of noticing something or an action of noticing something. Figure 10(A) shows an example where a light bulb image 402 is displayed near the character image 401 as an action of the character image 401 noticing something.

[0095] In the external display mode screen 420, a visual effect to notify the occupants of the ABC Tower 423 is performed using at least one of the character image 421 and the vehicle image 422. For example, a character image 421 can be displayed in a manner that points to the ABC Tower 423 with a finger, and a speech bubble image 424 can be displayed that indicates the character image 421 is saying "It's the ABC Tower." Alternatively, for example, a visual effect may be used in which light emanating from the vehicle image 422 points to the ABC Tower 423. Furthermore, in this switching effect, a marker sign 425 or an arrow sign 426 indicating the position of the ABC Tower 423 in the captured image may also be displayed.

[0096] Figure 10(B) shows an example of a transition when the display mode is automatically switched when another vehicle equipped with a display control device capable of executing the display control processing in this embodiment is present around vehicle 1 and that vehicle is included in the imaging range of the imaging device 20. The other vehicle shown here is a vehicle capable of displaying an in-vehicle display mode screen including character images, or an out-of-vehicle display mode screen including character images and vehicle images on the display device, as shown in Figures 5 to 7. Figure 10(B) also shows an example when vehicle 453 is detected as the other vehicle.

[0097] Figure 10(B) also shows an example in which each vehicle equipped with a display control device capable of executing the display control processing in this embodiment is assigned a different virtual object. The virtual objects assigned to each vehicle are managed by an external device, such as a management server 70. Furthermore, each of these vehicles can determine the location of other vehicles based on vehicle-to-vehicle communication. Each of these vehicles can determine the location of other vehicles by sequentially transmitting its own location information to an external device, such as a management server 70, and obtaining location information of other vehicles from the external device as needed. In addition, each of these vehicles can download and display information about the virtual objects assigned to other vehicles using vehicle-to-vehicle communication or from an external device.

[0098] Figure 10(B) shows an example where vehicle 453 is detected in front of vehicle 1. As shown in Figure 10(B), when the in-vehicle display mode is set and vehicle 453 is detected around vehicle 1, the external display mode is set to notify the occupants of the presence of vehicle 453. In this case, an external display mode switching screen 440 for notifying the occupants of the presence of vehicle 453 is temporarily displayed, and then an external display mode screen 450 including the captured image generated by the imaging device 20 is displayed.

[0099] In addition, similar to the example shown in Figure 10(A), a predetermined switching animation may be performed on the in-vehicle display mode screen 430 before the exterior display mode is set, when vehicle 453 is detected around vehicle 1. Figure 10(B) shows an example in which the character image 431 makes an action as if pointing outside with its right hand, and displays a message saying "There's something outside" (speech bubble image 433), and the light bulb image 432 is displayed.

[0100] In the external display mode screen 450, a visual effect to notify the occupants of the vehicle 453 is performed using at least one of the character image 451 and the vehicle image 452. For example, a character image 451 can be displayed in a manner that points to the vehicle 453 with a finger, and a speech bubble image 454 can be displayed that indicates the character image 451 is saying, "There are friends here." Alternatively, for example, a visual effect may be used in which light emanating from the vehicle image 422 points to the vehicle 453. Furthermore, in this switching effect, a marker sign 457 or an arrow sign 455 indicating the position of the vehicle 453 in the image may also be displayed.

[0101] Furthermore, in the external display mode screen 450, the virtual object (character image 456) assigned to the vehicle 453 is displayed in association with the vehicle 453. For example, it can be displayed as if the character image 456 is sitting on the roof of the vehicle 453.

[0102] In this way, whenever another vehicle equipped with the same display control device as vehicle 1 appears around vehicle 1, the occupants of vehicle 1 can see the virtual objects assigned to the other vehicle in association with those other vehicles. This allows them to enjoy the virtual objects assigned to other vehicles along with the virtual objects assigned to their own vehicle 1. This further enhances the enjoyment of the occupants viewing the captured images inside and outside the vehicle, thereby improving the user experience.

[0103] [Example of detecting a specific object] Figure 11 shows an example of a detection method for detecting specific objects present around vehicle 1. These specific objects include, for example, the landmarks mentioned above, or other vehicles equipped with a display control device capable of performing the display control processing described in this embodiment. Figure 11 also shows an example of detecting specific objects present around vehicle 1 as it moves in the direction of arrow AW1.

[0104] First, we will explain the case of detecting a specific object fixed to the ground. In this case, the specific object present around the vehicle 1 can be detected based on map information associated with the specific object and the position information of the vehicle 1 acquired by the position information acquisition unit of the vehicle system 110. The position information acquisition unit acquires position information related to the location where the vehicle 1 is located. For example, this can be achieved by a GNSS receiver that acquires position information using GNSS (Global Navigation Satellite System). This position information includes various position data such as latitude, longitude, and altitude at the time of receiving the GNSS signal. Position information may also be acquired by other methods of acquiring position information. For example, position information may be derived using information from access points or base stations in the vicinity. Alternatively, position information may be derived using position estimation technology by a navigation device.

[0105] Furthermore, map information associated with specific object information may be stored in the storage unit 150 and used, or it may be obtained from an external device, such as a map server and used.

[0106] The display mode determination unit 142 determines whether a specific object exists within a predetermined range PR1 around the vehicle 1, based on the vehicle system 110's position information and the map information stored in the storage unit 150. Here, the predetermined range PR1 can be set as appropriate using experimental data, etc. For example, the predetermined range PR1 can be a circle with a radius R1 of several tens of meters to several kilometers.

[0107] Next, the display mode determination unit 142 determines whether the detected specific object is included in the imaging range IR1 of the imaging device 20, based on the position information of the vehicle 1 acquired from the vehicle system 110 and the map information stored in the storage unit 150. Here, the imaging range IR1 can be set based on the direction of travel of the vehicle 1 and the horizontal imaging range (angle of view) of the imaging device 20.

[0108] The display mode determination unit 142 then determines that if a specific object exists within a predetermined range PR1 and that the specific object is included in the imaging range IR1 of the imaging device 20, it will switch from the in-vehicle display mode to the out-of-vehicle display mode.

[0109] For example, in the example shown in Figure 11, we assume that specific objects LM1 to LM5 exist around vehicle 1. Note that specific objects LM1 to LM3 are located within a predetermined range PR1, while specific objects LM4 and LM5 are located outside the predetermined range PR1. Furthermore, specific objects LM1 and LM5 are located within the imaging range IR1, while specific objects LM2 to LM4 are located outside the imaging range IR1. In this case, since specific object LM1 is located within the predetermined range PR1 and specific object LM1 is included in the imaging range IR1 of the imaging device 20, the display mode determination unit 142 determines to switch from in-vehicle display mode to out-of-vehicle display mode.

[0110] In this case, the display object determination unit 144 and the display control unit 146 acquire various information, such as location information and image information, related to the specific object LM1 detected by the display mode determination unit 142 from the storage unit 150. Then, based on the various information related to the specific object LM1, the display object determination unit 144 causes the vehicle image and object image to perform effects to notify the occupants of the specific object LM1 included in the captured image generated by the imaging device 20. For example, as shown in Figure 10(A), a character image 421 as a display mode pointing to the ABC tower 423 and a blown-out image 424 from the character image 421 can be displayed.

[0111] Furthermore, the display control unit 146 can perform various effects related to the specific object LM1 in the captured image generated by the imaging device 20, based on various information about the specific object LM1. For example, as shown in Figure 10(A), it can display a marker sign 425 indicating the position of the ABC tower 423 and an arrow sign 426.

[0112] The above example illustrates how to determine whether to switch from in-vehicle display mode to external display mode when a specific object exists within a predetermined range PR1 around vehicle 1, based on the position of vehicle 1, and that specific object is included in the imaging range IR1 of the imaging device 20. However, the relationship between the road on which vehicle 1 travels, the direction of travel of vehicle 1, and the imaging range of the imaging device 20 may be stored in map information in advance, and the determination may be made based on that map information.

[0113] For example, road A, direction of travel AW1, predetermined position B, and specific object LM1 are associated and stored in the map information. Then, when vehicle 1 is traveling on road A in the direction of arrow AW1, the display mode determination unit 142 may determine, based on the condition that it has passed predetermined position B on road A, that specific object LM1 is within a predetermined range PR1 and that specific object LM1 is included in the imaging range IR1 of the imaging device 20. In this case, the display mode determination unit 142 determines to switch from in-vehicle display mode to out-of-vehicle display mode.

[0114] Next, we will describe the case of detecting a specific object moving on the ground, for example, another vehicle equipped with the same display control device as vehicle 1. In this case, other vehicles present around vehicle 1 can be detected based on the location information of the other vehicles and the location information of vehicle 1 acquired by the location information acquisition unit of the vehicle system 110. Here, the location of the other vehicles can be obtained by various location estimation processes. These location estimation processes include, for example, location estimation using images of other vehicles included in the captured images generated by the imaging device 20, location estimation using sensors equipped in the vehicle system 110, such as LIDAR and sonar, location estimation using location information of other vehicles obtainable from the management server 70 via the communication unit 160, and location estimation using vehicle-to-vehicle communication.

[0115] The display mode determination unit 142 determines, based on the position estimation process described above, whether or not another vehicle exists within a predetermined range PR1 around the vehicle 1, with the position of the vehicle 1 as the reference point.

[0116] Next, the display mode determination unit 142 determines whether the other vehicle is included in the imaging range IR1 of the imaging device 20, based on the position of the other vehicle relative to the position of vehicle 1, the direction of the other vehicle relative to the position of vehicle 1, and the imaging range IR1 of the imaging device 20.

[0117] The display mode determination unit 142 then determines that if another vehicle is present within the predetermined range PR1 and that vehicle is included in the imaging range IR1 of the imaging device 20, it will switch from the in-vehicle display mode to the out-of-vehicle display mode.

[0118] In this case, the display object determination unit 144 and the display control unit 146 acquire various information about other vehicles detected by the display mode determination unit 142, such as location information and image information. Then, based on the various information about other vehicles, the display object determination unit 144 causes the vehicle image and object image to perform effects to notify the occupants of other vehicles included in the captured image generated by the imaging device 20. For example, as shown in Figure 10(B), a character image 451 as a display mode pointing to vehicle 453 and a speech bubble image 454 from the character image 451 can be displayed.

[0119] Furthermore, the display control unit 146 can perform various effects related to other vehicles in the captured image generated by the imaging device 20, based on various information about other vehicles. For example, as shown in Figure 10(B), it can display a marker sign 457 or an arrow sign 455 indicating the position of vehicle 453, and display a character image 456 assigned to vehicle 453 in association with vehicle 453.

[0120] The above example illustrates how to determine whether to switch from in-vehicle display mode to external display mode when a specific object exists within a predetermined range PR1 around vehicle 1, based on the position of vehicle 1, and that specific object is included in the imaging range IR1 of the imaging device 20. However, even if a specific object exists within a predetermined range PR1 around vehicle 1, based on the position of vehicle 1, but that specific object is not included in the imaging range IR1 of the imaging device 20, the system may automatically switch from in-vehicle display mode to external display mode. In this case, although the specific object is not displayed in the external image, the occupants may be notified that the specific object is nearby using voice information, character images, vehicle images, etc.

[0121] Here, we assume that the imaging device 20 is a 360-degree camera capable of capturing images in all directions of the vehicle 1. In this case, this can be addressed by switching the imaging range of the external image displayed on the display devices 30 and 40 to the direction of a specific object located within a predetermined range PR1. We also assume that the imaging device 20 is a movable camera that can move in at least one of the left-right and up-down directions. In this case, this can be addressed by driving the movable camera so that the imaging range of the external image displayed on the display devices 30 and 40 includes a specific object located within the predetermined range PR1.

[0122] Furthermore, other object detection techniques may be used as a detection method for detecting specific objects present around vehicle 1. For example, artificial intelligence (AI) may be used to detect specific objects present around vehicle 1. For instance, the number of occupants in vehicle 1, the vehicle's position on the road, the timing of the display mode switching, and the captured images generated by the imaging device 20 can be pre-trained, and this trained data can be used to detect specific objects. For example, for objects included in the captured image when the display mode switching operation is performed, an evaluation value for determining the presence or absence of a specific object can be calculated using the trained data, and the presence or absence of a specific object can be determined based on that evaluation value. The trained data can be appropriately set through experiments, etc.

[0123] [Example of display control device operation] Figure 12 is a flowchart illustrating an example of the display mode switching process in the display control device 100. This display mode switching process is executed by the controller 140 based on a program stored in the memory unit 150. This display mode switching process is executed continuously at each control cycle. This display mode switching process will be explained with reference to Figures 10 and 11 as appropriate.

[0124] In step S531, the display mode determination unit 142 determines whether or not a display mode switching operation has been performed by an occupant of vehicle 1. If a display mode switching operation has been performed, the process proceeds to step S532. On the other hand, if a display mode switching operation has not been performed, the process proceeds to step S533.

[0125] In step S532, the display mode determination unit 142 sets a display mode corresponding to the display mode switching operation performed by the occupant of the vehicle 1.

[0126] In step S533, the display mode determination unit 142 determines whether or not an in-vehicle display mode is set. If an in-vehicle display mode is set, the process proceeds to step S534. On the other hand, if an in-vehicle display mode is not set, the display mode switching process terminates.

[0127] In step S534, the display mode determination unit 142 determines whether a landmark exists within a predetermined range PR1 around the vehicle 1, based on the position of the vehicle 1. If a landmark exists within the predetermined range PR1, the process proceeds to step S535. On the other hand, if no landmark exists within the predetermined range PR1, the display mode switching process is terminated.

[0128] In step S535, the display mode determination unit 142 determines whether a landmark located within a predetermined range PR1 is included in the imaging range IR1 of the imaging device 20. If the landmark is included in the imaging range IR1, the process proceeds to step S538. On the other hand, if the landmark is not included in the imaging range IR1, the process proceeds to step S536.

[0129] In step S536, the display mode determination unit 142 determines whether there is another vehicle equipped with the same display control device as vehicle 1 within a predetermined range PR1 around vehicle 1, based on the position of vehicle 1. If another vehicle is present within the predetermined range PR1, the process proceeds to step S537. On the other hand, if no other vehicle is present within the predetermined range PR1, the display mode switching process is terminated.

[0130] In step S537, the display mode determination unit 142 determines whether any other vehicles located within the predetermined range PR1 are included in the imaging range IR1 of the imaging device 20. If the vehicle is included in the imaging range IR1, the process proceeds to step S538. On the other hand, if the vehicle is not included in the imaging range IR1, the display mode switching process is terminated.

[0131] In step S538, the display mode determination unit 142 sets the external display mode.

[0132] The above example illustrates how to automatically switch from in-vehicle display mode to exterior display mode when in-vehicle display mode is set. However, it is also possible to automatically switch from exterior display mode to in-vehicle display mode when exterior display mode is set. For example, the system can automatically switch from exterior display mode to in-vehicle display mode when a predetermined change occurs in the in-vehicle environment. For example, if the volume of the occupant's voice in vehicle 1 exceeds a certain threshold, or if laughter is detected from the occupant, it can be determined that a predetermined change has occurred in the in-vehicle environment. Laughter can be detected using known sound analysis techniques. Furthermore, if a predetermined facial expression, such as a smile or an angry face, is detected from the occupant of vehicle 1, it can be determined that a predetermined change has occurred in the in-vehicle environment.

[0133] Furthermore, automatic switching of the display mode may be performed based on other conditions. For example, if the display mode has not been switched for a predetermined period of time, it may be automatically switched to another display mode. This provides an opportunity for occupants who have been looking at exterior images for a long time to view interior images, and conversely, provides an opportunity for occupants who have been looking at interior images for a long time to view exterior images. In this way, the occupants' moods can be appropriately changed, providing a comfortable vehicle environment.

[0134] Furthermore, other methods may be used to determine the timing of the automatic switching of the display mode. For example, artificial intelligence may be used to determine the timing of the automatic switching of the display mode. For instance, the number of occupants in vehicle 1, the position of vehicle 1 on the road, and the timing of the display mode switching can be learned in advance, and this learned data can be used to determine the timing of the automatic switching of the display mode. The learned data can be set as appropriate through experiments, etc.

[0135] In this embodiment, an example is shown in which the display device 30 is provided on the passenger side 8, but other display media may be used. For example, if the windshield 4 functions as a display medium for the vehicle 1's HUD (Head Up Display), the above-described displays can be shown on the HUD display section of the windshield 4.

[0136] As described above, in this embodiment, imaging devices 10 and 20 installed in the vehicle 1 can be used to capture images of subjects inside and outside the vehicle, and at least one of the captured images from inside or outside the vehicle can be displayed on the display devices 30 and 40. In this case, virtual objects, such as character images or vehicle images, can be superimposed on the captured image. That is, the user experience can be improved by performing effects on characters, etc., in captured images from different angles within the vehicle 1. In other words, by superimposing the same character, etc., on both the captured images inside and outside the vehicle, an interesting effect can be achieved where the character appears to be moving along with the occupants inside the vehicle. This makes it possible to perform effects that enhance the user experience within the vehicle 1 by utilizing the multiple spaces, such as the interior and exterior, of the vehicle 1. That is, it is possible to improve the enjoyment of the occupants when they are riding in the vehicle by viewing the captured images inside and outside the vehicle, and to provide a comfortable vehicle space.

[0137] Furthermore, according to this embodiment, by appropriately setting the orientation of the virtual object, it is possible to create a user experience in which the occupants inside the vehicle and the virtual object move together as one, as well as a scenario in which the occupants are riding together. This enhances the enjoyment for occupants when viewing the captured images inside and outside the vehicle, and provides a comfortable vehicle space.

[0138] For example, let's assume that vehicle 1 is a shared car and that passengers use vehicle 1 for travel. In this case, by displaying character images and vehicle images on display devices 30 and 40, passengers in vehicle 1 can have a simulated experience of the character being in the vehicle with them. For example, by displaying an image of the character riding in a virtual vehicle, the feeling that the character is driving along with vehicle 1 can be enhanced. Passengers in vehicle 1 can also have a simulated experience of traveling with the character. This can lead to passengers developing an attachment to the character displayed on display devices 30 and 40, motivating them to want to ride the shared car again next time.

[0139] In this embodiment, an example is shown in which the display control processing and display mode determination processing are performed in the display control device 100. However, all or part of these processes may be performed in other devices. In this case, the display control system of the vehicle 1 is composed of devices that perform part of these processes. For example, at least part of the display control processing and display mode determination processing can be performed using electronic devices 80 (see Figure 3) used by the occupants inside the vehicle 1, various information processing devices such as servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0140] Furthermore, some (or all) of the display control system of the vehicle 1 capable of performing the functions of the display control device 100 may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0141] Furthermore, each process shown in this embodiment is executed based on a program that causes a computer to execute each processing procedure. For this reason, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, by performing an update process to add a new function to the display control device, the program can be stored in the storage device of the display control device. This makes it possible to have the updated display control device perform each of the processes shown in this embodiment.

[0142] [Configuration and Effects of This Embodiment] The display control method for the vehicle 1 according to this embodiment is a display control method for controlling the display devices 30 and 40 mounted on the vehicle 1. This display control method includes a switching process (steps S501, S531 to S538) that switches between an external display mode (an example of a first display mode) in which a virtual object having a form that allows the front and rear to be identified is superimposed on a first captured image of the outside of the vehicle 1, and an internal display mode in which a virtual object having a different display mode than that of the external display mode (an example of a first display mode) is superimposed on a second captured image of the inside of the vehicle 1, and a display control process (steps S502 to S506, S509 to S516, S521, S522) that causes the virtual object to be displayed in the external display mode so that the front of the virtual object faces the direction in which the vehicle 1 is moving forward, and in the internal display mode so that the front of the virtual object faces the direction in which the seats inside the vehicle 1 are facing.

[0143] This configuration allows for the creation of visual effects where a vehicle 1 with occupants and a virtual object are moving together (exterior display mode), and visual effects where the occupants are riding together (interior display mode), by appropriately setting the orientation of the virtual object. This enables a user experience where the occupants inside the vehicle and the virtual object are moving together as one. Furthermore, it enhances the enjoyment of the occupants while riding in the vehicle by allowing them to view the captured images inside and outside the vehicle, providing a comfortable vehicle space.

[0144] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (steps S505, S506, S513, S514, S516) displays an object image representing an object that can ride in the virtual vehicle as a virtual object in the in-vehicle display mode (an example of a second display mode), and displays a composite image in which an object image is placed on a vehicle image representing the virtual vehicle as a virtual object in the exterior display mode (an example of a first display mode). For example, as shown in Figure 5, a composite image in which character images 211, 221, 231 are placed on vehicle images 212, 222, 232 representing the virtual vehicle can be displayed as a virtual object. Also, as shown in Figure 6, character images 261, 271, 281 representing objects that can ride in the virtual vehicle can be displayed as virtual objects.

[0145] This configuration allows for the creation of an external display mode in which vehicle 1 with occupants, vehicle images 212, 222, 232, and character images 211, 221, 231 are shown driving together, and an internal display mode in which character images 261, 271, 281 are shown riding along with the occupants.

[0146] Furthermore, the display control method for the vehicle 1 according to this embodiment further includes an acquisition process (step S502) for acquiring control signals from the vehicle 1, and the display control process (step S506) changes the display mode of part or all of the vehicle image and object image based on the control signals from the vehicle 1 in the external display mode (an example of the first display mode). For example, as shown in Figure 7(C), when the vehicle 1 is moving in reverse, a character image 381 in which a character riding in the vehicle image 382 turns around can be displayed as a virtual object.

[0147] With this configuration, the display of some or all of the vehicle image and object image can be changed in accordance with the movement of vehicle 1, so that the occupants of vehicle 1 can enjoy the changes in the vehicle image and object image that correspond to the movement of vehicle 1.

[0148] Furthermore, in the display control method for the vehicle 1 according to this embodiment, the object image is a character image that imitates an object having a face and a body, and the display control process (step S506) changes the display manner of the character image based on the control signal of the vehicle 1 so that at least a part of the face and body faces the direction of travel of the vehicle 1 in the external display mode (an example of the first display mode). For example, as shown in Figure 7(C), when the vehicle 1 is moving in reverse, the character image 381 can be displayed as a virtual object so that the face of the character riding in the vehicle image 382 faces the direction of travel (rear) of the vehicle 1.

[0149] This configuration allows for the display of a character that is likely to be familiar to the occupants of vehicle 1, and enables the character's face and body to be turned in the direction of vehicle 1's movement in response to vehicle 1's movement, allowing the occupants of vehicle 1 to enjoy the changes in the character.

[0150] Furthermore, in the display control method of the vehicle 1 according to this embodiment, the display control process (step S513) places a virtual object on the seat in the second captured image in the in-vehicle display mode (an example of the second display mode). For example, as shown in image 270 in Figure 6, a character image 271 can be placed on the empty passenger seat 8.

[0151] This configuration allows for a more immersive experience, enhancing the feeling of riding together by displaying the characters as if they were sitting in the empty seats of vehicle 1.

[0152] Furthermore, the display control method for vehicle 1 according to this embodiment further includes a detection process (step S512) for detecting empty seats that are not occupied by occupants, and the display control process (steps S513, S514) in which, in the in-vehicle display mode (an example of the second display mode), if there are empty seats in the second captured image, a virtual object is placed so as to be superimposed on the empty seats, and if there are no empty seats in the second captured image, a virtual object is placed so as to be superimposed on the occupants who are seated in the seats in the second captured image.

[0153] This configuration allows for different arrangements of virtual objects depending on whether or not there are empty seats in vehicle 1, thereby improving the enjoyment for passengers viewing the captured images of the interior of the vehicle.

[0154] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (step S514) places a virtual object so as to be superimposed on the driver's face in the second captured image in the in-vehicle display mode (an example of the second display mode). For example, as shown in image 260 in Figure 6, a character image 261 can be superimposed on a part of the driver U1's face.

[0155] This configuration allows for an unexpected effect by placing a virtual object on the driver's face, further enhancing the enjoyment for passengers viewing the in-car images.

[0156] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (steps S521, S522) executes a predetermined switching animation using a virtual object when switching from an external display mode (an example of a first display mode) to an internal display mode (an example of a second display mode), or when switching from an internal display mode to an external display mode.

[0157] This configuration allows for the use of virtual objects in visual effects even when switching display modes, further enhancing the enjoyment for occupants viewing the captured images inside and outside the vehicle.

[0158] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (steps S521, S522) executes a switching effect that sequentially changes the display manner of the object image and the vehicle image so that the object image appears to be moving away from the vehicle image when switching from an external display mode (an example of a first display mode) to an internal display mode (an example of a second display mode), and executes a switching effect that sequentially changes the display manner of the object image and the vehicle image so that the object image appears to be moving onto the vehicle image when switching from an internal display mode to an external display mode.

[0159] With this configuration, when switching display modes, a transition animation using virtual objects corresponding to the switch can be performed, further enhancing the enjoyment for occupants viewing the captured images inside and outside the vehicle.

[0160] Furthermore, in the display control method for vehicle 1 according to this embodiment, the display control process (step S522) performs a switching effect when switching from an external display mode (an example of a first display mode) to an internal display mode (an example of a second display mode). This effect involves moving the object image to an empty seat in the second captured image where no occupant is seated, and then sequentially changing the display mode of the object image so that it appears as if it is wearing a seat belt in that empty seat.

[0161] This configuration allows for an unexpected transition effect when switching display modes, such as an object image appearing to fasten a seatbelt, further enhancing the enjoyment for occupants viewing the captured images both inside and outside the vehicle.

[0162] Furthermore, in the display control method for the vehicle 1 according to this embodiment, the switching process (steps S531 to S538) is performed when an in-vehicle display mode (an example of a second display mode) is set, and a specific object exists within a predetermined range PR1 around the vehicle 1 based on the position of the vehicle 1, the system switches from the in-vehicle display mode to the exterior display mode (an example of a first display mode).

[0163] This configuration allows occupants viewing the captured images inside the vehicle to recognize specific objects outside the vehicle, further enhancing the enjoyment of viewing both the interior and exterior images.

[0164] Furthermore, in the display control method for vehicle 1 according to this embodiment, the specific object is a landmark on the ground, and the switching process (steps S534, S535, S538) switches from the in-vehicle display mode (an example of the second display mode) to the out-of-vehicle display mode (an example of the first display mode) when it is determined that a landmark exists within a predetermined range PR1 and that the landmark is included in the imaging range IR1 of the first image. The display control process (steps S521, S505) switches from the in-vehicle display mode to the out-of-vehicle display mode due to the presence of a landmark in the switching process, and causes a virtual object to perform an effect to notify the occupants of the landmark included in the first image. For example, as shown in Figure 10(A), an effect can be performed to display a character image 421 in a display mode pointing to the ABC Tower 423 and a speech bubble image 424 that says "It's the ABC Tower" from the character image 421.

[0165] This configuration allows for the use of virtual objects to notify passengers of landmarks around vehicle 1, thereby enhancing the enjoyment for passengers who are interested in the landmarks in the area where vehicle 1 is traveling.

[0166] Furthermore, in the display control method for vehicle 1 according to this embodiment, the specific object is another vehicle capable of executing the display control processing described above. The switching process (steps S536 to S538) switches from the in-vehicle display mode (an example of the second display mode) to the out-of-vehicle display mode (an example of the first display mode) when it is determined that another vehicle exists within a predetermined range PR1 and that the other vehicle is included in the imaging range of the first image. The display control process (steps S521, S505) switches from the in-vehicle display mode to the out-of-vehicle display mode due to the presence of another vehicle in the switching process, and causes a virtual object to execute a performance to notify the occupants of the other vehicle included in the first image, and also displays the virtual object assigned to the other vehicle in association with the other vehicle. For example, as shown in Figure 10(B), it is possible to execute a performance that displays a character image 451 in a display mode pointing to vehicle 453 and a speech bubble image 454 that says "There are friends" from the character image 451. Additionally, it's possible to create the effect of having character image 456 superimposed on the roof of vehicle 453.

[0167] This configuration allows for the use of virtual objects to notify the user of other vehicles surrounding Vehicle 1. Furthermore, because virtual objects assigned to other vehicles can be displayed in association with those vehicles, the enjoyment of occupants interested in the virtual objects of each vehicle can be enhanced.

[0168] Furthermore, the display control device 100 is a display control device that controls the display devices 30 and 40 mounted on the vehicle 1. The display control device 100 includes a display mode switching unit 143 that switches between an external display mode (an example of a first display mode) which displays a virtual object having a form that allows the front and rear to be identified superimposed on a first captured image of the outside of the vehicle 1, and an internal display mode which displays a virtual object with a different display mode than that of the external display mode (an example of a first display mode) superimposed on a second captured image of the inside of the vehicle 1, and a display control unit 146 which, in the external display mode, displays the virtual object so that the front of the virtual object faces the direction in which the vehicle 1 is moving forward, and in the internal display mode (an example of a second display mode), displays the virtual object so that the front of the virtual object faces the direction in which the seats inside the vehicle 1 are facing.

[0169] This configuration allows for the creation of visual effects where a vehicle 1 with occupants and a virtual object are moving together (exterior display mode), and visual effects where the occupants are riding together (interior display mode), by appropriately setting the orientation of the virtual object. This enables a user experience where the occupants inside the vehicle and the virtual object are moving together as one. Furthermore, it enhances the enjoyment of the occupants while riding in the vehicle by allowing them to view the captured images inside and outside the vehicle, providing a comfortable vehicle space.

[0170] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of Symbols]

[0171] 1 Vehicle, 2 Dashboard, 3 Steering wheel, 4 Windshield, 5 Rearview mirror, 6 Car navigation system, 7 Driver's seat, 8 Passenger seat, 10, 20 Imaging device, 11, 12 Rear seats, 30, 40 Display device, 50 Vehicle management system, 60 Network, 70 Management server, 71 Communication unit, 72 Control unit, 73 Storage unit, 80 Electronic equipment, 100 Display control device, 110 Vehicle system, 120 UI unit, 130 Image acquisition unit, 140 Controller, 141 Occupant recognition unit, 142 Display mode determination unit, 143 Display mode switching unit, 144 Display object determination unit, 145 Shooting control unit, 146 Display control unit, 150 Storage unit, 160 Communication unit

Claims

1. A display control method for controlling a display device mounted on a vehicle, A switching process that switches between a first display mode in which a virtual object having a form that allows for front and rear identification is superimposed on a first image captured of the exterior of the vehicle, and a second display mode in which the virtual object is superimposed on a second image captured of the interior of the vehicle in a display manner different from that of the first display mode, The first display mode includes a display control process in which the orientation of the virtual object is determined based on first information relating to the orientation of the vehicle, and the second display mode includes a display control process in which the orientation of the virtual object is determined based on second information relating to the orientation of the seats inside the vehicle. Display control method.

2. A display control method according to claim 1, In the first display mode, the virtual object is displayed such that its front faces the direction in which the vehicle is moving forward. Display control method.

3. A display control method according to claim 1, In the second display mode, the virtual object is displayed such that its front faces the direction of the front of the seat inside the vehicle. Display control method.

4. A display control method according to any one of claims 1 to 3, In the second display mode, the display control process displays an object image representing an object that can ride in the virtual vehicle as the virtual object, and in the first display mode, it displays a composite image in which the object image is placed on a vehicle image representing the virtual vehicle as the virtual object. Display control method.

5. A display control method according to claim 4, The process further includes an acquisition process for acquiring control signals of the vehicle, In the first display mode, the display control process changes the display mode of part or all of the vehicle image and the object image based on the vehicle's control signal. Display control method.

6. A display control method according to claim 5, The aforementioned object image is a character image that imitates an object having a face and a body. In the first display mode, the display control process changes the display mode of the character image based on the control signal of the vehicle. Display control method.

7. A display control method according to any one of claims 1 to 6, The display control process executes a predetermined switching animation using the virtual object when switching from the first display mode to the second display mode, or when switching from the second display mode to the first display mode. Display control method.

8. A display control device that controls a display device mounted on a vehicle, A display mode switching unit switches between a first display mode in which a virtual object having a form that allows for front and rear identification is superimposed on a first image captured of the exterior of the vehicle, and a second display mode in which the virtual object is superimposed on a second image captured of the interior of the vehicle in a display manner different from that of the first display mode. The system comprises a display control unit which, in the first display mode, determines the orientation of the virtual object based on first information relating to the orientation of the vehicle, and in the second display mode, determines the orientation of the virtual object based on second information relating to the orientation of the seats inside the vehicle. Display control device.

Images