Information processing device, information processing method, information processing program, and storage medium

Abstract

[Problem] To provide an information processing device with which a user outside a vehicle such as an automobile can enjoy a virtual drive together with the occupants in the vehicle. [Solution] An information processing device according to the present invention is characterized by comprising: an acquisition unit that acquires current position information of a moving body and an external image of the moving body captured during movement of the moving body by an image capture unit disposed on the moving body; a transmission unit that performs a transmission process for transmitting, to a terminal device, a travel image in which information based on the current position information is superimposed on the external image acquired by the acquisition unit; and a voice call unit that, concurrently with the transmission process, performs a voice call process for an occupant of the moving body and the user of the terminal device to engage in a voice call with each other.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing program, and a storage medium, and more particularly to an information processing device, an information processing method, an information processing program, and a storage medium for communicating between a mobile body and the outside. [Background technology]

[0002] There is a communication system that performs communication between an onboard device mounted in a vehicle and an external terminal located outside the vehicle. For example, Patent Document 1 discloses a system in which a voice call is performed between a driver of the vehicle and an operator of the external terminal, and while the voice call is being performed, video data showing an image of the area in front of the vehicle is transmitted from the onboard device to the external terminal, and the video is displayed on the external terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-213791 Summary of the Invention [Problem to be solved by the invention]

[0004] Systems such as that described in Patent Document 1 are intended to address drivers' distracted driving, and one of the problems with this is that the user of the external terminal cannot enjoy driving with the driver of the vehicle as if they were actually in the vehicle.

[0005] The present invention has been made in consideration of the above-mentioned points, and aims to provide an information processing device that allows a user outside a vehicle such as an automobile to enjoy a virtual drive together with the passengers in the vehicle. [Means for solving the problem]

[0006] The invention described in claim 1 is an information processing device characterized by comprising: an acquisition unit that acquires current location information of a moving body and an external image of the moving body captured by a photographing unit disposed on the moving body while the moving body is moving; a transmission unit that performs a transmission process to transmit to a terminal device image information during travel in which information based on the current location information is superimposed on the external image acquired by the acquisition unit; and a voice call unit that, simultaneously with the transmission process, establishes a voice call connection for voice communication between an occupant of the moving body and a user of the terminal device.

[0007] The invention described in claim 4 is a method for detecting a current position of a moving body, route information relating to a planned route of the moving body, and a method for detecting a current position of the moving body by a photographing unit disposed in the moving body. While the moving object is moving The information processing device is characterized by comprising: an acquisition unit that acquires external images of the moving body photographed by a camera; a transmission unit that performs a transmission process that transmits the external images acquired by the acquisition unit and the route information; and a voice call unit that performs a voice call process that enables a voice call between an occupant of the moving body and a user of the terminal device simultaneously with the transmission process.

[0008] The invention described in claim 5 is an information processing method executed by an information processing device moving together with a moving body, characterized by having an acquisition step of acquiring current location information of the moving body and an external image of the moving body captured by a photographing unit disposed on the moving body while the moving body is moving; a transmission step of performing a transmission process of transmitting to a terminal device an image of the moving body in which information based on the current location information is superimposed on the external image acquired in the acquisition step; and a voice call step of performing a voice call process in which a passenger of the moving body and a user of the terminal device make a voice call simultaneously with the transmission process.

[0009] The invention described in claim 6 is an information processing program characterized by causing an information processing device moving together with a moving body to execute the following steps: an acquisition step for acquiring current position information of the moving body and an external image of the moving body captured by a photographing unit disposed on the moving body while the moving body is moving; a transmission step for performing a transmission process for transmitting to a terminal device an image of the moving body in which information based on the current position information is superimposed on the external image acquired in the acquisition step; and a voice call step for performing a voice call process for a voice call between an occupant of the moving body and a user of the terminal device simultaneously with the transmission process. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a virtual passenger communication system according to a first embodiment of the present invention. [Figure 2] 1 is a perspective view of a front seat portion of a vehicle equipped with an in-vehicle device. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of an in-vehicle device. [Figure 4] FIG. 2 is a block diagram illustrating an example of a configuration of a server. [Figure 5] FIG. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an external device. [Figure 7] FIG. 10 is a sequence diagram of the virtual passenger communication system during virtual passenger communication. [Figure 8] FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Example]

[0012] [1. System Configuration] A virtual ride-share communication system 100 according to a first embodiment of the present invention will be described below with reference to the accompanying drawings.

[0013] Fig. 1 shows a virtual passenger-share communication system 100 according to a first embodiment of the present invention. As shown in Fig. 1, the virtual passenger-share communication system 100 includes an in-vehicle device 10 as an information processing device, a relay server (hereinafter also simply referred to as a server) 40, and an external device 70. Fig. 1 shows a case in which the in-vehicle device 10 is mounted on an automobile M as an example of a moving body. Fig. 1 also shows a smartphone as an example of the external device 70.

[0014] The in-vehicle device 10, the relay server 40, and the external device 70 can transmit and receive data to and from each other via a network NW using communication protocols such as TCP / IP, UDP / IP, etc. The network NW can be constructed using internet communication including wireless communication such as a mobile communication network, Wi-Fi (registered trademark), and wired communication.

[0015] In the virtual passenger communication system 100 of this embodiment, a voice call is established between the in-vehicle device 10 and the external device 70 via the server 40, and then a video or image (hereinafter simply referred to as an image) captured by an in-vehicle camera mounted on the automobile M is transmitted from the in-vehicle device 10 to the external device 70 via the server 40. This communication mode in which a voice call is established between the in-vehicle device 10 and the external device 70 and an image captured by an in-vehicle camera mounted on the automobile M is transmitted from the in-vehicle device 10 to the external device 70 is referred to as audio / video communication or virtual passenger communication.

[0016] By performing such virtual passenger communication, the user of the external device 70 who is making a voice call while watching the video transmitted from the in-vehicle device 10 can feel as if he or she is riding in the vehicle M with the driver of the vehicle M. In other words, the virtual passenger communication can realize a virtual passenger ride in the vehicle M with the user of the external device 70.

[0017] In the following, in the embodiment, an example will be described in which the in-vehicle device 10 is a car navigation device that receives from the user a destination to which the user wishes to be guided and generates a route to the destination.

[0018] 2 is a perspective view showing the vicinity of the front seat of an automobile M equipped with an in-vehicle device 10 as an information processing device. In FIG. 2, as an installation example, the in-vehicle device 10 is shown installed in a dashboard DB of the front seat of the automobile M.

[0019] The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is disposed, for example, on a dashboard DB. The GPS receiver 11 may be disposed anywhere as long as it can receive GPS signals. The GPS receiver 11 is capable of transmitting the received GPS signals to the in-vehicle device 10.

[0020] The external camera 12 serving as the photographing unit is a photographing device that photographs the area in front of the automobile M. In this embodiment, the external camera 12 is arranged on the dashboard DB so that the photographing direction faces forward. For example, the external camera 12 is a wide-angle camera that can photograph a wide area in front of the automobile M through the windshield. During virtual passenger communication, the image photographed by the external camera 12 is distributed to the external device 70. Note that the external camera 12 may be arranged anywhere, such as behind the rearview mirror RM, as long as it can photograph the area in front of the vehicle.

[0021] The internal camera 13 is a photographing device that photographs the interior of the automobile M. In this embodiment, the internal camera 13 is provided at the top end of the windshield FG or on the ceiling near the top end, and is capable of photographing the driver of the automobile M.

[0022] The touch panel 14 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touchpad. The touch panel 14 is disposed, for example, on the center console CC of the dashboard DB. The touch panel 14 may be disposed in a location that is visible to the driver and within the driver's reach. For example, the touch panel 14 may be attached to the dashboard DB.

[0023] The touch panel 14 can display a screen based on the control of the in-vehicle device 10. The touch panel 14 can also transmit a signal representing an input operation to the touch panel 14 received from the user to the in-vehicle device 10. For example, the touch panel 14 may display car navigation guidance. Furthermore, operations related to the car navigation function, such as setting a destination, may be possible via the touch panel 14.

[0024] Furthermore, information about the virtual passenger communication or a screen for accepting operations for connecting the virtual passenger communication may be displayed on the touch panel 14. The passengers of the automobile M can perform the operation for connecting the virtual passenger communication by inputting an input operation on the touch panel 14.

[0025] The speaker 15 is provided, for example, on the interior side of the A-pillar AP. The speaker 15 can emit sounds such as music and voices under the control of the in-vehicle device 10. During virtual passenger communication, the speaker 15 emits voice from the external device 70 during the voice call.

[0026] The microphone 17 is a microphone device that picks up sounds inside the vehicle, and is arranged, for example, on the dashboard DB. The microphone 17 may be arranged anywhere, such as on the rearview mirror RM or the steering wheel, as long as it can pick up sounds inside the vehicle. During virtual passenger communication, the sound picked up by the microphone 17 is transmitted to the external device 70 as the sound of the voice call.

[0027] 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a large-capacity storage device 23, a control unit 25, an input unit 27, an output unit 29, an encoder unit 30, and a data communication unit 31 cooperate with each other via a system bus 21.

[0028] In the following description, it is assumed that the automobile M is equipped with an acceleration sensor 19. The acceleration sensor 19 may be, for example, a MEMS acceleration sensor that is capable of measuring the acceleration of the automobile M and outputting a signal indicating the measured speed. The acceleration sensor 19 may also be an inertial measurement unit (IMU). The acceleration sensor 19 may also be built into the in-vehicle device 10.

[0029] The mass storage device 23 is configured with, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc., and stores various programs such as an operating system and terminal software, etc. The mass storage device 23 also holds a map information database 23A that has map information including road maps.

[0030] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the mass storage device 23 (including an information processing program for executing processing in the in-vehicle device 10, which will be described later) can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0031] The control unit 25 is configured with a CPU (Central Processing Unit) 25A, a ROM (Read Only Memory) 25B, a RAM (Random Access Memory) 25C, etc., and functions as a computer. The CPU 25A reads and executes various programs stored in the ROM 25B and the large-capacity storage device 23, thereby realizing various functions. In this embodiment, the control unit 25 performs functions such as a video and audio communication function during virtual passenger communication and a car navigation function.

[0032] The control unit 25 can receive input of a destination from the occupant via the touch panel 14 or the microphone 17, and generate route information or navigation information to the destination, as well as a car navigation screen based on the information. Even if the control unit 25 does not receive input of a destination from the occupant, it may refer to the past driving history, estimate the route and destination along which the vehicle is scheduled to travel, and generate route information. Here, the route information can be said to be information regarding the planned driving route of the vehicle M.

[0033] Furthermore, the control unit 25 can generate a composite image by combining a camera image captured by the external camera 12 or the internal camera 13 with other information, such as the above-mentioned car navigation screen. In other words, the control unit 25 can generate a video in which an external image has been processed in some way, that is, an image based on the external image. In other words, the control unit 25 can generate a superimposed image in which a car navigation screen, an image showing the current location, an image showing the above-mentioned route, or the like is superimposed on the above-mentioned camera image.

[0034] The other information may include, for example, map image information in which a mark indicating the current position of automobile M is superimposed on a map image, such as the above-mentioned car navigation screen, or a route image in which an image indicating a route to a destination set in automobile M is added to the map image information. The map image information or route image information can be said to be image information that is generated based on current position information and can identify the position of automobile M. In other words, the composite image including the map image information or route image information can be said to include current position information as position identification information that identifies the current position of automobile M.

[0035] In addition, the other information may include identification information such as the name of the driver of automobile M or the registration number of automobile M, information indicating the driving status of automobile M such as the speed or acceleration of automobile M, or information indicating the operating status of the driver of automobile M, such as the steering angle of automobile M's steering wheel, the amount of depression of the accelerator or brake, or the operating status of the turn signal.

[0036] The control unit 25 can also generate a composite image that includes both images captured by the external camera 12 and the internal camera 13 (hereinafter also referred to as a captured image). For example, the control unit 25 can generate a composite image in which an image captured by the internal camera 13 (hereinafter also referred to as an internal image) is placed in the lower right corner of an image captured by the external camera 12 (hereinafter also referred to as an external image), or vice versa. It is also possible to generate a composite image in which, for example, half is an external image and the other half is an internal image. Of course, the control unit 25 can also transmit these to the external device 70.

[0037] The input unit 27 is an interface unit that communicatively connects the control unit 25 of the in-vehicle device 10 with the external camera 12, the internal camera 13, the touch panel 14, the microphone 17, and the acceleration sensor 19. The in-vehicle device 10 can sequentially acquire images captured by the external camera 12 via the input unit 27. In other words, the control unit 25 functions as an acquisition unit that sequentially acquires video data showing images of the surroundings of the automobile captured by the external camera 12 via the input unit 27.

[0038] The control unit 25 can successively acquire acceleration measurement signals from the acceleration sensor 19 via the input unit 27. The control unit 25 can calculate the acceleration and speed of the automobile M based on the acceleration measurement signals.

[0039] The in-vehicle device 10 can receive a signal indicating an input operation to the touch panel 14 via the input unit 27. For example, the in-vehicle device 10 can accept, via the input unit 27, a connection request for virtual passenger communication made by the user via the touch panel 14 and the microphone 17, or an input for setting a destination for car navigation.

[0040] The input unit 27 is an interface unit that communicatively connects the in-vehicle device 10 and the GPS receiver 11. The in-vehicle device 10 receives a GPS signal from the GPS receiver 11 via the input unit 27, and can acquire information about the current position of the in-vehicle device 10, that is, the current position of the automobile M in this embodiment, based on the GPS signal.

[0041] In other words, the control unit 25 functions as an acquisition unit that acquires current position information that identifies the current position of the automobile M based on the GPS signal from the GPS receiver 11. Note that the current position information may be acquired by using self-contained navigation in addition to or instead of acquiring the information based on the GPS signal.

[0042] The output unit 29 is communicatively connected to the touch panel 14 and the speaker 15, and is capable of transmitting video or image signals to the touch panel 14 to display the same, and transmitting audio signals to the speaker 15 to output sound.

[0043] The encoder unit 30 is a part that encodes (hereinafter also referred to as encoding processing) the image captured by the external camera 12 or the internal camera 13 or the composite image based on a command from the control unit 25. The encoder unit has a CPU for video encoding, a so-called GPU, and encoding may be performed by the GPU.

[0044] The encoder unit 30 generates encoded data as video encoded data by encoding the captured image or composite image using, for example, an MPEG-4 standard encoding method. For example, the encoder unit 30 generates encoded data from the captured image or composite image using a codec such as H.264, Xvid, DivX, VP8, or VP9. Hereinafter, this encoded data will also be simply referred to as image data.

[0045] The data communication unit 31 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The data communication unit 31 also transmits and receives various data to and from the external device 70 via the server 40.

[0046] For example, the control unit 25 of the in-vehicle device 10 can transmit, via the data communication unit 31, to the server 40, position identification information as position data that can identify the current position of the in-vehicle device 10, that is, information on the current position of the automobile M.

[0047] The control unit 25 can transmit the voice picked up by the microphone 17, for example, the voice data of the voice spoken by the passenger, to the external device 70 via the data communication unit 31 for the voice call in the virtual passenger communication. The control unit 25 can also receive the voice data of the voice input to the external device 70 via the data communication unit 31 for the voice call in the virtual passenger communication. This transmission and reception of the voice data is performed in real time while the automobile M is moving.

[0048] Furthermore, the control unit 25 transmits the image data to the external device 70 via the data communication unit 31. The control unit 25 may transmit the image data to the external device 70 via the data communication unit 31 while buffering the image data.

[0049] The control unit 25 transmits the image data without delay after the captured image is captured via the data communication unit 31. In other words, after the captured image is captured, the control unit 25 performs necessary processing and then transmits the image data including the captured image immediately in real time. In yet other words, the control unit 25 transmits the image data so that the captured image can be viewed in real time on the external device 70.

[0050] In addition, the control unit 25 can acquire external or internal images taken while the automobile M is moving via the data communication unit 31, and transmit image data of the external or internal images in real time while the automobile M is moving.

[0051] The above-mentioned period when the automobile M is moving refers to, for example, a period when the automobile M is traveling from a departure point to a destination, including a period when the automobile M is temporarily stopped. For example, the period when the automobile M is moving may be a period from when the ACC power of the automobile M is turned on until the ACC power of the automobile M is turned off.

[0052] 4 is a block diagram showing the configuration of the server 40. For example, the server 40 is a device in which a large-capacity storage device 43, a control unit 45, and a data communication unit 47 cooperate with each other via a system bus 41. The server 40 has a function similar to that of a SIP server that establishes a voice call between the in-vehicle device 10 and the external device 70 during virtual passenger communication and transfers data of the voice call.

[0053] The server 40 also has the function of transferring the captured image data sent from the in-vehicle device 10 to the external device 70, and processing the captured image data and sending it to the external device 70.

[0054] The mass storage device 43 is configured by, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as an operating system and software for the server 40.

[0055] The control unit 45 is configured with a CPU (Central Processing Unit) 45A, a ROM (Read Only Memory) 45B, a RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads and executes various programs stored in the ROM 45B and the large-capacity storage device 43, thereby realizing various functions.

[0056] The data communication unit 47 is connected to the network NW, and transmits and receives various data between the in-vehicle device 10 and the external device 70.

[0057] The control unit 45 transfers image data including captured images, current position information, route information, and audio information received from the in-vehicle device 10 to the external device 70 via the data communication unit 47. The control unit 45 transfers audio received from the external device 70 to the in-vehicle device 10 via the data communication unit 47. The control unit 45 may also generate a superimposed image by superimposing (combining) an image indicating the current position generated based on the current position information received from the in-vehicle device 10 and an image indicating the route generated based on the route information received from the in-vehicle device 10 onto the captured image data received from the in-vehicle device 10, and transmit the superimposed image to the external device 70.

[0058] Fig. 5 is a front view showing the appearance of the external device 70. As described above, in Example 1, the external device 70 is a smartphone. In Fig. 5, the external device 70 displays a composite image that combines the above-described external image and a car navigation screen. The example shown in Fig. 5 is an example of a portrait screen display in which the longitudinal direction of the touch panel 71 is the up-down direction.

[0059] The touch panel 71 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touch pad. The touch panel 71 is capable of generating a signal that represents an input operation to the touch panel 71 received from the user.

[0060] In this embodiment, the touch panel 71 displays an image transmitted from the in-vehicle device 10, the speed information SI of the automobile M, an end icon IC that is an icon to be touched to end the virtual passenger communication, and the like. In the example of FIG. 5, an external image is displayed in an upper area AR1 of the touch panel 71, and a car navigation screen is displayed in a lower area AR2. The speed information SI of the automobile M is displayed in the lower right of the area AR1 as seen from the user. The end icon IC is displayed in the lower left of the area AR2 as seen from the user. This is to give the user of the external device 70 a sense of being in the passenger seat of the automobile M, or a so-called passenger feeling, during the virtual passenger communication. That is, assuming that the automobile M is a Japanese vehicle with a right-hand drive (a vehicle made in Japan), the speed information SI of the automobile M is displayed on the right side where the driver's seat is located (the right side in the forward traveling image), and the icon for ending the virtual passenger communication is displayed on the left side where the door through which the passenger exits the passenger seat is located (the left side in the forward traveling image), thereby giving a stronger sense of being a passenger.

[0061] Furthermore, information about the virtual ride-along communication or a screen for accepting operations for connecting the virtual ride-along communication may be displayed on the touch panel 71. The user of the external device 70 may perform an operation for connecting the virtual ride-along communication by inputting information to the touch panel 71.

[0062] The speaker 73 is capable of emitting sounds such as music and voices. During virtual passenger communication, the speaker 73 emits voice from the in-vehicle device 10 during voice communication.

[0063] The microphone 75 is a microphone device that receives sounds emitted toward the external device 70. During virtual passenger communication, the voices collected by the microphone 75 are transmitted to the external device 70 as voice calls.

[0064] 6 is a block diagram showing the configuration of the external device 70. For example, the external device 70 is a device in which a large-capacity storage device 83, a control unit 84, an input unit 85, an output unit 86, a data communication unit 87, and a decoder unit 88 cooperate with each other via a system bus 81.

[0065] The mass storage device 83 is configured by, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc., and stores various programs such as an operating system and software for the terminal.

[0066] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the mass storage device 83 (including a program for executing processing in the external device 70, which will be described later) can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0067] The control unit 84 is configured with a CPU (Central Processing Unit) 84A, a ROM (Read Only Memory) 84B, a RAM (Random Access Memory) 84C, etc., and functions as a computer. The CPU 84A reads and executes various programs stored in the ROM 84B and the large-capacity storage device 83, thereby realizing various functions.

[0068] The input unit 85 is an input interface unit for the touch panel 71 and the microphone 75. The control unit 84 can receive, via the input unit 85, signals indicating input operations to the touch panel 71 and audio input signals from the microphone 75. For example, the control unit 84 can accept, via the input unit 85, a connection request for virtual ride-along communication made by the user via the touch panel 71 and the microphone 75.

[0069] The output unit 86 is an output interface unit to the touch panel 71 and the speaker 73. The control unit 84 can transmit a video or image signal via the output unit 86 to the touch panel 71 to display it, or transmit an audio signal to the speaker 73 to output sound.

[0070] The data communication unit 87 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The data communication unit 87 also transmits and receives various data to and from the in-vehicle device 10 via the server 40, including image data and current location information transmitted from the in-vehicle device 10. For example, the control unit 84 can receive route information of the automobile M, speed information of the automobile M, or navigation information from the in-vehicle device 10 via the data communication unit 87.

[0071] Furthermore, the control unit 84 can transmit audio data of the audio picked up by the microphone 75 to the in-vehicle device 10 via the data communication unit 87 for the audio call in the virtual passenger-sharing communication. Furthermore, the control unit 84 can receive audio data transmitted from the in-vehicle device 10 via the data communication unit 87 for the audio call in the virtual passenger-sharing communication.

[0072] The decoder unit 88 is a part that decodes, plays back, and outputs image data received from the in-vehicle device 10 based on commands from the control unit 84. The decoder unit 88 decodes the image data using a codec used when encoding the image data, for example, an encoding method conforming to the MPEG-4 standard, and plays back and outputs the video. The played-back image is displayed on the touch panel 71 by the control unit 84.

[0073] [2. System operation sequence when performing virtual passenger communication] The operation of the virtual passenger communication system 100 of the first embodiment when performing virtual passenger communication will be described below with reference to Fig. 7. Fig. 7 is a diagram showing the processing sequence of the virtual passenger communication system 100 during virtual passenger communication. Here, an example will be described in which an external device 70 requests the start of virtual passenger communication and an external device 70 requests the end of virtual passenger communication.

[0074] First, the external device 70 receives a request to start virtual passenger communication from the user (step S1). This start request is received, for example, by receiving a user's operation on the touch panel 71 of the external device 70 or a voice input operation via the microphone 75. Upon receiving the start request, the external device 70 transmits the start request for virtual passenger communication to the server 40 (step S2).

[0075] When the server 40 receives the start request from the external device 70, it transmits a start request acceptance notice to the in-vehicle device 10 to notify the in-vehicle device 10 that a request for virtual passenger communication has been made (step S3). When the in-vehicle device 10 receives the start request acceptance notice, it notifies the occupant of the automobile M of the reception of the start request via, for example, the touch panel 14 or the speaker 15.

[0076] The in-vehicle device 10 receives the start request acceptance notification, notifies the occupant of the acceptance, and then accepts consent to start the virtual passenger-sharing communication from the occupant, such as the driver of the automobile M (step S4). This acceptance is accepted, for example, by accepting an operation by the occupant on the touch panel 14 of the in-vehicle device 10, or by accepting a voice input operation by the occupant via the microphone 17. Upon accepting consent to start the virtual passenger-sharing communication, the in-vehicle device 10 transmits a consent notification to the server 40 (step S5).

[0077] When the server 40 receives the acceptance notification, it executes a virtual passenger communication start process (S6) including the establishment of voice call communication between the in-vehicle device 10 and the external device 70, the establishment of video communication for transferring images transmitted from the in-vehicle device to the external device, and the start of transfer of current location information transmitted from the in-vehicle device 10 to the external device 70, and the video and audio communication of the virtual passenger communication is started. When this virtual passenger communication start process is completed and the video and audio communication is started, the in-vehicle device 10 starts transmitting to the server 40 external images captured by the external camera 12, internal images, current location information, route information, etc.

[0078] Thereafter, when the virtual passenger-only communication is to end, the external device 70 receives a request to end the virtual passenger-only communication from the user (step S7). This request to end is received, for example, by receiving a user's operation on the touch panel 71 of the external device 70 or a voice input operation via the microphone 75. Upon receiving the end request, the external device 70 transmits a request to end the virtual passenger-only communication to the server 40 (step S8).

[0079] When the server 40 receives the end request from the external device 70, it transmits an end notification to the in-vehicle device 10 to notify the end of the virtual passenger communication (step S9). When the in-vehicle device 10 receives the end notification, it notifies the occupant of the automobile M of the end notification via, for example, the touch panel 14 or the speaker 15.

[0080] After transmitting the end notification, the server 40 executes an end process of the virtual passenger communication, such as disconnecting the voice call communication and the video distribution communication (S10), and the video and audio communication of the virtual passenger communication is ended.

[0081] [3. Operation routine of the in-vehicle device when performing virtual passenger communication] 8 is a flowchart showing the virtual passenger communication processing routine RT1 executed by the control unit 25 of the in-vehicle device 10. The control unit 25 starts the virtual passenger communication processing routine RT1 when the in-vehicle device 10 is powered on, for example, and repeatedly executes this routine. This powering on is performed, for example, by turning on the ACC power of the automobile M.

[0082] First, the control unit 25 determines whether the in-vehicle device 10 has started virtual passenger-share communication with the external device 70 (step S101). This determination is made, for example, based on whether the in-vehicle device 10 has accepted a request to start virtual passenger-share communication and established communication with the external device 70. If the control unit 25 determines that virtual passenger-share communication has not started (step S101: NO), it ends routine RT1.

[0083] When the control unit 25 determines that virtual passenger communication has begun (step S101: YES), it acquires an external image, which is an image of the outside of the automobile M, from the external camera 12, and acquires current location information indicating the location of the automobile M based on the GPS signal from the GPS receiver 11, and once these have been acquired, it starts a transmission process to transmit them in real time to the external device 70 via the server 40 without delay (step S102).

[0084] In this transmission process, the external image itself or an image obtained by processing the external image in some way, i.e., an image based on the external image, is transmitted. For example, in the transmission process, a composite image in which other information about the automobile M as described above is visibly superimposed on the external image may be transmitted. In addition, in this transmission process, a composite image generated by the in-vehicle device 10 may be transmitted in which a navigation image in which a mark indicating the current position is displayed on at least a map image of the area around the current position is superimposed on the external image. In addition, when the image superimposition (composite) process is performed by the external device 70 or the server 40, which is the transmission destination, in this transmission process, the captured image (external image, internal image) and other information to be superimposed on the captured image (current position information, route information, etc.) are transmitted from the in-vehicle device 10 to the external device 10 (or the server 40).

[0085] In other words, when the transmission process is started in step S102, the control unit 25 functions as an acquisition unit that executes an acquisition process or an acquisition step to acquire current location information of the automobile M and acquire external images from the external camera 12. Also, when the transmission process is started in step S102, the control unit 25 functions as a transmission unit that executes a transmission process or a transmission step to transmit the acquired current location information and the external images themselves or traveling image information based on the external images to the external device 70 as a terminal device without delay after acquisition.

[0086] In addition, if the external image contains information that can identify the location of the automobile M, such as the map information or map route information described above, i.e., if the image contains current location information, it is not necessary to transmit the current location information separately from the image in the above transmission process.

[0087] After step S102, the control unit 25 starts a voice call process for transmitting and receiving voice to and from the external device 70 via the server 40 (step S103). Specifically, the control unit 25 starts a process for transmitting voice input via the microphone 17 to the external device 70 and outputting voice received from the external device 70 via the speaker 15.

[0088] In other words, after the voice call processing is started in step S103, the control unit 25 functions as a voice call unit that executes voice call processing or a voice call step that performs voice call processing in a voice call connection in which a voice call is made between an occupant of the automobile M as a moving body and a user of the external device 70 as a terminal device, simultaneously with the above-mentioned transmission processing.

[0089] When steps S102 and S103 are executed and the image and current location information transmission process and the voice call process are started, the control unit 25 determines whether the virtual passenger communication between the in-vehicle device 10 and the external device 70 has ended (step S104). If the control unit 25 determines that the virtual passenger communication has not ended, i.e., that the virtual passenger communication is continuing (step S104: NO), it repeatedly executes step S104. In other words, the image and current location information transmission process and the voice call process continue as long as the virtual passenger communication does not end.

[0090] When the control unit 25 determines in step S104 that the virtual ride-along communication with the external device 70 has ended (step S104: YES), it ends the process of acquiring and transmitting external images and current location information (step S105), then ends the voice call process (step S106), and then routine R1 ends. In other words, when the control unit 25 determines that the virtual ride-along communication with the external device 70 has ended, it ends the process of acquiring and transmitting images and current location information and the voice call process.

[0091] In this way, according to the virtual passenger communication system 100 or the in-vehicle device 10 of the present invention, external images of the automobile M are transmitted to the external device 70 in real time, voice communication is possible between the in-vehicle device 10 and the external device 70, and current location information that can identify the location of the automobile M is transmitted to the external device 70.

[0092] By enabling voice communication between the in-vehicle device 10 and the external device 70 while transmitting external images to the external device 70 in real time, the user of the external device 70 can talk with the occupants of the car M while viewing the same scenery as the occupants of the car M in real time, and can have a virtual riding experience as if he or she were actually riding in the car M.

[0093] The user of the external device 70 can further obtain the current location information, route information, or information based thereon transmitted from the in-vehicle device 10, thereby understanding where the car M is currently traveling, where it is scheduled to travel in the future, and so on, and can have a pleasant virtual ride-along experience by sharing with the occupants of the car M further information about the car M other than the scenery.

[0094] In the virtual passenger communication of the above embodiment, an external image of the automobile M is transmitted in real time from the in-vehicle device 10, but the external device 70 may be equipped with a camera, and images captured by the camera may also be transmitted from the external device 70 to the in-vehicle device 10. In this case, the images transmitted from the external device 70 are displayed on the touch panel 14 of the in-vehicle device 10, for example.

[0095] In the above embodiment, the in-vehicle device 10 is an in-vehicle navigation device, but the in-vehicle device 10 may simply be capable of distributing video from the external camera 12 or the internal camera 13. The in-vehicle device 10 may also be a terminal, such as a smartphone or a tablet, that incorporates the external camera 12 or the internal camera 13. In this case, the in-vehicle device 10 may be attached to the dashboard DB by, for example, a cradle so that the built-in camera can capture an image of the area ahead of the automobile M through the windshield of the automobile M.

[0096] In the above embodiment, the in-vehicle device 10 and the external device 70 perform virtual passenger communication via the server 40, but the virtual passenger communication may be performed directly between the in-vehicle device 10 and the external device 70. For example, the virtual passenger communication may be performed directly between the in-vehicle device 10 and the external device 70 by P2P (Peer to Peer) communication or the like, without using the server 40.

[0097] Furthermore, voice communication is not necessarily possible between the in-vehicle device 10 and the external device 70. For example, like the above-mentioned YouTube (registered trademark) and Niconico Live Broadcast (registered trademark), a mode in which video and audio are simply distributed from the in-vehicle device 10 to the external device 70 without two-way voice communication may be used.

[0098] In the above embodiment, the in-vehicle device 10 starts video streaming after establishing communication between the in-vehicle device 10 and the external device 70. However, video streaming may be performed in a manner similar to the video streaming provided by YouTube (registered trademark) or Niconico Live Broadcast (registered trademark) described above. That is, the video streaming may be started even if communication with a viewer's terminal such as the external device 70 is not established. Specifically, the in-vehicle device 10 may start uploading video data to the server 40 even if communication between the in-vehicle device 10 and the viewer's terminal such as the external device 70 is not established.

[0099] For example, the video distribution operation by the in-vehicle device 10 may be started after communication between the in-vehicle device 10 and the server 40 is established, without establishing a communication connection with the external device 70. In this case, an unspecified or authorized specific external device 70 can connect to the server 40 to receive the video distributed from the in-vehicle device 10, and the user of the external device 70 can watch the video.

[0100] Furthermore, the information processing device of the present invention may be configured by integrating a terminal device having a configuration similar to that of the in-car device 10 of this embodiment, the external camera 12 (or the external camera 12 and the internal camera 13), and the touch panel 14. Specifically, for example, the information processing device of the present invention may be a smartphone, tablet, or PC with a camera that is equipped with an app that performs the same functions as the in-car device 10.

[0101] Furthermore, the in-vehicle device 10 may not have an image display unit such as the touch panel 14 for the driver of the automobile M. For example, the in-vehicle device 10 may have a configuration similar to that of a drive recorder, and may be a device integrated with the external camera 12 or the internal camera 13. Specifically, the in-vehicle device 10 may be a device in which hardware that performs the virtual passenger communication function of the in-vehicle device 10 is built into the housing of the external camera 12. In this case, the in-vehicle device 10 may not provide the various display outputs described above.

[0102] In this embodiment, a case has been described in which an external image, which is an image of the outside of the automobile M, is transmitted from the in-vehicle device 10 to the external device 70, but the present invention is not limited to this. For example, the image transmitted from the in-vehicle device 10 to the external device 70 may be switchable between an image captured by the external camera 12 of the automobile M and an image of the interior of the automobile M captured by the internal camera 13. When an image of the interior of the automobile M is being transmitted, the user of the external device 70 can, for example, communicate with the driver of the automobile M while viewing the interior of the automobile M.

[0103] The switching operation for switching the video to be transmitted to the external device 70 between the video from the external camera 12 and the video from the internal camera 13 may be performed by the in-vehicle device 10. The switching operation may also be performed remotely by a user of the external device 70 operating the external device 70.

[0104] In this embodiment, the in-vehicle device 10 transmits an external image or an internal image of the automobile M, or a composite image obtained by processing these, and current location information to the external device 70 via the server 40.

[0105] However, the in-vehicle device 10 may transmit to the external device 70, in addition to the external image, internal image, and current location information, a map image showing the planned travel route, the name of the driver of the automobile M, the traveling speed of the automobile M, etc., and the external device 70 may generate an image by superimposing the information transmitted from the in-vehicle device 10 on the image of the exterior of the automobile M. In other words, the external device 70 may generate a composite image like the one generated by the in-vehicle device 10 in the above embodiment.

[0106] For example, the control unit 84 of the external device 70 may display on the touch panel 71 a map image received from the in-vehicle device 10, identification information such as the name of the driver of the vehicle M and the vehicle registration number, and the traveling speed of the vehicle M, superimposed on an image of the outside of the vehicle M. Furthermore, the external device 70 may perform processing to display the map image received from the in-vehicle device 10, identification information such as the name of the driver of the vehicle M and the vehicle registration number, and the traveling speed of the vehicle M side by side in a display area different from the image of the outside of the vehicle M. The map image, the name of the driver of the vehicle M, and the traveling speed of the vehicle M may be freely switched between display and non-display by the operator of the external device 70. Note that similarly, the external device 70 may be configured to be able to change whether to display or not information included in a composite image generated and transmitted by the in-vehicle device 10 through an operation from the external device 70.

[0107] In the above embodiment, an example has been described in which the in-vehicle device 10 is mounted on an automobile M, but the in-vehicle device 10 may be mounted on other moving objects such as a bicycle or a motorcycle. Also, the in-vehicle device 10 may be a device with a built-in camera like a smartphone, which may be held by a person, and the in-vehicle device 10 may perform the above-mentioned virtual passenger communication while walking, for example, to deliver video.

[0108] The configurations, routines, etc. of the in-vehicle device 10, the server 40, and the external device 70 in the above-described embodiment are merely examples, and can be appropriately selected or changed depending on the application, etc. [Explanation of symbols]

[0109] 100 Virtual passenger communication system 10 Onboard equipment 12 External Camera 13 Internal Camera 25, 84 Control section 31, 87 Data Communications Department 40 Relay Server 70 External device

Claims

1. An acquisition unit that acquires current location information indicating the current location of a moving vehicle, and acquires an external image of the vehicle taken by a photographing unit disposed in the vehicle while the vehicle is moving, and route information to the destination of the vehicle; a transmitting unit that performs a transmitting process to transmit to a terminal device, while the vehicle is moving, a traveling image in which information based on the route information and the current position information is superimposed on the external image acquired by the acquiring unit; a voice communication unit that performs a voice communication process for a voice communication between an occupant of the vehicle and a user of the terminal device simultaneously with the transmission process; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the acquisition unit acquires internal images taken by a photographing unit disposed in the vehicle while the vehicle is moving, and the transmission unit transmits the internal images to the terminal device in addition to the images taken while the vehicle is moving during the transmission process.

3. An acquisition unit that acquires current location information indicating the current location of a vehicle while the vehicle is moving, while acquiring route information regarding the planned route of the vehicle and an external image of the vehicle taken by a photographing unit disposed in the vehicle while the vehicle is moving; a transmitting unit that performs a transmission process of transmitting the external image, the route information, and the current position information acquired by the acquiring unit to a terminal device while the vehicle is moving; and a voice communication unit that performs voice communication processing for a voice call between an occupant of the vehicle and a user of the terminal device simultaneously with the transmission processing.

4. An information processing method executed by an information processing device moving with a vehicle, comprising: an acquisition step of acquiring current position information indicating a current position of the vehicle while the vehicle is moving, and acquiring an external image of the vehicle captured by an image capturing unit disposed in the vehicle while the vehicle is moving, and route information to a destination of the vehicle; a transmitting step of performing a transmitting process of transmitting a traveling image, in which information based on the route information and the current position information is superimposed on the external image acquired in the acquiring step, to a terminal device while the vehicle is moving; a voice call step of performing a voice call process for a voice call between an occupant of the vehicle and a user of the terminal device simultaneously with the transmission process; An information processing method comprising:

5. An information processing device that moves with a vehicle, an acquisition step of acquiring current position information indicating a current position of the vehicle while the vehicle is moving, and acquiring an external image of the vehicle captured by an image capturing unit disposed in the vehicle while the vehicle is moving, and route information to a destination of the vehicle; a transmitting step of performing a transmitting process of transmitting a traveling image, in which information based on the route information and the current position information is superimposed on the external image acquired in the acquiring step, to a terminal device while the vehicle is moving; a voice call step of performing a voice call process for a voice call between an occupant of the vehicle and a user of the terminal device simultaneously with the transmission process; An information processing program characterized by causing the program to execute the above.

6. An information processing device that moves with a vehicle, an acquisition step of acquiring current position information indicating a current position of the vehicle while the vehicle is moving, and acquiring an external image of the vehicle captured by an image capturing unit disposed in the vehicle while the vehicle is moving, and route information to a destination of the vehicle; a transmitting step of performing a transmitting process of transmitting a traveling image, in which information based on the route information and the current position information is superimposed on the external image acquired in the acquiring step, to a terminal device while the vehicle is moving; a voice call step of performing a voice call process for a voice call between an occupant of the vehicle and a user of the terminal device simultaneously with the transmission process; A computer-readable storage medium storing an information processing program for executing the above.

Images