Communication system, communication device, server device, communication method, communication program, and storage medium

Abstract

To provide a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium that make it possible, in communication between a first device inside a mobile and a second device outside the mobile, to suppress wasteful connection trial from the second device to the first device while considering privacy of an operator of the mobile, to implement efficient smooth communication.SOLUTION: A communication system for performing audio and video communication to transmit audio and video in a mobile to a second device, the audio and video acquired by a first device moving together with the mobile, comprises: a connection information provision unit for providing connection information that is to be used for connection of the audio and video communication and has a validity period; and a display control unit that when receiving access to the connection information from the second device, causes the second device to display boarding status information indicating whether or not an operator is boarding the mobile in a period corresponding to the validity period.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium, and more particularly, to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium capable of providing video and audio from a mobile body to a user.

Background Art

[0002] There is a communication system that performs communication between an in-vehicle device mounted on a vehicle and an out-of-vehicle terminal located outside the vehicle.

[0003] For example, Patent Document 1 discloses a system in which a voice call is made between a driver of a vehicle and an operator of an out-of-vehicle terminal, and when the voice call is being made, video data representing a video in front of the vehicle is transmitted from the in-vehicle device to the out-of-vehicle terminal and the video is displayed on the out-of-vehicle terminal. In Patent Document 1, based on receiving a call start notification, the in-vehicle device starts a process of performing a call and transmitting data such as an image, transmits a voice signal of the speech content of the driver of the vehicle to the out-of-vehicle terminal, and also transmits vehicle information including front image data to the out-of-vehicle terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a system such as Patent Document 1, for example, when a call start notification is transmitted from the out-of-vehicle terminal to the in-vehicle device, there may be a case where the driver is not in the vehicle on which the in-vehicle device is mounted. In that case, the connection attempt by the transmission of the call start notification is wasted, and it takes unnecessary time and effort, which is one of the problems.

[0006] Therefore, it is conceivable to notify the user of the in-vehicle terminal of the driver's boarding status before the connection attempt. However, it is also cited as one of the problems that there is an issue from the perspective of the driver's privacy if the boarding status is known without limitation.

[0007] The present invention has been made in view of the above points, and in communication between a first device inside a moving body and a second device outside the moving body, while taking into account the privacy of the operator of the moving body, it suppresses unnecessary connection attempts from the second device to the first device and realizes efficient and smooth communication. One of the objectives is to provide a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium.

Means for Solving the Problems

[0008] The invention according to claim 1 is a communication system that performs audio-visual communication for transmitting the audio and video in the moving body acquired by a first device that moves with the moving body to a second device, and is used for connecting the audio-visual communication and provides connection information having an expiration date. A connection information providing unit, and when receiving access to the connection information from the second device, a display control unit that causes the second device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date. It is characterized by having.

[0009] The invention according to claim 7 is a communication method executed by a communication system that performs audio-visual communication for transmitting the audio and video in the moving body acquired by a first device that moves with the moving body to a second device, and is used for connecting the audio-visual communication and provides connection information having an expiration date. A connection information providing step, and when receiving access to the connection information from the second device, a display control step of causing the second device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date. It is characterized by including.

[0010] The invention according to claim 8 is a communication program executed by a communication system that includes a computer and performs audio-visual communication for transmitting the audio and video in the moving body acquired by a first device that moves together with the moving body to a second device. The computer is caused to execute a connection information providing step of providing connection information that is used for connection of the audio-visual communication and has an expiration date, and a display control step of causing the second device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date when access to the connection information from the second device is received.

[0011] The invention according to claim 9 is a computer-readable storage medium that stores a communication program for causing a communication system that includes a computer and performs audio-visual communication for transmitting the audio and video in the moving body acquired by a first device that moves together with the moving body to a second device to execute a connection information providing step of providing connection information that is used for connection of the audio-visual communication and has an expiration date, and a display control step of causing the second device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date when access to the connection information from the second device is received.

[0012] The invention according to claim 10 is a communication device that moves together with a moving body and performs audio-visual communication for acquiring the audio and video in the moving body and transmitting them to another device, the communication device including a connection information providing unit that provides connection information that is used for connection of the audio-visual communication and has an expiration date, and a display control unit that causes the other device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date when access to the connection information from the other device is received.

[0013] The invention according to claim 11 is a server device that performs audio-visual communication for transmitting the audio and video in the moving body acquired by a first device that moves with the moving body to a second device, the server device comprising: a connection information providing unit that provides connection information that is used for connection of the audio-visual communication and has an expiration date; and a display control unit that, when receiving access to the connection information from the second device, causes the second device to display boarding status information indicating whether or not an operator is boarding the moving body during a period corresponding to the expiration date.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described in detail. In the following description and the accompanying drawings, the same reference numerals are given to substantially the same or equivalent parts.

Example

[0016] [1. Configuration of the System] The configuration of a communication system 100 including an in-vehicle device 10 according to Example 1 will be described with reference to the accompanying drawings.

[0017] FIG. 1 shows a communication system 100 which is Example 1 of the present invention. As shown in FIG. 1, the communication system 100 includes an in-vehicle device 10, a server 30, and an external device 40. In FIG. 1, a case where the in-vehicle device 10 is mounted on an automobile M as an example of a moving body is shown. Also, in FIG. 1, a smartphone is shown as an example of the external device 40.

[0018] The in-vehicle device 10, the server 30, and the external device 40 can mutually transmit and receive data via a network NW using a communication protocol such as TCP / IP or UDP / IP. Note that the network NW can be constructed by, for example, a mobile communication network, wireless communication such as Wi-Fi (registered trademark), and Internet communication including wired communication.

[0019] In the communication system 100, audio-visual communication for transmitting the audio and video in the automobile M acquired by the in-vehicle device 10 as the first device moving with the automobile M to the external device 40 as the second device is possible.

[0020] In the communication system 100 of this embodiment, after a voice call is established between the in-vehicle device 10 and the external device 40, the video captured in the vehicle M from the in-vehicle device 10 is distributed to the external device 40.

[0021] In the following description, as described above, a communication mode in which while establishing a voice call between the in-vehicle device 10 and the external device 40, the video captured in the vehicle M is distributed from the in-vehicle device 10 to the external device 40 is referred to as video communication as one mode of audio-video communication. In this embodiment, the in-vehicle device 10 and the external device 40 perform video communication via the server 30.

[0022] By performing such video communication, the user of the external device 40 who is viewing the video transmitted from the in-vehicle device 10 can obtain a feeling as if riding in the vehicle M with the driver of the vehicle M as the operator of the moving body. In other words, video communication can realize the virtual ride of the user of the external device 40 in the vehicle M. Also, a system such as the communication system 100 of this embodiment that can realize such video communication is also referred to as a virtual ride system.

[0023] Hereinafter, in the first embodiment, the case where the in-vehicle device 10 is a car navigation device will be described as an example. Also, in the first embodiment, the case where the in-vehicle device 10 is a terminal device of a so-called cloud-type car navigation device that receives from the user a destination that the user desires to be guided to, transmits the destination to the server 30, and the server 30 generates a route to the destination will be described as an example.

[0024] Also, in this embodiment, as an example of the mode of starting video communication, the case where first an invitation for video communication from the in-vehicle device 10 to the external device 40 is made, and connection information for video communication is provided at the time of the invitation will be described. In this embodiment, when a connection attempt from the external device 40 using the connection information is made, the process of starting video communication proceeds.

[0025] FIG. 2 is a perspective view showing the vicinity of the front seat of the automobile M equipped with the in-vehicle device 10 as a video transmission device or a video generation device. In FIG. 1, as an example of the mounting, the case where the in-vehicle device 10 is mounted in the dashboard DB of the front seat of the automobile M is shown.

[0026] The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is arranged, for example, on the dashboard DB. Note that the GPS receiver 11 may be arranged anywhere as long as GPS signals can be received. The GPS receiver 11 can transmit the received GPS signals to the in-vehicle device 10.

[0027] The outside vehicle camera 13 is an imaging device that captures the front of the automobile M. In the present embodiment, the outside vehicle camera 13 is arranged on the dashboard DB so that the front is the imaging direction. For example, the outside vehicle camera 13 is a wide-angle camera and can capture a wide area in front of the automobile M through the windshield. Note that the outside vehicle camera 13 may be provided on the rearview mirror RM or may be attached inside the windshield FG.

[0028] The inside vehicle camera 15 is an imaging device that captures the situation inside the automobile M. In the present embodiment, the inside vehicle camera 15 is provided at the upper end of the windshield FG or in the ceiling portion near the upper end, and can capture the driver of the automobile M.

[0029] During video communication, the video captured by the outside vehicle camera 13 or the inside vehicle camera 15 is distributed to the external device 40.

[0030] The touch panel 17 is, for example, a touch panel monitor in which a display such as a liquid crystal display capable of displaying an image and a touch pad are combined. The touch panel 17 is arranged, for example, on the center console CC of the dashboard DB. The touch panel 17 only needs to be arranged in a place where it can be visually recognized by the driver and within reach of the driver's hand. For example, the touch panel 17 may be mounted on the dashboard DB.

[0031] The touch panel 17 is capable of performing screen display based on the control of the in-vehicle device 10. Further, the touch panel 17 can transmit a signal representing an input operation to the touch panel 17 received from the user to the in-vehicle device 10. For example, a guidance display for car navigation may be provided on the touch panel 17. Also, operations related to the car navigation function such as setting a destination may be possible via the touch panel 17.

[0032] Further, information related to video communication may be displayed on the touch panel 17, and a reception screen for an operation for making a connection for video communication (hereinafter also referred to as a video connection) may be displayed. An occupant of the automobile M may perform a connection operation for video communication by an input operation on the touch panel 17.

[0033] The speaker 19 is provided, for example, on the interior side of the A-pillar AP. The speaker 19 is capable of emitting sounds such as music and voices based on the control of the in-vehicle device 10. During video communication, the voice from the external device 40 in a voice call is emitted from the speaker 19.

[0034] The microphone 21 is a microphone device that receives sounds inside the vehicle, and is arranged, for example, on the dashboard DB. The microphone 21 may be provided anywhere, such as on the rearview mirror RM or the steering wheel, as long as it can receive sounds inside the vehicle. During video communication, the sound picked up by the microphone 21 is transmitted to the external device 40 as the voice of the voice call. Also, an operation input related to the start of video communication may be made by voice via the microphone 21.

[0035] FIG. 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a storage unit 24, a control unit 25, and a communication unit 27 cooperate via a system bus.

[0036] The seating sensor 22 can actually detect whether or not the driver of the automobile M is in the vehicle (hereinafter also referred to as "boarding"), and is, for example, a sensor that can output a signal indicating the presence or absence of the detected driver. The seating sensor 22 is provided, for example, on the seat surface or the back surface of the driver's seat of the automobile M, and is a sensor that detects a load. Note that the detection of the driver may be performed by image recognition of the video of the in-vehicle camera 15, or may be performed by directly detecting the driver himself / herself with a photoelectric sensor such as another infrared sensor.

[0037] In addition, the automobile M is equipped with an acceleration sensor 23. The acceleration sensor 23 can measure the acceleration of the automobile M and may be, for example, a MEMS acceleration sensor that can output a signal indicating the measured acceleration. Note that the acceleration sensor 23 may be an inertial measurement unit (IMU: Inertial Measurement Unit). Further, the acceleration sensor 23 may be built into the in-vehicle device 10.

[0038] The storage unit 24 is a storage device composed of, for example, a hard disk device, an SSD (solid state drive), a flash memory, etc. The storage unit 24 stores various programs executed in the in-vehicle device 10, such as an operating system and software for the terminal.

[0039] The various programs may be acquired from other server devices or the like via a network, for example, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the storage unit 24 can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred.

[0040] In addition, the storage unit 24 stores map information including road maps. The storage unit 24 also stores information indicating the contact information of the external device 40 as a video communication partner, such as a telephone number or an email address.

[0041] The control unit 25 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and functions as a computer. The control unit 25 realizes various functions by reading and executing various programs stored in the ROM and the storage unit 24 by the CPU. In this embodiment, the control unit 25 exhibits functions such as presenting information regarding the start and end of video communication to the driver, a video distribution function during video communication, and a car navigation function.

[0042] The control unit 25 is communicably connected to an external camera 13, an in-vehicle camera 15, a touch panel 17, a speaker 19, a microphone 21, a seating sensor 22, and an acceleration sensor 23 provided in the in-vehicle device 10.

[0043] The control unit 25 sequentially acquires the videos captured by the external camera 13 and the in-vehicle camera 15. The control unit 25 also sequentially acquires information indicating whether or not the driver detected by the seating sensor 22 is in the vehicle. The control unit 25 also sequentially acquires a signal from the GPS receiver 11 and a signal indicating the acceleration measured by the acceleration sensor 23.

[0044] The communication unit 27 is a communication device that transmits and receives data to and from external devices according to the instructions of the control unit 25. The communication unit 27 is, for example, a NIC (Network Interface Card) for connecting to the network NW.

[0045] The communication unit 27 is connected to the above-described network NW and transmits and receives various data to and from the server 30. The communication unit 27 also transmits and receives various data to and from the external device 40 via the server 30.

[0046] The communication unit 27 transmits information such as images outside or inside the vehicle M, current position information, and movement routes to the server 30, for example. Further, the communication unit 27 receives information for display on the touch panel 17 regarding video communication from the server 30, for example.

[0047] During video communication, the control unit 25 sequentially transmits the images captured by the external camera 13 or the in-vehicle camera 15 to the server 30 via the communication unit 27.

[0048] The control unit 25 receives display information regarding the start and end of video communication from the server 30 via the communication unit 27 and displays it on the touch panel 17. Further, the control unit 25 may output information regarding the start and end of video communication as audio from the speaker 19.

[0049] For example, the control unit 25 of the in-vehicle device 10 can transmit, as position identification information as position data capable of identifying the current position of the in-vehicle device 10, that is, information on the current position of the vehicle M in this embodiment, to the server 30 via the communication unit 27. Further, for example, the control unit 25 can transmit information including a destination input by the user to the server 30 via the communication unit 27 and receive route information or navigation information to the destination from the server 30.

[0050] The control unit 25 can transmit the audio data of the audio picked up by the microphone 21 to an external device 40 for voice calls in video communication via the communication unit 27. Further, the control unit 25 can receive the audio data of the audio input to the external device 40 for voice calls in video communication via the communication unit 27.

[0051] FIG. 4 is a block diagram showing the configuration of the server 30. For example, the server 30 is a device in which the storage unit 31, the communication unit 33, and the control unit 35 cooperate via a system bus.

[0052] During video communication, the server 30 has a function such as a SIP server that establishes a voice call between the in-vehicle device 10 and the external device 40 and transfers the data of the voice call.

[0053] In addition, the server 30 has a function of receiving the position identification information of the automobile M and the information of the destination set by the user who is the occupant of the automobile M from the in-vehicle device 10, and generating a route to the destination based on the position identification information and the destination information.

[0054] The storage unit 31 is composed of, for example, a hard disk device and an SSD (solid state drive), and stores various programs such as an operating system and software for the server 30.

[0055] The various programs may be obtained via a network from other server devices or the like, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the storage unit 31 can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred.

[0056] The storage unit 31 stores, for example, a communication program for the server 30 to control the connection of video communication in the communication system 100.

[0057] In addition, the storage unit 31 stores a map information database (not shown) in which map information including a road map is stored. The map information of the map information database is a database having information equivalent to the map information used in, for example, a navigation device.

[0058] The communication unit 33 is connected to the above-described network NW and transmits and receives various data to and from the in-vehicle device 10 and the external device 40.

[0059] The control unit 35 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and functions as a computer. Then, the CPU reads and executes various programs stored in the ROM and the storage unit 31 to realize various functions.

[0060] By executing the above-mentioned communication program, the control unit 35 realizes the control related to the connection of video communication in the communication system 100.

[0061] The control unit 35 includes a connection information providing unit 36, a boarding information acquisition unit 37, and a display control unit 38.

[0062] The connection information providing unit 36 is a functional unit that executes a connection information providing step of providing, to the external device 40, a URL (Uniform Resource Locator) used for connection when the external device 40 performs video communication with the in-vehicle device 10 in response to a request from the in-vehicle device 10 and having an expiration date.

[0063] For example, when the connection information providing unit 36 receives a transmission request for a URL for inviting the external device 40 to video communication from the in-vehicle device 10, it generates a URL having an expiration date, and transmits the URL having the expiration date to the external device 40, thereby providing the URL to the external device 40.

[0064] The connection information providing unit 36, for example, generates a URL used for the connection of video communication, associates the URL with the expiration date of the URL, stores them in a database, and performs management of rejecting access to the page by the URL when the expiration date arrives. The database is stored in the storage unit 31, for example.

[0065] Note that the connection information providing unit 36 may acquire a URL having an expiration date from a device such as an external server.

[0066] The expiration date of the above URL is set, for example, at the time when a predetermined period has elapsed since a request was made from in-vehicle device 10. For example, the expiration date may be several hours to 24 hours, or even several days, from the time when the request was made.

[0067] Also, the expiration date of the URL may be settable by the driver for each user of external device 40. Thereby, for example, the driver of motor vehicle M can set a longer expiration date for a user who frequently conducts video communication, such as a friendly user, and can save the trouble of making an invitation each time video communication is conducted. Also, for a user who does not conduct video communication very frequently, the expiration date can be set short, and unnecessary access from such a user to the driver can be prevented.

[0068] The boarding information acquisition unit 37 is a functional unit that acquires boarding status information indicating whether or not a driver is boarding on motor vehicle M.

[0069] The boarding status information is, for example, information based on the result of actually detecting whether or not the driver is boarding on motor vehicle M. For example, the boarding information acquisition unit 37 acquires, from in-vehicle device 10, a video of an area including the driver's seat of motor vehicle M taken by in-vehicle camera 15, and actually detects whether or not the driver is present in the driver's seat by performing image recognition on the video. The boarding information acquisition unit 37 acquires boarding status information indicating whether or not the driver is boarding based on the detection result.

[0070] Also, for example, the boarding information acquisition unit 37 actually detects whether or not a driver is present in the driver's seat by acquiring the detection result of seat sensor 22 provided in motor vehicle M, and acquires boarding status information indicating whether or not the driver is boarding based on the detection result.

[0071] Also, the boarding status information is, for example, information based on whether or not in-vehicle device 10 is connected to network NW, which is a communication network used for video communication. The boarding information acquisition unit 37 acquires the boarding status information by detecting whether or not in-vehicle device 10 is connected to network NW.

[0072] In addition, the boarding status information may be information based on, for example, whether or not the accessory power supply of the vehicle M is turned on.

[0073] For example, a monitoring device capable of detecting the ON / OFF of the accessory power supply may be installed in the vehicle M, and the boarding information acquisition unit 37 may acquire information indicating whether or not the accessory power supply is turned on from the monitoring device.

[0074] In addition, the boarding status information is, for example, information indicating the driver's state of relaxation during driving. The state of relaxation is, in other words, a state determined by the busyness of driving or the level of workload. The boarding information acquisition unit 37 acquires boarding status information indicating the driver's state of relaxation with respect to driving by using information from, for example, the in-vehicle camera 15, the out-of-vehicle camera 13, the acceleration sensor 23, etc. Further, in other words, it can be said that the state of relaxation is a state as to whether or not the driver has enough time to conduct video communication and have a conversation during driving.

[0075] The display control unit 38 is a functional unit that, when receiving access to the URL provided by the connection information providing unit 36, executes a display control step of causing an external device 40 to display boarding status information indicating whether or not a driver is on board the vehicle M during a period corresponding to the expiration date of the URL.

[0076] For example, in the display control step, the display control unit 38 determines whether or not it is within the expiration date of the accessed URL, and if it is within the expiration date, causes the external device 40 to display the boarding status information. For example, when it is outside the expiration date of the URL, the display control unit 38 causes the external device 40 to display a message indicating that the URL is invalid.

[0077] For example, if it is within the expiration period of the above URL, the display control unit 38 refers to the boarding status information acquired by the boarding information acquisition unit in the display control step and determines whether or not a driver is on board the vehicle M. As a result of this determination, if a driver is on board the vehicle M, the display control unit 38 causes, for example, an external device 40 to display a display indicating that the driver is on board. As a result of this determination, if a driver is not on board the vehicle M, the display control unit 38 causes, for example, an external device 40 to display a display indicating that the driver is getting off the vehicle.

[0078] The user of the external device 40 can know, before making a connection attempt, whether or not a driver is on board, that is, whether or not video communication between the in-vehicle device 10 and the external device 40 is possible. Therefore, when the driver is getting off the vehicle, it is possible to avoid situations where a connection cannot be made even if a connection attempt is made, or unnecessary efforts such as repeatedly making connection attempts.

[0079] In addition, the control unit 35 acquires position information indicating the current position of the vehicle M from the in-vehicle device 10 via the communication unit 33. The control unit 35 also acquires information on the destination input by the passenger of the vehicle M to the in-vehicle device 10 from the in-vehicle device 10 via the communication unit 33. Based on the position information and the destination information, the control unit 35 generates a route to the destination and transmits information indicating the route to the in-vehicle device 10.

[0080] In addition, the control unit 35 transfers the video and audio received from the in-vehicle device 10 to the external device 40 via the communication unit 33. The control unit 35 transfers the audio received from the external device 40 to the in-vehicle device 10 via the communication unit 33.

[0081] FIG. 5 is a front view of the external device 40. As described above, in the present embodiment, the external device 40 is a smartphone.

[0082] The touch panel 41 is, for example, a touch panel monitor in which a display such as a liquid crystal display capable of displaying images and a touch pad are combined. The touch panel 41 can generate a signal representing an input operation to the touch panel 41 received from a user. In the present embodiment, an image distributed from the in-vehicle device 10 is displayed on the touch panel 41.

[0083] Further, information regarding video communication is displayed on the touch panel 41, and a reception screen for an operation to start video communication is displayed. A user of the external device 40 can perform an input continuation operation for starting video communication by an input operation on the touch panel 41.

[0084] The speaker 43 can emit sounds such as music and voices. During video communication, voices from the in-vehicle device 10 in a voice call are emitted from the speaker 43.

[0085] The microphone 45 is a microphone device that receives sounds emitted toward the external device 40. During video communication, the sounds picked up by the microphone 45 are transmitted to the in-vehicle device 10 via the server 30 as the voices of the voice call.

[0086] FIG. 6 is a block diagram showing an example of the configuration of the external device 40. For example, the external device 40 is a device in which a storage unit 47, a control unit 49, and a communication unit 51 cooperate via a system bus (not shown).

[0087] The storage unit 47 is constituted by, for example, a hard disk device, an SSD (solid state drive), a flash memory, etc., and stores various programs such as an operating system and software for the external device 40.

[0088] Note that various programs may be acquired, for example, from other server devices or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the storage unit 47 can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred.

[0089] The control unit 49 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and functions as a computer. Then, the CPU reads and executes various programs stored in the ROM and the storage unit 47 to realize various functions.

[0090] The control unit 49 is communicably connected to the touch panel 41, the speaker 43, and the microphone 45.

[0091] The control unit 49 can receive a signal indicating an input operation to the touch panel 41 and an audio input signal from the microphone 45. For example, the control unit 49 can receive a request to start a video communication made by the user via the touch panel 41 and the microphone 45.

[0092] Also, the control unit 49 can transmit a video or image signal to the touch panel 41 to display it, or transmit an audio signal to the speaker 43 to output sound.

[0093] For example, the control unit 49 causes the touch panel 41 to display a start screen for receiving a request for a video communication connection attempt, a screen indicating whether the driver of the automobile M is in the vehicle, or a screen indicating that the URL is invalid.

[0094] Also, for example, the control unit 49 causes the touch panel 41 to display the video during the video communication, and causes the speaker 43 to output the audio during the video communication recorded in the automobile M.

[0095] The communication unit 51 is connected to the above-described network NW and transmits and receives various data to and from the server 30. Further, the communication unit 51 transmits and receives various data including video transmitted from the in-vehicle device 10 to and from the in-vehicle device 10 via the server 30.

[0096] The control unit 49 of the external device 40 can receive, via the communication unit 51, information for displaying a start screen for receiving a request for a connection attempt of video communication, a screen indicating whether or not the driver of the automobile M is in the vehicle, or a screen indicating that the URL is invalid.

[0097] Further, the control unit 49 can transmit, via the communication unit 51, voice data of the voice collected by the microphone 45 to the in-vehicle device 10 for a voice call in video communication. Further, the control unit 49 can receive, via the communication unit 51, voice data transmitted from the in-vehicle device 10 for a voice call in video communication.

[0098] For example, the control unit 49 of the external device 40 can receive, via the communication unit 51, position identification information capable of identifying the current position of the automobile M transmitted from the in-vehicle device 10 from the server 30. Further, for example, the control unit 49 can receive, via the communication unit 51, route information or navigation information of the automobile M from the server 30.

[0099] [2. Sequence when the URL is valid] FIG. 7 is a sequence diagram showing an example of communication performed between the in-vehicle device 10 and the external device 40 via the server 30 when video communication is started in the communication system 100 with a valid URL. Hereinafter, a case will be described in which a driver who is a user of the in-vehicle device 10 plans to perform video communication with a virtual co-rider who is a user of the external device 40 when driving the automobile M.

[0100] As shown in FIG. 7, the driver registers in advance the contact information of the virtual co-rider who is the partner of the video communication to be performed with the external device 40 (step S11).

[0101] In the registration of a virtual passenger, for example, registration information including a name that can identify the virtual passenger such as a nickname and information indicating a destination such as an email address or a phone number necessary for sending information to the virtual passenger is transmitted to the server 30 and held by the server 30. The registration information includes, for example, an identifier that can identify the in-vehicle device 10.

[0102] The registration of the virtual passenger is performed via the touch panel 41 of the in-vehicle device 10, but is not limited thereto, and may be performed by transmitting input information to a terminal device such as a smartphone used by the driver of the automobile M other than the in-vehicle device 10 to the server 30.

[0103] The timing at which the driver pre-registers the contact information of the virtual passenger may be, for example, before the day before the day when video communication during the driving of the automobile M is planned, or may be immediately before the video communication.

[0104] After that, for example, when the driver starts driving the automobile M, the driver performs an input operation for instructing the in-vehicle device 10 to send an invitation email to the external device 40. The input operation may be performed via the touch panel 17 of the in-vehicle device 10 or may be performed by voice via the microphone 21. Further, the input operation for instructing the sending of the invitation email may be performed before starting the driving of the automobile M.

[0105] An invitation email sending request for requesting the server 30 to send an invitation email to the external device 40 is transmitted by the driver's input operation (step S12). The invitation email sending request includes, for example, information indicating a destination such as an email address used by the pre-registered virtual passenger or a phone number of the external device 40 and a nickname that can identify the virtual passenger.

[0106] When the server 30 receives an invitation email sending request, it sends an invitation email to the destination of the external device 40 included in the invitation email sending request (step S13). The invitation email includes a URL for opening a start screen for the virtual passenger to conduct a video communication.

[0107] For example, the connection information providing unit 36 of the server 30 generates a URL with an expiration date, generates an invitation email including a message inviting the user of the external device 40 to a video communication and the URL with the expiration date, and sends the invitation email to the external device 40.

[0108] After the invitation email is sent from the server 30, when the virtual passenger opens the invitation email using the external device 40 and taps the URL (step S14A), the server 30 checks the expiration date of the URL.

[0109] If the URL is within the valid period and the URL is valid (step S15A), a browser starts on the touch panel 41 of the external device 40, and a boarding status screen showing the driver's boarding status is displayed (step S16A).

[0110] For example, when the driver is in the vehicle, on the boarding status screen, together with a message indicating that the driver is in the vehicle, a start button, which is a display area for accepting a request to start a video communication when the virtual passenger wishes to start a video communication and operates it, may be displayed.

[0111] When the virtual passenger taps the start button, the operation of the start button by the user is accepted (step S17), and a request to start a video communication is sent to the server 30 (step S18).

[0112] When the server 30 receives a request to start a video communication, it notifies the in-vehicle device 10 that there has been a request to start a video communication from the external device 40 (step S19).

[0113] In step S19, for example, when in-vehicle device 10 receives a notification from server 30, a selection button for whether to approve the start of video communication is displayed on touch panel 17. For example, when the driver taps the approval button, information indicating that the driver has approved the start of video communication is transmitted to server 30 (step S20).

[0114] When server 30 receives information indicating that the driver has approved the start of video communication, while establishing an audio call between in-vehicle device 10 and external device 40, it starts a video communication to distribute the video captured in motor vehicle M from in-vehicle device 10 to external device 40 (step S21).

[0115] In this way, when the URL is within the valid period and the driver is in the vehicle, in-vehicle device 10 and external device 40 start video communication via server 30.

[0116] [3. Sequence when the URL is invalid] FIG. 8 is a diagram showing an example of a sequence when the expiration date of the URL accessed from external device 40 has passed. In FIG. 8, steps S11 to S13 proceed in the same manner as the sequence in FIG. 7, and after the expiration date has passed, the steps after step S14B will be described for the case where they proceed. In FIG. 8, for example, the expiration date of the URL is set to 24 hours.

[0117] In step S14B, when the virtual co-rider uses external device 40 to open the invitation email received in step S13 and taps the URL, server 30 checks the expiration date of the URL.

[0118] When it is confirmed that the URL is outside the expiration date and it is determined that the URL is invalid in the confirmation of the deadline (step S15B), the server 30 causes the external device 40 to display a page indicating that the URL is invalid (step S16B). In step S16B, for example, an expired screen indicating that the expiration date of the URL has passed is displayed on the touch panel 41 of the external device 40. Thereafter, the sequence of FIG. 8 ends.

[0119] [4. Server Operation] Hereinafter, a control routine of the server 30 for realizing the operation of the communication system 100 of the embodiment will be described. Hereinafter, the boarding information acquisition unit 37 will be described as always acquiring boarding status information and holding the latest boarding status information, for example, after a request to send an invitation email (FIG. 7, step S12) is received by the server 30.

[0120] FIG. 9 is a flowchart showing a display control routine RT1 executed by the display control unit 38 of the server 30. The display control unit 38 starts the display control routine RT1, for example, when an invitation email is sent from the server 30 to the external device 40.

[0121] When starting the display control routine RT1, the display control unit 38 determines whether there has been an access to the URL included in the invitation email (step S101). In step S101, for example, when the virtual co-rider opens the invitation email using the external device 40 and taps the URL included in the invitation email, the external device 40 accesses the URL.

[0122] In step S101, when the display control unit 38 determines that there is no access to the URL (step S101: NO), it ends the display control routine RT1, starts the next display control routine RT1, and executes step S101 again.

[0123] In step S101, when the display control unit 38 determines that there has been an access to the URL (step S101: YES), it determines whether the URL is within the expiration date (step S102).

[0124] In step S102, for example, the display control unit 38 refers to a database in which a URL and an expiration date are associated and stored, and determines whether the accessed URL is within the expiration date.

[0125] In step S102, when the display control unit 38 determines that the URL is within the expiration date (step S102: YES), it refers to the boarding status information sequentially acquired and held by the boarding information acquisition unit 37 (step S103). In step S103, for example, information indicating whether the driver is in the vehicle is referred to.

[0126] Also, for example, in step S103, information indicating whether the in-vehicle device 10 is connected to the network NW may be referred to, or information indicating the ON / OFF state of a seating sensor provided in the seat of the driver's seat of the automobile M may be referred to.

[0127] After the execution of step S103, the display control unit 38 determines whether the driver is in the vehicle based on the boarding status information referred to in step S102 (step S104).

[0128] In step S104, when the display control unit 38 determines that the driver is in the vehicle (step S104: YES), it causes an external device 40 to display a screen including a message indicating that the driver is in the vehicle (step S105).

[0129] FIG. 10 is a diagram showing an example of an in-vehicle screen displayed on the touch panel 41 of the external device 40 in step S105. As shown in FIG. 10, a message indicating that the driver is in the vehicle is displayed on the in-vehicle screen.

[0130] In addition, on the in-vehicle screen shown in FIG. 10, in addition to the message indicating that the driver is in the vehicle, a message prompting the start of a video call and a start button may be displayed. Note that the start button may be displayed on a screen different from the in-vehicle screen. For example, a button such as "Go to Start Screen" for proceeding to a screen for starting a video call may be displayed on the in-vehicle screen. When the button is pressed, a start screen including the start button may be displayed.

[0131] Note that a PIN input field for identity verification may be displayed on the in-vehicle screen or the start screen. For example, identity verification may be performed by inputting the last four digits of the phone number into the input field.

[0132] In addition, it may be possible to check the terms of use and privacy policy regarding video calls from the in-vehicle screen or the start screen. For example, when the start button is pressed, a display indicating that the user is deemed to have agreed to the terms of use and privacy policy may be shown.

[0133] In step S104, when the display control unit 38 determines that the driver is not in the vehicle (step S104: NO), it causes the external device 40 to display a screen including a message indicating that the driver is getting off the vehicle (step S106).

[0134] FIG. 11 is a diagram showing an example of the getting-off screen displayed on the touch panel 41 of the external device 40 in step S106. As shown in FIG. 11, a message indicating that the driver is getting off the vehicle is displayed on the getting-off screen. Also, as shown in FIG. 11, a message indicating that a video call cannot be started may be displayed on the getting-off screen.

[0135] In the example shown in FIG. 11, the display related to the start of a video call such as the start button is shaded and the start button is disabled. Note that on the getting-off screen, there may be no display related to the start of a video call including the start button at all.

[0136] When the display control unit 38 determines in step S102 that the URL is not within the expiration date (step S102: NO), it causes the external device 40 to display a screen including a message indicating that the expiration date of the URL has passed (step S107).

[0137] FIG. 12 is a diagram showing an example of an expired screen displayed on the touch panel 41 of the external device 40 in step S107. As shown in FIG. 12, in step S107, since access to the screen related to video communication is not possible, a message "The expiration date of the URL has passed" is displayed on the expired screen as a message indicating that the URL is invalid.

[0138] For example, on the expired screen, a message indicating that video communication cannot be started, or a message indicating that it is necessary to newly receive an invitation email to start video communication may be further displayed.

[0139] In steps S104 to S107, the display control unit 38 executes a display control step of causing the external device 40 to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date of the URL as connection information.

[0140] After the execution of step S105, step S106, or step S107, the display control unit 38 ends the display control routine RT1 and starts the next display control routine RT1.

[0141] According to the control routine RT1, by displaying a boarding screen as shown in FIG. 10 or an alighting screen as shown in FIG. 11, the virtual co-rider can know whether the driver is on board the vehicle M before attempting to connect for video communication. Therefore, the virtual co-rider can avoid making a futile connection attempt.

[0142] Also, according to the control routine RT1, an expiration date can be set for the URL, and the boarding status information can be displayed on the external device 40 only within the expiration date, and control can be performed so that the boarding status information is not displayed on the external device 40 outside the expiration date. Therefore, the boarding status information will not be known to the virtual co-rider without limit, and the privacy of the driver can be protected. For example, it is possible to prevent the boarding status information from being repeatedly known even after the scheduled video communication ends.

[0143] Furthermore, outside the expiration date, by displaying a message indicating that the URL as shown in FIG. 12 is invalid, the virtual co-rider can avoid accessing unnecessary connection information.

[0144] In step S103 of the control routine RT1 described above, in accordance with the driver's boarding status, or instead of this, when the driver's spare state during driving is referred to and the driver is in a situation without spare even while boarding, a message indicating that the video communication cannot be started may be displayed.

[0145] As described above, the communication system 100 of the present embodiment can perform video communication that transmits the video in the automobile M equipped with the in-vehicle device 10 to the external device 40 while establishing a voice call between the in-vehicle device 10 as the first device and the external device 40 as the second device via the server 30.

[0146] The communication system 100 of the present embodiment includes a connection information providing unit that is used for connecting audio-visual communication including video communication and provides connection information having an expiration date, and when receiving access to the connection information from the external device 40, a display control unit that displays boarding status information indicating whether or not a driver is boarding on the mobile body on the external device 40 during a period corresponding to the expiration date of the connection information.

[0147] According to this embodiment, before attempting to establish an audio-visual communication connection, the user of the external device 40 can determine whether the operator is on board. If the operator is not on board, it is clear that an audio-visual communication connection cannot be established, and the user of the external device 40 can avoid making a futile connection attempt when the operator is not on board.

[0148] According to this embodiment, for example, the display control unit can display the boarding status information only within the validity period of the connection information. This can prevent the boarding status of the operator from being known to the user of the external device 40 without limit, and can protect the privacy of the operator.

[0149] Therefore, according to this embodiment, in the communication between the first device inside the moving body and the second device outside the moving body, it is possible to provide a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium that can suppress futile connection attempts from the second device to the first device while taking into account the privacy of the operator of the moving body and realize efficient and smooth communication.

[0150] [Modification Example] FIG. 13 is a block diagram showing an example of the configuration of the in-vehicle device 60 in a modification of the communication system 100 of the first and second embodiments. In this modification, the in-vehicle device 60 has the functions of the connection information providing unit 36, the boarding information acquisition unit 37, and the display control unit 38 that the server 30 has in the above embodiments.

[0151] Specifically, the control unit 61 of the in-vehicle device 60 has a connection information providing unit 63, a boarding information acquisition unit 64, and a display control unit 65.

[0152] The storage unit 67 stores a communication program that is executed by the control unit 61, provides a URL that is used for establishing a video communication connection and has an expiration date, and when receiving an access from the external device 40 to the URL, performs control to display boarding status information indicating whether a driver is in the vehicle M during a period corresponding to the expiration date of the ULR.

[0153] For example, when the connection information providing unit 63 of the in-vehicle device 60 receives an operation input by the driver instructing the transmission of an invitation email to the external device 40, it generates a URL with an expiration date and transmits an invitation email including the URL to the external device 40 via the communication unit 27.

[0154] The boarding information acquisition unit 64 sequentially acquires boarding status information indicating whether or not the driver is boarding the vehicle M based on, for example, the detection results by the in-vehicle camera 15 or the seating sensor 22.

[0155] When the display control unit 65 receives an access to the URL from the external device 40, it performs control to cause the external device 40 to display the boarding status information indicating whether or not the driver is boarding the vehicle M during a period corresponding to the expiration date of the ULR.

[0156] According to this modification, regarding video communication between the external device 40 and the in-vehicle device 60, it is possible to display the boarding status information without passing through the server 30. For example, the control regarding the display of the boarding status information and the video communication in this modification may be directly performed between the in-vehicle device 10 and the external device 40 by P2P (Peer to Peer) communication or the like without passing through the server 30.

[0157] The configurations, routines, etc. of the in-vehicle device 10, the in-vehicle device 60, the server 30, and the external device 40 in the above-described embodiments and modifications are merely examples, and can be appropriately selected or changed according to the application, etc.

[0158] For example, in the above embodiments and modifications, the boarding status information may include information indicating whether or not the situation is such that video communication cannot be started even when the driver is boarding, in addition to the information indicating whether or not the driver is boarding.

[0159] For example, when in video communication with a device other than the external device 40, when there are passengers in the vehicle M, when the driving load is high, or when in conversation with a passenger or communication partner, it is presumed that the driver is in a situation where they cannot respond even if they start video communication, or a situation where they do not want to transmit the video inside or outside the vehicle or the audio inside the vehicle M to the external device 40.

[0160] In such a case, the driver's boarding situation may be treated as a situation where video communication cannot be started even when in the vehicle. In this case, for example, a display indicating that video communication cannot be started may be made on the in-vehicle screen showing that the driver is in the vehicle.

[0161] For example, the vehicle M may be equipped with seat sensors on seats other than the driver's seat, and the presence or absence of passengers used for determining the driver's boarding situation may be detected by the seat sensors.

[0162] For example, the magnitude of the driving load used for determining the driver's boarding situation may be calculated by workload estimation using the video of the outside camera 13 or the in-vehicle camera 15.

[0163] Also, in the above embodiments, it has been described that when the URL is outside the expiration date, a screen indicating that the URL is invalid is displayed on the external device 40, but it is not limited to this, and access may be made impossible by deleting the page specified by the URL.

[0164] Also, in the above embodiments and modified examples, as an aspect of receiving a start request from the external device 40, an example of tapping the start button displayed on the touch panel 41 has been described, but it is not limited to this. The start request may be received, for example, by an input operation using voice.

[0165] In the above-described embodiments and modified examples, an example of performing video communication in the communication system 100 has been described, but the present invention is not limited thereto. The present invention is applicable to a case where audio and video in the vehicle M acquired by the in-vehicle device 10 as the first device moving with the vehicle M are transmitted to the external device 40 as the second device in the communication system 100. That is, the present invention is also applicable to communication in a mode where no audio is transmitted from the external device 40.

[0166] Further, the in-vehicle device 10 and the external device 40 do not necessarily need to be able to make a voice call. For example, the communication mode between the in-vehicle device 10 and the external device 40 may be a mode in which video and audio are simply distributed from the in-vehicle device 10 to the external device 40 without a two-way voice call, such as YouTube (registered trademark) or Nico Nico Douga (registered trademark).

[0167] Also, in the above-described embodiment, the in-vehicle device 10 and the external device 40 perform video communication via the server 30, but the video communication may be directly performed between the in-vehicle device 10 and the external device 40.

[0168] Also, while the above video distribution is being performed from the in-vehicle device 10 to the external device 40 in the communication system 100, a voice call between the in-vehicle device 10 and the external device 40 may be established in parallel via a route different from the communication system 100.

[0169] In the above-described embodiments and modified examples, the in-vehicle device 10 is assumed to be an in-vehicle navigation device, but the in-vehicle device 10 only needs to be able to distribute a moving image from the vehicle exterior camera 13 or the vehicle interior camera 15.

[0170] For example, the in-vehicle device 10 may have a configuration in which a terminal device having the same configuration as the in-vehicle device 10, an in-vehicle camera 13, and a touch panel 17 are integrated. Specifically, for example, the in-vehicle device 10 may be a terminal device such as a smartphone, tablet, or PC with a camera that installs an application that exhibits the same functions as the above in-vehicle device 10. In this case, the in-vehicle device 10 can be attached on the dashboard DB with a cradle or the like so that the built-in camera can capture the front of the vehicle M through the windshield of the vehicle M.

[0171] Also, the in-vehicle device 10 may have a configuration that does not display a screen presented to the driver of the vehicle M. For example, the in-vehicle device 10 may have a configuration like a drive recorder, or may be a device integrated with the in-vehicle camera 13. Specifically, the in-vehicle device 10 may be a device that incorporates hardware that performs the video communication function of the above in-vehicle device 10 in the housing of the in-vehicle camera 13, for example. In this case, the in-vehicle device 10 may not perform various display outputs as described above.

[0172] In the above embodiments and modified examples, the case where the external device 40 is a smartphone has been described, but it is not limited thereto. The external device 40 is a terminal device that a virtual passenger can use for video communication, and only needs to be configured to be able to present a display or message related to video communication, receive operation inputs necessary for performing video communication, transmit and receive audio data, and receive and display video. For example, the external device 40 may be a terminal device such as a tablet, PC, or wearable terminal.

[0173] In the above-described embodiments and modified examples, an example in which the in-vehicle device 10 and the in-vehicle device 60 are mounted on the automobile M has been described. However, the in-vehicle device 10 may be mounted on other moving bodies such as bicycles, motorcycles, and ships. Further, the in-vehicle device 10 may be a terminal device incorporating a camera such as a smartphone, which is held by a person, and for example, video communication may be performed while walking to distribute video. In that case, for example, instead of whether the driver is on board the moving body, information indicating whether the user of the terminal device is holding the terminal device may be displayed on the external device 40.

Explanation of Reference Numerals

[0174] 10 In-vehicle device 13 External camera 15 In-vehicle camera 17 Touch panel 19 Speaker 21 Microphone 23 Storage unit 25 Control unit 27 Communication unit 30 Server 31 Storage unit 33 Communication unit 35 Control unit 36, 63 Connection information providing unit 37, 64 Occupancy status information acquisition unit 38, 65 Display control unit 40 External device 41 Touch panel 43 Speaker 45 Microphone 47 Storage unit 49 Control unit 51 Communication unit

Claims

【Claim 1】 A communication system that performs audio-visual communication for transmitting audio and video in the moving body acquired by a first device that moves with the moving body to a second device, a connection information providing unit that provides connection information used for connection of the audio-visual communication and has an expiration date; a display control unit that, when accepting access to the connection information from the second device, causes the second device to display boarding status information indicating whether or not an operator is on board the moving body during a period corresponding to the expiration date; A communication system characterized by comprising the above.

Images