Communication systems, communication devices, server devices, communication methods, communication programs, and storage media

Abstract

One of the objectives is to provide a communication system, communication device, server device, communication method, communication program, and storage medium that enable efficient and smooth communication between a first device inside a mobile body and a second device outside the mobile body, while respecting the privacy of the mobile body operator and suppressing unnecessary connection attempts from the second device to the first device. [Solution] A communication system for audio-video communication that transmits audio and video from a mobile body acquired by a first device moving with the mobile body to a second device, comprising: a connection information providing unit that provides connection information used for connecting the audio-video communication and which has an expiration date; and a display control unit that, upon receiving access to the connection information from the second device, causes the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date.

Description

[[Technical Field]]

[0001] The present invention relates to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium, and for example, relates to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium that can provide users with video and audio from a moving body. [[Background Art]]

[0002] There is a communication system that performs communication between an in-vehicle device mounted on a vehicle and an off-vehicle terminal located outside the vehicle.

[0003] For example, Patent Document 1 discloses a system in which a voice call is performed between a driver of a vehicle and an operator of an off-vehicle terminal, and when the voice call is in progress, video data representing video in front of the vehicle is transmitted from the in-vehicle device to the off-vehicle terminal, and the video is displayed on the off-vehicle terminal. In Patent Document 1, it is disclosed that the in-vehicle device starts processing for performing a call and transmitting data such as images based on receiving a call start notification, transmits an audio signal of the utterance content of the vehicle driver to the off-vehicle terminal, and also transmits vehicle information including front image data to the off-vehicle terminal. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] Japanese Patent Application Laid-Open No. 2016-213791 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0005] In a system such as that disclosed in Patent Document 1, for example, when a call start notification is transmitted from the off-vehicle terminal to the in-vehicle device, there may be cases where no driver is in the vehicle equipped with the in-vehicle device. In such cases, the connection attempt resulting from transmitting the call start notification is wasted, and one of the problems is that unnecessary work is required.

[0006] Therefore, one possible approach is to inform the user of the external terminal of the driver's occupancy status before attempting to connect. However, one challenge is that allowing unlimited access to the occupancy status could raise concerns from the driver's privacy perspective.

[0007] The present invention has been made in view of the above-mentioned points, and one of its objectives is to provide a communication system, communication device, server device, communication method, communication program, and storage medium that enable efficient and smooth communication between a first device inside a mobile body and a second device outside the mobile body, while giving consideration to the privacy of the operator of the mobile body and suppressing unnecessary connection attempts from the second device to the first device. [Means for solving the problem]

[0008] The invention described in claim 1 is a communication system for performing audio-video communication, which transmits audio and video from a mobile body acquired by a first device moving together with the mobile body to a second device, and is characterized by comprising: a connection information providing unit that provides connection information used for connecting the audio-video communication and which has an expiration date; and a display control unit that, upon receiving access to the connection information from the second device, causes the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date.

[0009] The invention described in claim 7 is a communication method performed by a communication system that performs audio-video communication, transmitting audio and video from a mobile body acquired by a first device moving together with the mobile body to a second device, the method comprising: a connection information provision step of providing connection information used for connecting the audio-video communication and having an expiration date; and a display control step of, upon receiving access to the connection information from the second device, causing the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date.

[0010] The invention described in claim 8 is a communication program executed by a communication system that performs audio-video communication, which includes a computer and transmits audio and video from a mobile body acquired by a first device that moves together with the mobile body to a second device, the communication program causing the computer to perform a connection information provision step of providing connection information used for connecting the audio-video communication and having an expiration date, and a display control step of causing the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date when it receives access to the connection information from the second device.

[0011] The invention described in claim 9 is a computer-readable storage medium that stores a communication program for causing a communication system, which includes a computer and performs audio-video communication, transmitting audio and video acquired on a mobile body by a first device that moves together with the mobile body to a second device, to execute a connection information provision step that provides connection information used for connecting the audio-video communication and which has an expiration date, and a display control step that, upon receiving access to the connection information from the second device, causes the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date.

[0012] The invention described in claim 10 is a communication device that moves together with a mobile body and performs audio-video communication, acquiring audio and video from the mobile body and transmitting them to another device, and is characterized by comprising: a connection information providing unit that provides connection information used for connecting the audio-video communication and which has an expiration date; and a display control unit that, upon receiving access to the connection information from the other device, causes the other device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date.

[0013] The invention described in claim 11 is a server device that performs audio-video communication for transmitting audio and video from a mobile body to a second device, which is acquired by a first device that moves together with the mobile body, the server device comprising: a connection information providing unit that provides connection information used for connecting the audio-video communication and which has an expiration date; and a display control unit that, upon receiving access to the connection information from the second device, causes the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date. [Brief explanation of the drawing]

[0014] [Figure 1] This is a schematic diagram showing an overview of a communication system according to Embodiment 1 of the present invention. [Figure 2] This diagram shows the configuration of the front seat area of ​​the vehicle according to Example 1. [Figure 3] This is a block diagram showing an example of the configuration of an in-vehicle device according to Example 1. [Figure 4] This is a block diagram showing an example of the configuration of a server device according to Example 1. [Figure 5] This is a front view of the external device according to Example 1. [Figure 6] This is a block diagram showing an example of the configuration of the external device according to Example 1. [Figure 7] This is a sequence diagram showing an example of communication performed in the communication system according to Example 1. [Figure 8] This is a sequence diagram showing an example of communication performed in the communication system according to Example 1. [Figure 9] This flowchart shows an example of a routine executed by the server device according to Example 1. [Figure 10] This figure shows an example of a display screen shown on an external device according to Example 1. [Figure 11] This figure shows an example of a display screen shown on an external device according to Example 1. [Figure 12] This figure shows an example of a display screen shown on an external device according to Example 1. [Figure 13] It is a block diagram showing an example configuration of an in-vehicle device according to a modification of Embodiment 1. MODE FOR CARRYING OUT THE INVENTION

[0015] Embodiments of the present invention will be described in detail below. In the following description and the accompanying drawings, substantially identical or equivalent parts are denoted by the same reference numerals. [Embodiment]

[0016] [1. System Configuration] The configuration of a communication system 100 including the in-vehicle device 10 according to Embodiment 1 will be described with reference to the accompanying drawings.

[0017] Fig. 1 shows a communication system 100 which is Embodiment 1 of the present invention. As shown in Fig. 1, the communication system 100 is configured to include an in-vehicle device 10, a server 30, and an external device 40. Fig. 1 shows a case where the in-vehicle device 10 is mounted in an automobile M as an example of a mobile body. In addition, Fig. 1 shows a smartphone as an example of the external device 40.

[0018] The in-vehicle device 10, the server 30, and the external device 40 can mutually transmit and receive data via the network NW using, for example, a communication protocol such as TCP / IP or UDP / IP. The network NW can be constructed by, for example, Internet communication including wireless communication such as mobile communication networks, Wi-Fi (registered trademark), and wired communication.

[0019] In the communication system 100, audio-video communication is possible in which audio and video in an automobile M acquired by the in-vehicle device 10 serving as a first device moving together with the automobile M are transmitted to the external device 40 serving as a second device.

[0020] In the communication system 100 of this embodiment, after a voice call is established between the in-vehicle device 10 and the external device 40, the video captured in the vehicle M is distributed from the in-vehicle device 10 to the external device 40.

[0021] In the following description, the communication method described above, which establishes voice communication between the in-vehicle device 10 and the external device 40 while simultaneously distributing video footage captured in the vehicle M from the in-vehicle device 10 to the external device 40, is referred to as video communication as a form of voice-video communication. In this embodiment, the in-vehicle device 10 and the external device 40 communicate via the server 30.

[0022] Through this type of video communication, the user of the external device 40, who is viewing the video transmitted from the in-vehicle device 10, can get the feeling that they are riding in the car M with the driver of the vehicle M, who is the operator of the moving object. In other words, video communication enables the user of the external device 40 to virtually ride in the car M. A system like the communication system 100 of this embodiment, which is capable of realizing such video communication, is also referred to as a virtual ride-along system.

[0023] In the following example, Embodiment 1 will be described using the case where the in-vehicle device 10 is a car navigation system. Furthermore, in Embodiment 1, the example will be described using the case where the in-vehicle device 10 is a terminal device for a so-called cloud-type car navigation system, which receives a destination that the user wishes to be guided to from the user, transmits the destination to the server 30, and the server 30 generates a route to the destination.

[0024] Furthermore, in this embodiment, as an example of how video communication is initiated, we will describe a case in which an invitation for video communication is first made from the in-vehicle device 10 to the external device 40, and connection information for video communication is provided at the time of the invitation. In this embodiment, when the external device 40 attempts to connect using the said connection information, the process of initiating video communication proceeds.

[0025] Figure 2 is a perspective view showing the area around the front seats of a car M equipped with an in-vehicle device 10, which functions as a video transmission or video generation device. Figure 1 shows an example of installation where the in-vehicle device 10 is mounted inside the dashboard DB of the front seats of car M.

[0026] The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is located, for example, on the dashboard DB. However, the GPS receiver 11 can be located anywhere as long as it can receive GPS signals. The GPS receiver 11 is capable of transmitting the received GPS signals to the in-vehicle device 10.

[0027] The external camera 13 is an imaging device that captures images of the area in front of the vehicle M. In this embodiment, the external camera 13 is positioned on the dashboard DB so that the forward direction is the shooting direction. For example, the external camera 13 is a wide-angle camera and can capture a wide area in front of the vehicle M through the windshield. The external camera 13 may also be provided on the rearview mirror RM or mounted on the inside of the windshield FG.

[0028] The in-vehicle camera 15 is an imaging device that captures images of the interior of the automobile M. In this embodiment, the in-vehicle camera 15 is installed at the upper edge of the windshield FG or on the ceiling near the upper edge, and is capable of capturing images of the driver of the automobile M.

[0029] During video communication, video footage captured by the external camera 13 or the internal camera 15 is distributed to the external device 40.

[0030] The touch panel 17 is a touch panel monitor that combines a display, such as an LCD capable of displaying images, with a touchpad. The touch panel 17 is located, for example, on the center console CC of the dashboard DB. The touch panel 17 only needs to be located in a place that is visible to the driver and within the driver's reach. For example, the touch panel 17 may be mounted on the dashboard DB.

[0031] The touch panel 17 can display screen information based on the control of the in-vehicle device 10. The touch panel 17 can also transmit signals to the in-vehicle device 10 representing input operations received from the user. For example, the touch panel 17 may display navigation instructions. Furthermore, it may be possible to perform operations related to the navigation function, such as setting a destination, via the touch panel 17.

[0032] Furthermore, the touch panel 17 may display information related to video communication, and may also display a screen for accepting operations to establish a connection for video communication (hereinafter also referred to as a video connection). The occupant of the vehicle M may perform the video communication connection operation by inputting information on the touch panel 17.

[0033] The speaker 19 is installed, for example, on the interior side of the A-pillar AP. The speaker 19 is capable of emitting sounds such as music and voices based on the control of the in-vehicle device 10. During video communication, the speaker 19 emits voice from the external device 40 for voice calls.

[0034] Microphone 21 is a microphone device that receives sounds from inside the vehicle and is located, for example, on the dashboard DB. Microphone 21 may be installed anywhere, such as the rearview mirror RM or the steering wheel, as long as it can receive sounds from inside the vehicle. During video communication, the audio picked up by microphone 21 is transmitted to the external device 40 as the audio for the voice call. Operation inputs related to starting video communication may also be made by voice via microphone 21.

[0035] Figure 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a storage unit 24, a control unit 25, and a communication unit 27 cooperate via a system bus.

[0036] The seating sensor 22 is capable of actually detecting whether or not a driver is seated in (hereinafter also referred to as "occupied") the vehicle M, and is a sensor capable of outputting a signal indicating the presence or absence of the detected driver. The seating sensor 22 is, for example, installed on the seat surface or back of the driver's seat of the vehicle M, and is a sensor that detects load. The driver may be detected by image recognition of the video from the in-vehicle camera 15, or by directly detecting the driver himself using other photoelectric sensors such as infrared sensors.

[0037] Furthermore, the vehicle M is equipped with an acceleration sensor 23. The acceleration sensor 23 is capable of measuring the acceleration of the vehicle M and outputting a signal indicating the measured acceleration. For example, it may be a MEMS acceleration sensor. The acceleration sensor 23 may also be an inertial measurement unit (IMU). In addition, the acceleration sensor 23 may be built into the in-vehicle device 10.

[0038] The storage unit 24 is a storage device composed of, for example, a hard disk drive, an SSD (solid state drive), or flash memory. The storage unit 24 stores various programs executed in the in-vehicle device 10, such as the operating system and terminal software.

[0039] Various programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the storage unit 24 can be transmitted via a network, and they can also be recorded on a computer-readable recording medium and transferred.

[0040] Furthermore, the memory unit 24 also stores map information, including road maps. In addition, the memory unit 24 stores information indicating the contact details of the external device 40 as the video communication partner, such as a telephone number or email address.

[0041] The control unit 25 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The control unit 25 realizes various functions by having the CPU read and execute various programs stored in the ROM and memory unit 24. In this embodiment, the control unit 25 provides functions such as presenting information to the driver regarding the start and end of video communication, video distribution during video communication, and car navigation.

[0042] The control unit 25 is connected to communicate with the external camera 13, internal camera 15, touch panel 17, speaker 19, microphone 21, seat sensor 22, and acceleration sensor 23, which are located within the in-vehicle device 10.

[0043] The control unit 25 sequentially acquires video images captured by the exterior camera 13 and the interior camera 15. The control unit 25 also sequentially acquires information indicating whether or not a driver is in the vehicle, as detected by the seat sensor 22. Furthermore, the control unit 25 sequentially acquires signals from the GPS receiver 11 and signals indicating acceleration measured by the acceleration sensor 23.

[0044] The communication unit 27 is a communication device that sends and receives data with external devices according to instructions from the control unit 25. The communication unit 27 is, for example, a NIC (Network Interface Card) for connecting to a network NW.

[0045] The communication unit 27 is connected to the network NW described above and sends and receives various data to and from the server 30. The communication unit 27 also sends and receives various data to and from the external device 40 via the server 30.

[0046] The communication unit 27 transmits information such as, for example, external or internal video of the automobile M, current location information, and travel route to the server 30. The communication unit 27 also receives information from the server 30 for display on the touch panel 17, for example, regarding video communication.

[0047] During video communication, the control unit 25 sequentially transmits the video captured by the external camera 13 or the internal camera 15 to the server 30 via the communication unit 27.

[0048] The control unit 25 receives display information regarding the start and end of video communication from the server 30 via the communication unit 27 and displays it on the touch panel 17. The control unit 25 may also output information regarding the start and end of video communication as audio from the speaker 19.

[0049] For example, the control unit 25 of the in-vehicle device 10 can transmit location identification information, which is location data that can identify the current location of the in-vehicle device 10, to the server 30 via the communication unit 27. In this embodiment, this information pertains to the current location of the automobile M. Also, for example, the control unit 25 can transmit information including a destination entered by the user to the server 30 via the communication unit 27, and receive route information or navigation information to that destination from the server 30.

[0050] The control unit 25 can transmit audio data of voice picked up by the microphone 21 to an external device 40 via the communication unit 27 for voice communication in video communication. The control unit 25 can also receive audio data of voice input to the external device 40 via the communication unit 27 for voice communication in video communication.

[0051] Figure 4 is a block diagram showing the configuration of server 30. For example, server 30 is a device in which a storage unit 31, a communication unit 33, and a control unit 35 cooperate via a system bus.

[0052] Server 30 has functions similar to a SIP server, which establishes a voice call between the in-vehicle device 10 and the external device 40 during video communication and transfers the data of the voice call.

[0053] Furthermore, the server 30 has the function of receiving location information of the vehicle M and destination information set by the user, who is an occupant of the vehicle M, from the in-vehicle device 10, and generating a route to the destination based on the location information and destination information.

[0054] The storage unit 31 is composed of, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as the operating system and software for the server 30.

[0055] Various programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the storage unit 31 can be transmitted via a network, and they can also be recorded on a computer-readable recording medium and transferred.

[0056] The memory unit 31 stores, for example, a communication program that the server 30 uses to control the connection of video communication in the communication system 100.

[0057] Furthermore, the memory unit 31 stores a map information database (not shown) in which map information, including road maps, is stored. The map information in the map information database is a database that contains information equivalent to, for example, the map information used in a navigation device.

[0058] The communication unit 33 is connected to the network NW described above and transmits and receives various data between the in-vehicle device 10 and the external device 40.

[0059] The control unit 35 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The CPU then reads and executes various programs stored in the ROM and memory unit 31 to realize various functions.

[0060] The control unit 35 executes the above-described communication program to realize control over the video communication connection in the communication system 100.

[0061] The control unit 35 includes a connection information provision unit 36, a boarding information acquisition unit 37, and a display control unit 38.

[0062] The connection information provision unit 36 ​​is a functional unit that, in response to a request from the in-vehicle device 10, performs a connection information provision step by providing the external device 40 with a URL (Uniform Resource Locator) as connection information that is used for connection when the external device 40 performs video communication with the in-vehicle device 10 and has an expiration date.

[0063] For example, when the connection information providing unit 36 ​​receives a request from the in-vehicle device 10 to send a URL to invite the external device 40 to video communication, it generates a URL with an expiration date and provides the URL to the external device 40 by sending the URL with an expiration date to the external device 40.

[0064] The connection information provision unit 36, for example, generates a URL used for video communication connections, stores the URL and its expiration date in a database, and manages access to the page via the URL when the expiration date arrives. This database is stored, for example, in the storage unit 31.

[0065] The connection information provision unit 36 ​​may also obtain a URL with an expiration date from an external server or other device.

[0066] The expiration date of the above URL is set to a predetermined time after a request is made from the in-vehicle device 10. For example, the expiration date may be a few hours to 24 hours from the time of the request, or it may be several days.

[0067] Furthermore, the expiration date of the URL may be set by the driver for each user of the external device 40. This allows, for example, the driver of car M to set a longer expiration date for users with whom they frequently communicate via video, such as close friends, thus saving the trouble of sending an invitation each time they communicate via video. Conversely, a shorter expiration date can be set for users who do not communicate via video very often, preventing unnecessary access from those users to the driver.

[0068] The passenger information acquisition unit 37 is a functional unit that acquires passenger status information indicating whether or not a driver is in the vehicle M.

[0069] The passenger status information is, for example, information based on the actual detection results of whether or not a driver is in the vehicle M. For example, the passenger information acquisition unit 37 acquires video footage from the in-vehicle device 10, which includes the driver's seat area of ​​the vehicle M, captured by the in-vehicle camera 15, and actually detects whether or not a driver is in the driver's seat by performing image recognition on the video footage. Based on this detection result, the passenger information acquisition unit 37 acquires passenger status information indicating whether or not a driver is in the vehicle.

[0070] Furthermore, for example, the passenger information acquisition unit 37 actually detects whether or not a driver is present in the driver's seat by acquiring the detection result from the seat sensor 22 installed in the automobile M, and based on the detection result, acquires passenger status information indicating whether or not a driver is in the vehicle.

[0071] Furthermore, the passenger status information is based, for example, on whether or not the in-vehicle device 10 is connected to the network NW, which is a communication network used for video communication. The passenger information acquisition unit 37 acquires the passenger status information by detecting whether or not the in-vehicle device 10 is connected to the network NW.

[0072] Furthermore, the passenger status information may also be based on, for example, whether or not the accessory power of vehicle M is turned on.

[0073] For example, a monitoring device capable of detecting whether the accessory power is ON or OFF may be installed inside the vehicle M, and the passenger information acquisition unit 37 may acquire information from the monitoring device indicating whether or not the accessory power is turned on.

[0074] Furthermore, the passenger status information is information that indicates, for example, the driver's level of composure while driving. In other words, this level of composure is determined by the busyness or workload of the driving. The passenger information acquisition unit 37 acquires passenger status information that indicates the driver's level of composure while driving, for example, by using information from the in-vehicle camera 15, the exterior camera 13, the acceleration sensor 23, etc. In other words, this level of composure can be said to be the state of whether the driver has enough time to communicate via video while driving.

[0075] The display control unit 38 is a functional unit that, upon receiving access to a URL provided by the connection information provision unit 36, executes a display control step that causes the external device 40 to display occupancy status information indicating whether or not a driver is in the vehicle M for a period corresponding to the expiration date of the URL.

[0076] For example, in the display control step, the display control unit 38 determines whether the URL that was accessed is within its validity period, and if it is within its validity period, it causes the boarding status information to be displayed on the external device 40. For example, if the URL is outside its validity period, the display control unit 38 causes the external device 40 to display a message indicating that the URL is invalid.

[0077] For example, if the above URL is still valid, the display control unit 38, in the display control step, refers to the passenger status information acquired by the passenger information acquisition unit and determines whether or not a driver is in the vehicle M. If the result of this determination is that a driver is in the vehicle M, the display control unit 38 causes the external device 40 to display, for example, a display indicating that the driver is in the vehicle. If the result of this determination is that the driver is not in the vehicle M, the display control unit 38 causes the external device 40 to display, for example, a display indicating that the driver is getting out of the vehicle.

[0078] The user of the external device 40 can know whether the driver is in the vehicle, that is, whether video communication between the in-vehicle device 10 and the external device 40 is impossible, before attempting to connect. Therefore, if the driver is out of the vehicle, it is possible to avoid situations where a connection attempt fails or the unnecessary hassle of repeatedly attempting to connect.

[0079] Furthermore, the control unit 35 acquires location information indicating the current location of the vehicle M from the in-vehicle device 10 via the communication unit 33. The control unit 35 also acquires destination information entered into the in-vehicle device 10 by the occupants of the vehicle M via the communication unit 33. Based on the location information and destination information, the control unit 35 generates a route to the destination and transmits information indicating the route to the in-vehicle device 10.

[0080] Furthermore, the control unit 35 transmits the video and audio received from the in-vehicle device 10 to the external device 40 via the communication unit 33. The control unit 35 also transmits the audio received from the external device 40 to the in-vehicle device 10 via the communication unit 33.

[0081] Figure 5 is a front view of the external device 40. As described above, in this embodiment, the external device 40 is a smartphone.

[0082] The touch panel 41 is a touch panel monitor that combines a display, such as a liquid crystal display capable of displaying images, with a touch pad. The touch panel 41 is capable of generating signals that represent input operations received from the user. In this embodiment, the touch panel 41 displays images distributed from the in-vehicle device 10.

[0083] Furthermore, the touch panel 41 displays information related to video communication and shows a screen for accepting operations to initiate a video communication connection. The user of the external device 40 can perform the necessary input operations to initiate video communication by inputting information to the touch panel 41.

[0084] Speaker 43 is capable of emitting sounds such as music and voice. During video communication, speaker 43 emits voice from the in-vehicle device 10 for voice communication.

[0085] Microphone 45 is a microphone device that receives sound emitted toward the external device 40. During video communication, the audio picked up by microphone 45 is transmitted to the in-vehicle device 10 via server 30 as audio for the voice call.

[0086] Figure 6 is a block diagram showing an example of the configuration of the external device 40. For example, the external device 40 is a device in which a storage unit 47, a control unit 49, and a communication unit 51 cooperate via a system bus (not shown).

[0087] The storage unit 47 is composed of, for example, a hard disk drive, an SSD (solid state drive), flash memory, etc., and stores various programs such as the operating system and software for external devices 40.

[0088] Furthermore, various programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the storage unit 47 can be transmitted via a network, and they can also be recorded on a computer-readable recording medium and transferred.

[0089] The control unit 49 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The CPU then reads and executes various programs stored in the ROM and memory unit 47 to realize various functions.

[0090] The control unit 49 is communicatively connected to the touch panel 41, speaker 43, and microphone 45.

[0091] The control unit 49 is capable of receiving signals indicating input operations to the touch panel 41 and audio input signals from the microphone 45. For example, the control unit 49 can receive requests from the user to start video communication via the touch panel 41 and the microphone 45.

[0092] Furthermore, the control unit 49 can transmit video or image signals to the touch panel 41 for display, or transmit audio signals to the speaker 43 for sound output.

[0093] For example, the control unit 49 causes the touch panel 41 to display a start screen for accepting a request to attempt a video communication connection, a screen indicating whether or not the driver of the vehicle M is in the vehicle, or a screen indicating that the URL is invalid.

[0094] Furthermore, for example, the control unit 49 displays the video during video communication on the touch panel 41 and outputs the audio recorded in the automobile M during video communication to the speaker 43.

[0095] The communication unit 51 is connected to the network NW described above and sends and receives various data to and from the server 30. The communication unit 51 also sends and receives various data, including video transmitted from the in-vehicle device 10, to and from the in-vehicle device 10 via the server 30.

[0096] The control unit 49 of the external device 40 can receive information via the communication unit 51 to display a start screen for accepting a request to attempt a video communication connection, a screen indicating whether or not the driver of the vehicle M is in the vehicle, or a screen indicating that the URL is invalid.

[0097] Furthermore, the control unit 49 can transmit audio data of voices picked up by the microphone 45 to the in-vehicle device 10 via the communication unit 51 for voice communication in video communication. Also, the control unit 49 can receive audio data transmitted from the in-vehicle device 10 for voice communication in video communication via the communication unit 51.

[0098] For example, the control unit 49 of the external device 40 can receive location information that can identify the current location of the vehicle M transmitted from the server 30 to the in-vehicle device 10 via the communication unit 51. Also, for example, the control unit 49 can receive route information or navigation information of the vehicle M from the server 30 via the communication unit 51.

[0099] [2. Sequence when the URL is valid] Figure 7 is a sequence diagram showing an example of communication that takes place between the in-vehicle device 10 and the external device 40 via the server 30 when the URL is valid and video communication is initiated in the communication system 100. The following describes a case where the driver, who is the user of the in-vehicle device 10, plans to conduct video communication with a virtual passenger, who is the user of the external device 40, while driving the automobile M.

[0100] As shown in Figure 7, the driver registers in advance the contact information of the virtual passenger with whom they will communicate via video with the external device 40 (step S11).

[0101] In registering a virtual passenger, registration information is sent to the server 30 and held by the server 30. This registration information includes, for example, an identifier that can identify the virtual passenger, such as a nickname, and information indicating the recipient, such as an email address or telephone number, necessary to send information to the virtual passenger.

[0102] The registration of the virtual passenger is performed via the touch panel 41 of the in-vehicle device 10, but is not limited to this; it may also be performed by transmitting input information from a terminal device other than the in-vehicle device 10, such as a smartphone used by the driver of the automobile M, to the server 30.

[0103] The timing at which the driver registers the contact information of the virtual passenger in advance may be, for example, the day before or before the scheduled video communication while driving car M, or it may be immediately before the video communication.

[0104] Subsequently, when the driver starts driving, for example, car M, the driver performs an input operation on the in-vehicle device 10 to instruct it to send an invitation email to the external device 40. This input operation may be performed via the touch panel 17 of the in-vehicle device 10, or by voice via the microphone 21. Alternatively, the input operation to instruct the sending of the invitation email may be performed before starting to drive car M.

[0105] The driver's input operation sends an invitation email request to the server 30, asking it to send an invitation email to the external device 40 (step S12). The invitation email request includes, for example, information indicating the recipient, such as the email address or telephone number of the external device 40 used by the virtual passenger that was registered in advance, and a nickname that can identify the virtual passenger.

[0106] When server 30 receives an invitation email sending request, it sends an invitation email to the destination of the external device 40 included in the invitation email sending request (step S13). The invitation email contains a URL to open the start screen for the virtual passenger to perform video communication.

[0107] For example, the connection information provision unit 36 ​​of the server 30 generates a URL with an expiration date, generates an invitation email containing a message inviting the user of the external device 40 to participate in video communication and the URL with an expiration date, and sends the invitation email to the external device 40.

[0108] After the server 30 sends the invitation email, the virtual passenger opens the invitation email using the external device 40 and taps the URL (step S14A), at which point the server 30 checks the URL's expiration date.

[0109] If the URL is within its validity period and is valid (step S15A), a browser will be launched on the touch panel 41 of the external device 40, and a passenger status screen showing the driver's passenger status will be displayed (step S16A).

[0110] For example, if the driver is in the vehicle, the vehicle status screen may display a message indicating that the driver is in the vehicle, along with a start button, which is a display area for accepting requests to start video communication if a virtual passenger wishes to initiate video communication.

[0111] When the virtual passenger taps the start button, the user's operation of the start button is acknowledged (step S17), and a request to start video communication is sent to the server 30 (step S18).

[0112] When the server 30 receives a request to start video communication, it notifies the in-vehicle device 10 that there has been a request to start video communication from the external device 40 (step S19).

[0113] In step S19, for example, when the in-vehicle device 10 receives a notification from the server 30, a selection button is displayed on the touch panel 17 indicating whether or not to consent to the start of video communication. For example, if the driver taps the consent button, information indicating that the driver has consented to the start of video communication is sent to the server 30 (step S20).

[0114] When the server 30 receives information indicating that the driver has consented to the start of video communication, it establishes an audio call between the in-vehicle device 10 and the external device 40, and starts video communication to distribute the video footage captured in the vehicle M from the in-vehicle device 10 to the external device 40 (step S21).

[0115] In this way, if the URL is valid and the driver is in the vehicle, the in-vehicle device 10 and the external device 40 initiate video communication via the server 30.

[0116] [3. Sequence when the URL is invalid] Figure 8 shows an example of a sequence when the URL accessed from the external device 40 has expired. In Figure 8, steps S11 to S13 proceed in the same way as the sequence in Figure 7, and then the steps from step S14B onwards proceed after the expiration date has passed. In Figure 8, for example, the URL's expiration date is assumed to be 24 hours.

[0117] In step S14B, when the virtual passenger uses the external device 40 to open the invitation email received in step S13 and taps the URL, the server 30 checks the URL's expiration date.

[0118] If, during the expiration check, it is determined that the URL has expired and is therefore invalid (step S15B), the server 30 causes the external device 40 to display a page indicating that the URL is invalid (step S16B). In step S16B, for example, an expired URL screen indicating that the URL has expired is displayed on the touch panel 41 of the external device 40. The sequence in Figure 8 then ends.

[0119] [4. Server Operation] The following describes the control routines of the server 30 for realizing the operation of the communication system 100 of the embodiment. Hereinafter, the boarding information acquisition unit 37 will be described as, for example, acquiring boarding status information as needed after the server 30 receives a request to send an invitation email (Figure 7, step S12), and maintaining the latest boarding status information.

[0120] Figure 9 is a flowchart showing the display control routine RT1 executed by the display control unit 38 of the server 30. The display control unit 38 starts the display control routine RT1, for example, when an invitation email is sent from the server 30 to the external device 40.

[0121] When the display control unit 38 starts the display control routine RT1, it determines whether or not there has been access to the URL included in the invitation email (step S101). In step S101, for example, if a virtual passenger opens the invitation email using the external device 40 and taps the URL included in the invitation email, the external device 40 accesses the URL.

[0122] In step S101, if the display control unit 38 determines that there is no access to the URL (step S101: NO), it terminates the display control routine RT1, starts the next display control routine RT1, and executes step S101 again.

[0123] In step S101, the display control unit 38 determines that there has been access to the URL (step S101: YES), and then determines whether the URL is still valid (step S102).

[0124] In step S102, for example, the display control unit 38 refers to a database in which URLs and expiration dates are stored in association, and determines whether or not the accessed URL is within its expiration date.

[0125] In step S102, the display control unit 38 determines that the URL is within its validity period (step S102: YES), and then refers to the passenger status information that the passenger information acquisition unit 37 has acquired and holds sequentially (step S103). In step S103, for example, information indicating whether or not the driver is on board is referred to.

[0126] Furthermore, for example, in step S103, information indicating whether or not the in-vehicle device 10 is connected to the network NW may be referenced, or information indicating the ON / OFF state of the seat occupancy sensor provided on the driver's seat of the automobile M may be referenced.

[0127] After step S103 is executed, the display control unit 38 determines whether or not the driver is in the vehicle based on the passenger status information referenced in step S102 (step S104).

[0128] In step S104, the display control unit 38 determines that the driver is in the vehicle (step S104: YES), and causes the external device 40 to display a screen containing a message indicating that the driver is in the vehicle (step S105).

[0129] Figure 10 shows an example of the in-ride screen displayed on the touch panel 41 of the external device 40 in step S105. As shown in Figure 10, the in-ride screen displays a message indicating that the driver is in the vehicle.

[0130] Furthermore, the in-ride screen shown in Figure 10 may display a message indicating that the driver is in the vehicle, as well as a message prompting the user to start a video call and a start button. The start button may be displayed on a screen different from the in-ride screen; for example, a button such as "Go to Start Screen" may be displayed on the in-ride screen to proceed to the screen for starting video communication, and when this button is pressed, the start screen including the start button may be displayed.

[0131] Furthermore, a field for entering a PIN for identity verification may be displayed on the boarding screen or the start screen. For example, identity verification may be performed by entering the last four digits of a phone number into the input field.

[0132] Furthermore, the terms of service and privacy policy regarding video communication may be available on the in-ride screen or the start screen. For example, a message may be displayed stating that pressing the start button constitutes agreement to the said terms of service and privacy policy.

[0133] In step S104, if the display control unit 38 determines that the driver is not in the vehicle (step S104: NO), it causes the external device 40 to display a screen containing a message indicating that the driver is getting out of the vehicle (step S106).

[0134] Figure 11 shows an example of the "Disembarking" screen displayed on the touch panel 41 of the external device 40 in step S106. As shown in Figure 11, the "Disembarking" screen displays a message indicating that the driver is disembarking. Alternatively, as shown in Figure 11, the "Disembarking" screen may also display a message indicating that video communication cannot be started.

[0135] In the example shown in Figure 11, the display related to starting video communication, such as the start button, is shaded, and the start button is not functional. Note that the display related to starting video communication, including the start button, does not necessarily need to be shown at all on the screen while the passenger is disembarking.

[0136] If the display control unit 38 determines in step S102 that the URL is not within its validity period (step S102: NO), it causes the external device 40 to display a screen containing a message indicating that the URL has expired (step S107).

[0137] Figure 12 shows an example of an expiration screen displayed on the touch panel 41 of the external device 40 in step S107. As shown in Figure 12, in step S107, the screen related to video communication is not accessible, so the expiration screen displays the message "The URL has expired" as a message indicating that the URL is invalid.

[0138] For example, the expiration screen may also display a message indicating that video communication cannot be started, or a message indicating that a new invitation email must be received in order to start video communication.

[0139] In steps S104 to S107, the display control unit 38 performs a display control step that causes the external device 40 to display onboard status information, indicating whether or not a pilot is on board the mobile body, for a period corresponding to the expiration date of the URL as connection information.

[0140] After step S105, step S106, or step S107 is executed, the display control unit 38 terminates the display control routine RT1 and starts the next display control routine RT1.

[0141] According to control routine RT1, the virtual passenger can know whether the driver is in the vehicle M before attempting to connect to video communication, by displaying either the in-car screen shown in Figure 10 or the alighting screen shown in Figure 11. Therefore, the virtual passenger can avoid unnecessary connection attempts.

[0142] Furthermore, according to control routine RT1, an expiration date can be set for the URL, and the boarding status information can be displayed on the external device 40 only within the expiration date, and the boarding status information cannot be displayed on the external device 40 outside of the expiration date. Therefore, the boarding status information cannot be known to virtual passengers indefinitely, and the driver's privacy can be protected. For example, it is possible to prevent the boarding status information from being repeatedly known even after the scheduled video communication has ended.

[0143] Furthermore, outside of the expiration period, a message indicating that the URL is invalid, as shown in Figure 12, will be displayed, preventing the virtual passenger from accessing unnecessary connection information.

[0144] In addition, in step S103 of the control routine RT1 described above, the driver's occupancy status, or alternatively, the driver's available time while driving, may be referenced. If the driver is in a situation where there is no available time even though they are in the vehicle, a message indicating that video communication cannot be started may be displayed.

[0145] As described above, the communication system 100 of this embodiment is capable of establishing voice communication between the in-vehicle device 10, which is the first device, and the external device 40, which is the second device, via the server 30, while simultaneously performing video communication to transmit video from the automobile M on which the in-vehicle device 10 is installed to the external device 40.

[0146] The communication system 100 of this embodiment includes a connection information providing unit that provides connection information used for connecting audio-video communications, including video communications, and which has an expiration date, and a display control unit that, upon receiving access to the connection information from an external device 40, causes the external device 40 to display onboard status information indicating whether or not a pilot is on board the mobile vehicle for a period corresponding to the expiration date of the connection information.

[0147] According to this embodiment, the user of the external device 40 can know whether or not a pilot is on board before attempting to connect to audio and video communication. It is clear that audio and video communication is impossible if no pilot is on board, and the user of the external device 40 can avoid making futile connection attempts when no pilot is on board.

[0148] According to this embodiment, for example, the display control unit can display the boarding status information only within the validity period of the connection information. This prevents the pilot's boarding status from being known to the user of the external device 40 indefinitely, thereby protecting the pilot's privacy.

[0149] Accordingly, according to this embodiment, it is possible to provide a communication system, communication device, server device, communication method, communication program, and storage medium that enable efficient and smooth communication between a first device inside a mobile body and a second device outside the mobile body, while giving consideration to the privacy of the operator of the mobile body and suppressing unnecessary connection attempts from the second device to the first device.

[0150] [Differentiation] Figure 13 is a block diagram showing an example of the configuration of an in-vehicle device 60 in a modified version of the communication system 100 of Embodiments 1 and 2. In this modified version, the in-vehicle device 60 has the functions of the connection information provision unit 36, the boarding information acquisition unit 37, and the display control unit 38, which are located in the server 30 in the above embodiments.

[0151] Specifically, the control unit 61 of the in-vehicle device 60 includes a connection information provision unit 63, a passenger information acquisition unit 64, and a display control unit 65.

[0152] The memory unit 67 stores a communication program that is executed by the control unit 61, provides a URL used for video communication connection and which has an expiration date, and when it receives access to the URL from an external device 40, it controls the display of occupancy status information indicating whether or not a driver is in the car M for a period corresponding to the expiration date of the URL.

[0153] For example, when the connection information provision unit 63 of the in-vehicle device 60 receives an operation input from the driver instructing it to send an invitation email to the external device 40, it generates a URL with an expiration date and sends an invitation email containing the URL to the external device 40 via the communication unit 27.

[0154] The passenger information acquisition unit 64 sequentially acquires passenger status information indicating whether or not the driver is in the vehicle M, for example, based on detection results from the in-vehicle camera 15 or the seating sensor 22.

[0155] When the display control unit 65 receives access to the URL from the external device 40, it controls the display to show onboard status information, indicating whether or not a driver is in the vehicle M, on the external device 40 for a period corresponding to the expiration date of the URL.

[0156] According to this modified example, video communication between the external device 40 and the in-vehicle device 60 is possible without going through the server 30, and the display of passenger status information is possible. For example, the control and video communication related to the display of passenger status information in this modified example may be performed directly between the in-vehicle device 10 and the external device 40 by P2P (Peer to Peer) communication or the like, without going through the server 30.

[0157] The configurations, routines, etc., of the in-vehicle device 10, in-vehicle device 60, server 30, and external device 40 in the above-described embodiments and modifications are merely illustrative examples and can be appropriately selected or modified depending on the application.

[0158] For example, in the above embodiment and modified examples, the passenger status information may include not only information indicating whether or not the driver is in the vehicle, but also information indicating whether or not the driver is in a situation where video communication cannot be initiated even if they are in the vehicle.

[0159] For example, if the driver is communicating via video with a device other than the external device 40, if there is a passenger in the vehicle M, if the driver is under heavy driving load, or if the driver is in conversation with a passenger or the person they are communicating with, it is presumed that the driver is in a situation where they cannot respond even if video communication is initiated, or do not want to transmit video from inside or outside the vehicle or audio from inside the vehicle M to the external device 40.

[0160] In such cases, the driver's status may be treated as being unable to initiate video communication, even if they are in the vehicle. In this case, for example, the in-vehicle screen, which indicates that the driver is in the vehicle, may display a message indicating that video communication cannot be initiated.

[0161] For example, the automobile M may be equipped with seat occupancy sensors in seats other than the driver's seat, and the presence or absence of passengers used to determine the driver's occupancy status may be detected by these seat occupancy sensors.

[0162] For example, the magnitude of the driving load used to determine the driver's occupancy status may be calculated by workload estimation using images from an external camera 13 or an internal camera 15.

[0163] Furthermore, while the above embodiment describes displaying a screen on the external device 40 indicating that the URL is invalid when the URL has expired, the invention is not limited to this, and access may also be made impossible by deleting the page identified by the URL.

[0164] Furthermore, in the above embodiments and modifications, an example was described in which a start request from an external device 40 is received by tapping a start button displayed on the touch panel 41, but this is not limited to this. The start request may also be received, for example, by voice input.

[0165] The above embodiments and modifications describe examples in which video communication is performed in the communication system 100, but the invention is not limited thereto. The present invention is applicable when the communication system 100 performs audio-video communication in which audio and video from the automobile M, acquired by an in-vehicle device 10 which is a first device that moves with the automobile M, are transmitted to an external device 40 which is a second device. In other words, the present invention is also applicable to communication in which no audio is transmitted from the external device 40.

[0166] Furthermore, the in-vehicle device 10 and the external device 40 do not necessarily have to be able to communicate via voice. For example, the communication between the in-vehicle device 10 and the external device 40 may be such that video and audio are simply distributed from the in-vehicle device 10 to the external device 40 without involving two-way voice communication, like YouTube® or Nico Nico Live Broadcasting®.

[0167] Furthermore, in the above embodiment, the in-vehicle device 10 and the external device 40 perform video communication via the server 30, but video communication may also be performed directly between the in-vehicle device 10 and the external device 40.

[0168] Furthermore, while the video is being distributed from the in-vehicle device 10 to the external device 40 via the communication system 100, a voice call may also be established in parallel between the in-vehicle device 10 and the external device 40 via a separate route from the communication system 100.

[0169] In the above embodiments and modifications, the in-vehicle device 10 is described as an in-vehicle navigation device, but the in-vehicle device 10 only needs to be capable of distributing video from the external camera 13 or the internal camera 15.

[0170] For example, the in-vehicle device 10 may be a configuration in which a terminal device having a similar configuration to the in-vehicle device 10, an external camera 13, and a touch panel 17 are integrated. Specifically, for example, the in-vehicle device 10 may be a terminal device such as a smartphone, tablet, or PC with a camera that has an application that performs the same functions as the in-vehicle device 10 installed. In this case, the in-vehicle device 10 may be mounted on the dashboard DB, for example, in a cradle, so that the built-in camera can capture images of the front of the car M through the windshield of the car M.

[0171] Furthermore, the in-vehicle device 10 may be configured not to display a screen for the driver of the vehicle M. For example, the in-vehicle device 10 may have a configuration similar to a drive recorder, or it may be a device integrated with an external camera 13. Specifically, the in-vehicle device 10 may be a device in which the hardware performing the video communication function of the in-vehicle device 10 described above is built into the housing of the external camera 13. In this case, the in-vehicle device 10 may not perform the various display outputs described above.

[0172] In the above embodiments and modifications, the case where the external device 40 is a smartphone has been described, but it is not limited to this. The external device 40 is a terminal device that a virtual passenger can use for video communication, and it is sufficient that it is configured to display or present messages related to video communication, receive operation inputs necessary for performing video communication, send and receive audio data, and receive and display video. For example, the external device 40 may be a terminal device such as a tablet, PC, or wearable device.

[0173] In the above embodiments and modifications, examples were described in which the in-vehicle device 10 and the in-vehicle device 60 are mounted on an automobile M, but the in-vehicle device 10 may be mounted on other mobile devices such as bicycles, motorcycles, or ships. Alternatively, the in-vehicle device 10 may be a terminal device with a built-in camera, such as a smartphone, which is held by a person and used to transmit video while walking, for example. In that case, instead of indicating whether the driver is on board the mobile device, information indicating whether the user of the terminal device is holding the terminal device may be displayed on the external device 40. [Explanation of Symbols]

[0174] 10 Onboard equipment 13. Exterior car camera 15 In-car camera 17 Touch panel 19 speakers 21 Mike 23 Memory section 25 Control Unit 27 Communications Department 30 servers 31 Storage section 33 Communications Department 35 Control Unit 36, 63 Connection Information Provision Department 37, 64 Boarding Status Information Acquisition Unit 38, 65 Display Control Unit 40 External device 41 Touch panel 43 speakers 45 Mike 47 Memory section 49 Control Unit 51 Communications Department

Claims

[Claim 1] A communication system that performs audio-video communication, which transmits audio and video acquired from a mobile object by a first device that moves together with the mobile object to a second device, A connection information providing unit that provides connection information used for connecting the aforementioned audio-video communication and having an expiration date, Upon receiving access to the connection information from the second device, a display control unit causes the second device to display onboard status information indicating whether or not a pilot is on board the mobile body for a period corresponding to the expiration date, A communication system characterized by having the following features.

Images