Communication system, communication device, server device, communication method, communication program, and storage medium

Abstract

To provide a communication system, communication device, server device, communication method, communication program, and storage medium that in communication between a first device inside a mobile and a second device outside the mobile, make it possible to give, to a user of the second device outside the mobile, information obtained when actually existing inside the mobile.SOLUTION: A communication system for performing voice video communication for transmitting voice and video of a mobile acquired by a first device moving with the mobile to a second device comprises: a video acquisition unit for sequentially acquiring video imaged by the mobile; a manipulation operation information acquisition unit for sequentially acquiring manipulation operation information showing manipulation operation to the mobile by a manipulator of the mobile; and a display control unit for performing display control of making the video and display based on the manipulation operation information be displayed on the second device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium, and more particularly to a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium that can provide a user with video and audio from a mobile object, for example. [Background technology]

[0002] 2. Description of the Related Art There is a communication system that performs communication between an on-board device mounted in a vehicle and an external terminal located outside the vehicle.

[0003] For example, Patent Document 1 discloses a system in which a voice call is made between a vehicle driver and an operator of an external terminal, and while the voice call is being made, video data showing an image in front of the vehicle is transmitted from an in-vehicle device to the external terminal, and the video is displayed on the external terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-213791 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when using a system such as that disclosed in Patent Document 1 to show voice calls and video data showing the image in front of the vehicle to the operator of an external terminal, one of the issues is that the operator of the external terminal cannot obtain sufficient information that a person actually riding in the vehicle would obtain through their five senses, such as sight, hearing, and touch.

[0006] The present invention has been made in consideration of the above-mentioned points, and one of its objectives is to provide a communication system, communication device, server device, communication method, communication program, and storage medium that enable communication between a first device inside a mobile body and a second device outside the mobile body to provide a user of a second device outside the mobile body with information that would be available if the user were actually inside the mobile body. [Means for solving the problem]

[0007] The invention described in claim 1 is a communication system for audio-visual communication in which audio and video of a moving body acquired by a first device moving together with the moving body is transmitted to a second device, and is characterized by having an image acquisition unit that sequentially acquires images captured by the moving body, a steering operation information acquisition unit that sequentially acquires steering operation information indicating steering operations performed on the moving body by a driver of the moving body, and a display control unit that performs display control to display both the image and a display based on the steering operation information on the second device.

[0008] The invention described in claim 9 is a communication method executed by a communication system that performs audio-video communication in which audio and video of a moving body acquired by a first device moving together with the moving body is transmitted to a second device, and is characterized in that it includes an image acquisition step of sequentially acquiring images captured by the moving body, a steering operation information acquisition step of sequentially acquiring steering operation information indicating steering operations performed on the moving body by a driver of the moving body, and a display control step of performing display control to display both the image and a display based on the steering operation information on the second device.

[0009] The invention described in claim 10 is a communication program executed by a communication system that includes a computer and performs audio-video communication to transmit audio and video of a moving body acquired by a first device moving together with the moving body to a second device, the communication program causing the computer to execute an image acquisition step of sequentially acquiring images captured by the moving body, a steering operation information acquisition step of sequentially acquiring steering operation information indicating steering operations performed on the moving body by a pilot of the moving body, and a display control step of performing display control to display both the image and a display based on the steering operation information on the second device.

[0010] The invention described in claim 11 is a computer-readable storage medium that stores a communication program for causing a communication system that includes a computer and performs audio-video communication, in which audio and video of a moving body acquired by a first device moving together with the moving body is transmitted to a second device, to execute the following steps: an image acquisition step that sequentially acquires images captured on the moving body; a steering operation information acquisition step that sequentially acquires steering operation information indicating steering operations performed on the moving body by a pilot of the moving body; and a display control step that performs display control to display both the images and a display based on the steering operation information on the second device.

[0011] The invention described in claim 12 is a communication device that moves together with a moving body and performs audio-video communication by acquiring audio and video from the moving body and transmitting the audio and video to another device, characterized in that it has an image acquisition unit that sequentially acquires video captured by the moving body, a steering operation information acquisition unit that sequentially acquires steering operation information indicating steering operations performed on the moving body by a driver of the moving body, and a display control unit that performs display control to display both the video and a display based on the steering operation information on the other device.

[0012] The invention described in claim 13 is a server device that performs audio-video communication to transmit audio and video of a moving body acquired by a first device moving together with the moving body to a second device, and is characterized in having: a video acquisition unit that sequentially acquires video captured at the moving body from the first device via a communication path; a steering operation information acquisition unit that sequentially acquires steering operation information indicating steering operations performed on the moving body by a driver of the moving body from the first device via the communication path; and a display control unit that performs display control to display both the video and a display based on the steering operation information on the second device. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram illustrating an overview of a communication system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing a configuration of a front seat portion of a vehicle according to an embodiment; [Figure 3] 1 is a block diagram showing an example of a configuration of an in-vehicle device according to an embodiment; [Figure 4] FIG. 2 is a block diagram illustrating an example of a configuration of a server device according to an embodiment. [Figure 5] FIG. 2 is a front view of an external device according to an embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an example of a configuration of an external device according to an embodiment. [Figure 7] 4 is a flowchart illustrating an example of a routine executed by the in-vehicle device according to the embodiment. [Figure 8] 10 is a flowchart illustrating an example of a routine executed by a server device according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a display screen displayed on an external device according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating another example of a display screen displayed on an external device according to an embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of the configuration of an in-vehicle device according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description and accompanying drawings, substantially the same or equivalent parts are designated by the same reference numerals. [Example]

[0015] [1. System Configuration] The configuration of a communication system 100 including an in-vehicle device 10 according to an embodiment will be described with reference to the accompanying drawings.

[0016] Fig. 1 shows a communication system 100 according to an embodiment of the present invention. As shown in Fig. 1, the communication system 100 includes an in-vehicle device 10, a server 40, and an external device 50. Fig. 1 shows a case in which the in-vehicle device 10 is mounted on an automobile M, which is an example of a moving body. Fig. 1 also shows a smartphone as an example of the external device 50.

[0017] The in-vehicle device 10, the server 40, and the external device 50 can transmit and receive data to and from each other via a network NW using a communication protocol such as TCP / IP or UDP / IP. The network NW can be constructed using, for example, a mobile communication network, wireless communication such as Wi-Fi (registered trademark), and internet communication including wired communication.

[0018] In the communication system 100, audio and video communication is possible in which audio and video captured by an in-vehicle device 10 as a first device traveling together with the automobile M is transmitted to an external device 50 as a second device.

[0019] In the communication system 100 of this embodiment, after a voice call is established between the in-vehicle device 10 and the external device 50, the in-vehicle device 10 transmits a video image captured in the automobile M to the external device 50.

[0020] In the following description, the communication mode in which a voice call is established between the in-vehicle device 10 and the external device 50 and an image captured in the automobile M is transmitted from the in-vehicle device 10 to the external device 50 as described above will be referred to as video communication as one mode of audio / video communication. In this embodiment, the in-vehicle device 10 and the external device 50 perform video communication via the server 40.

[0021] By performing such video communication, the user of the external device 50 watching the video transmitted from the in-vehicle device 10 can feel as if he or she is riding in the vehicle M with the driver of the vehicle M who is operating the moving object. In other words, the video communication can realize a virtual ride-along of the user of the external device 50 in the vehicle M. A system such as the communication system 100 of this embodiment that can realize such video communication is also referred to as a virtual ride-along system. The user of the external device 50 is also referred to as a virtual ride-along passenger.

[0022] In the following embodiments, the in-vehicle device 10 is described as a car navigation device. Also, the in-vehicle device 10 is described as a terminal device of a so-called cloud-based car navigation device that receives a destination to which the user wants guidance from the user, transmits the destination to the server 40, and the server 40 generates a route to the destination.

[0023] In this embodiment, a case will be described in which a virtual passenger provides route guidance to the driver of the automobile M during video communication via the communication system 100. The virtual passenger provides route guidance using the video and audio output by the external device 50 via video communication as clues.

[0024] Fig. 2 is a perspective view showing the vicinity of the front seat of an automobile M equipped with an in-vehicle device 10 as a video transmission device or a video generation device. Fig. 1 shows, as an example of installation, a case where the in-vehicle device 10 is installed inside a dashboard DB of the front seat of the automobile M.

[0025] The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is disposed, for example, on a dashboard DB. The GPS receiver 11 may be disposed anywhere as long as it can receive GPS signals. The GPS receiver 11 is capable of transmitting the received GPS signals to the in-vehicle device 10.

[0026] The exterior camera 13 is an imaging device that captures images in front of the automobile M. In this embodiment, the exterior camera 13 is disposed on the dashboard DB so that the imaging direction faces forward. For example, the exterior camera 13 is a wide-angle camera that can capture images of a wide area in front of the automobile M through the windshield. The exterior camera 13 may be provided on the rearview mirror RM or attached to the inside of the windshield FG.

[0027] The in-vehicle camera 15 is an imaging device that captures images of the interior of the automobile M. In this embodiment, the in-vehicle camera 15 is provided on the upper end of the windshield FG or on the ceiling near the upper end, and is capable of capturing images of the driver of the automobile M.

[0028] During video communication, the video captured by the exterior camera 13 or the interior camera 15 is transmitted to the external device 50.

[0029] The touch panel 17 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touchpad. The touch panel 17 is disposed, for example, in the center console of the dashboard DB. The touch panel 17 may be disposed in a location that is visible to the driver and within the driver's reach. For example, the touch panel 17 may be attached to the dashboard DB.

[0030] The touch panel 17 can display a screen based on the control of the in-vehicle device 10. The touch panel 17 can also transmit a signal representing an input operation to the touch panel 17 received from a user to the in-vehicle device 10. For example, the touch panel 17 may display car navigation guidance. Furthermore, operations related to the car navigation function, such as setting a destination, may be possible via the touch panel 17.

[0031] Furthermore, the touch panel 17 may display information about video communication, or may display a screen for accepting operations to start or end a connection for video communication (hereinafter also referred to as a video connection). The occupant of the automobile M may perform an operation to connect video communication by inputting an operation to the touch panel 17.

[0032] The speaker 19 is provided, for example, on the interior side of the A-pillar AP. The speaker 19 is capable of emitting sounds such as music and voices under the control of the in-vehicle device 10. During video communication, the speaker 19 emits sound from the external device 50 during a voice call.

[0033] The microphone 21 is a microphone device that picks up sounds inside the vehicle, and is arranged, for example, on the dashboard DB. The microphone 21 may be arranged anywhere, such as on the rearview mirror RM or the steering wheel, as long as it can pick up sounds inside the vehicle. During video communication, the sound picked up by the microphone 21 is transmitted to the external device 50 as the sound of the voice call. Furthermore, an operation input for starting or ending the video communication may be made by voice via the microphone 21.

[0034] The steering wheel 23 is a member that receives a steering operation for steering the automobile M. The steering wheel 23 is provided rotatably with respect to the dashboard DB. When the steering wheel 23 is turned, the wheels that serve as the steered wheels of the automobile M turn accordingly.

[0035] The turn signal lever 24 is provided near the steering wheel 23 and is a member that receives an operation to turn on a turn signal (directional indicator). The turn signal system including the turn signal lever 24 and the turn signal is configured so that when the turn signal lever 24 is operated, the left or right turn signal is turned on depending on the direction of operation.

[0036] The accelerator pedal 25 is provided under the driver's feet and is a member that receives, when depressed by the driver, operations related to control of a driving motor such as an engine or a motor of the automobile M. In other words, the accelerator pedal 25 is a member that receives steering operations related to acceleration and deceleration of the automobile M.

[0037] The brake pedal 26 is provided under the driver's feet and is a member that receives an operation related to controlling the brakes of the automobile M when the driver depresses the brake pedal 26. In other words, the brake pedal 26 is a member that receives a steering operation related to deceleration of the automobile M.

[0038] The shift lever 27 is provided on the center console of the automobile M, and is a member that receives operations for shift changes according to the driving conditions of the automobile M.

[0039] 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a storage unit 31, a control unit 33, and a communication unit 35 cooperate with each other via a system bus.

[0040] The automobile M is equipped with an acceleration sensor 29. The acceleration sensor 29 may be, for example, a MEMS acceleration sensor that is capable of measuring the acceleration of the automobile M and outputting a signal indicating the measured acceleration. The acceleration sensor 29 may also be an inertial measurement unit (IMU). The acceleration sensor 29 may also be built into the in-vehicle device 10.

[0041] The storage unit 31 is a storage device configured by, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc. The storage unit 31 stores various programs executed in the in-vehicle device 10, such as an operating system and software for the terminal.

[0042] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the storage unit 31 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0043] The storage unit 31 also stores map information including road maps. The storage unit 31 also stores information indicating contact information of the external device 50 as a party with which the in-vehicle device 10 will perform video communication, such as a telephone number or an email address. The contact information may be associated with an identifier such as a nickname that can identify the virtual passenger who is the user of the external device 50.

[0044] The control unit 33 is configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The control unit 33 realizes various functions by the CPU reading and executing various programs stored in the ROM and storage unit 31. In this embodiment, the control unit 33 performs functions such as a function of presenting information regarding the start and end of video communication to the driver, a function of transmitting video during video communication, and a car navigation function.

[0045] The control unit 33 is connected to various devices provided inside the automobile M, such as the exterior camera 13, the interior camera 15, the touch panel 17, the speaker 19, the microphone 21, and the acceleration sensor 29, so as to be able to communicate with them.

[0046] The control unit 33 sequentially acquires images captured by the exterior camera 13 and the interior camera 15. The control unit 33 also sequentially acquires a signal from the GPS receiver 11 and a signal indicating acceleration measured by the acceleration sensor 29.

[0047] In addition, the control unit 33 is communicably connected to various sensors that detect the driving operation state of the automobile M, and can acquire sensor information from the various sensors as information indicating the driving operation state of the automobile M.

[0048] Specifically, the control unit 33 can acquire sensor information from the steering sensor 23S, which is a sensor that detects physical quantities related to the operation of the steering wheel 23, such as the amount of rotation, the direction of rotation, and the rotational force.

[0049] The control unit 33 can acquire sensor information from the winker sensor 24S, which is a sensor that detects the operating state, such as the operating direction of the winker lever.

[0050] The control unit 33 can acquire sensor information from the accelerator sensor 25S, which is a sensor that detects physical quantities related to the operation of the accelerator pedal 25, such as the amount of movement and the speed of movement.

[0051] The control unit 33 can acquire sensor information from the brake sensor 26S, which is a sensor that detects physical quantities related to the operation of the brake pedal 26, such as the amount of movement and the speed of movement.

[0052] The control unit 33 can acquire sensor information from the shift lever sensor 27S, which is a sensor that detects which shift position the shift lever 27 is in, such as P (parking), R (reverse), N (neutral), or D (drive).

[0053] The communication unit 35 is a communication device that transmits and receives data to and from external devices in accordance with instructions from the control unit 33. The communication unit 35 is, for example, a network interface card (NIC) for connecting to the network NW.

[0054] The communication unit 35 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The communication unit 35 also transmits and receives various data to and from the external device 50 via the server 40.

[0055] The communication unit 35 transmits, for example, images of the exterior or interior of the automobile M, information indicating input operations performed on the in-vehicle device 10, current location information, travel route, and other information to the server 40. The communication unit 35 also receives, for example, information to be displayed on the touch panel 17 regarding video communication from the server 40.

[0056] During the video communication, the control unit 33 sequentially transmits the images captured by the exterior camera 13 or the interior camera 15 to the server 40 via the communication unit 35 .

[0057] During the video communication, the control unit 33 sequentially transmits sensor information acquired from various sensors that detect the driving operation state of the automobile M to the server 40 via the communication unit 35.

[0058] The control unit 33 receives display information related to the start and end of video communication from the server 40 via the communication unit 35 and displays it on the touch panel 17. The control unit 33 may also output information related to the start and end of video communication by sound from the speaker 19.

[0059] The control unit 33 can transmit information indicating instructions relating to the start or end of video communication received via the touch panel 17 or the microphone 21, for example, to the server 40 via the communication unit .

[0060] For example, the control unit 33 of the in-vehicle device 10 can transmit location identification information as location data that can identify the current location of the in-vehicle device 10, i.e., information on the current location of the automobile M in this embodiment, to the server 40 via the communication unit 35.

[0061] Furthermore, for example, the control unit 33 can transmit information including a destination input by the user to the server 40 via the communication unit 35, and receive route information or navigation information to the destination from the server 40.

[0062] The control unit 33 can transmit audio data of the audio picked up by the microphone 21 to the external device 50 via the communication unit 35 for audio calls in video communication. The control unit 33 can also receive audio data of the audio input to the external device 50 via the communication unit 35 for audio calls in video communication.

[0063] 4 is a block diagram showing the configuration of the server 40. For example, the server 40 is a device in which a storage unit 41, a communication unit 43, and a control unit 45 cooperate with each other via a system bus.

[0064] The server 40 has a function similar to that of a SIP server that establishes a voice call between the in-car device 10 and the external device 50 during video communication and transfers data of the voice call.

[0065] The server 40 also has the function of receiving from the in-vehicle device 10 the location identification information of the automobile M and information on the destination set by the user who is an occupant of the automobile M, and generating a route to the destination based on the location identification information and the destination information.

[0066] The storage unit 41 is configured with, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as an operating system and software for the server 40.

[0067] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the storage unit 41 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0068] The storage unit 41 stores, for example, a communication program for the server 40 to transmit the video and driving operation information acquired by the in-vehicle device 10 to the external device 50.

[0069] The storage unit 41 also stores a map information database (not shown) that stores map information including road maps. The map information in the map information database is a database that contains information equivalent to the map information used in, for example, a navigation device.

[0070] The communication unit 43 is connected to the network NW, and transmits and receives various data between the in-vehicle device 10 and the external device 50.

[0071] The control unit 45 is configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The CPU reads and executes various programs stored in the ROM and the storage unit 41, thereby realizing various functions.

[0072] The control unit 45 executes the above-mentioned communication program to realize audio-video communication for transmitting audio and video in the automobile M acquired by the in-vehicle device 10 to the external device 50.

[0073] For example, the control unit 45 realizes video communication by transferring the video and audio received from the in-vehicle device 10 to the external device 50 via the communication unit 43, and transferring the audio received from the external device 50 to the in-vehicle device 10.

[0074] The control unit 45 includes an image acquisition unit 46, a driving operation information acquisition unit 47, and a display control unit 48.

[0075] The image acquisition unit 46 is a functional unit that executes an image acquisition step of sequentially acquiring images captured in the automobile M.

[0076] The image acquisition unit 46 sequentially acquires images, for example, of the area in front of the automobile M captured by an exterior camera 13 installed on the automobile M, by sequentially receiving the images from the in-vehicle device 10 via the communication unit 43.

[0077] For example, the image acquisition unit 46 may sequentially acquire images captured in the automobile M by sequentially receiving images of the driver of the automobile M captured by the in-vehicle camera 15 along with images captured by the exterior camera 13 via the communication unit 43.

[0078] The driving operation information acquisition unit 47 is a functional unit that executes a driving operation information acquisition step of successively acquiring driving operation information as driving operation information indicating the driving operation of the vehicle M by the driver of the vehicle M.

[0079] The driving operation information acquisition unit 47 sequentially acquires driving operation information, for example, by sequentially receiving signals from various sensors that detect the driving operation state of the automobile M from the in-vehicle device 10 via the communication unit 43.

[0080] For example, the driving operation information acquisition unit 47 sequentially acquires, as driving operation information, information indicating at least one of the rotation direction and rotation angle of the steering wheel 23, the operation state of the turn signal lever 24, the operation state of the accelerator pedal 25 such as the movement amount and movement speed, the operation state of the brake pedal 26 such as the movement amount and movement speed, and the operation state of the shift lever 27.

[0081] Further, for example, the driving operation information acquisition unit 47 may sequentially acquire a signal from the GPS receiver 11 and a signal indicating acceleration measured by the acceleration sensor 29 as the driving operation information.

[0082] For example, the driving operation information acquisition unit 47 may acquire information indicating the driving conditions of the automobile M, such as location information acquired based on a GPS signal or the speed of the automobile M acquired based on a signal indicating acceleration from the acceleration sensor 29.

[0083] The display control unit 48 is a functional unit that executes a display control step to control the display of both the image acquired by the image acquisition unit 46 and a display based on the driving operation information acquired by the driving operation information acquisition unit 47 on the external device 50.

[0084] The display control unit 48 generates display information that combines, for example, an image of the area in front of the automobile M captured by the exterior camera 13 with a display based on driving operation information, and sequentially transmits the generated display information to the external device 50.

[0085] The display control unit 48 generates display information for superimposing and displaying, for example, an image of the area ahead of the automobile M on a display based on the driving operation information.

[0086] The display control unit 48 may, for example, generate display information for displaying a video of the area ahead of the automobile M and a display based on driving operation information side by side in separate areas.

[0087] For example, when generating display information for displaying both an image and a display based on driving operation information, the display control unit 48 matches the time when the image was acquired with the time when a signal was detected in the in-vehicle device 10.

[0088] When the display control unit 48 executes the display control step, for example, a display showing the operation state of the steering wheel 23, the operation state of the turn signal lever 24, the operation state of the accelerator pedal 25, the operation state of the brake pedal 26, and the operation state of the shift lever 27 is displayed on the external device 50 together with the image captured in the automobile M.

[0089] In addition to the display based on the driving operation information, the display may be based on information indicating the driving conditions of the automobile M, such as the position information and speed of the automobile M.

[0090] In this way, by displaying information based on driving operation information and driving conditions, the user of the external device 50 can supplement information that can be obtained, for example, if the user is actually riding in the passenger seat of the automobile M, but cannot be obtained through video footage of the area in front of the automobile M or through a conversation with the driver of the automobile M when video communication is being performed using the external device 50.

[0091] For example, a real passenger can visually know the direction and angle of rotation of the steering wheel 23 through the driver's operation, audibly know whether the turn signal is on through the operation sound of the turn signal lever 24 or the sound of the turn signal, and visually know whether the turn signal lever 24 is being operated to the left or right. However, this information is difficult to obtain from video communication images, making it difficult for a virtual passenger to grasp this information.

[0092] By supplementing information that is difficult for the virtual passenger to grasp, the virtual passenger can feel as if he or she is actually riding in the automobile M.

[0093] The control unit 45 also acquires location information indicating the current location of the automobile M from the in-vehicle device 10 via the communication unit 43. The control unit 45 also acquires destination information input to the in-vehicle device 10 by an occupant of the automobile M from the in-vehicle device 10 via the communication unit 43. The control unit 45 generates a route to the destination based on the location information and the destination information, and transmits information indicating the route to the in-vehicle device 10.

[0094] 5 is a front view of the external device 50. As described above, in this embodiment, the external device 50 is a smartphone.

[0095] The touch panel 51 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touch pad. The touch panel 51 is capable of generating signals that represent input operations to the touch panel 51 received from a user. In this embodiment, an image transmitted from the in-vehicle device 10 is displayed on the touch panel 51.

[0096] Furthermore, information about the video communication and a screen for accepting an operation to start a video communication connection are displayed on the touch panel 51. The user of the external device 50 can perform an input operation to start the video communication by inputting information to the touch panel 51.

[0097] The speaker 53 is capable of emitting sounds such as music and voices. During video communication, the speaker 53 emits the sound from the in-car device 10 during voice communication.

[0098] The microphone 55 is a microphone device that receives sounds emitted toward the external device 50. During video communication, the sounds picked up by the microphone 55 are transmitted to the in-car device 10 via the server 40 as voice calls.

[0099] 6 is a block diagram showing an example of the configuration of the external device 50. For example, the external device 50 is a device in which a storage unit 57, a control unit 59, and a communication unit 61 cooperate with each other via a system bus (not shown).

[0100] The storage unit 57 is configured by, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc., and stores various programs such as an operating system and software for the external device 50.

[0101] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via a drive device. That is, the various programs stored in the storage unit 57 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0102] The control unit 59 is configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. The CPU reads and executes various programs stored in the ROM and the storage unit 57, thereby realizing various functions.

[0103] The control unit 59 is communicably connected to the touch panel 51, the speaker 53, and the microphone 55.

[0104] The control unit 59 can receive signals indicating input operations on the touch panel 51 and audio input signals from the microphone 55. For example, the control unit 59 can accept a request to start or end video communication made by the user via the touch panel 51 or the microphone 55.

[0105] Furthermore, the control unit 59 can transmit a video or image signal to the touch panel 51 to display it, and can transmit an audio signal to the speaker 53 to output sound.

[0106] For example, the control unit 59 causes the touch panel 51 to display a start screen for accepting a request for a video communication connection attempt.

[0107] Furthermore, for example, the control unit 59 causes the touch panel 51 to display an image during the video communication, and causes the speaker 53 to output audio collected in the automobile M during the video communication.

[0108] The communication unit 61 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The communication unit 61 also transmits and receives various data, including video transmitted from the in-vehicle device 10, to and from the in-vehicle device 10 via the server 40.

[0109] The control unit 59 of the external device 50 can receive, via the communication unit 61, information for displaying a start screen for accepting a request for a video communication connection attempt.

[0110] Furthermore, the control unit 59 can transmit audio data of the audio picked up by the microphone 55 to the in-car device 10 via the communication unit 61 for audio communication in video communication. The control unit 59 can also receive audio data transmitted from the in-car device 10 via the communication unit 61 for audio communication in video communication.

[0111] For example, the control unit 59 of the external device 50 can receive, via the communication unit 61, location identification information capable of identifying the current location of the automobile M transmitted from the in-vehicle device 10 from the server 40. Furthermore, for example, the control unit 59 can receive, via the communication unit 61, route information or navigation information for the automobile M from the server 40.

[0112] [2. Operation of in-vehicle device] The control routine of the in-vehicle device 10 for realizing the operation of the communication system 100 of this embodiment will be described below.

[0113] 7 is a flowchart showing a video transmission routine RT1 executed by the control unit 33 of the in-vehicle device 10. For example, when the in-vehicle device 10 is powered on, the control unit 33 starts the video transmission routine RT1.

[0114] When the video transmission routine RT1 starts, the control unit 33 determines whether video communication has started (step S101). The determination in step S101 is made based on whether a video communication connection has been established in the in-vehicle device 10 and communication has been established with the external device 50, for example.

[0115] In step S101, when the control unit 33 determines that video communication has not started (step S101: NO), it ends the video transmission routine RT1 and starts the next video transmission routine RT1 anew.

[0116] In step S101, when the control unit 33 determines that video communication has started (step S101: YES), it starts transmitting the video and driving operation information acquired in the automobile M (step S102).

[0117] In step 102, for example, the control unit 33 starts transmitting the video captured by the exterior camera 13 to the server 40. Also in step 102, for example, the control unit 33 sequentially transmits sensor information from various sensors that detect the driving operation state of the automobile M.

[0118] In step 102, for example, the control unit 33 sequentially transmits information indicating at least one of the rotation direction and rotation angle of the steering wheel 23, the operation state of the turn signal lever 24, the operation state of the accelerator pedal 25 such as the movement amount and movement speed, the operation state of the brake pedal 26 such as the movement amount and movement speed, and the operation state of the shift lever 27.

[0119] Also, in step 102, for example, the control unit 33 may sequentially transmit a signal from the GPS receiver 11 and a signal indicating the acceleration measured by the acceleration sensor 29.

[0120] After executing step S102, the control unit 33 determines whether the video communication has ended (step S103). In step S103, for example, when the in-car device 10 is not connected for video communication and communication with the external device 50 is not established, it is determined that the video communication has ended.

[0121] In step S103, if the control unit 33 determines that the video communication has not ended (step S103: NO), it repeats step S103 and determines again whether or not the video communication has ended.

[0122] In step S103, when the control unit 33 determines that the video communication has ended (step S103: YES), it ends the transmission of the video and driving operation information (step S104).

[0123] After executing step S104, the control unit 33 ends the video transmission routine RT1 and starts the next video transmission routine RT1 anew.

[0124] [3. Server Operation] The control routine of the server 40 for realizing the operation of the communication system 100 of the embodiment will be described below.

[0125] 8 is a flowchart showing a display control routine RT2 executed by the display control unit 48 of the control unit 45 of the server 40. For example, when a notification indicating that video communication will be performed between the in-vehicle device 10 and the external device 50 is received via the server 40, the display control unit 48 starts the display control routine RT2.

[0126] When the display control routine RT2 starts, the display control unit 48 determines whether video communication has started (step S201). The determination in step S201 is made based on, for example, whether a video communication connection has been established between the in-vehicle device 10 and the external device 50, and whether communication with the external device 50 has been established.

[0127] In step S201, when the display control unit 48 determines that video communication has not started (step S201: NO), it repeats step S201 and determines again whether video communication has started.

[0128] In step S201, when the display control unit 48 determines that video communication has started (step S201: YES), it starts receiving video and driving operation information from the in-vehicle device 10 (step S202). In step S202, for example, the display control unit 48 starts receiving video captured by the exterior camera 13 of the automobile M, and also starts receiving sensor information from various sensors that detect the driving operation state of the automobile M.

[0129] In step S202, for example, the display control unit 48 sequentially receives, as driving operation information, information indicating at least one of the rotation direction and rotation angle of the steering wheel 23, the operation state of the turn signal lever 24, the operation state of the accelerator pedal 25 such as the movement amount and movement speed, the operation state of the brake pedal 26 such as the movement amount and movement speed, and the operation state of the shift lever 27.

[0130] In step S202, for example, the display control unit 48 may successively receive a signal from the GPS receiver 11 and a signal indicating acceleration measured by the acceleration sensor 29 as driving operation information.

[0131] After executing step S202, the display control unit 48 starts generating and transmitting a driving operation display image that is an image that combines the received image and a display based on the driving operation information (step S203).

[0132] In step S203, the display control unit 48 sequentially generates display information for superimposing a display showing the operation status of the steering wheel 23, the turn signal lever 24, the accelerator pedal 25, the brake pedal 26, or the shift lever 27 on an image captured in front of the automobile M, to create a driving operation display image, and sequentially transmits the sequentially generated driving operation display image to the external device 50.

[0133] 9 is a diagram showing an example of a driving operation display image displayed on the touch panel 51 of the external device 50 when step S203 is executed. As shown in FIG. 9, the touch panel 51 displays a so-called landscape screen, in which the longitudinal direction of the touch panel 51 is the left-right direction.

[0134] In the example shown in FIG. 9, the touch panel 51 is divided into two display areas: a first display area AR1 and a second display area AR2 that occupies a smaller area than the first display area AR1.

[0135] In the first display area AR1, a display based on driving operation information is superimposed on an image captured in front of the automobile M. Specifically, in FIG. 9, an area A surrounded by a dashed line displays an arrow-shaped icon indicating the direction of the turn signal lever 24 as the operation state of the turn signal lever 24. In the example shown in FIG. 9, the icon indicates that the turn signal lever 24 is operated so that the left turn signal is turned on.

[0136] 9, an icon indicating the operation state of the steering wheel 23 is displayed in an area B surrounded by a dashed line. In the example shown in FIG. 9, the icon indicates an operation state in which the steering wheel 23 is rotated to the left.

[0137] In Fig. 9, an area C surrounded by a dashed line displays an indication of the operation state of the accelerator pedal 25. In the example shown in Fig. 9, a gauge indicating the accelerator opening is displayed together with an icon indicating the accelerator pedal.

[0138] In Fig. 9, an area D surrounded by a dashed line displays an indication of the operation state of the brake pedal 26. In the example shown in Fig. 9, an icon representing the brake pedal is displayed together with a gauge indicating the amount of depression of the brake pedal.

[0139] In Fig. 9, an area E surrounded by a dashed line displays an indication of the operating state of the shift lever 27. As shown in Fig. 9, area E displays an icon representing the shift lever, as well as a letter indicating the shift position that has been reached by operation of the shift lever 27. In the example shown in Fig. 9, "D" is displayed, indicating the drive shift position.

[0140] In Fig. 9, an area F surrounded by a dashed line displays the speed of the automobile M. In the example shown in Fig. 9, the current speed of the automobile M is displayed together with an icon showing a speedometer.

[0141] The second display area AR2 displays a map image showing the current position PL of the automobile M and the route R of the automobile M. For example, a virtual passenger can check the route R of the automobile M by looking at the map image displayed in the second display area AR2, and can receive route guidance using the image of the area ahead of the automobile M and driving operation information as clues.

[0142] In this way, by displaying both the image captured in automobile M and the display based on the driving operation information, the virtual passenger can obtain information that would be available if the passenger were actually riding in automobile M, but that would not be available if the passenger were not riding in automobile M and was performing video communication using external device 50.

[0143] For example, when a real passenger is riding in automobile M with the driver of automobile M and giving directions, the real passenger can grasp the state of the driver's driving operation by sight, hearing, etc. For example, the real passenger can collect information inside automobile M such as how much the driver is turning the steering wheel 23 to the left or right, whether the driver has operated the turn signal lever 24, and if so, which direction the driver is indicating to the left or right.

[0144] Therefore, for example, if the steering wheel 23 is not rotated enough or the driver does not operate the turn signal lever 24 when it should be operated, the real passenger can advise the driver to rotate the steering wheel 23 more or to operate the turn signal lever 24.

[0145] However, when a user of the external device 50 as a virtual passenger is not in the automobile M and is providing directions via video communication while watching the video footage captured by the exterior camera 13 outside the automobile M, it is difficult for the user to grasp the driving operation status of the driver, and for example, the user cannot grasp the driving operation status such as the operation status of the steering wheel 23 or the operation status of the turn signal lever 24 by the driver.

[0146] 9, the virtual passenger can be provided with information that is lacking by having the external device 50 display, based on driving operation information, such as a display showing the rotation state of the steering wheel 23 and a display showing the direction of the turn signal lever 24, based on the driving device information. This allows the virtual passenger to smoothly receive route guidance with a feeling similar to that of actually riding in the automobile M.

[0147] In addition, the display control unit 48 sequentially acquires the images and driving operation information, instantly generates driving operation display images, and sequentially transmits them to the external device 50, so that the driving operation status of the automobile M can be displayed on the external device 50 in real time.

[0148] Therefore, the virtual passenger can give advice on routes without any time lag. For example, if the driver of automobile M is approaching a point where he should turn left but has not turned on his blinker to the left, it is possible to prevent situations where automobile M passes by the point where he should turn left without making a left turn, even if the virtual passenger warns him to "turn left there."

[0149] Furthermore, even if the virtual passenger does not provide route guidance, the virtual passenger can easily get the feeling that he or she is riding in the car M, or in other words, a sense of being together in the car, and can get the feeling that he or she is driving together with the driver of the car M, for example.

[0150] For example, in Figure 9, a case has been described in which a display based on driving operation information is superimposed on an image captured in front of the automobile M, but this is not limited to this, and a display based on driving operation information may be displayed in an area other than the area in which the image captured in the automobile M is displayed.

[0151] After executing step S203, the display control unit 48 determines whether the video communication has ended (step S204).

[0152] In step S204, if the display control unit 48 determines that the video communication has not ended (step S204: NO), it repeats step S204 and determines again whether or not the video communication has ended.

[0153] In step S204, if the display control unit 48 determines that the video communication has ended (step S204: YES), it terminates the generation and transmission of the driving operation display image (step S205), terminates the reception of the image and driving operation information (step S206), and terminates the display control routine RT2.

[0154] 10 is a diagram showing another example of a driving operation display image displayed on the touch panel 51 of the external device 50 when step S203 of the display control routine RT2 is executed. As shown in FIG. 10, the touch panel 51 displays a so-called vertical screen, in which the longitudinal direction of the touch panel 51 is the up-down direction.

[0155] 10, the touch panel 51 is divided into two display areas: a third display area AR3 displaying an image of the area ahead of the automobile M, and a fourth display area AR4 displaying a map image and occupying a larger area than the third display area AR3. The portrait screen display shown in FIG. 10 allows the user of the external device 50 to see the map image clearly.

[0156] For example, the user of the external device 50 may be able to switch between a landscape screen display shown in Fig. 9 and a portrait screen display shown in Fig. 10. For example, by setting the display of the touch panel 51 to a landscape screen display as shown in Fig. 9, the user of the external device 50 can receive route guidance that is close to reality while feeling like they are riding along. Furthermore, for example, by setting the display of the touch panel 51 to a portrait screen display as shown in Fig. 10, the user of the external device 50 can receive route guidance while carefully checking the map.

[0157] As explained in detail above, the communication system 100 of this embodiment is capable of establishing a voice call via the server 40 between the in-vehicle device 10 as a first device that moves together with the automobile M as a moving body and the external device 50 as a second device, while also performing video communication in which an image of the moving body on which the in-vehicle device 10 is mounted is transmitted to the external device 50.

[0158] The communication system 100 of this embodiment includes an image acquisition unit that sequentially acquires images captured by a moving body, and a steering operation information acquisition unit that sequentially acquires steering operation information indicating steering operations performed on the moving body by a driver of the moving body. The communication system 100 of this embodiment also includes a display control unit that performs display control to display both the images and a display based on the steering operation information on a second device.

[0159] With this configuration, it is possible for a user of the second device who is outside the moving body to supplement the steering operation information that can be obtained when inside the moving body but cannot be obtained when outside the moving body.

[0160] The display control unit can, for example, sequentially display both the sequentially acquired image and a display based on the sequentially acquired steering operation information, and can present the steering operation information to the user of the second device in real time.

[0161] Therefore, according to this embodiment, it is possible to provide a communication system, a communication device, a server device, a communication method, a communication program, and a storage medium that enable communication between a first device inside a mobile body and a second device outside the mobile body to provide the user of the second device outside the mobile body with information that would be available when the user was actually inside the mobile body.

[0162] [Variations] 11 is a block diagram showing an example of the configuration of an in-vehicle device 70 as a communication device in a modification of the communication system 100 of the embodiment. In this modification, the in-vehicle device 70 has the functions of the video acquisition unit 46, the driving operation information acquisition unit 47, and the display control unit 48 that the server 40 has in the above embodiment.

[0163] Specifically, the control unit 71 of the in-vehicle device 70 has an image acquisition unit 73 , a driving operation information acquisition unit 74 , and a display control unit 75 .

[0164] The video acquisition unit 73 sequentially acquires video captured by the exterior camera 13. For example, the video acquisition unit 73 may sequentially acquire video captured by the interior camera 15 together with video captured by the exterior camera 13.

[0165] The driving operation information acquisition unit 74 sequentially acquires signals from various sensors that detect the driving operation state of the automobile M, for example, as driving operation information.

[0166] The display control unit 75 executes a display control step that performs display control to display both the image acquired by the image acquisition unit 73 and a display based on the driving operation information acquired by the driving operation information acquisition unit 74 on the external device 50.

[0167] For example, the display control unit 75 generates display information that combines an image of the area in front of the automobile M captured by the exterior camera 13 and sequentially acquired by the image acquisition unit 73 with a display based on driving operation information sequentially acquired by the driving operation information acquisition unit 74, and sequentially transmits the generated display information to the external device 50.

[0168] The memory unit 77 stores a communication program that allows the control unit 71 to execute display control that sequentially acquires images captured in the automobile M, sequentially acquires driving operation information indicating driving operations performed on the automobile M by the driver of the automobile M, and displays the images and a display based on the driving operation information on the external device 50.

[0169] According to this modified example, during video communication between the external device 50 and the in-vehicle device 70, it is possible to control the external device 50 to display both the video captured in the automobile M and sequentially acquired by the video acquisition unit 73, and a display based on the driving operation information sequentially acquired by the driving operation information acquisition unit 74, without going through the server 40.

[0170] For example, the video communication in this modification may be performed directly between the in-vehicle device 70 and the external device 50 by P2P (Peer to Peer) communication or the like, without going through the server 40.

[0171] The configurations, routines, etc. of the in-vehicle device 10, the in-vehicle device 70, the server 40, and the external device 50 in the above-described embodiment and modified examples are merely examples, and can be appropriately selected or changed depending on the application, etc.

[0172] For example, in the above-mentioned embodiments and variants, the image displayed on the touch panel 51 of the external device 50 may be switchable between the image captured by the outside-vehicle camera 13 and the image captured by the inside-vehicle camera 15 by operating the external device 50 or the in-vehicle device 10.

[0173] For example, when the display screen of the external device 50 is displaying an image captured by the in-vehicle camera 15, at least some of the displays based on driving operation information, such as displays showing the steering wheel operation state and the turn signal lever operation state, may be hidden. For example, with regard to displays suitable for display together with an image of the area ahead of the vehicle captured by the exterior camera 13, displays that are less important to display together with an image of the driver captured by the in-vehicle camera 15 may be hidden when the image captured by the in-vehicle camera 15 is being displayed. For example, when the image captured by the in-vehicle camera 15 is being displayed, if the importance of route guidance is low, only displays showing, for example, the speed of the automobile M and the turn signal operation state may be displayed, and the rest may be hidden.

[0174] In the above-described embodiment and modified examples, for example, the external device 50 may be configured to emit a turn signal sound together with a display showing the operation state of the turn signal lever. For example, the turn signal sound picked up by the microphone 21 provided in the automobile M may be controlled to be output from the external device 50 in synchronization with the display showing the operation state of the turn signal lever. In this case, for example, a learning model or the like may be used to distinguish between the turn signal sound and other sounds.

[0175] Furthermore, for example, at the timing when a turn signal lever is operated in the automobile M to emit a turn signal sound, a predetermined turn signal sound prepared in advance may be output from the external device 50 together with a display showing the operation state of the turn signal lever. In this case, for example, when sound in the automobile M is output from the external device 50, a process may be performed to cancel the actual turn signal sound picked up by the microphone 21 provided in the automobile M.

[0176] For example, when the turn signal lever is in a position to make the turn signals flash, a display showing the operation status of the turn signals may be made to flash, and the sound of the turn signal may be synchronized with the flashing of the display.

[0177] Furthermore, the driving operation information in the above embodiments is not limited to the information exemplified in the embodiments, and may include other information indicating driving operations for the automobile M. For example, the driving operation information may include information indicating operation of the parking brake, operation of the wipers, operation of the headlight switch, etc.

[0178] In the above embodiment, the case where the virtual passenger provides route guidance to the driver of the automobile M has been described, but this is not limited to this. Even if the virtual passenger does not provide route guidance, if the driving operation information is displayed together with the image captured in the automobile M, the virtual passenger can grasp the details of the driving operation by the driver of the automobile M, thereby enhancing the feeling of being a passenger in the automobile.

[0179] In the above embodiment and modified example, an example of performing video communication in the communication system 100 has been described, but the present invention is not limited to this. The present invention is applicable to a case where audio-video communication is performed in the communication system 100, in which audio and video in the automobile M acquired by an in-vehicle device 10 as a first device traveling together with the automobile M is transmitted to an external device 50 as a second device. In other words, the present invention is also applicable to communication in a mode in which audio is not transmitted from the external device 50.

[0180] Furthermore, voice communication is not necessarily possible between the in-vehicle device 10 and the external device 50. For example, the communication mode between the in-vehicle device 10 and the external device 50 may be a mode in which video and audio are simply distributed from the in-vehicle device 10 to the external device 50 without two-way voice communication, such as YouTube (registered trademark) or Niconico Live Broadcast (registered trademark).

[0181] In the above embodiment, the in-vehicle device 10 and the external device 50 perform video communication via the server 40. However, the video communication may be performed directly between the in-vehicle device 10 and the external device 50.

[0182] In addition, while the above-mentioned video distribution is being performed from the in-vehicle device 10 to the external device 50 via the communication system 100, a voice call may be established between the in-vehicle device 10 and the external device 50 in parallel via a path separate from the communication system 100.

[0183] In the above embodiment and modified example, the in-vehicle device 10 is an in-vehicle navigation device. However, the in-vehicle device 10 may simply be capable of distributing video from the exterior camera 13 or the interior camera 15.

[0184] For example, the in-vehicle device 10 may be configured such that a terminal device having a configuration similar to that of the in-vehicle device 10, the exterior camera 13, and the touch panel 17 are integrated together. Specifically, the in-vehicle device 10 may be a terminal device such as a smartphone, tablet, or PC with a camera that is equipped with an application that performs the same functions as the in-vehicle device 10. In this case, the in-vehicle device 10 may be attached to the dashboard DB by, for example, a cradle or the like so that the built-in camera can capture an image of the area ahead of the automobile M through the windshield of the automobile M.

[0185] Furthermore, the in-vehicle device 10 may be configured not to display a screen to be presented to the driver of the automobile M. For example, the in-vehicle device 10 may have a configuration similar to that of a drive recorder, and may be a device integrated with the exterior camera 13. Specifically, the in-vehicle device 10 may be a device in which hardware that performs the video communication function of the in-vehicle device 10 is built into the housing of the exterior camera 13. In this case, the in-vehicle device 10 may not perform the various display outputs described above.

[0186] In the above embodiment and modified example, the external device 50 is described as a smartphone, but is not limited thereto. The external device 50 is a terminal device that can be used by a virtual passenger for video communication, and is configured to be able to display or present messages related to video communication, accept operation inputs necessary for video communication, send and receive audio data, and receive and display video. For example, the external device 50 may be a terminal device such as a tablet, PC, or wearable device.

[0187] In the above embodiment and modified example, the in-vehicle device 10 and the in-vehicle device 70 are mounted on the automobile M, but the in-vehicle device 10 may be mounted on other moving objects such as a bicycle, a motorbike, a boat, etc. Also, the in-vehicle device 10 may be a terminal device with a built-in camera, such as a smartphone, which may be held by a person and used for video communication while walking, for example, to transmit images. [Explanation of symbols]

[0188] 10, 70 On-vehicle equipment 13. Exterior camera 15 In-car camera 17 Touch Panel 19 Speaker 21. Mike 23S Steering sensor 24S turn signal sensor 25S Accelerator sensor 26S Brake sensor 27S Shift lever sensor 31 Storage section 33 Control Unit 35 Communications Department 40 servers 41 Storage section 43 Communications Department 45 Control Unit 46, 73 Video acquisition unit 47, 74 Driving operation information acquisition unit 48, 75 Display control unit 50 External device 51 Touch Panel 53 Speaker 55 Mike 57 Memory section 59 Control Unit 61 Communications Department

Claims

1. A communication system for performing audio-video communication in which audio and video captured by a first device moving together with a mobile object are transmitted to a second device, an image acquisition unit that sequentially acquires images captured by the moving object; a steering operation information acquisition unit that sequentially acquires steering operation information indicating a steering operation performed by a driver of the moving body on the moving body; a display control unit that performs display control to display a maneuvering operation display image, which includes the captured image and a display based on the maneuvering operation information and has a longitudinal direction, and a map image related to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device included in the second device; The display control unit controls switching between a landscape screen display in which the maneuvering operation display video and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the maneuvering operation display video and the map image are displayed with the longitudinal direction of the display screen as the up-down direction. A communication system comprising:

2. The communication system according to claim 1, characterized in that the display control unit causes the second device to display, together with the captured image, a display based on the steering operation information indicating the steering operation at the time of capturing the captured image.

3. 3. The communication system according to claim 1, wherein the steering operation information includes turn signal information indicating an operation state of a turn signal lever provided on the mobile object.

4. 4. The communication system according to claim 1, wherein the steering operation information includes information indicating an operation state of an accelerator pedal provided on the mobile object.

5. 5. The communication system according to claim 1, wherein the steering operation information includes information indicating an operation state of a brake pedal provided on the mobile object.

6. 6. The communication system according to claim 1, wherein the steering operation information includes information indicating an operation state of a steering wheel provided on the mobile object.

7. 7. The communication system according to claim 1, wherein the steering operation information includes information indicating an operating state of a shift lever provided on the mobile object.

8. 8. The communication system according to claim 1, wherein the steering operation information includes information indicating an operation state of a side brake provided on the mobile object.

9. 1. A communication method executed by a communication system for performing audio-video communication in which audio and video captured by a first device moving together with a mobile object are transmitted to a second device, the method comprising: an image acquisition step of sequentially acquiring images captured by the moving body; a steering operation information acquisition step of sequentially acquiring steering operation information indicating a steering operation performed by a driver of the moving body on the moving body; a display control step of performing display control to display a maneuvering operation display image, which includes the captured image and a display based on the maneuvering operation information and has a longitudinal direction, and a map image relating to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device included in the second device, a communication method characterized in that, in the display control step, control is performed to switch between a landscape screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the up-down direction.

10. A communication program executed by a communication system for performing audio-video communication, the communication system including a computer, the first device moving together with a mobile object, and transmitting audio and video captured by the first device at the mobile object to a second device, the program comprising: an image acquisition step of sequentially acquiring images captured by the moving body; a steering operation information acquisition step of sequentially acquiring steering operation information indicating a steering operation performed by a driver of the moving body on the moving body; a display control step of performing display control to display a maneuvering operation display image including the captured image and a display based on the maneuvering operation information and having a longitudinal direction, and a map image relating to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device included in the second device; A communication program for controlling switching between a landscape screen display in which the steering operation display image and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the steering operation display image and the map image are displayed with the longitudinal direction of the display screen as the up-down direction, in the display control step.

11. A communication system for performing audio-video communication includes a computer, the system including a first device moving together with a mobile object, and the first device acquiring audio and video of the mobile object and transmitting the audio and video to a second device. an image acquisition step of sequentially acquiring images captured by the moving body; a steering operation information acquisition step of sequentially acquiring steering operation information indicating a steering operation performed by a driver of the moving body on the moving body; a display control step of performing display control to display a maneuvering operation display image including the captured image and a display based on the maneuvering operation information and having a longitudinal direction, and a map image relating to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device included in the second device; A computer-readable storage medium that stores a communication program for controlling switching, in the display control step, between a landscape screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the up-down direction.

12. A communication device that moves together with a mobile object, acquires audio and video from the mobile object, and transmits the audio and video to another device, an image acquisition unit that sequentially acquires images captured by the moving object; a steering operation information acquisition unit that sequentially acquires steering operation information indicating a steering operation performed by a driver of the moving body on the moving body; a display control unit that performs display control to display a maneuvering operation display image, which includes the captured image and a display based on the maneuvering operation information and has a longitudinal direction, and a map image relating to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device of the other device, The communication device is characterized in that the display control unit controls switching between a landscape screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the up-down direction.

13. A server device for performing audio / video communication in which audio and video of a moving object, which is acquired by a first device moving together with the moving object, is transmitted to a second device, an image acquisition unit that sequentially acquires captured images captured at the moving object from the first device via a communication path; a steering operation information acquisition unit that sequentially acquires steering operation information indicating a steering operation of the moving body by a driver of the moving body from the first device via a communication path; a display control unit that performs display control to display a maneuvering operation display image, which includes the captured image and a display based on the maneuvering operation information and has a longitudinal direction, and a map image related to the movement of the moving object, side by side along the longitudinal direction of a display screen on a display device included in the second device; The server device is characterized in that the display control unit controls switching between a landscape screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the left-right direction, and a portrait screen display in which the steering operation display video and the map image are displayed with the longitudinal direction of the display screen as the up-down direction.

Images