Video transmission device, video output device, video output system, video transmission method, video transmission program, and storage medium

Abstract

To provide a video transmission device, a video output device, and a video distribution system capable of appropriately sharing the situation around a vehicle between an operator of a terminal of a vehicle who watches a video sent from the vehicle and a driver of the vehicle while maintaining the stability of video distribution.SOLUTION: A video transmission device according to the present invention includes a video acquisition unit that sequentially acquires video data representing a video around a moving object, a position information acquisition unit that acquires the position of the moving body, and a data transmission unit that encodes the video data in a delay mode based on the position of the moving body to generate encoded video data and transmit the encoded video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video transmission device, a video output device, a video output system, a video transmission method, a video transmission program, and a storage medium, and more particularly to a video transmission device, a video output device, a video distribution system, a video transmission method, a video transmission program, and a storage medium for providing a user with video from a mobile object. [Background technology]

[0002] There is a communication system that performs communication between an onboard device mounted in a vehicle and an external terminal located outside the vehicle. For example, Patent Document 1 discloses a system in which a voice call is performed between a driver of the vehicle and an operator of the external terminal, and while the voice call is being performed, video data showing an image of the area in front of the vehicle is transmitted from the onboard device to the external terminal, and the video is displayed on the external terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-213791 Summary of the Invention [Problem to be solved by the invention]

[0004] In a system such as that described in Patent Document 1, for example, it is conceivable to distribute video in a mode with sufficient delay to stabilize distribution. However, there are also cases where a small delay in the video, i.e., high real-timeness, is required, such as when a user of an external terminal receives route guidance while viewing video of the area ahead of the vehicle.

[0005] The present invention has been made in consideration of the above points, and aims to provide a video transmission device, a video output device, and a video distribution system that can appropriately share the situation around the vehicle between the operator of an external terminal that watches the video sent from the vehicle and the driver of the vehicle, while maintaining stability in the distribution of video. [Means for solving the problem]

[0006] The invention described in claim 1 is a video transmission device having a video acquisition unit that sequentially acquires video data showing an image of the surroundings of a moving body, a position information acquisition unit that acquires the position of the moving body, and a data transmission unit that encodes the video data in a delay mode based on the position of the moving body to generate video encoded data and transmits the video encoded data.

[0007] The invention described in claim 6 is a video output device having a video encoded data receiving unit that sequentially receives video encoded data captured and encoded by an imaging device that moves along with a moving body, a location information acquisition unit that acquires location information indicating the location of the moving body, and a video output unit that decodes the video encoded data in a delay mode based on the position of the moving body and outputs video.

[0008] The invention described in claim 9 is a video output system comprising: a video acquisition unit that sequentially acquires video data showing a video around a moving body; a first position information acquisition unit that acquires the position of the moving body and transmits position data indicating the position; a data transmission unit that encodes the video data in a delay mode based on the position of the moving body to generate video encoded data and transmits the video encoded data; a data transmission unit that transmits the video encoded data; a receiving unit that receives the video encoded data; a second position information acquisition unit that receives the position data and acquires the position of the moving body indicated by the position data; and a video output unit that decodes the video encoded data in a delay mode based on the position of the moving body and outputs video.

[0009] The invention described in claim 10 is a video transmission method executed by a video transmission device that transmits video from a moving body, characterized in that it includes a video acquisition step of sequentially acquiring video data showing video around the moving body, a location information acquisition step of acquiring the position of the moving body, and a data transmission step of encoding the video data in a delay mode based on the position of the moving body to generate video encoded data and transmitting the video encoded data.

[0010] The invention described in claim 11 is a video transmission program that causes a video transmission device that transmits video from a moving body to execute a video acquisition step of sequentially acquiring video data showing video around the moving body, a location information acquisition step of acquiring the position of the moving body, and a data transmission step of encoding the video data in a delay mode based on the position of the moving body to generate video encoded data and transmit the video encoded data.

[0011] The invention described in claim 12 is a computer-readable storage medium that stores a video transmission program that causes a video transmission device that transmits video from a moving body to execute a video acquisition step of sequentially acquiring video data showing video around the moving body, a location information acquisition step of acquiring the location of the moving body, and a data transmission step of encoding the video data in a delay mode based on the location of the moving body to generate video encoded data and transmit the video encoded data. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating a video distribution system according to a first embodiment of the present invention. [Figure 2] 1 is a perspective view of a front seat portion of a vehicle equipped with an in-vehicle device. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of an in-vehicle device. [Figure 4] FIG. 2 is a block diagram illustrating an example of a configuration of a server. [Figure 5]FIG. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an external device. [Figure 7] FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. [Figure 8] FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. [Figure 9] FIG. 4 is a flow diagram of an operation routine of the external device. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0013] [1. System Configuration] A video distribution system 100 according to a first embodiment of the present invention will be described below with reference to the accompanying drawings.

[0014] Fig. 1 shows a video distribution system 100 according to a first embodiment of the present invention. As shown in Fig. 1, the video distribution system 100 includes an in-vehicle device 10, a relay server 40, and an external device 70. Fig. 1 shows a case in which the in-vehicle device 10 is mounted on an automobile M, which is an example of a moving body. Fig. 1 also shows a smartphone as an example of the external device 70.

[0015] The in-vehicle device 10, the relay server 40, and the external device 70 can transmit and receive data to and from each other via a network NW using communication protocols such as TCP / IP, UDP / IP, etc. The network NW can be constructed using internet communication including wireless communication such as a mobile communication network, Wi-Fi (registered trademark), and wired communication.

[0016] In the video distribution system 100 of this embodiment, a voice call is established between the in-vehicle device 10 and the external device 70, and then the video captured in the automobile M is distributed from the in-vehicle device 10 to the external device 70. This communication mode in which the video captured in the automobile M is distributed from the in-vehicle device 10 to the external device 70 while the voice call is established between the in-vehicle device 10 and the external device 70 is referred to as video communication.

[0017] By performing such video communication, the user of the external device 70 watching the video transmitted from the in-vehicle device 10 can feel as if he or she is riding in the vehicle M with the driver of the vehicle M. In other words, the video communication can realize a virtual ride-along of the user of the external device 70 in the vehicle M. A system such as the video distribution system 100 of this embodiment that can realize such video communication is also called a virtual ride-along system.

[0018] In the following, in the first embodiment, the in-vehicle device 10 is described as a car navigation device. Also, in the first embodiment, the in-vehicle device 10 is described as a terminal device of a so-called cloud-type car navigation device, which receives a destination to which the user wants guidance from the user, transmits the destination to the server 40, and the server 40 generates a route to the destination.

[0019] Fig. 2 is a perspective view showing the vicinity of the front seat of an automobile M equipped with an in-vehicle device 10 as a video transmission device. Fig. 1 shows, as an example of installation, a case where the in-vehicle device 10 is installed inside a dashboard DB of the front seat of the automobile M.

[0020] The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is disposed, for example, on a dashboard DB. The GPS receiver 11 may be disposed anywhere as long as it can receive GPS signals. The GPS receiver 11 is capable of transmitting the received GPS signals to the in-vehicle device 10.

[0021] The exterior camera 12 serving as the photographing unit is an imaging device that photographs the area ahead of the automobile M. In this embodiment, the exterior camera 12 is disposed on the dashboard DB so that the photographing direction faces forward. For example, the exterior camera 12 is a wide-angle camera that can photograph a wide area ahead of the automobile M through the windshield.

[0022] The in-vehicle camera 13 is an imaging device that captures images of the interior of the automobile M. In this embodiment, the in-vehicle camera 13 is provided on the upper end of the windshield FG or on the ceiling near the upper end, and is capable of capturing images of the driver of the automobile M.

[0023] During video communication, video captured by the vehicle exterior camera 12 or the vehicle interior camera 13 is distributed to the external device 70. The following mainly describes the case where video captured by the vehicle exterior camera 12 is distributed to the external device 70.

[0024] The touch panel 14 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touchpad. The touch panel 14 is disposed, for example, on the center console CC of the dashboard DB. The touch panel 14 may be disposed in a location that is visible to the driver and within the driver's reach. For example, the touch panel 14 may be attached to the dashboard DB.

[0025] The touch panel 14 can display a screen based on the control of the in-vehicle device 10. The touch panel 14 can also transmit a signal representing an input operation to the touch panel 14 received from the user to the in-vehicle device 10. For example, the touch panel 14 may display car navigation guidance. Furthermore, operations related to the car navigation function, such as setting a destination, may be possible via the touch panel 14.

[0026] Furthermore, information about video communication or a screen for accepting operations to connect to video communication may be displayed on the touch panel 14. The passenger of the automobile M may perform an operation to connect to video communication by inputting an input operation to the touch panel 14.

[0027] The speaker 15 is provided, for example, on the interior side of the A-pillar AP. The speaker 15 is capable of emitting sounds such as music and voices under the control of the in-vehicle device 10. During video communication, the speaker 15 emits sound from the external device 70 during the voice call.

[0028] The microphone 17 is a microphone device that picks up sounds inside the vehicle and is arranged, for example, on the dashboard DB. The microphone 17 may be arranged anywhere, such as on the rearview mirror RM or the steering wheel, as long as it can pick up sounds inside the vehicle. During video communication, the sound picked up by the microphone 17 is transmitted to the external device 70 as the sound of the voice call.

[0029] 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a large-capacity storage device 23, a control unit 25, an input unit 27, an output unit 29, an encoder unit 30, and a data communication unit 31 cooperate with each other via a system bus 21.

[0030] The mass storage device 23 is configured with, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc., and stores various programs such as an operating system and terminal software, etc. The mass storage device 23 also holds map information including road maps.

[0031] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the mass storage device 23 (including a program for executing processing in the in-vehicle device 10, which will be described later) can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0032] The control unit 25 is configured with a CPU (Central Processing Unit) 25A, a ROM (Read Only Memory) 25B, a RAM (Random Access Memory) 25C, etc., and functions as a computer. The CPU 25A reads and executes various programs stored in the ROM 25B and the mass storage device 23, thereby realizing various functions. In this embodiment, the control unit 25 performs functions such as a video distribution function during video communication connection and a car navigation function.

[0033] The input unit 27 is an interface unit that communicatively connects the in-vehicle device 10 with the exterior camera 12, the interior camera 13, the touch panel 14, and the microphone 17. The in-vehicle device 10 can sequentially acquire images captured by the exterior camera 12 and the interior camera 13 via the input unit 27. In other words, the control unit 25 functions as an image acquisition unit that sequentially acquires image data showing images of the surroundings of the vehicle captured by the exterior camera 12 via the input unit 27.

[0034] The in-vehicle device 10 can receive, via the input unit 27, a signal indicating an input operation to the touch panel 14. For example, the in-vehicle device 10 can accept, via the input unit 27, a video communication connection request made by the user via the touch panel 14 and the microphone 17, or an input to set a destination for car navigation.

[0035] The input unit 27 is an interface unit that communicatively connects the in-vehicle device 10 and the GPS receiver 11. The in-vehicle device 10 receives a GPS signal from the GPS receiver 11 via the input unit 27 and can acquire information on the current position of the in-vehicle device 10, that is, the current position of the automobile M in this embodiment, from the GPS signal. In other words, the control unit 25 functions as a position information acquisition unit that acquires position information of the automobile M from the GPS receiver 11.

[0036] The output unit 29 is communicatively connected to the touch panel 14 and the speaker 15, and is capable of transmitting video or image signals to the touch panel 14 to display the same, and transmitting audio signals to the speaker 15 to output sound.

[0037] Encoder unit 30 is a part that encodes (hereinafter also referred to as encoding process) the video captured by camera 12 or camera 13 (also referred to as captured video) based on instructions from control unit 25. The encoder unit has a CPU for video encoding, a so-called GPU, and encoding may be performed by the GPU.

[0038] The encoder unit 30 generates encoded data as video encoded data by encoding the captured video using, for example, an encoding method conforming to the MPEG-4 standard. For example, the encoder unit 30 generates encoded data from the captured video using a codec such as H.264, Xvid, DivX, VP8, or VP9.

[0039] The data communication unit 31 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The data communication unit 31 also transmits and receives various data to and from the external device 70 via the server 40.

[0040] For example, the control unit 25 of the in-vehicle device 10 can transmit, via the data communication unit 31, to the server 40, location identification information as location data that can identify the current location of the in-vehicle device 10, that is, in this embodiment, information on the current location of the automobile M. Furthermore, for example, the control unit 25 can transmit, via the data communication unit 31, to the server 40, information including a destination input by a user, and receive, from the server 40, route information or navigation information to the destination.

[0041] The control unit 25 can transmit audio data of the audio picked up by the microphone 17 to the external device 70 via the data communication unit 31 for audio calls in video communication. The control unit 25 can also receive audio data of the audio input to the external device 70 via the data communication unit 31 for audio calls in video communication.

[0042] Furthermore, the control unit 25 transmits the encoded data encoded by the encoder unit 30 to the external device 70 via the data communication unit 31. The control unit 25 transmits the encoded data to the external device 70 via the data communication unit 31 while buffering it.

[0043] The control unit 25, the encoder unit 30, and the data communication unit 31 cooperate to function as an encoding / transmission unit 32 as a data transmission unit in the process of transmitting encoded data during video communication.

[0044] The encoding / transmission unit 32 encodes the captured video and transmits the encoded data in a plurality of different processing modes. The encoding / transmission unit 32 has a plurality of operation modes for the process from the encoder unit 30 encoding the video from the cameras 12 and 13 to the data communication unit 31 transmitting the encoded data (hereinafter also referred to as the encoding / transmission process). Specifically, for example, the encoding / transmission unit 32 has a normal mode as a first delay mode and a low delay mode as a second delay mode as operation modes for the encoding / transmission process.

[0045] In this embodiment, the low latency mode is an operating mode in which the time from when video is acquired from the camera to when encoded data of the video is transmitted via the data communication unit 31 is shorter than in the normal mode. In other words, the low latency mode is an operating mode in which there is less delay in delivering video to the external device 70 than in the normal mode. This low latency mode may be similar to the low latency modes in video delivery services such as YouTube (registered trademark) and Niconico Live Broadcast (registered trademark).

[0046] For example, when operating in low-latency mode, the data communication unit 31 operates with a smaller transmission buffer than when operating in normal mode, thereby reducing the time from acquiring video from the outside camera 12 to sending encoded data (hereinafter also simply referred to as transmission delay). Also, for example, when operating in low-latency mode, the encoder unit 30 performs encoding processing so that the frame rate and resolution are lower than when operating in normal mode, thereby shortening the time required for encoding processing and reducing transmission delay.

[0047] Furthermore, for example, when the encoder unit 30 operates in the low-delay mode, the in-vehicle device 10 may allocate more processing resources to the encoding process than in the normal mode, thereby speeding up the encoding process. For example, when the encoder unit 30 operates in the low-delay mode, the in-vehicle device 10 may allocate more resources of the CPU 25A to the encoding process than in the normal mode.

[0048] Furthermore, for example, when the in-vehicle device 10 operates the encoder unit 30 in the low-delay mode, the in-vehicle device 10 may use a codec that requires a shorter encoding process time than the codec used in the normal mode.

[0049] The in-vehicle device 10 changes the operation mode of the encoding / transmission unit 32 between the normal mode and the low-delay mode depending on the position of the automobile M indicated by the above-mentioned position specifying information.

[0050] For example, when an occupant of the automobile M (hereinafter simply referred to as an occupant) and a user of the external device 70 (hereinafter simply referred to as an external user) need to talk with each other about the scenery or conditions ahead of the automobile M with little time lag in the video, that is, when real-time video is required, the in-vehicle device 10 operates the encoding / transmission unit 32 in the low-delay mode. An example of an example of a conversation with little time lag is when the external user gives directions to the occupant while watching the video from the outside camera 12 displayed on the external device 70.

[0051] When an external user provides route guidance to a passenger, a time lag in the image can be a problem, particularly at important points where guidance is required, such as road intersections and forks, or at points such as intersections where the navigation device provides voice guidance when a guidance route is being generated (hereinafter simply referred to as guidance points). Specifically, the time lag in the image can cause a problem in that the external user may pass the guidance point before providing guidance, making it impossible to provide appropriate guidance.

[0052] Therefore, in this embodiment, the in-vehicle device 10 reduces video delay by operating the encoding transmission unit 32 in low-delay mode from the time the automobile M approaches a guidance point on the road until it passes the guidance point.

[0053] Generally, video streaming in the low latency mode described above is less stable than video streaming in the normal mode described above, because, for example, a small transmission buffer can cause even a slight delay in the encoding process to lead to data transmission interruptions.

[0054] In the in-vehicle device 10, the encoding / transmission unit 32 operates in low-latency mode only when real-time video is required. This allows stable video distribution under normal circumstances and highly real-time video distribution when necessary, thereby enabling smooth communication between external users and occupants, for example, when providing route guidance, while maintaining the stability of video distribution.

[0055] 3 is a block diagram showing the configuration of the server 40. For example, the server 40 is a device in which a mass storage device 43, a control unit 45, and a data communication unit 47 cooperate with each other via a system bus 41. The server 40 has a function similar to that of a SIP server that establishes a voice call between the in-vehicle device 10 and the external device 70 during video communication and transfers data of the voice call.

[0056] The server 40 also has the function of receiving from the in-vehicle device 10 the location identification information of the automobile M and information on the destination set by the user who is an occupant of the automobile M, and generating a route to the destination based on the location identification information and the destination information.

[0057] The server 40 also has a function of transferring the encoded data sent from the in-vehicle device 10 to the external device 70 .

[0058] The mass storage device 43 is configured by, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as an operating system and software for the server 40.

[0059] A map information database (shown as map information DB in the drawing) 43A in which map information including road maps is stored is also included in the large-capacity storage device 43. The map information in the map information database 55A is a database that contains information equivalent to the map information used in, for example, a navigation device.

[0060] The control unit 45 is configured with a CPU (Central Processing Unit) 45A, a ROM (Read Only Memory) 45B, a RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads and executes various programs stored in the ROM 45B and the large-capacity storage device 43, thereby realizing various functions.

[0061] The data communication unit 47 is connected to the network NW, and transmits and receives various data between the in-vehicle device 10 and the external device 70.

[0062] The control unit 45 acquires location identification information indicating the current location of the automobile M from the in-vehicle device 10 via the data communication unit 47. The control unit 45 also acquires destination information input to the in-vehicle device 10 by an occupant of the automobile M from the in-vehicle device 10 via the data communication unit 47. The control unit 45 generates a route to the destination based on the location identification information and the destination information, and transmits information indicating the route to the in-vehicle device 10.

[0063] The control unit 45 also transfers the encoded data and audio received from the in-vehicle device 10 to the external device 70 via the data communication unit 47. The control unit 45 transfers the audio received from the external device 70 to the in-vehicle device 10 via the data communication unit 47.

[0064] 4 is a front view showing the appearance of the external device 70. As described above, in the first embodiment, the external device 70 is a smartphone.

[0065] The touch panel 71 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touch pad. The touch panel 71 is capable of generating signals that represent input operations received from a user onto the touch panel 71. In this embodiment, images distributed from the in-vehicle device 10 are displayed on the touch panel 71.

[0066] Furthermore, information about video communication or a screen for accepting operations for connecting to video communication may be displayed on the touch panel 71. The user of the external device 70 may perform an operation for connecting to video communication by inputting information to the touch panel 71.

[0067] The speaker 73 is capable of emitting sounds such as music and voices. During video communication, the speaker 73 emits the sound from the in-car device 10 during voice communication.

[0068] The microphone 75 is a microphone device that receives sounds emitted toward the external device 70. During video communication, the sounds collected by the microphone 75 are transmitted to the external device 70 as the sounds of the voice call.

[0069] 6 is a block diagram showing the configuration of the external device 70. For example, the external device 70 is a device in which a large-capacity storage device 83, a control unit 84, an input unit 85, an output unit 86, a data communication unit 87, and a decoder unit 88 cooperate with each other via a system bus 81.

[0070] The mass storage device 83 is configured by, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc., and stores various programs such as an operating system and software for the terminal.

[0071] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. That is, the various programs stored in the mass storage device 83 (including a program for executing processing in the external device 70, which will be described later) can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0072] The control unit 84 is configured with a CPU (Central Processing Unit) 84A, a ROM (Read Only Memory) 84B, a RAM (Random Access Memory) 84C, etc., and functions as a computer. The CPU 84A reads and executes various programs stored in the ROM 84B and the large-capacity storage device 83, thereby realizing various functions.

[0073] The input unit 85 is an input interface unit for the touch panel 71 and the microphone 75. The control unit 84 can receive, via the input unit 85, signals indicating input operations to the touch panel 71 and audio input signals from the microphone 75. For example, the control unit 84 can accept, via the input unit 85, a connection request for a video communication connection made by the user via the touch panel 71 and the microphone 75.

[0074] The output unit 86 is an output interface to the touch panel 71 and the speaker 73. The control unit 84 can transmit a video or image signal to the touch panel 14 via the output unit 86 to display the image, or transmit an audio signal to the speaker 15 to output sound.

[0075] The data communication unit 87 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The data communication unit 87 also transmits and receives various data, including encoded data of video transmitted from the in-vehicle device 10, to and from the in-vehicle device 10 via the server 40. That is, the control unit 84 functions as an encoded data receiving unit that receives encoded data via the data communication unit 87.

[0076] For example, the control unit 84 of the external device 70 can receive, via the data communication unit 31, location identification information (location information) capable of identifying the current location of the automobile M transmitted from the in-vehicle device 10 from the server 40. Furthermore, for example, the control unit 84 can receive route information or navigation information for the automobile M from the server 40 via the data communication unit 87.

[0077] Furthermore, the control unit 84 can transmit audio data of the audio picked up by the microphone 75 to the in-vehicle device 10 via the data communication unit 87 for audio communication in video communication. Furthermore, the control unit 84 can receive audio data transmitted from the in-vehicle device 10 via the data communication unit 87 for audio communication in video communication.

[0078] The decoder unit 88 as a video output unit decodes, plays back, and outputs encoded data received from the in-vehicle device 10 based on commands from the control unit 84. The decoder unit 88 decodes the encoded data using a codec used when encoding the encoded data, for example, an encoding method conforming to the MPEG-4 standard, and plays back and outputs the video. The played-back video is displayed on the touch panel 71 by the control unit 84.

[0079] The decoder unit 88 decodes encoded data in a plurality of different operation modes. Specifically, for example, the decoder unit 88 has, as operation modes for decoding, a normal mode as a first delay mode and a low delay mode as a second delay mode.

[0080] In this embodiment, the low latency mode is an operation mode in which the time from receiving encoded data to reproducing the encoded data as video and displaying it on the touch panel 71 is shorter than in the normal mode. In other words, the low latency mode is an operation mode in which there is less delay in video reproduction on the external device 70 than in the normal mode. This low latency mode may be similar to the low latency modes in video distribution services such as YouTube (registered trademark) and Niconico Live Broadcast (registered trademark).

[0081] For example, when operating in low delay mode, the decoder unit 88 operates with a reduced playback buffer compared to when operating in normal mode, thereby reducing the time from receiving or acquiring encoded data to playing back video (hereinafter simply referred to as playback delay).

[0082] Furthermore, for example, when the decoder unit 88 operates in the low-delay mode, the external device 70 may allocate more processing resources to the decoding process than in the normal mode, thereby speeding up the decoding process. For example, when the decoder unit 88 operates in the low-delay mode, the external device 70 may allocate more resources of the CPU 84A to the decoding process than in the normal mode.

[0083] The external device 70 changes the operation mode of the encoding / transmission unit 32 between the normal mode and the low-delay mode depending on the position of the automobile M indicated by the above-mentioned position specifying information.

[0084] For example, similar to the operating mode of the encoding process of the in-vehicle device 10 described above, the external device 70 operates the encoding transmission unit 32 in a low-delay mode when the occupants of the automobile M and an external user need to talk about the scenery or conditions ahead of the automobile M with little video time lag.

[0085] As described in the description of the in-vehicle device 10, a situation in which it is necessary to talk with little time lag is, for example, when an external user is giving directions to the occupants while watching the image from the outside camera 12 displayed on the external device 70.

[0086] As described in the description of the in-vehicle device 10, when an external user provides route guidance to a passenger, a time lag in the image at the guidance point becomes a problem. Therefore, in this embodiment, the external device 70 reduces the delay in the image by operating the decoder unit 88 in low-delay mode from the time the automobile M approaches the guidance point on the road until it passes the guidance point.

[0087] Generally, video playback in the low latency mode described above is less stable than video playback in the normal mode described above, because, for example, a small playback buffer can cause even a slight delay in the decoding process to stop playback.

[0088] In the external device 70, the decoder unit 88 operates in low-delay mode only when real-time video is required. This allows stable video playback under normal circumstances and high-real-time video playback when necessary, thereby enabling smooth communication between external users and passengers when providing route guidance, for example, while maintaining the stability of video distribution.

[0089] [2. Operation of the video distribution system] The operation of the video distribution system 100 including the in-vehicle device 10, the server 40, and the external device 70 will be described below.

[0090] A description will be given of control routines of the in-vehicle device 10 and the external device 70 for realizing the operation of the system 100 of the first embodiment. In the following description, it is assumed that when video distribution starts through video communication, the encoding / transmission unit 32 operates in the normal mode, and encoding and transmission processes of the video start.

[0091] 7 is a flowchart showing a video distribution routine RT1 executed by the control unit 25 of the in-vehicle device 10. For example, when the in-vehicle device 10 is powered on, the control unit 25 starts the video distribution routine RT1 and repeatedly executes it.

[0092] First, the control unit 25 determines whether the in-vehicle device 10 has started video communication with the external device 70 (step S101). This determination is made, for example, based on whether video communication has been initiated in the in-vehicle device 10 and communication has been established with the external device 70. If the control unit 25 determines that video communication has not started (step S101: NO), it ends routine RT1.

[0093] When the control unit 25 determines that the in-vehicle device 10 has started video communication (step S101: YES), it acquires location information indicating the location of the automobile M based on a signal from the GPS receiver 11, acquires video from the exterior camera 12 or the interior camera 13, encodes the acquired video, and distributes it together with the location information to the external device 70 via the server 40, starting a video distribution operation (step S102).

[0094] In other words, in step S102, control unit 25 functions as an image acquisition unit that acquires image from camera 12 or camera 13. Also in step S102, control unit 25 functions as a location information acquisition unit that acquires location information indicating the location of automobile M. Also in step S102, control unit 25 functions as a data transmission unit that encodes image data and transmits the encoded encoded data.

[0095] Step S102 corresponds to the position information acquiring step, the image acquiring step, and the data transmitting step in the video transmission method, the program, and the storage medium of the present invention. Note that in step S102, the position information does not necessarily have to be transmitted to the external device 70.

[0096] When step S102 is executed and the video distribution operation is started, the control unit 25 determines whether the video communication between the in-car device 10 and the external device 70 has ended (step S103). If the control unit 25 determines that the video communication has not ended, that is, that the video communication is continuing (step S103: NO), it repeatedly executes step S103. That is, the video distribution operation continues as long as the video communication does not end.

[0097] When the control unit 25 determines in step S103 that the video communication with the external device 70 has ended, the control unit 25 ends the video distribution operation (step S104), and the routine R1 ends. In other words, when the control unit 25 determines that the video communication with the external device 70 has ended, the control unit 25 ends the acquisition of the location information, the acquisition of the video, and the encoding process and transmission of the video.

[0098] 8 is a flowchart showing the encoding / transmission process control routine RT2 executed by the control unit 25 of the in-vehicle device 10. The control unit 25 starts the encoding / transmission process control routine RT2, for example, when the in-vehicle device 10 is powered on, and executes this routine repeatedly. Note that in this embodiment, it is assumed that when a video distribution operation is started, encoding is initially performed in normal mode.

[0099] First, the control unit 25 determines whether the in-vehicle device 10 is performing a video communication video distribution operation with the external device 70 (step S201). If the control unit 25 determines that the in-vehicle device 10 is not performing a video communication video distribution operation (step S201: NO), the control unit 25 ends the routine RT2.

[0100] When the control unit 25 determines that the in-vehicle device 10 is performing a video communication video distribution operation (step S201: YES), it determines whether the automobile M is approaching or has arrived just before the guide point (step S202).

[0101] This determination may be made, for example, by determining whether or not the automobile M has reached a point on the map that is a predetermined distance before the guide point, based on map information and the current position of the automobile M. This determination may also be made, for example, by determining whether or not the automobile M has reached a point on the map where the arrival time to the guide point is a predetermined time, based on map information and the current position and current speed of the automobile M.

[0102] This determination may also be made based on an operation by the driver of the automobile M to the in-vehicle device 10. For example, before the automobile M approaches the guide point, the driver may perform an operation on the in-vehicle device 10, for example, via the touch panel 14 or the microphone 17, and when this operation is performed, it may be determined that the automobile M has arrived just before a predetermined guide point or a guide point to which the driver wishes to receive guidance. Similarly, this determination may be made based on an operation by the user of the external device 70 to the external device 70.

[0103] Furthermore, when the determination is made based on an operation by the driver or the user of the external device 70, for example, log data of the position of the automobile M at the time the operation was performed may be stored in the in-vehicle device 10 or the server 40. Using the log data, for example, it may be determined based on the point indicated by the log data whether the automobile M has arrived just before the guidance point.

[0104] Furthermore, based on the log data, it may be possible to use AI to determine whether the automobile M has reached a guide point. For example, a learning model may be constructed based on the log data, and the learning model may be used to determine whether the automobile M has reached a guide point.

[0105] The log data may be an image or video captured by the camera 12 at the time the operation was performed. Using such log data or a learning model based on the log data, it is possible to automatically determine from the video of the camera 12 whether or not the vehicle has reached the guide point.

[0106] When the control unit 25 determines that the automobile M is approaching or has arrived just before the guide point (step S202: YES), it switches the operation of the encoding / transmission unit 32 to the low-latency mode (step S203). When switching to the low-latency mode, the control unit 25 may transmit a low-latency mode switching signal to the external device 70 to notify that the operation of the encoding / transmission unit 32 has been switched to the low-latency mode.

[0107] When the control unit 25 determines that the automobile M is not approaching the guide point or has not yet reached the guide point (step S202: NO), the control unit 25 ends the routine RT2.

[0108] After executing step S203, the control unit 25 determines whether the automobile M has passed the guide point (step S204). This determination may be made, for example, by determining whether the automobile M has passed the guide point on the map based on map information and the current position of the automobile M.

[0109] If the control unit 25 determines that the automobile M has not passed the guide point (step S204: NO), it repeatedly executes step S204. That is, the encoding / transmission unit 32 operates in the low-delay mode until the automobile M passes the guide point.

[0110] When the control unit 25 determines that the automobile M has passed the guidance point (step S204: YES), it switches the operation of the encoding / transmission unit 32 to the normal mode (step S205), and then ends routine RT2. Note that when switching to the normal mode, the control unit 25 may transmit a normal mode switching signal to the external device 70 to notify that the operation of the encoding / transmission unit 32 has been switched to the normal mode.

[0111] 9 is a flowchart showing the decoding process control routine RT3 executed by the control unit 84 of the external device 70. The control unit 84 starts the decoding process control routine RT3 when the external device 70 is powered on, for example, and executes this routine repeatedly. Note that in this embodiment, the decoding process is initially performed in normal mode when video playback operation is started.

[0112] First, the control unit 84 determines whether the external device 70 is in video communication with the in-vehicle device 10 and is playing a video distributed from the in-vehicle device 10 (step S301). If the control unit 84 determines that the external device 70 is in video communication and is not playing a video (step S301: NO), it ends routine RT3.

[0113] When the control unit 84 determines that the external device 70 is engaged in video communication and is playing back a video stream being distributed from the in-vehicle device 10 (step S301: YES), it determines whether the automobile M is approaching the guidance point or has arrived just before the guidance point (step S302).

[0114] This determination may be made, for example, by determining whether or not the automobile M has reached a point on the map that is a predetermined distance before the guide point, based on map information that is stored by the automobile M itself or obtained from an external device such as the server 40, and the current position of the automobile M. This determination may also be made, for example, by determining whether or not the automobile M has reached a point on the map at which the arrival time to the guide point is a predetermined time, based on map information and the current position and current speed of the automobile M.

[0115] This determination may be made based on the low-latency mode switching signal from the in-vehicle device 10. For example, in this determination, the control unit 84 may determine that the automobile M is approaching or has arrived just before the guidance point when the low-latency mode switching signal is received.

[0116] When the control unit 84 determines that the automobile M is approaching or has arrived just before the guide point (step S302: YES), it switches the operation of the decoder unit 88 to the low delay mode (step S303).

[0117] When the control unit 84 determines that the automobile M is not approaching the guide point or has not yet reached the guide point (step S302: NO), the control unit 84 ends the routine RT3.

[0118] After executing step S303, the control unit 84 determines whether the automobile M has passed the guidance point (step S304). This determination may be made, for example, by determining whether the automobile M has passed the guidance point on the map based on map information and the current position of the automobile M.

[0119] This determination may be made based on the normal mode switching signal from the above-mentioned in-vehicle device 10. For example, in this determination, the control unit 84 may determine that the automobile M has passed the guidance point when the normal mode switching signal is received.

[0120] When the control unit 84 determines that the automobile M has not passed the guide point (step S304: NO), it repeatedly executes step S304. That is, the decoder unit 88 operates in the low-delay mode until the automobile M passes the guide point.

[0121] When the control unit 84 determines that the automobile M has passed the guidance point (step S304: YES), it switches the operation of the decoder unit 88 to the normal mode (step S305), and then ends the routine RT3.

[0122] In the above embodiment, the in-vehicle device 10 is an in-vehicle navigation device, but the in-vehicle device 10 may be various terminals such as a smartphone or a tablet.

[0123] The configurations, routines, etc. of the in-vehicle device 10, the server 40, and the external device 70 in the above-described embodiments are merely illustrative and can be appropriately selected or changed depending on the application, etc. For example, the determination of whether the vehicle is approaching the guidance point in steps S202 and S302 of routines RT2 and RT3, and the determination of steps S204 and S304 may be performed by any of the in-vehicle device 10, the server 40, and the external device 70.

[0124] For example, when these steps are performed by the server 40, the judgment results of each step may be notified to the in-vehicle device 10 and the external device 70, and the in-vehicle device 10 and the external device 70 may switch between the normal mode and the low-latency mode accordingly.

[0125] Also, for example, if these steps are performed by the external device 70, the judgment results of each step may be notified to the in-vehicle device 10, and the in-vehicle device 10 may switch between the normal mode and the low-latency mode accordingly.

[0126] In the above embodiment, the in-vehicle device 10 and the external device 70 perform video communication via the server 40. However, the video communication may be performed directly between the in-vehicle device 10 and the external device 70.

[0127] In the above embodiment, it is determined whether the vehicle M is approaching a guide point based on the position information and map information, and the operating mode of the encoding / transmission unit 32 or the decoder unit 88 is switched to the low-latency mode based on the determination result. However, it may also be determined whether the vehicle M is approaching a guide point based on the map information by performing image recognition on the video from the exterior camera 12.

[0128] In the above embodiment, the operating mode of the encoding / transmission unit 32 or the decoder unit 88 is switched to low-latency mode when the vehicle approaches a guidance point. However, the operating mode may be switched to low-latency mode even when real-time video is required, even if the vehicle is not approaching a guidance point. For example, when approaching a scenic spot, the vehicle may be switched to low-latency mode if the occupant and an external user need to talk about the scenery while sharing high-real-time video. Furthermore, when the video communication mode is switched from a video sharing mode to a guidance mode in which the party viewing the video provides guidance, the video may always be shared in low-latency mode in the guidance mode regardless of the location information.

[0129] In the above embodiment, an example has been described in which the in-vehicle device 10 is mounted on an automobile M, but the in-vehicle device 10 may be mounted on other moving objects such as a bicycle or a motorcycle. Also, the in-vehicle device may be held by a person, and video communication may be performed while the person is walking, for example, to distribute video.

[0130] In the above embodiment, the in-vehicle device 10 starts video distribution operation after communication between the in-vehicle device 10 and the external device 70 is established. However, video distribution may be performed in a manner similar to the video distribution of YouTube (registered trademark) or Niconico Live Broadcast (registered trademark) described above. That is, the video distribution operation may be started even if communication with a viewer's terminal such as the external device 70 is not established. Specifically, the in-vehicle device 10 may start uploading video data to the server 40 even if communication between the in-vehicle device 10 and the viewer's terminal such as the external device 70 is not established.

[0131] For example, the video distribution operation by the in-vehicle device 10 may be started after communication between the in-vehicle device 10 and the server 40 is established, without establishing a communication connection with the external device 70. In this case, an unspecified or authorized specific external device 70 can connect to the server 40 to receive the video distributed from the in-vehicle device 10, and the user of the external device 70 can watch the video.

[0132] In this embodiment, the in-vehicle device 10 and the external device 70 perform video communication via the server 40, but the video communication may also be performed directly between the in-vehicle device 10 and the external device 70 by P2P (Peer to Peer) communication or the like, without going through the server 40.

[0133] In this embodiment, the case where the in-vehicle device 10 is connected to the touch panel 14 has been described, but the present invention is not limited to this. For example, the in-vehicle device 10 may communicate with a smartphone carried by the driver of the automobile M, and display a screen related to video communication or the like on the display of the smartphone instead of the touch panel 14.

[0134] Furthermore, the in-vehicle device 10 may be configured not to display a screen to be presented to the driver of the automobile M. For example, the in-vehicle device 10 may have a configuration similar to that of a drive recorder, and may be a device integrated with the exterior camera 12. Specifically, the in-vehicle device 10 may be a device in which hardware that performs the video communication function of the in-vehicle device 10 is built into the housing of the exterior camera 12. In this case, the in-vehicle device 10 may not perform the various display outputs described above.

[0135] Furthermore, the video transmission device of the present invention may be configured by integrating a terminal device having a configuration similar to that of the in-vehicle device 10 of this embodiment with the outside-vehicle camera 12 and the touch panel 14. Specifically, for example, the video transmission device of the present invention may be a smartphone, tablet, or PC with a camera that is equipped with an app that performs the same functions as the in-vehicle device 10.

[0136] In this embodiment, a case has been described in which an image of the outside of the automobile M is transmitted from the in-vehicle device 10 to the external device 70, but the image transmitted from the in-vehicle device 10 to the external device 70 may be switchable from an image of the outside of the automobile M to an image of the inside of the automobile M captured by the in-vehicle camera 13. When an image of the inside of the automobile M is being transmitted, the user of the external device 70 can, for example, communicate with the driver of the automobile M while viewing the interior of the automobile M.

[0137] The switching operation for switching the video to be transmitted to the external device 70 between the video from the exterior camera 12 and the video from the interior camera 13 may be performed by the in-vehicle device 10. The switching operation may also be performed remotely by a user of the external device 70 operating the external device 70.

[0138] In this embodiment, the control unit 84 of the external device 70 receives the image of the outside of the automobile M transmitted from the in-vehicle device 10 via the server 40, but in addition to the image, it may also receive a map image showing the current location of the automobile M and the planned route of travel, the name of the driver of the automobile M, the traveling speed of the automobile M, etc.

[0139] For example, the control unit 84 of the external device 70 may display on the touch panel 71 a map image, the name of the driver of the automobile M, and the traveling speed of the automobile M received from the in-vehicle device 10, superimposed on an image of the exterior of the automobile M, or may display them in a display area different from the image of the exterior of the automobile M. In this case, the map image, the name of the driver of the automobile M, and the traveling speed of the automobile M may be freely switchable between display and non-display by the operator of the external device 70. [Explanation of symbols]

[0140] 100 Video Distribution System 10 Onboard equipment 25, 84 Control section 30 Encoder section 31, 87 Data Communications Department 88 Decoder section 40 Relay Server 70 External device

Claims

1. an image acquisition unit that sequentially acquires image data showing an image of the surroundings of the moving object; a location information acquisition unit that acquires the location of the moving object; a data transmitting unit that generates encoded video data by encoding the video data in a delay mode based on the position of the moving object, and transmits the encoded video data; A video transmission device comprising:

2. the data transmission unit has, as operation modes, a first delay mode and a second delay mode in which a time from when the video data is acquired until when the data transmission unit transmits the video data is shorter than that in the first delay mode; 2. The video transmission device according to claim 1, wherein the data transmission unit operates in the second delay mode when the position indicates that the mobile object is approaching a guide point on a guide route.

3. 3. The video transmission device according to claim 2, wherein the transmission buffer when the data transmission unit operates in the second delay mode is smaller than the transmission buffer when the data transmission unit operates in the first delay mode.

4. 4. The video transmission device according to claim 2, wherein the frame rate of the video encoded data generated by the data transmission unit in the second delay mode is lower than the frame rate of the video encoded data output by the data transmission unit in the first delay mode.

5. 5. The video transmitting device according to claim 2, wherein the resolution of the video encoded data generated by the data transmitting unit in the second delay mode is smaller than the resolution of the video encoded data output by the data transmitting unit in the first delay mode.

6. an encoded data receiving unit that sequentially receives encoded video data captured and encoded by an imaging device that moves with the moving object; a location information acquisition unit that acquires location information indicating the location of the moving object; a video output unit that decodes the encoded video data in a delay mode based on the position of the moving object and outputs the video.

7. the video output unit has, as operation modes, a first delay mode and a second delay mode in which a time from acquisition of the encoded video data to output of the video is shorter than that in the first delay mode; 7. The video output device according to claim 6, wherein the video output unit operates in the second delay mode when the position indicates that the moving object is approaching a guide point on a guide route.

8. 8. The video output device according to claim 7, wherein the playback buffer when the video output unit operates in the second delay mode is smaller than the playback buffer when the video output unit operates in the first delay mode.

9. an image acquisition unit that sequentially acquires image data showing an image of the surroundings of the moving object; a first location information acquisition unit that acquires the location of the mobile object and transmits location data indicating the location; a data transmitting unit that generates encoded video data by encoding the video data in a delay mode based on the position of the mobile unit, and transmits the encoded video data; a receiving unit that receives the encoded video data; a second location information acquisition unit that receives the location data and acquires the location of the moving object indicated by the location data; a video output unit that decodes the encoded video data in a delay mode based on the position of the moving object and outputs the video.

10. A video transmission method executed by a video transmission device that transmits video from a mobile object, comprising: an image acquisition step of sequentially acquiring image data showing an image of the surroundings of the moving object; a position information acquisition step of acquiring a position of the moving body; a data transmitting step of generating encoded video data by encoding the video data in a delay mode based on the position of the mobile unit and transmitting the encoded video data; A video transmission method comprising:

11. A video transmission device that transmits video from a mobile object, an image acquisition step of sequentially acquiring image data showing an image of the surroundings of the moving object; a position information acquisition step of acquiring a position of the moving body; a data transmitting step of generating encoded video data by encoding the video data in a delay mode based on the position of the mobile unit and transmitting the encoded video data; A video transmission program characterized by causing the program to execute the above.

12. A video transmission device that transmits video from a mobile object, an image acquisition step of sequentially acquiring image data showing an image of the surroundings of the moving object; a position information acquisition step of acquiring a position of the moving body; a data transmitting step of generating encoded video data by encoding the video data in a delay mode based on the position of the mobile unit and transmitting the encoded video data; A computer-readable storage medium storing a video transmission program for executing the above.

Images