Video transmission device, video output device, video output system, video transmission method, video transmission program, and storage medium

Abstract

To provide a video transmission device, a video output device, and a video distribution system capable of appropriately sharing the situation around a vehicle between an operator of a terminal of a vehicle who watches a video sent from the vehicle and a driver of the vehicle while maintaining the stability of video distribution.SOLUTION: A video transmission device according to the present invention includes a video acquisition unit that sequentially acquires video data representing a video around a moving object, a position information acquisition unit that acquires the position of the moving body, and a data transmission unit that encodes the video data in a delay mode based on the position of the moving body to generate encoded video data and transmit the encoded video data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a video transmission device, a video output device, a video output system, a video transmission method, a video transmission program, and a storage medium. For example, it relates to a video transmission device, a video output device, a video distribution system, a video transmission method, a video transmission program, and a storage medium for providing video from a moving body to a user.

Background Art

[0002] There is a communication system that communicates between an in-vehicle device mounted on a vehicle and an out-of-vehicle terminal located outside the vehicle. For example, Patent Document 1 discloses a system in which a voice call is made between a driver of a vehicle and an operator of an out-of-vehicle terminal, and when the voice call is being made, video data representing the video in front of the vehicle is transmitted from the in-vehicle device to the out-of-vehicle terminal and the video is displayed on the out-of-vehicle terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a system such as Patent Document 1, for example, it is conceivable to distribute video in a mode with a sufficient delay in order to stabilize the distribution. However, there are cases where it is necessary that the delay of the video is small, that is, the real-time property is high, such as when the user of the out-of-vehicle terminal guides the way while viewing the video in front of the vehicle.

[0005] The present invention has been made in view of the above-mentioned points, and aims to provide a video transmission device, a video output device, and a video distribution system that can appropriately share the situation around the vehicle between the operator of an external terminal viewing video transmitted from the vehicle and the driver of the vehicle, while maintaining the stability of video distribution. [Means for solving the problem]

[0006] The invention described in claim 1 is a video transmission device comprising: a video acquisition unit that sequentially acquires video data showing images of the surroundings of a moving object; a position information acquisition unit that acquires the position of the moving object; and a data transmission unit that encodes the video data in a delayed mode based on the position of the moving object to generate video encoded data and transmits the video encoded data.

[0007] The invention described in claim 6 is a video output device comprising: a video encoded data receiving unit that sequentially receives video encoded data captured and encoded by an imaging device that moves together with a moving object; a position information acquisition unit that acquires position information indicating the position of the moving object; and a video output unit that decodes the video encoded data in a delayed mode based on the position of the moving object and outputs a video.

[0008] Furthermore, the invention described in claim 9 is a video output system characterized by comprising: a video acquisition unit that sequentially acquires video data showing images of the surroundings of a moving object; a first position information acquisition unit that acquires the position of the moving object and transmits position data indicating the position; a data transmission unit that encodes the video data in a delayed mode based on the position of the moving object to generate video encoded data and transmits the video encoded data; a data transmission unit that transmits the video encoded data; a receiving unit that receives the video encoded data; a second position information acquisition unit that receives the position data and acquires the position of the moving object indicated by the position data; and a video output unit that decodes the video encoded data in a delayed mode based on the position of the moving object and outputs an image.

[0009] Furthermore, the invention described in claim 10 is a video transmission method performed by a video transmission device that transmits video from a moving body, characterized by comprising: a video acquisition step of sequentially acquiring video data showing video of the surroundings of the moving body; a position information acquisition step of acquiring the position of the moving body; and a data transmission step of encoding the video data in a delayed mode based on the position of the moving body to generate video encoded data and transmitting the video encoded data.

[0010] Furthermore, the invention described in claim 11 is a video transmission program characterized by causing a video transmission device that transmits video from a moving object to execute a video acquisition step of sequentially acquiring video data showing video of the surroundings of the moving object, a position information acquisition step of acquiring the position of the moving object, and a data transmission step of encoding the video data in a delayed mode based on the position of the moving object to generate video encoded data and transmitting the video encoded data.

[0011] Furthermore, the invention described in claim 12 is a computer-readable storage medium that stores a video transmission program that causes a video transmission device that transmits video from a moving object to execute a video acquisition step of sequentially acquiring video data showing video of the surroundings of the moving object, a position information acquisition step of acquiring the position of the moving object, and a data transmission step of encoding the video data in a delayed mode based on the position of the moving object to generate video encoded data and transmitting the video encoded data. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram of a video distribution system which is one embodiment of the present invention. [Figure 2] This is a perspective view of the front seat area of ​​a vehicle equipped with an in-vehicle device. [Figure 3] This is a block diagram showing an example of the configuration of an in-vehicle device. [Figure 4] An example of a server configuration is shown in the block diagram. [Figure 5]This is a front view of the external device. [Figure 6] This is a block diagram showing an example of the configuration of an external device. [Figure 7] This is a flowchart of the operating routine of the in-vehicle device. [Figure 8] This is a flowchart of the operating routine of the in-vehicle device. [Figure 9] This is a flowchart of the operating routine of an external device. [Modes for carrying out the invention] [Examples]

[0013] [1. System Configuration] Below, a video distribution system 100, which is Embodiment 1 of the present invention, will be described with reference to the attached drawings.

[0014] Figure 1 shows a video distribution system 100, which is an embodiment 1 of the present invention. As shown in Figure 1, the video distribution system 100 is composed of an in-vehicle device 10, a relay server 40, and an external device 70. In Figure 1, the in-vehicle device 10 is shown mounted on an automobile M, which is an example of a mobile device. Also in Figure 1, a smartphone is shown as an example of the external device 70.

[0015] The in-vehicle device 10, the relay server 40, and the external device 70 can send and receive data from each other via a network NW using communication protocols such as TCP / IP and UDP / IP. The network NW can be constructed using, for example, a mobile communication network, wireless communication such as Wi-Fi (registered trademark), and internet communication including wired communication.

[0016] In the video distribution system 100 of this embodiment, after a voice call is established between the in-vehicle device 10 and the external device 70, the video captured in the vehicle M from the in-vehicle device 10 is distributed to the external device 70. In this way, the communication mode of distributing the video captured in the vehicle M from the in-vehicle device 10 to the external device 70 while establishing a voice call between the in-vehicle device 10 and the external device 70 is called video communication.

[0017] By performing such video communication, the user of the external device 70 who is viewing the video transmitted from the in-vehicle device 10 can obtain a feeling as if he / she is riding in the vehicle M with the driver of the vehicle M. In other words, video communication can realize the virtual ride of the user of the external device 70 in the vehicle M. Also, a system such as the video distribution system 100 of this embodiment that can realize such video communication is also called a virtual ride system.

[0018] Hereinafter, in the first embodiment, the case where the in-vehicle device 10 is a car navigation device will be described as an example. Also, in the first embodiment, the case where the in-vehicle device 10 is a terminal device of a so-called cloud-type car navigation device that receives a destination desired by the user from the user, transmits the destination to the server 40, and the server 40 generates a route to the destination will be described as an example.

[0019] FIG. 2 is a perspective view showing the vicinity of the front seat of the vehicle M equipped with the in-vehicle device 10 as a video transmission device. In FIG. 1, as an attachment example, the case where the in-vehicle device 10 is attached in the dashboard DB of the front seat of the vehicle M is shown.

[0020] The GPS receiver 11 is a device that receives a signal (GPS signal) from GPS (Global Positioning System) satellites. The GPS receiver 11 is arranged, for example, on the dashboard DB. Note that the GPS receiver 11 may be arranged anywhere as long as a GPS signal can be received. The GPS receiver 11 can transmit the received GPS signal to the in-vehicle device 10.

[0021] The external camera 12, acting as the imaging unit, is an imaging device that captures images of the area in front of the vehicle M. In this embodiment, the external camera 12 is positioned on the dashboard DB so that the forward direction is the shooting direction. For example, the external camera 12 is a wide-angle camera, capable of capturing a wide area in front of the vehicle M through the windshield.

[0022] The in-vehicle camera 13 is an imaging device that captures images of the interior of the automobile M. In this embodiment, the in-vehicle camera 13 is installed at the upper edge of the windshield FG or on the ceiling near the upper edge, and is capable of capturing images of the driver of the automobile M.

[0023] During video communication, the video captured by the external camera 12 or the internal camera 13 is distributed to the external device 70. The following description mainly explains the case where the video from the external camera 12 is distributed to the external device 70.

[0024] The touch panel 14 is a touch panel monitor that combines a display, such as an LCD capable of displaying images, with a touchpad. The touch panel 14 is located, for example, on the center console CC of the dashboard DB. The touch panel 14 only needs to be located in a place that is visible to the driver and within the driver's reach. For example, the touch panel 14 may be mounted on the dashboard DB.

[0025] The touch panel 14 can display screen information based on the control of the in-vehicle device 10. The touch panel 14 can also transmit signals to the in-vehicle device 10 representing input operations received from the user. For example, the touch panel 14 may display navigation directions. Furthermore, it may be possible to perform operations related to the navigation function, such as setting a destination, via the touch panel 14.

[0026] Furthermore, the touch panel 14 may display information related to video communication or a screen for accepting operations to connect to video communication. The occupant of the vehicle M may perform the video communication connection operation by inputting information on the touch panel 14.

[0027] The speaker 15 is installed, for example, on the interior side of the A-pillar AP. The speaker 15 is capable of emitting sounds such as music and voices based on the control of the in-vehicle device 10. During video communication, the speaker 15 emits voice from the external device 70 for voice calls.

[0028] Microphone 17 is a microphone device that receives sounds from inside the vehicle and is located, for example, on the dashboard DB. Microphone 17 can be installed anywhere, such as on the rearview mirror RM or the steering wheel, as long as it can receive sounds from inside the vehicle. During video communication, the audio picked up by microphone 17 is transmitted to the external device 70 as the audio for the voice call.

[0029] Figure 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a large-capacity storage device 23, a control unit 25, an input unit 27, an output unit 29, an encoder unit 30, and a data communication unit 31 cooperate via a system bus 21.

[0030] The large-capacity storage device 23 is composed of, for example, a hard disk drive, an SSD (solid state drive), flash memory, etc., and stores various programs such as the operating system and terminal software. The large-capacity storage device 23 also holds map information, including road maps.

[0031] Various programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, various programs stored in the mass storage device 23 (including programs for executing processing in the in-vehicle device 10 described later) can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred.

[0032] The control unit 25 is composed of a CPU (Central Processing Unit) 25A, ROM (Read Only Memory) 25B, RAM (Random Access Memory) 25C, etc., and functions as a computer. The CPU 25A reads and executes various programs stored in the ROM 25B and the mass storage device 23 to realize various functions. In this embodiment, the control unit 25 provides functions such as video distribution when connected to video communication and car navigation.

[0033] The input unit 27 is an interface unit that enables communication between the in-vehicle device 10 and the exterior camera 12, interior camera 13, touch panel 14, and microphone 17. The in-vehicle device 10 can sequentially acquire video footage captured by the exterior camera 12 and interior camera 13 via the input unit 27. In other words, the control unit 25 functions as a video acquisition unit that sequentially acquires video data showing the surroundings of the vehicle captured by the exterior camera 12 via the input unit 27.

[0034] The in-vehicle device 10 can receive signals indicating input operations to the touch panel 14 via the input unit 27. For example, the in-vehicle device 10 can receive video communication connection requests and destination setting inputs for the car navigation system from the user via the touch panel 14 and microphone 17 via the input unit 27.

[0035] Furthermore, the input unit 27 is an interface unit that enables communication between the in-vehicle device 10 and the GPS receiver 11. The in-vehicle device 10 receives GPS signals from the GPS receiver 11 via the input unit 27 and can acquire the current location of the in-vehicle device 10, that is, the current location of the automobile M in this embodiment, from the GPS signals. In other words, the control unit 25 functions as a location information acquisition unit that acquires the location information of the automobile M from the GPS receiver 11.

[0036] The output unit 29 is connected to the touch panel 14 and the speaker 15 in a communicative manner, and can transmit video or image signals to the touch panel 14 for display, or transmit audio signals to the speaker 15 for sound output.

[0037] The encoder unit 30 is the part that encodes (hereinafter also referred to as encoding processing) the video captured by the camera 12 or camera 13 (also referred to as captured video) based on commands from the control unit 25. The encoder unit has a CPU for video encoding, so-called GPU, and encoding may be performed by the GPU.

[0038] The encoder unit 30 encodes the captured video using, for example, the MPEG-4 encoding method to generate encoded data as video encoded data. For example, the encoder unit 30 generates encoded data from the captured video using codecs such as H.264, Xvid, DivX, VP8, and VP9.

[0039] The data communication unit 31 is connected to the network NW described above and sends and receives various data to and from the server 40. The data communication unit 31 also sends and receives various data to and from an external device 70 via the server 40.

[0040] For example, the control unit 25 of the in-vehicle device 10 can transmit location identification information, which is location data that can identify the current location of the in-vehicle device 10, to the server 40 via the data communication unit 31. In addition, for example, the control unit 25 can transmit information including a destination entered by the user to the server 40 via the data communication unit 31 and receive route information or navigation information to that destination from the server 40.

[0041] The control unit 25 can transmit audio data of voice picked up by the microphone 17 to an external device 70 via the data communication unit 31 for voice communication in video communication. The control unit 25 can also receive audio data of voice input to the external device 70 via the data communication unit 31 for voice communication in video communication.

[0042] Furthermore, the control unit 25 transmits the encoded data encoded by the encoder unit 30 to the external device 70 via the data communication unit 31. The control unit 25 transmits the encoded data to the external device 70 via the data communication unit 31 while buffering it.

[0043] The control unit 25, the encoder unit 30, and the data communication unit 31 cooperate in the transmission process of encoded data during video communication to function as an encoded transmission unit 32.

[0044] The encoding transmission unit 32 encodes captured video and transmits the encoded data using multiple different processing modes. The encoding transmission unit 32 has multiple operating modes for the process from when the encoder unit 30 encodes the video from cameras 12 and 13 until the data communication unit 31 transmits the encoded data (hereinafter also referred to as the encoding transmission process). Specifically, for example, the encoding transmission unit 32 has a normal mode as the first delay mode and a low-latency mode as the second delay mode as operating modes for the encoding transmission process.

[0045] In this embodiment, the low-latency mode is an operating mode in which the time from when video is acquired from the camera until the encoded data of the video is transmitted via the data communication unit 31 is shorter than in the normal mode. In other words, the low-latency mode is an operating mode that reduces the delay in video distribution to the external device 70 compared to the normal mode. This low-latency mode may be the same as the low-latency mode used in video distribution services such as YouTube® and Nico Nico Live Broadcasting®.

[0046] For example, when the data communication unit 31 operates in low-latency mode, it reduces the transmission buffer compared to when it operates in normal mode, thereby reducing the time from acquiring video from the external camera 12 to sending encoded data (hereinafter also simply referred to as transmission delay). Also, for example, when the encoder unit 30 operates in low-latency mode, it reduces the time required for encoding processing and thus reduces transmission delay by performing encoding processing at a lower frame rate and resolution than when it operates in normal mode.

[0047] Furthermore, for example, when the in-vehicle device 10 operates the encoder unit 30 in low-latency mode, it may speed up the encoding process by allocating more processing resources to the encoding process than in normal mode. For example, when the in-vehicle device 10 operates the encoder unit 30 in low-latency mode, it may allocate more resources of the CPU 25A to the encoding process than in normal mode.

[0048] Furthermore, for example, when the in-vehicle device 10 operates the encoder unit 30 in low-latency mode, it may use a codec that shortens the encoding processing time compared to the codec used in normal mode.

[0049] The in-vehicle device 10 changes the operating mode of the encoded transmission unit 32 between normal mode and low-latency mode according to the position of the automobile M indicated by the position identification information described above.

[0050] For example, the in-vehicle device 10 operates the encoding and transmission unit 32 in low-latency mode when the occupants of the vehicle M (hereinafter simply referred to as "occupants") and the user of the external device 70 (hereinafter simply referred to as "external user") need to talk about the scenery or conditions in front of the vehicle M with minimal video lag, that is, when real-time video is required. An example of a situation where they need to talk with minimal time lag is when the external user gives directions to the occupants while viewing the video from the external camera 12 displayed on the external device 70.

[0051] When an external user provides directions to the occupants, a time lag in the video becomes a problem, particularly at points where guidance is necessary or important, such as road intersections and junctions, or at points where the navigation system provides voice guidance when a guidance route has been generated (hereinafter also simply referred to as guidance points). Specifically, the time lag in the video can lead to problems such as the occupants passing the guidance point before the external user can provide guidance, making it impossible to provide appropriate directions.

[0052] Therefore, in this embodiment, the in-vehicle device 10 reduces video delay by operating the encoding transmission unit 32 in low-latency mode from the time the vehicle M approaches a road guidance point until it passes the guidance point.

[0053] Generally speaking, video streaming using the low-latency mode described above is less stable than video streaming using the normal mode described above. This is because, for example, by reducing the transmit buffer, even a slight delay in encoding processing can lead to the interruption of transmitted data.

[0054] In the above-described in-vehicle device 10, the encoding and transmission unit 32 operates in low-latency mode only when real-time video is required. This enables stable video distribution under normal circumstances, while also allowing for highly real-time video distribution when necessary. For example, it enables smooth communication between external users and occupants, such as during navigation, while maintaining the stability of video distribution.

[0055] Figure 3 is a block diagram showing the configuration of server 40. For example, server 40 is a device in which a mass storage device 43, a control unit 45, and a data communication unit 47 cooperate via a system bus 41. Server 40 has a function like a SIP server that establishes a voice call between the in-vehicle device 10 and an external device 70 during video communication and transfers the data of the voice call.

[0056] Furthermore, the server 40 has the function of receiving location information of the vehicle M and destination information set by the user, who is an occupant of the vehicle M, from the in-vehicle device 10, and generating a route to the destination based on the location information and destination information.

[0057] Furthermore, the server 40 has the function of transferring encoded data sent from the in-vehicle device 10 to the external device 70.

[0058] The large-capacity storage device 43 is composed of, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as the operating system and software for the server 40.

[0059] Furthermore, the large-capacity storage device 43 contains a map information database (indicated as map information DB in the figure) 43A, which stores map information including road maps. The map information in the map information database 55A is a database that contains information equivalent to, for example, the map information used in a navigation device.

[0060] The control unit 45 is composed of a CPU (Central Processing Unit) 45A, a ROM (Read Only Memory) 45B, a RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads and executes various programs stored in the ROM 45B and the mass storage device 43 to realize various functions.

[0061] The data communication unit 47 is connected to the network NW described above and transmits and receives various data between the in-vehicle device 10 and the external device 70.

[0062] The control unit 45 obtains location information indicating the current location of the vehicle M from the in-vehicle device 10 via the data communication unit 47. The control unit 45 also obtains destination information entered into the in-vehicle device 10 by the occupants of the vehicle M via the data communication unit 47. Based on the location information and destination information, the control unit 45 generates a route to the destination and transmits information indicating the route to the in-vehicle device 10.

[0063] Furthermore, the control unit 45 transfers the encoded data and audio received from the in-vehicle device 10 to the external device 70 via the data communication unit 47. The control unit 45 also transfers the audio received from the external device 70 to the in-vehicle device 10 via the data communication unit 47.

[0064] Figure 4 is a front view showing the external appearance of the external device 70. As described above, in Embodiment 1, the external device 70 is a smartphone.

[0065] The touch panel 71 is a touch panel monitor that combines a display, such as a liquid crystal display capable of displaying images, with a touchpad. The touch panel 71 is capable of generating signals that represent input operations received from the user. In this embodiment, the touch panel 71 displays images distributed from the in-vehicle device 10.

[0066] Furthermore, the touch panel 71 may display information related to video communication or a screen for accepting operations to connect to video communication. The user of the external device 70 may perform the video communication connection operation by inputting to the touch panel 71.

[0067] Speaker 73 is capable of emitting sounds such as music and voice. During video communication, speaker 73 emits voice from the in-vehicle device 10 for voice calls.

[0068] Microphone 75 is a microphone device that receives sound emitted toward the external device 70. During video communication, the sound picked up by microphone 75 is transmitted to the external device 70 as the audio for the voice call.

[0069] Figure 6 is a block diagram showing the configuration of the external device 70. For example, the external device 70 is a device in which a large-capacity storage device 83, a control unit 84, an input unit 85, an output unit 86, a data communication unit 87, and a decoder unit 88 cooperate via a system bus 81.

[0070] The large-capacity storage device 83 is composed of, for example, a hard disk drive, an SSD (solid state drive), flash memory, etc., and stores various programs such as the operating system and terminal software.

[0071] Furthermore, various programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, various programs stored in the mass storage device 83 (including programs for executing processing in the external device 70 described later) can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred.

[0072] The control unit 84 is composed of a CPU (Central Processing Unit) 84A, a ROM (Read Only Memory) 84B, a RAM (Random Access Memory) 84C, etc., and functions as a computer. The CPU 84A reads and executes various programs stored in the ROM 84B and the mass storage device 83 to realize various functions.

[0073] The input unit 85 is an input interface unit for the touch panel 71 and the microphone 75. The control unit 84 can receive signals indicating input operations to the touch panel 71 and audio input signals from the microphone 75 via the input unit 85. For example, the control unit 84 can receive connection requests for video communication connections made by the user via the touch panel 71 and the microphone 75 via the input unit 85.

[0074] The output unit 86 is an output interface unit for the touch panel 71 and the speaker 73. The control unit 84 and output unit 86 can transmit video or image signals to the touch panel 14 for display, or transmit audio signals to the speaker 15 for sound output.

[0075] The data communication unit 87 is connected to the network NW described above and sends and receives various data to and from the server 40. The data communication unit 87 also sends and receives various data to and from the in-vehicle device 10 via the server 40, including encoded video data transmitted from the in-vehicle device 10. In other words, the control unit 84 functions as an encoded data receiving unit that receives encoded data via the data communication unit 87.

[0076] For example, the control unit 84 of the external device 70 can receive location information that allows it to identify the current location of the vehicle M transmitted from the server 40 to the in-vehicle device 10 via the data communication unit 31. Also, for example, the control unit 84 can receive route information or navigation information of the vehicle M from the server 40 via the data communication unit 87.

[0077] Furthermore, the control unit 84 can transmit audio data of voices picked up by the microphone 75 to the in-vehicle device 10 via the data communication unit 87 for voice calls in video communication. Also, the control unit 84 can receive audio data transmitted from the in-vehicle device 10 for voice calls in video communication via the data communication unit 87.

[0078] The decoder unit 88, acting as a video output unit, decodes and reproduces the encoded data received from the in-vehicle device 10 based on commands from the control unit 84, and outputs it. The decoder unit 88 decodes the encoded data using the encoding codec used for encoding the data, for example, the MPEG-4 encoding standard, and reproduces and outputs the video. The reproduced video is then displayed on the touch panel 71 by the control unit 84.

[0079] The decoder unit 88 performs decoding of encoded data in multiple different operating modes. Specifically, for example, the decoder unit 88 has a normal mode as the first delay mode and a low-latency mode as the second delay mode as operating modes for decoding.

[0080] In this embodiment, the low-latency mode is an operating mode in which the time from receiving the encoded data to playing the encoded data back into video and displaying it on the touch panel 71 is shorter than in the normal mode. In other words, the low-latency mode is an operating mode in which the delay of video playback on the external device 70 is reduced compared to the normal mode. This low-latency mode may be the same as the low-latency mode used in video distribution services such as YouTube® and Nico Nico Live®.

[0081] For example, when the decoder unit 88 operates in low-latency mode, it reduces the playback buffer compared to when it operates in normal mode, thereby reducing the time from receiving or acquiring encoded data to playing back the video (hereinafter also simply referred to as playback delay).

[0082] Furthermore, for example, when the decoder unit 88 is operated in low-latency mode, the external device 70 may speed up the decoding process by allocating more processing resources to the decoding process than in normal mode. For example, when the decoder unit 88 is operated in low-latency mode, the external device 70 may allocate more resources of the CPU 84A to the decoding process than in normal mode.

[0083] The external device 70 changes the operating mode of the encoded transmission unit 32 between normal mode and low-latency mode according to the position of the automobile M indicated by the position identification information described above.

[0084] For example, similar to the encoding processing operation mode of the in-vehicle device 10 described above, the external device 70 operates the encoding transmission unit 32 in low-latency mode when the occupants of the vehicle M and an external user need to talk about the view or condition in front of the vehicle M with minimal video time lag.

[0085] As described in the explanation of the in-vehicle device 10, an example of a situation where communication must be conducted with minimal time lag is when an external user provides directions to the occupants while viewing the video feed from the external camera 12 displayed on the external device 70.

[0086] As described in the explanation of the in-vehicle device 10, when an external user provides directions to the occupants, a time lag in the video at the directions point becomes a problem. Therefore, in this embodiment, the external device 70 reduces the video delay by operating the decoder unit 88 in low-latency mode from the time the vehicle M approaches the directions point on the road until it passes the directions point.

[0087] Generally speaking, video playback using the low-latency mode described above is less stable than video playback using the normal mode described above. This is because, for example, by reducing the playback buffer, even a slight delay in the decoding process can lead to playback stopping.

[0088] In the external device 70 described above, the decoder unit 88 operates in low-latency mode only when real-time video is required. This ensures stable video playback under normal circumstances while providing highly real-time video playback when necessary, enabling smooth communication between external users and crew members, such as during navigation, while maintaining the stability of video distribution.

[0089] [2. Operation of the video streaming system] The operation of the video distribution system 100, which includes the in-vehicle device 10, server 40, and external device 70, will be described below.

[0090] The control routines for the in-vehicle device 10 and external device 70 that realize the operation of the system 100 of Example 1 will be described below.In the following description, it will be assumed that when video distribution is started via video communication, the encoding and transmission unit 32 operates in normal mode and starts the video encoding and transmission processes.

[0091] Figure 7 is a flowchart showing the video distribution routine RT1 executed in the control unit 25 of the in-vehicle device 10. For example, when power is turned on to the in-vehicle device 10, the control unit 25 starts the video distribution routine RT1 and executes it repeatedly.

[0092] First, the control unit 25 determines whether the in-vehicle device 10 has started video communication with the external device 70 (step S101). This determination is based, for example, on whether the in-vehicle device 10 has initiated video communication and established communication with the external device 70. If the control unit 25 determines that video communication has not started (step S101: NO), it terminates routine RT1.

[0093] When the control unit 25 determines that the in-vehicle device 10 has started video communication (step S101: YES), it acquires location information indicating the position of the vehicle M based on the signal from the GPS receiver 11, acquires video from the external camera 12 or the internal camera 13, encodes the acquired video, and starts a video distribution operation to distribute it to the external device 70 via the server 40 along with the location information (step S102).

[0094] In other words, in step S102, the control unit 25 functions as an image acquisition unit that acquires images from camera 12 or camera 13. Also in step S102, the control unit 25 functions as a location information acquisition unit that acquires location information indicating the position of automobile M. Also in step S102, the control unit 25 functions as a data transmission unit that encodes the image data and transmits the encoded data.

[0095] Step S102 corresponds to the video transmission method, program, and storage medium of the present invention, specifically the steps for acquiring location information, acquiring video, and transmitting data. In step S102, the location information does not necessarily have to be transmitted to the external device 70.

[0096] Once step S102 is executed and the video streaming operation begins, the control unit 25 determines whether or not video communication between the in-vehicle device 10 and the external device 70 has ended (step S103). If the control unit 25 determines that video communication has not ended, i.e., that video communication is continuing (step S103: NO), it repeatedly executes step S103. In other words, the video streaming operation continues as long as video communication does not end.

[0097] In step S103, when the control unit 25 determines that video communication with the external device 70 has ended, it terminates the video distribution operation (step S104) and routine R1 ends. In other words, when the control unit 25 determines that video communication with the external device 70 has ended, it terminates the acquisition of location information, the acquisition of video, and the encoding and transmission of said video.

[0098] Figure 8 is a flowchart showing the encoding transmission processing control routine RT2 executed in the control unit 25 of the in-vehicle device 10. For example, when power is turned on to the in-vehicle device 10, the control unit 25 starts the encoding transmission processing control routine RT2 and executes it repeatedly. In this embodiment, it is explained that when the video distribution operation starts, the encoding process is initially performed in normal mode.

[0099] First, the control unit 25 determines whether the in-vehicle device 10 is performing video distribution operations via video communication with the external device 70 (step S201). If the control unit 25 determines that it is not performing video distribution operations via video communication (step S201: NO), it terminates routine RT2.

[0100] When the control unit 25 determines that the in-vehicle device 10 is performing video distribution operations for video communication (step S201: YES), it determines whether the vehicle M is approaching the guidance point or has reached the point just before the guidance point (step S202).

[0101] This determination may be made, for example, by determining whether the vehicle M has reached a point at a predetermined distance before the designated destination on the map, based on map information and the vehicle's current position. Alternatively, this determination may be made, for example, by determining whether the vehicle M has reached a point where the estimated time to reach the designated destination on the map is predetermined, based on map information, the vehicle's current position, and its current speed.

[0102] Furthermore, this determination may be made based on the driver's operation of the in-vehicle device 10. For example, the driver may operate the in-vehicle device 10, for example, via the touch panel 14 or microphone 17, before the vehicle M approaches the guidance point, and at the time of such operation, it may be determined that the vehicle has reached a predetermined guidance point or a guidance point that the driver wishes to be guided to. Similarly, this determination may be made based on the user's operation of the external device 70.

[0103] Furthermore, if the determination is made based on an operation by the driver or the user of the external device 70, for example, log data of the position of the vehicle M at the time the operation was made may be stored in the in-vehicle device 10 or the server 40. Using this log data, for example, it may be determined whether the vehicle M has reached the point just before the guidance point based on the location indicated by the log data.

[0104] Furthermore, based on the log data, AI may be used to determine whether or not the vehicle M has reached the destination. For example, a learning model may be constructed based on the log data, and this learning model may be used to determine whether or not the vehicle M has reached the destination.

[0105] The log data mentioned above may be images or videos captured by camera 12 at the time the above operation was performed. Using such log data or a learning model based on such log data, it is possible to automatically determine from the video footage of camera 12 whether or not the vehicle has reached the destination.

[0106] When the control unit 25 determines that the vehicle M is approaching or has reached the guidance point (step S202: YES), it switches the operation of the encoding transmission unit 32 to low-latency mode (step S203). When switching to this low-latency mode, the control unit 25 may also transmit a low-latency mode switching signal to the external device 70 to notify it that the operation of the encoding transmission unit 32 has been switched to low-latency mode.

[0107] If the control unit 25 determines that the vehicle M is not approaching the guide point or has not reached the point just before the guide point (step S202: NO), it terminates routine RT2.

[0108] After step S203 is executed, the control unit 25 determines whether or not the vehicle M has passed the guidance point (step S204). This determination may be made, for example, by determining whether or not the vehicle M has passed the guidance point on the map based on map information and the current position of the vehicle M.

[0109] If the control unit 25 determines that the vehicle M has not passed the guide point (step S204: NO), it repeatedly executes step S204. In other words, the encoded transmission unit 32 operates in low-latency mode until the vehicle M passes the guide point.

[0110] When the control unit 25 determines that the vehicle M has passed the guidance point (step S204: YES), it switches the operation of the encoding transmission unit 32 to normal mode (step S205), and then terminates routine RT2. Note that when switching to normal mode, the control unit 25 may send a normal mode switching signal to the external device 70 to notify that the operation of the encoding transmission unit 32 has been switched to normal mode.

[0111] Figure 9 is a flowchart showing the decode processing control routine RT3 executed in the control unit 84 of the external device 70. The control unit 84 starts the decode processing control routine RT3 when power is turned on to the external device 70, for example, and executes it repeatedly. In this embodiment, it is explained that when video playback operation starts, the decoding process is initially performed in normal mode.

[0112] First, the control unit 84 determines whether the external device 70 is communicating with the in-vehicle device 10 via video and playing back a video being distributed from the in-vehicle device 10 (step S301). If the control unit 84 determines that the external device is not communicating with the in-vehicle device 70 via video and is not playing back a video (step S301: NO), it terminates routine RT3.

[0113] If the control unit 84 determines that the external device 70 is in video communication and is playing a video streamed from the in-vehicle device 10 (step S301: YES), it determines whether the vehicle M is approaching the guidance point or has reached the point just before it (step S302).

[0114] This determination may be made, for example, by determining whether the vehicle M has reached a point at a predetermined distance before the guidance point on the map, based on map information held by the vehicle itself or obtained from an external device such as a server 40, and the current position of the vehicle M. Alternatively, this determination may be made by determining whether the vehicle M has reached a point where the time to reach the guidance point on the map is predetermined, based on map information, the current position of the vehicle M, and its current speed.

[0115] This determination may also be made based on the low-latency mode switching signal from the in-vehicle device 10 described above. For example, in this determination, the control unit 84 may determine that the vehicle M is approaching the guidance point or has reached the point just before the guidance point based on the receipt of the low-latency mode switching signal.

[0116] When the control unit 84 determines that the vehicle M is approaching or has reached the point just before the guide point (step S302: YES), it switches the operation of the decoder unit 88 to low-latency mode (step S303).

[0117] If the control unit 84 determines that the vehicle M is not approaching the guide point or has not reached the point just before it (step S302: NO), it terminates routine RT3.

[0118] After step S303 is executed, the control unit 84 determines whether or not the vehicle M has passed the guidance point (step S304). This determination may be made, for example, by determining whether or not the vehicle M has passed the guidance point on the map based on map information and the current position of the vehicle M.

[0119] This determination may also be made based on the normal mode switching signal from the in-vehicle device 10 described above. For example, in this determination, the control unit 84 may determine that the vehicle M has passed the guidance point when it receives the normal mode switching signal.

[0120] If the control unit 84 determines that the vehicle M has not passed the guide point (step S304: NO), it repeatedly executes step S304. In other words, the decoder unit 88 operates in low-latency mode until the vehicle M passes the guide point.

[0121] When the control unit 84 determines that the automobile M has passed the guide point (step S304: YES), it switches the operation of the decoder unit 88 to normal mode (step S305), and then terminates routine RT3.

[0122] In the above embodiment, the in-vehicle device 10 was described as an in-vehicle navigation system, but the in-vehicle device 10 could be any other device, such as a smartphone or tablet.

[0123] The configurations, routines, etc., of the in-vehicle device 10, server 40, and external device 70 in the above-described embodiment are merely illustrative and can be appropriately selected or modified depending on the application. For example, the determination of whether the vehicle is approaching the guidance point in steps S202 and S302 of routines RT2 and RT3, and the determination in steps S204 and S304, may be performed by any of the in-vehicle device 10, server 40, or external device 70.

[0124] For example, if these steps are performed on the server 40, the determination result of each step is notified to the in-vehicle device 10 and the external device 70, and accordingly, the in-vehicle device 10 and the external device 70 may switch between normal mode and low-latency mode.

[0125] Furthermore, if these steps are performed by an external device 70, the determination result of each step may be notified to the in-vehicle device 10, and the in-vehicle device 10 may switch between normal mode and low-latency mode accordingly.

[0126] Furthermore, in the above embodiment, the in-vehicle device 10 and the external device 70 perform video communication via the server 40, but video communication may also be performed directly between the in-vehicle device 10 and the external device 70.

[0127] Furthermore, in the above embodiment, it is determined whether or not the guidance point is approaching based on the location information and map information of the vehicle M, and the operating mode of the encoded transmission unit 32 or the decoder unit 88 is switched to low-latency mode based on the determination result. However, the determination of whether or not the guidance point is approaching based on the map information may also be performed by image recognition of the video from the external camera 12.

[0128] Furthermore, in the above embodiment, the operating mode of the encoded transmission unit 32 or the decoder unit 88 is switched to low-latency mode when approaching a guidance point. However, even when not approaching a guidance point, if real-time video is required, the system may switch to low-latency mode. For example, when approaching a scenic spot, if the crew and an external user need to share highly real-time video of the scenery while talking, the system may switch to low-latency mode. Also, when the video communication mode is switched from a mode for sharing video to a guidance mode where the party viewing the video provides guidance, the system may always share in low-latency mode regardless of location information in the guidance mode.

[0129] In the above embodiment, an example was described in which the in-vehicle device 10 is mounted on an automobile M, but the in-vehicle device 10 may also be mounted on other mobile devices such as bicycles or motorcycles. Furthermore, the in-vehicle device may be held by a person, and video communication may be performed and video streamed while, for example, walking.

[0130] In the above embodiment, the video distribution operation was described as starting after communication between the in-vehicle device 10 and the external device 70 has been established. However, video distribution may be performed in a manner similar to the video distribution of YouTube® and Nico Nico Live® mentioned above. In other words, the video distribution operation may start even without establishing communication with the viewer's terminal, such as the external device 70. Specifically, the uploading of video data to the server 40 by the in-vehicle device 10 may start even without establishing communication between the in-vehicle device 10 and the viewer's terminal, such as the external device 70.

[0131] For example, the video distribution operation by the in-vehicle device 10 may be started after communication between the in-vehicle device 10 and the server 40 has been established, without establishing a communication connection with the external device 70. In this case, an unspecified or authorized specific external device 70 can connect to the server 40 to receive the video distributed from the in-vehicle device 10, and the user of the external device 70 will be able to view the video.

[0132] In this embodiment, it has been described that the in-vehicle device 10 and the external device 70 perform video communication via the server 40. However, this video communication may also be performed directly between the in-vehicle device 10 and the external device 70 by P2P (Peer to Peer) communication or the like, without going through the server 40.

[0133] In this embodiment, the case in which the in-vehicle device 10 is connected to the touch panel 14 has been described, but it is not limited to this. For example, the in-vehicle device 10 may communicate with a smartphone carried by the driver of the automobile M and display a screen related to video communication on the smartphone's display instead of the touch panel 14.

[0134] Furthermore, the in-vehicle device 10 may be configured not to display a screen for the driver of the vehicle M. For example, the in-vehicle device 10 may have a configuration similar to a drive recorder, or it may be a device integrated with an external camera 12. Specifically, the in-vehicle device 10 may be a device in which the hardware performing the video communication function of the in-vehicle device 10 is built into the housing of the external camera 12. In this case, the in-vehicle device 10 may not perform the various display outputs described above.

[0135] Furthermore, the video transmission device of the present invention may have a configuration in which a terminal device having the same configuration as the in-vehicle device 10 in this embodiment, an external camera 12, and a touch panel 14 are integrated. Specifically, for example, the video transmission device of the present invention may be a smartphone, tablet, or PC with a camera equipped with an application that performs the same functions as the in-vehicle device 10 described above.

[0136] In this embodiment, the case in which external video of the automobile M is transmitted from the in-vehicle device 10 to the external device 70 has been described. However, the video transmitted from the in-vehicle device 10 to the external device 70 may be switchable from external video of the automobile M to internal video of the automobile M captured by the in-vehicle camera 13. When internal video of the automobile M is being transmitted, the user of the external device 70 can, for example, communicate with the driver of the automobile M while viewing the inside of the automobile M.

[0137] The switching operation to transmit video to the external device 70 between the video from the external camera 12 and the video from the internal camera 13 may be performed in the in-vehicle device 10. Alternatively, this switching operation may be performed remotely by the user of the external device 70 through an operation on the external device 70.

[0138] In this embodiment, the control unit 84 of the external device 70 receives external video of the automobile M transmitted from the in-vehicle device 10 via the server 40. However, in addition to the video, it may also receive a map image showing the current location and planned route of the automobile M, the name of the driver of the automobile M, the speed of the automobile M, and so on.

[0139] For example, the control unit 84 of the external device 70 may display the map image, the name of the driver of the vehicle M, and the driving speed of the vehicle M received from the in-vehicle device 10 on the touch panel 71, or it may display them in a different display area from the external image of the vehicle M. In this case, the display / hide of the map image, the name of the driver of the vehicle M, and the driving speed of the vehicle M may be freely switched by the operator of the external device 70. [Explanation of Symbols]

[0140] 100 Video Distribution Systems 10 Onboard equipment 25, 84 Control Unit 30 Encoder section 31, 87 Data Communications Department 88 Decoder section 40 relay servers 70 External device

Claims

1. A video acquisition unit that sequentially acquires video data showing the surroundings of a moving object, A position information acquisition unit that acquires the position of the moving object, A data transmission unit that encodes the video data in a delay mode based on the position of the moving object to generate video encoded data and transmits the video encoded data, A video transmission device characterized by having the following features.

2. The data transmission unit has a first delay mode and a second delay mode in which the time from when the video data is acquired until the data transmission unit transmits it is shorter than the time from when the video data is acquired until the data transmission unit transmits it, as operating modes. The video transmission device according to claim 1, characterized in that the data transmission unit operates in the second delay mode when the position approaches a guide point on the guide path of the moving body.

3. The video transmission device according to claim 2, characterized in that the transmission buffer when the data transmission unit operates in the second delay mode is smaller than the transmission buffer when the data transmission unit operates in the first delay mode.

4. The video transmission device according to claim 2 or 3, characterized in that the frame rate of the video encoded data generated by the data transmission unit in the second delay mode is lower than the frame rate of the video encoded data output by the data transmission unit in the first delay mode.

5. The video transmission device according to any one of claims 2 to 4, characterized in that the resolution of the video encoded data generated by the data transmission unit in the second delay mode is smaller than the resolution of the video encoded data output by the data transmission unit in the first delay mode.

6. An encoded data receiving unit that sequentially receives video encoded data captured and encoded by an imaging device that moves along with a moving object, A position information acquisition unit that acquires position information indicating the position of the moving object, A video output device characterized by having a video output unit that decodes the video encoded data in a delay mode based on the position of the moving object and outputs a video.

7. The video output unit has a first delay mode and a second delay mode in which the time from when the video encoded data is acquired until the video is output is shorter than the time from when the video encoded data is acquired than in the first delay mode. The video output device according to claim 6, characterized in that the video output unit operates in the second delay mode when the position approaches a guide point on the guide path of the moving body.

8. The video output device according to claim 7, characterized in that the playback buffer when the video output unit operates in the second delay mode is less than the playback buffer when the video output unit operates in the first delay mode.

9. A video acquisition unit that sequentially acquires video data showing the surroundings of a moving object, A first position information acquisition unit that acquires the position of the moving object and transmits position data indicating the position, A data transmission unit that encodes the video data in a delay mode based on the position of the moving object to generate video encoded data and transmits the video encoded data, A receiving unit that receives the aforementioned video encoded data, A second position information acquisition unit receives the position data and acquires the position of the moving object indicated by the position data, A video output system characterized by having a video output unit that decodes the video encoded data in a delayed mode based on the position of the moving object and outputs a video.

10. A video transmission method performed by a video transmission device that transmits video from a mobile device, A video acquisition step involves sequentially acquiring video data showing the surroundings of the moving object, A position information acquisition step to acquire the position of the moving object, A data transmission step which involves encoding the video data in a delay mode based on the position of the moving object to generate video encoded data and transmitting the video encoded data, A video transmission method characterized by having the following features.

11. A video transmission device that transmits video from a mobile device, A video acquisition step involves sequentially acquiring video data showing the surroundings of the moving object, A position information acquisition step to acquire the position of the moving object, A data transmission step which involves encoding the video data in a delay mode based on the position of the moving object to generate video encoded data and transmitting the video encoded data, A video transmission program characterized by causing the execution of a specific action.

12. A video transmission device that transmits video from a mobile device, A video acquisition step involves sequentially acquiring video data showing the surroundings of the moving object, A position information acquisition step to acquire the position of the moving object, A data transmission step which involves encoding the video data in a delay mode based on the position of the moving object to generate video encoded data and transmitting the video encoded data, A computer-readable storage medium that stores a video transmission program that executes a video transmission program.

Images