Video transmitting device, video transmitting method, program, and recording medium

Abstract

To provide a video transmitting device and a video transmitting method that can transmit a video of the outside of a vehicle to an operator of an external terminal in an appropriate manner according to a situation while reducing the amount of communication during communication.SOLUTION: A video transmitting device that moves together with a moving body includes a video acquisition unit that sequentially acquires a video in front of the moving body, a visibility state determination unit that sequentially acquires visibility information relating to the degree of openness of left and right visibility in the video in front of the moving body, and a video transmitting unit that transmits a video of a frame rate according to the degree of openness. The degree of openness is determined based on whether or not there are, in the video, objects that are continuously present or a plurality of objects that are intermittently located, along the side of a road along which the moving body travels.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a video transmission device, a video transmission method, a program, and a recording medium.

Background Art

[0002] There is a communication system that performs communication between an in-vehicle device mounted on a vehicle and an out-of-vehicle terminal located outside the vehicle. For example, Patent Document 1 discloses a system in which a voice call is made between a driver of a vehicle and an operator of an out-of-vehicle terminal, and when the voice call is being made, video data representing a video in front of the vehicle is transmitted from the in-vehicle device to the out-of-vehicle terminal and the video is displayed on the out-of-vehicle terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a system such as Patent Document 1, for example, in order to avoid an excessive increase in communication volume during video transmission, the video may be transmitted to the out-of-vehicle terminal in a mode that minimizes the communication volume. Even in such a case, it is preferable that the operator of the out-of-vehicle terminal viewing the video does not feel discomfort.

[0005] The present invention has been made in view of the above points, and an object thereof is to provide a video transmission device and a video transmission method capable of transmitting a video outside the vehicle in an appropriate manner according to the situation while reducing the communication volume during communication for an operator of the out-of-vehicle terminal.

Means for Solving the Problems

[0006] The video transmission device according to the present invention is a video transmission device that moves together with a moving body, and includes a video acquisition unit that sequentially acquires video in front of the moving body, a field-of-view state determination unit that sequentially acquires field-of-view information regarding the degree of opening of the left and right fields of view in the video in front of the moving body, and a video transmission unit that transmits video at a frame rate according to the degree of opening. The degree of opening is determined based on whether there are objects that continuously exist along the side of the road on which the moving body travels or a plurality of objects that are intermittently located in the video.

[0007] The video transmission method according to the present invention is a video transmission method executed by a video transmission device that moves together with a moving body, and includes a video acquisition step of sequentially acquiring video in front of the moving body, a field-of-view state determination step of sequentially acquiring information regarding the degree of opening of the left and right fields of view in the video in front of the moving body, and a video transmission step of transmitting video at a frame rate according to the degree of opening. The degree of opening is determined based on whether there are objects that continuously exist along the side of the road on which the moving body travels or a plurality of objects that are intermittently located in the video.

[0008] The program according to the present invention is a program for causing a computer to execute a video transmission method executed by a video transmission device that moves together with a moving body, and includes a video acquisition step of sequentially acquiring video in front of the moving body, a field-of-view state determination step of sequentially acquiring information regarding the degree of opening of the left and right fields of view in the video in front of the moving body, and a video transmission step of transmitting video at a frame rate according to the degree of opening. The degree of opening is determined based on whether there are objects that continuously exist along the side of the road on which the moving body travels or a plurality of objects that are intermittently located in the video.

[0009] The recording medium according to the present invention is a recording medium on which the above program is recorded.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In the drawings, the same reference numerals are assigned to the same components, and the description of overlapping components is omitted.

Embodiment

[0012] FIG. 1 is a diagram showing the configuration of a video distribution system 100 according to Embodiment 1. As shown in FIG. 1, in the video distribution system 100, an in-vehicle device 10 mounted on a vehicle M as a moving body, a terminal device T, and a server S can transmit and receive data to and from each other via a network NW using a communication protocol such as TCP / IP or UDP / IP.

[0013] The network NW can be constructed by mobile communication networks such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5G mobile communication system), wireless communication such as Wi-Fi (registered trademark), and Internet communication including wired communication.

[0014] FIG. 2 is a perspective view showing the front seat portion of the vehicle M on which the in-vehicle device 10 according to Example 1 is mounted. The in-vehicle device 10 is connected to each of an external shooting camera 12, an in-vehicle shooting camera 13, a GPS receiver 14, a microphone 15, a touch panel display 16, and a speaker 17 installed in the vehicle M, and includes a control unit that controls these. The in-vehicle device 10 is disposed, for example, at the center in the dashboard DB of the front seat of the vehicle M.

[0015] The external shooting camera 12 is an imaging device that shoots the situation outside the vehicle M. In this embodiment, the external shooting camera 12 is a wide-angle camera that shoots a wide area in front of the vehicle M through the front glass FG. In this embodiment, the external shooting camera 12 is disposed on the dashboard DB.

[0016] The in-vehicle shooting camera 13 is an imaging device that shoots the situation inside the vehicle M. In this embodiment, the in-vehicle shooting camera 13 is a camera that shoots the driver of the vehicle M. In this embodiment, the in-vehicle shooting camera 13 is provided at the upper end of the front glass FG or in the ceiling portion near the upper end.

[0017] The GPS receiver 14 is a receiver that receives signals (GPS signals) from GPS (Global Positioning System) satellites. In this embodiment, the GPS receiver 14 is disposed on the dashboard DB.

[0018] The microphone 15 is a voice input device that receives sounds inside the vehicle, for example, the voice uttered by the driver, and converts it into an electrical signal. In this embodiment, the microphone 15 is disposed on the dashboard DB.

[0019] The touch panel display 16 is a display device in which a display that performs screen display based on the control of the in-vehicle device 10 and a touch panel that receives an input operation from a passenger (e.g., a driver) of the vehicle M are combined. In the present embodiment, the touch panel display 16 is arranged at the center of the dashboard DB.

[0020] For example, a navigation image in which the current position and the planned travel route of the vehicle M are superimposed on a map can be displayed on the touch panel display 16. Also, operations related to the navigation function, such as setting a destination via the touch panel display 16, may be possible.

[0021] The speaker 17 is an audio output device that outputs sound based on an electrical signal transmitted from the outside. In the present embodiment, the speakers 17 are provided on each of the two A-pillars AP.

[0022] In the present embodiment, the driver of the vehicle M can make a voice call with the operator of the terminal device T via the microphone 15 and the speaker 17 connected to the in-vehicle device 10.

[0023] The positions of the in-vehicle device 10, the outside vehicle camera 12, the inside vehicle camera 13, the GPS receiver 14, the microphone 15, the touch panel display 16, and the speaker 17 in the front seat portion of the vehicle M described above are merely examples, and these may be arranged at other positions.

[0024] For example, the outside vehicle camera 12 may be provided anywhere as long as it can capture the situation in front of the vehicle M, and may be provided at the upper end of the windshield FG or in the ceiling portion near the upper end.

[0025] In the present embodiment, the in-vehicle device 10 sequentially transmits video information indicating the video of the outside (front) of the vehicle M captured by the outside vehicle camera 12 to the terminal device T via the server S.

[0026] FIG. 3 is a diagram showing the configuration of the terminal device T and the display modes of the video and images displayed on the terminal device T. FIG. 3 shows a case where a landscape video in which no object is approaching the side of the road and the view ahead of the vehicle M is open is displayed on the terminal device T. The terminal device T is a communication device including a frame F, a microphone 21, a speaker 22, and a touch panel display 23 housed in the frame F. In the present embodiment, the terminal device T is a smartphone capable of making a call with others via communication.

[0027] The microphone 21 is a voice input device that receives the voice uttered by the user of the terminal device T and converts it into an electrical signal. The microphone 21 is provided on one end side of the terminal device T.

[0028] The speaker 22 is a voice output device that outputs voice based on an electrical signal transmitted from the outside. The speaker 22 is provided on the other end side of the terminal device T.

[0029] The touch panel display 23 is a display device in which a display for performing screen display based on the control of the terminal device T and a touch panel for receiving an input operation from the user of the terminal device T are combined. The touch panel display 23 is provided at the center of the terminal device T.

[0030] In the present embodiment, the operator of the terminal device T is configured to be able to make a voice call with the driver of the vehicle M on which the in-vehicle device 10 is mounted via the above-described microphone 21 and speaker 22.

[0031] In the present embodiment, the terminal device T sequentially receives video information showing the video outside the vehicle M sequentially transmitted from the in-vehicle device 10 of the vehicle M via the server S. Specifically, the terminal device T sequentially receives video information showing the video ahead of the vehicle M imaged by the out-of-vehicle camera 12 via the server S.

[0032] As described above, in this embodiment, in the video distribution system 100, the driver of the vehicle M equipped with the in-vehicle device 10 and the operator of the terminal device T can communicate with each other by voice. Also, in this embodiment, in the video distribution system 100, the video outside the vehicle M captured by the outside vehicle camera 12 can be displayed in real time on the touch panel display 23 of the terminal device T. In other words, the video distribution system 100 is configured such that the in-vehicle device 10 can perform live streaming to the terminal device T.

[0033] In this way, a communication form in which, while establishing voice communication between the in-vehicle device 10 and the terminal device T, video such as the video outside the vehicle M, the current position of the vehicle M, and the moving route are transmitted in real time from the in-vehicle device 10 to the terminal device T is referred to as video communication. In this embodiment, the in-vehicle device 10 and the terminal device T perform video communication via the server S.

[0034] By performing such video communication, the user of the terminal device T who is viewing the video transmitted from the in-vehicle device 10 can obtain a feeling as if riding in the vehicle M with the driver of the vehicle M. In other words, video communication can realize virtual boarding of the user of the terminal device T on the vehicle M. A system configured by such video communication is referred to as a virtual boarding system.

[0035] Hereinafter, with reference to FIGS. 4 to 7, the configurations of the in-vehicle device 10, the server S, and the terminal device T that constitute the video distribution system 100 will be described.

[0036] FIG. 4 is a block diagram showing the configuration of the in-vehicle device 10. The control unit 25 is a processing device including a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). In this embodiment, the control unit 25 functions as a computer, and the control unit 25 exhibits a video transmission function during video communication.

[0037] The encoding unit 26 is a part that encodes video (also referred to as captured video) captured by the external camera 12 or the in-vehicle camera 13 based on an instruction from the control unit 25. The encoding unit 26 has a CPU for video encoding, a so-called GPU, and the encoding may be performed by the GPU.

[0038] The encoding unit 26 encodes the captured video, for example, using an encoding method of the MPEG-4 standard to generate encoded data as video data. For example, the encoding unit 26 generates encoded data from the captured video using a codec such as H.264, Xvid, DivX, VP8, VP9, etc.

[0039] The communication unit 27 is a communication device that transmits and receives data to and from an external device according to an instruction from the control unit 25. The communication unit 27 is, for example, a NIC (Network Interface Card) for connecting to the network NW.

[0040] The communication unit 27 is connected to the above-described network NW and transmits and receives various data between the in-vehicle device 10 and the terminal device T. In this embodiment, the control unit 25, the encoding unit, and the communication unit 27 can be, for example, a video transmission unit that transmits the encoded data of the captured video generated by the encoding unit 26 to the terminal device T via the server S.

[0041] The storage unit 28 is a storage device that stores and manages data necessary for the processing of the control unit 25. The storage unit 28 is, for example, a storage device such as a hard disk, a flash memory, an SSD (Solid State Drive), etc. The storage unit 28 stores, for example, the contact information of the terminal device T as a partner for video communication. Also, the storage unit 28 stores, for example, map data for display on the above-described touch panel display 16.

[0042] Hereinafter, the functional blocks of the control unit 25 will be described.

[0043] The image acquisition unit 31 is a part that acquires the captured image captured by the above-described outside-vehicle camera 12 or inside-vehicle camera 13. The control unit 25 encodes the captured image acquired by the image acquisition unit 31 via the encoding unit 26, and transmits the encoded captured image to the server S via the communication unit 27.

[0044] The image recognition unit 32 is a part that performs image recognition on the image showing the captured image acquired by the image acquisition unit 31, and acquires information indicating the recognition result of the object in the field of view in front of the vehicle M. For example, the image recognition unit 32 detects the contour of the object to be recognized from the image, separates the object from other objects, etc., and then compares it with the image as reference data to recognize what the object is. The image as the reference data is stored in the storage unit 28, for example.

[0045] In the present embodiment, the image recognition unit 32 recognizes an object that can be involved in the degree of opening of the field of view in front of the vehicle M in the captured image described above. Specifically, the image recognition unit 32 recognizes an object that exists along the side of the road on which the vehicle M travels in front of the vehicle M.

[0046] More specifically, the image recognition unit 32 recognizes an object that is continuously located along the side of the road (hereinafter referred to as a continuous object), such as a windbreak, a sound barrier, a stone wall, a concrete wall, a hedge, or a guardrail. Further, the image recognition unit 32 recognizes an object that exists at intervals along the side of the road (hereinafter referred to as a discontinuous object), such as a plurality of guide poles for lane separation, trees, or crops.

[0047] The field of view state determination unit 33 is a part that determines the degree of opening of the field of view in front of the vehicle M based on the image recognition result obtained by the image recognition unit 32. That is, the field of view state determination unit 33 is a field of view information acquisition unit that acquires field of view information regarding the degree of opening of the field of view in front of the vehicle M based on the image recognition result.

[0048] For example, the field of view state determination unit 33 determines the degree of opening of the field of view in front of the vehicle M based on the presence or absence of an object recognized by the image recognition unit 32 and the position of the object, and acquires field of view information.

[0049] For example, when the field of view state determination unit 33 obtains an image recognition result such that the above-described continuous object is located along the side of the road on which the vehicle M travels, it determines that the degree of opening of the field of view in front of the vehicle M is small, that is, it is determined that the field of view in front of the vehicle M is not open. Further, when the field of view state determination unit 33 obtains an image recognition result in which there is nothing located on the side of the road on which the vehicle M travels or there are almost no continuous or discontinuous objects, it determines that the degree of opening of the field of view in front of the vehicle M is large, that is, it is determined that the field of view in front of the vehicle M is open.

[0050] The control unit 25 changes the encoding mode of the captured video by the encoding unit 26 based on the degree of opening of the field of view in front of the vehicle M determined by the field of view state determination unit 33.

[0051] Here, the encoding mode at the time of transmitting the captured video performed by the encoding unit 26 in the present embodiment will be described. Hereinafter, the outside vehicle camera 12 captures images at a frame rate sufficiently higher than the frame rate of the video data after the encoding process described below, for example, 27.5 fps constantly, and the frame rate is reduced by the encoding process, and it will be described that the encoded data is generated with a variable frame rate.

[0052] The encoding unit 26 performs encoding of the captured video in a plurality of different encoding modes based on an instruction from the control unit 25. Specifically, the encoding unit 26 has a normal mode as a first encoding mode and a high frame rate mode as a second encoding mode as operation modes during encoding.

[0053] The normal mode as the first encoding mode is an encoding mode that encodes imaging video with a lower frame rate than the second encoding mode. In this embodiment, the normal mode is applied, for example, when the view in front of the vehicle M is clear.

[0054] The high frame rate mode as the second encoding mode is an encoding mode that encodes imaging video in a manner that increases the frame rate compared to the first encoding mode. In this embodiment, the high frame rate mode is applied, for example, when the view in front of the vehicle M is not clear.

[0055] For example, when the front of the vehicle M is clear, the encoding unit 26 encodes the imaging video at a frame rate of 10 fps in the normal mode. And, for example, when the vehicle M is passing through a road where the front of the vehicle M is not clear, the encoding mode is changed from the normal mode to the high frame rate mode, and the imaging video is encoded at a frame rate of 24 fps.

[0056] In this embodiment, when video communication is established between the in-vehicle device 10 and the terminal device T, the encoding mode of the video outside the vehicle M transmitted from the in-vehicle device 10 is automatically changed according to the degree of openness of the view determined based on the presence or absence of an object located along the road on which the vehicle M is passing.

[0057] FIG. 5 is a diagram showing the terminal device T in which a video in front of the vehicle M different from that in FIG. 3 is displayed in the video communication between the in-vehicle device 10 and the terminal device T. In FIG. 5, on the touch panel display 23 of the terminal device T, a state in which the vehicle M is passing through a road provided with a concrete wall CW along the side as a video in front of the vehicle M is displayed. That is, in FIG. 5, in the view in front of the vehicle M, the above-described continuous object exists along the side of the road on which the vehicle M is passing.

[0058] In the encoding unit 26 of the in-vehicle device 10, for example, when all the videos outside the vehicle M are encoded and transmitted in the normal mode with a low frame rate, the operator of the terminal device T that has received the video, that is, the viewer, may feel uncomfortable depending on the degree of opening of the field of view in front of the vehicle M.

[0059] Specifically, for example, as shown in FIG. 5, when there is a continuous object such as a concrete wall CW along the side of the road on which the vehicle M is traveling, the operator of the terminal device T will visually perceive a speed sensation from the video, that is, a so-called perceived speed, that is faster than the actual speed of the vehicle M when viewing the video encoded in the above-described normal mode with a low frame rate.

[0060] In particular, when the height of an object located along the side of the road on which the vehicle M is traveling is higher than the camera's line of sight, the operator of the terminal device T will feel that the perceived speed is fast. Also, even when the height of the object is not higher than the camera's line of sight, if the vehicle M is traveling on a narrow road such as a one-way road and the distance from the object is close, the operator of the terminal device T may feel that the perceived speed is fast. This speed of the perceived speed may cause discomfort to the operator of the terminal device T viewing the video.

[0061] Also, due to the difference between the speed of the vehicle M or the actual perceived speed of the driver of the vehicle M and the perceived speed of the operator of the terminal device T, there is a risk that the sense of sharing of the video will fade between the driver of the vehicle M and the operator of the terminal device T. Also, for example, it may cause fear in the operator of the terminal device T.

[0062] According to the present embodiment, while the encoding unit 26 encodes the video at a low frame rate in the normal mode to suppress the communication volume during video communication, when the field of view in front of the vehicle M is not open, it encodes the video in the high frame rate mode. Thereby, the in-vehicle device 10 can transmit the video outside the vehicle M to the terminal device T in an appropriate manner according to the situation while reducing the communication volume during communication.

[0063] Therefore, the operator of the terminal device T that has received the video outside the vehicle M can view the video without discomfort. Thus, the sense of sharing of the video can be enhanced between the driver of the vehicle M and the operator of the terminal device T, and information can be shared smoothly.

[0064] FIG. 6 is a block diagram showing the configuration of the server S. The server S has a function such as a SIP server that establishes a voice call between the in-vehicle device 10 and the terminal device T during video communication and transfers the data of the voice call.

[0065] The control unit 35 is a processing device including a CPU, a ROM, and a RAM. In the present embodiment, the control unit 35 transfers the captured video transmitted from the in-vehicle device 10 to the terminal device T during video communication.

[0066] The communication unit 36 is a communication device that transmits and receives data to and from an external device according to an instruction from the control unit 25. The communication unit 36 is, for example, a NIC for connecting to the network NW. In the present embodiment, the communication unit 36 can be a receiving unit that receives the encoded video outside the vehicle M from the in-vehicle device 10. Also, the communication unit 36 can be a transmitting unit that transmits the encoded video outside the vehicle M to the terminal device T.

[0067] FIG. 7 is a block diagram showing the configuration of the terminal device T. The control unit 37 is a processing device including a CPU, a ROM, and a RAM.

[0068] The communication unit 38 is a communication device that transmits and receives data to and from an external device according to an instruction from the control unit 37. The communication unit 38 is, for example, a NIC for connecting to the network NW. In the present embodiment, the communication unit 38 can be a receiving unit that receives the encoded video outside the vehicle M transferred from the server S.

[0069] The decoding unit 39 is a part that decodes, reproduces, and outputs the encoded video outside the vehicle M received from the in-vehicle device 10 via the server S based on the instruction from the control unit 37.

[0070] The decoding unit 39 decodes the encoded data with the codec used at the time of encoding the encoded data, for example, decodes the encoded data with the encoding method of the MPEG-4 standard to reproduce and output the video. The reproduced video is displayed on the touch panel display 23 by the control unit 37.

[0071] In this way, the terminal device T can receive the video (encoded data) outside the vehicle M transmitted from the in-vehicle device 10 in video communication, decode it, and display the video on the touch panel display 23. Further, the terminal device T can display videos with different frame rates according to the degree of opening of the field of view in front of the vehicle M by a plurality of different encoding modes at the time of decoding the in-vehicle device 10 described above.

[0072] Hereinafter, the specific operation of the in-vehicle device 10 in this embodiment will be described.

[0073] FIG. 8 is a flowchart showing a video distribution routine RT1 executed in the control unit 25 of the in-vehicle device 10. The control unit 25 starts the video distribution routine RT1 using, for example, the establishment of a connection for video communication (hereinafter referred to as a video connection) between the in-vehicle device 10 and the terminal device T via the server S as a start trigger.

[0074] Specifically, for example, when the touch panel display 16 of the in-vehicle device 10 is powered on by the driver of the vehicle M, a video call reception screen is displayed on the touch panel display 16. Then, when the terminal device T is selected as the video connection partner on the screen, a video connection is established between the in-vehicle device 10 and the terminal device T. Thereby, the control unit 25 starts the video distribution routine RT1.

[0075] The control unit 25 determines whether video communication has started by determining whether a video connection has been established (step S101). If the control unit 25 determines that video communication has not started (step S101: NO), it ends the video distribution routine RT1.

[0076] If the control unit 25 determines that video communication has started (step S101: YES), it acquires the captured video, starts a video distribution operation of encoding the video and transmitting it to the terminal device T via the server S (step S102).

[0077] Specifically, in step S102, the video acquisition unit 31 of the control unit 25 acquires the captured video outside the vehicle M from the outside vehicle camera 12, the image recognition unit 32 of the control unit 25 processes the image showing the captured video, and recognizes the objects in the image. Then, the field of view state determination unit 33 acquires the recognition result of the objects in the image showing the captured video by the image recognition unit 32, that is, the field of view information. The control unit 25 instructs the encoding unit 26 and the communication unit 27 to transmit the encoded data obtained by encoding the captured video to the terminal device T via the server S.

[0078] After step S102, the control unit 25 determines whether the video communication has ended (step S103). If the control unit 25 determines that the video communication has ended (step S103: YES), it ends the video distribution operation (step S104). In other words, when the control unit 25 determines that the video communication with the external device 70 has ended, it ends the acquisition of the captured video, image recognition, encoding and transmission of the captured video.

[0079] If the control unit 25 determines that the video communication has not ended (step S103: NO), it repeatedly executes step S103.

[0080] While the video communication between the in-vehicle device 10 and the terminal device T continues according to the above-described video distribution routine RT1, the control unit 25 of the in-vehicle device 10 continues to encode the video outside the vehicle M and transmit it to the terminal device T via the server S.

[0081] FIG. 9 is a flowchart showing a frame rate control routine RT2 executed in the control unit 25 of the in-vehicle device 10. The control unit 25 starts the frame rate control routine RT2, for example, using the start of the above-described video distribution operation in the in-vehicle device 10 by video communication as a start trigger.

[0082] First, the control unit 25 determines whether the view in front of the vehicle M is open based on the presence or absence of the above-described continuous object (or discontinuous object) as the recognition result of the video in front of the vehicle M (step S201). When the control unit 25 determines that the view in front of the vehicle M is open (step S201: YES), the frame rate control routine RT2 ends.

[0083] When the control unit 25 determines that the view in front of the vehicle M is not open (step S201: NO), it changes the encoding mode of the encoding unit 26 to the high frame rate mode (step S202). Specifically, the control unit 25 changes the encoding mode of the encoding unit 26 to the high frame rate mode, which is a mode that increases the frame rate of the captured video during encoding compared to the normal mode.

[0084] After step S202, the control unit 25 determines whether the view in front of the vehicle M has opened (step S203). When the control unit 25 determines that the view in front of the vehicle M has opened (step S203: YES), it changes the encoding mode of the encoding unit 26 to the normal mode (step S204).

[0085] When the control unit 25 determines that the view in front of the vehicle M is not open (step S203: NO), it repeats step S203.

[0086] In addition, in the frame rate control routine RT2, when changing from the normal mode to the high frame rate mode and when changing from the high frame rate mode to the normal mode, the control unit 25 may transmit a mode change signal to the terminal device T notifying that the encoding mode during encoding of the encoding unit 26 has been changed.

[0087] According to this embodiment, as described above, during video communication between the in-vehicle device 10 and the terminal device T, when the view in front of the vehicle M is considered not to be open for the video outside the vehicle M transmitted from the in-vehicle device 10, the encoding mode is automatically changed.

[0088] Therefore, according to this embodiment, the in-vehicle device 10 can transmit the video outside the vehicle M in an appropriate manner according to the situation while reducing the communication volume during communication. Thus, according to this embodiment, it is possible to smoothly share information between the driver of the vehicle M and the operator of the terminal device T while reducing the communication volume during communication.

[0089] In this embodiment, the field of view state determination unit 33 of the control unit 25 determines the degree of opening of the field of view based on the presence or absence of continuous (or discontinuous) objects by image recognition as the field of view information. However, the method for determining the degree of opening of the field of view is not limited to this.

[0090] For example, the vehicle M may be equipped with a distance sensor such as LiDAR (Light Detection And Ranging) or a millimeter-wave radar, and the distance between the vehicle M and an object located from the side of the vehicle M to the front may be measured by the distance sensor. Further, the control unit 25 of the in-vehicle device 10 may function as a distance information acquisition unit that acquires the distance between the vehicle M and an object located from the side of the vehicle M to the front measured by the distance sensor.

[0091] As a result, the field of view state determination unit 33 acquires, as field of view information, image data indicating an object located from the side to the front of the vehicle M represented by the point cloud data based on the above-described distance, and determines the degree of opening of the field of view based on the presence or absence of continuous objects (or discontinuous objects) in the field of view information. For example, when there are continuous objects (or discontinuous objects) in the field of view in front of the vehicle M, the field of view state determination unit 33 determines that the field of view is not open.

[0092] Therefore, the control unit 25 can determine the degree of opening of the field of view in front of the vehicle M based on the distance between the vehicle M and a continuous object (or discontinuous object) located along the side of the road on which the vehicle M travels using a distance sensor, and change the encoding mode according to the degree of opening of the field of view.

[0093] Also, as another method for determining the degree of opening of the field of view, the degree of opening of the field of view may be determined using the current position of the vehicle M and map information.

[0094] For example, the control unit 25 may function as a position information acquisition unit that acquires the current position of the vehicle M via the GPS receiver 14, and attribute information may be assigned to the map data stored in the storage unit 28 for each region on the map. Specifically, the map data may be provided with information indicating attributes such as mountains, residential areas, along the coast, industrial areas, rural areas, etc. as the environment of the surrounding area for each link or node. The field of view state determination unit 33 acquires the current position of the vehicle M on the map and the above-described attribute information as field of view information.

[0095] As a result, based on the current position of the vehicle M on the map and the above-described attribute information, when the attribute information of the region including the current position of the vehicle M is specific attribute information, for example, when it is attribute information in which continuous objects (or discontinuous objects) may exist on the side of the road such as in the mountains or in a residential area, the field of view state determination unit 33 determines that the field of view is not open.

[0096] Therefore, the control unit 25 can determine the degree of opening of the field of view in front of the vehicle M based on the attribute information of the area including the current position of the vehicle M on the map, and can change the encoding mode according to the degree of opening of the field of view.

[0097] Alternatively, the degree of opening of the field of view may be determined by inputting the video data in front of the vehicle M captured by the in-vehicle camera 12 into an AI having a learning model for determining the degree of opening of the field of view from the video data and obtaining its output.

[0098] In addition, in the first embodiment, the control unit 25 of the in-vehicle device 10 has been described with regard to changing the encoding mode of the encoding unit 26 when the field of view in front of the vehicle M is not open, that is, an example of changing the frame rate in two steps. However, this change in mode may be made in three or more steps. That is, there may be three or more encoding modes for changing the frame rate of the video.

[0099] Note that depending on the shooting environment, shooting target, or type or specification of the shooting device in the vehicle M, or depending on the type or specification of the terminal device T, contrary to the above description, when the field of view in front of the vehicle M is open in the normal mode, the operator of the terminal device T may feel that the perceived speed is faster than the speed of the vehicle M or the actual perceived speed of the driver of the vehicle M. In such a case, the above encoding mode is set in the reverse manner.

[0100] For example, the control unit 25 of the in-vehicle device 10 may perform encoding in the normal mode when the field of view in front of the vehicle M is not open, and may change to a high frame rate mode, which is a mode of increasing the frame rate compared to the normal mode, when the field of view in front of the vehicle M is open.

[0101] In this embodiment, the resolution of the video during encoding by the encoding unit 26 may be changed according to the above-described encoding mode. For example, when encoding in the high frame rate mode when the view in front of the vehicle M is blocked, the control unit 25 may lower the resolution of the video. Thereby, for example, an increase in communication volume due to an increase in the frame rate of the video caused by a change from the normal mode to the high frame rate mode can be suppressed.

[0102] In this embodiment, the video communication between the in-vehicle device 10 and the terminal device T via the server S has been described. However, the video communication may be directly performed between the in-vehicle device 10 and the terminal device T by P2P (Peer to Peer) communication or the like.

[0103] In this embodiment, the case where the in-vehicle device 10 is connected to the touch panel display 16 has been described. However, the present invention is not limited to this. For example, the in-vehicle device 10 may communicate with a smartphone carried by the driver of the vehicle M and display a screen related to video communication on the display of the smartphone instead of the touch panel display 16.

[0104] Further, the in-vehicle device 10 may be configured not to display a screen presented to the driver of the vehicle M. For example, the in-vehicle device 10 may have a configuration such as a drive recorder, or may be an apparatus integrated with the vehicle exterior camera 12. Specifically, the in-vehicle device 10 may be an apparatus that incorporates hardware that performs the functions of the in-vehicle device 10 described above in the housing of the vehicle exterior camera 12. In this case, the in-vehicle device 10 may not perform various display outputs as described above.

[0105] Furthermore, the video transmission device of the present application may have a configuration in which a terminal device (for example, FIG. 4) having the same configuration as the in-vehicle device 10 in this embodiment, the vehicle exterior camera 12, and the touch panel display 16 are integrated.

[0106] In this embodiment, the case where the video of the outside of the vehicle M is transmitted from the in-vehicle device 10 to the terminal device T has been described. However, the video transmitted from the in-vehicle device 10 to the terminal device T may be switched from the video of the outside of the vehicle M to the video of the inside of the vehicle M taken by the in-vehicle camera 13. When the video of the inside of the vehicle M is being transmitted, the user of the terminal device T can communicate with the driver of the vehicle M while, for example, viewing the state inside the vehicle M.

[0107] Note that the switching operation for switching the video transmitted to the terminal device T between the video of the outside camera 12 and the video of the in-vehicle camera 13 may be performed in the in-vehicle device 10. Further, the switching operation may be remotely performed by an operation of the terminal device T by the user of the terminal device T.

[0108] In this embodiment, it has been assumed that the control unit 37 of the terminal device T receives the video of the outside of the vehicle M transmitted from the in-vehicle device 10 via the server S. However, in addition to the video, a map image showing the current location and the planned travel route of the vehicle M, the name of the driver of the vehicle M, the traveling speed of the vehicle M, etc. may be received.

[0109] For example, the control unit 37 of the terminal device T may display the map image received from the in-vehicle device 10, the name of the driver of the vehicle M, and the traveling speed of the vehicle M on the video of the outside of the vehicle M on the touch panel display 23, or may display them in a display area different from the video of the outside of the vehicle M. At this time, the map image, the name of the driver of the vehicle M, and the traveling speed of the vehicle M may be freely switched between display and non-display by the operator of the terminal device T.

[0110] In this embodiment, an example where the in-vehicle device 10 is mounted on the vehicle M has been described. However, the in-vehicle device 10 may be mounted on other moving bodies such as bicycles and motorcycles. Further, a person may hold the in-vehicle device 10 and, for example, perform video communication while walking to distribute the video.

[0111] In this embodiment, the case where the terminal device T is a smartphone has been described, but the present invention is not limited to this. For example, the terminal device T may be a tablet terminal capable of video communication with the in-vehicle device 10.

[0112] In this embodiment, the case where the control unit 25 of the in-vehicle device 10 starts the video distribution operation after the video communication between the in-vehicle device 10 and the terminal device T is established has been described as an example. However, the control unit 25 may perform video distribution in a manner such as the video distribution of YouTube (registered trademark) or Nico Nico Douga (registered trademark). That is, the video distribution operation may be started even without establishing communication with the terminal on the operator side of the terminal device T. Specifically, even without establishing communication between the in-vehicle device 10 and the terminal on the operator side of the terminal device T, the upload of video data from the in-vehicle device 10 to the server S may be started.

[0113] For example, the video distribution operation by the in-vehicle device 10 may be started without establishing a communication connection with the terminal device T after the communication between the in-vehicle device 10 and the server S is established. In this case, an unspecified or a specific terminal device T that has received permission connects to the server S, thereby receiving the video distributed from the in-vehicle device 10, and the operator of the terminal device T can view the video.

[0114] Note that the series of processes in the control unit 25 of the in-vehicle device 10 described in Embodiment 1 may also be a program to be executed by a computer. The program may be recorded on a computer-readable recording medium. The type of the recording medium is not particularly limited, and for example, it may be an optical disk, a hard disk, or a semiconductor memory such as a flash memory or an SSD. Further, the above program may be downloaded and installed in the in-vehicle device 10, the server S, and the terminal device T via communication.

[0115] The control routine shown in the above-described Embodiment 1 is merely an example, and can be appropriately selected and changed according to the application, usage conditions, etc.

Explanation of Reference Numerals

[0116] 100 Information processing system Vehicle M 10 In-vehicle device Server S Terminal device T 12 Outdoor camera 13 Indoor camera 14 GPS receiver 15, 21 Microphone 16, 23 Touch panel display 17, 22 Speaker 25, 35, 37 Control unit 26 Encoding unit 27, 36, 38 Communication unit 28 Memory unit 31 Video acquisition unit 32 Image recognition unit 33 Field of view state determination unit 39 Decoding unit

Claims

1. A video transmission device that moves with a moving object, an image acquisition unit that sequentially acquires images of a scene ahead of the moving object; a visibility state determination unit that sequentially acquires visibility information related to a degree of visibility in a left-right direction in an image ahead of the moving body; a video transmission unit that transmits the video at a frame rate according to the degree of opening, A video transmission device characterized in that the degree of opening is determined based on whether or not there are objects that are continuously located along the side of the road on which the mobile body travels or multiple objects that are located intermittently within the image.

2. The video transmission device according to claim 1 , wherein the degree of opening is determined based on a distance to the object.

3. 3. The video transmission device according to claim 1, wherein the degree of opening is determined based on attribute information of an area on a map that includes the position of the mobile object.

4. 4. The video transmission device according to claim 1, wherein the video transmission unit transmits the video at a high frame rate when the degree of opening is small.

5. the video transmission unit has an encoding unit having a first encoding mode and a second encoding mode for encoding the video in a manner that increases a frame rate compared to the first encoding mode as operation modes when encoding the video, 5. The video transmission device according to claim 1, wherein the encoding unit operates in the second encoding mode when the field of view is not open.

6. 6. The video transmission device according to claim 5, wherein a resolution of the video in the second encoding mode is smaller than a resolution of the video in the first encoding mode.

7. A video transmission method performed by a video transmission device moving together with a moving object, comprising: an image capturing step of sequentially capturing images of a scene ahead of the moving object; a visibility state determination step of sequentially acquiring information regarding a degree of visibility in the left and right directions in an image ahead of the moving body; and a video transmission step of transmitting the video at a frame rate according to the degree of opening, A video transmission method characterized in that the degree of opening is determined based on whether or not an object that exists continuously or a plurality of objects that are located intermittently along the side of the road on which the mobile body travels is present in the video.

8. A program executed by a computer, an image capturing step of sequentially capturing images of a scene ahead of the moving object; a visibility state determination step of sequentially acquiring information regarding a degree of visibility in the left and right directions in an image ahead of the moving body; and a video transmission step of transmitting the video at a frame rate according to the degree of opening, The program is characterized in that the degree of opening is determined based on whether or not there are objects that exist continuously or multiple objects that are located intermittently along the side of the road on which the mobile body is traveling within the image.

9. A recording medium storing the program according to claim 8.

Images