Streaming distribution system, distribution server, and cameraman terminal

Abstract

Provided are a distribution server, a cameraman terminal, and a streaming distribution system capable of performing, with high quality even in an environment with a poor communication environment, archive recording when streaming distribution is performed. The cameraman terminal records the captured video in a video data recording unit at a plurality of resolutions, and streams transmission of the video data with the optimum resolution according to the speed of communication with the streaming distribution server. The distribution server stores the received video data in an archive recording unit while performing streaming distribution of the video data to a viewer terminal. After the end of the streaming distribution, the time slot is detected in which the resolution of the video data streaming-transmitted by the streaming distribution server or the cameraman terminal was not the maximum, and the cameraman terminal is requested to transmit the data at the maximum resolution for the time slot. The cameraman terminal uploads and transmits the video data for the requested time slot and resolution, and the distribution server stores the uploaded / transmitted video data in the archive recording unit.

Description

Streaming distribution system, distribution server and cameraman terminal

[0001] The present invention relates to a streaming distribution system, a distribution server, and a photographer terminal that relays video from a photographer terminal and distributes live streaming broadcasts to multiple viewer terminals.

[0002] In recent years, with the expansion of network environments such as mobile phone networks and the widespread use of imaging devices, anyone can stream moving images regardless of the type of device or location. In particular, with the development of online videoconferencing systems, online talk broadcasts and online lessons are now being held using online videoconferencing systems, in which captured video is relayed and streamed from multiple devices remotely connected via a network. Furthermore, streaming video data is archived and stored in a distribution server for on-demand distribution to viewers who cannot participate in the streaming distribution in real time, or for content creation using the video streamed during streaming distribution.

[0003] Patent document 1 discloses that in a recording device that transmits captured moving image data in a streaming manner, the moving image data is recorded in response to a user's instruction to continuously record the moving image data, and the continuously recorded moving image data is transmitted using a transfer method different from streaming transmission in response to an instruction to end recording.

[0004] Patent Document 2 discloses that a transmitting device transmits captured video data to a receiving device in real time while the data is being shot, and during real time transmission, the transmitting device selects and transmits either low-quality MPEG data or high-quality MPEG data depending on the line conditions, and after shooting is completed, the transmitting device additionally transmits high-quality MPEG data in place of the low-quality MPEG data transmitted, and the receiving device merges the MPEG data received during and after shooting to record high-quality video data, and the high-quality video data is recorded for viewing after the fact.

[0005] Note that "streaming transmission" and "real-time transmission" are used as they are described in the above-mentioned prior art patent documents, and in this specification, they are terms with the same meaning, meaning "instantaneous distribution of video being filmed on the cameraman's terminal to the viewer's terminal via a distribution server."

[0006] Japanese Patent No. 6102647 Japanese Patent Laid-Open No. 2007-274443

[0007] The technology described in Patent Literature 1 allows high-quality video data to be transmitted via FTP for a separately specified period while streaming. However, the transmitted high-quality video data is video data for only a partial time range instructed by a user to be continuously recorded, and is therefore insufficient for creating archived videos to be viewed later.

[0008] In the technology described in Patent Document 2, video data is transmitted at a quality that corresponds to the communication conditions during real-time transmission, and frames that could not be transmitted in high quality after real-time transmission are subsequently transmitted to the server, so that only high-quality archived video data is recorded in the server.

[0009] As described in Patent Document 2, videos stored in a streaming distribution server are used for subsequent viewing. In this case, if the state of the communication line between the viewer's terminal and the streaming distribution server is insufficient to transmit high-quality video data, problems arise such as the viewing on the viewer's terminal stopping or the image becoming distorted.

[0010] A known solution to this problem is a conversion technique that generates low-resolution video data from high-resolution video data. This involves generating low-resolution video data from high-resolution video data stored on a server, or generating stored video data of multiple resolutions through a so-called conversion process.

[0011] During streaming, if the communication state between the viewer terminal and the distribution server is poor and the distribution server does not have low-resolution video data, the low-resolution video data is generated at regular intervals and transmitted as a stream. The conversion process places a heavy processing load on the distribution server, and because part of the video data is temporarily stored in a buffer, read and converted before being transmitted, it is known that there is a time lag of several seconds in the time it takes for streaming to be transmitted from the shooter terminal to the viewer terminal.

[0012] To further solve this problem, it is conceivable to create low-resolution archive video data by converting high-resolution archive video data stored in a distribution server. However, when multiple camera terminals are used and multiple streaming distributions are performed simultaneously, multiple conversion processes also occur simultaneously. Because video data conversion processes require high memory and CPU specifications, when multiple conversion processes occur, the server load increases, and preparing a server capable of handling this load becomes expensive. The inventors have discovered a new problem regarding optimal archive video storage on a server when live streaming distribution, such as online talk distribution or online lessons, in which multiple video distributors distributed over a network participate, is performed using conventional systems.

[0013] The means for solving these problems will be described below, but other problems and novel features will become apparent from the description of this specification and the accompanying drawings.

[0014] According to one embodiment of the present invention, the following is true.

[0015] The streaming distribution system according to the present invention includes a photographer terminal and a distribution server connected to each other via a communication line, wherein the photographer terminal includes: a video capture unit that generates video data from a captured video; a video data recording unit that records the video data at a plurality of resolutions for each predetermined time period; a measurement unit that measures the transmission rate of the communication line for streaming transmission to the distribution server; a streaming transmission determination unit that determines, from the plurality of resolutions for each predetermined time period, a combination of resolutions that can be streamed within the range of the measured transmission rate; a streaming transmission unit that streams video data of the determined combination of resolutions to the distribution server; an upload request receiving unit that receives an upload request from the distribution server after streaming transmission; and a video data upload unit that uploads the video data of the time period and resolution specified in the upload request from the video data recording unit to the distribution server in accordance with the upload request. the distribution server comprises a streaming receiving unit that receives video data streamed from the photographer's terminal, a streaming distribution unit that streams at least a portion of the received video data to the viewer's terminal, an archive recording unit that records the received video data by time period and resolution, an upload request unit that, after streaming distribution, detects time periods that are missing for each of a plurality of resolutions from the video data in the archive recording unit and transmits an upload request to the photographer's terminal to request transmission of video data for the detected time period and resolution, and an upload video receiving unit that receives the video data streamed from the photographer's terminal and records it in the archive recording unit.

[0016] The photographer terminal of the present invention is characterized by comprising a video imaging unit that generates video data from captured video; a video data recording unit that records the video data at multiple resolutions for each predetermined time period; a measurement unit that measures the transmission rate of a communication line for streaming transmission to a distribution server; a streaming transmission determination unit that determines a combination of resolutions that can be streamed from multiple resolutions for each predetermined time period within the range of the measured transmission rate; a streaming transmission unit that streams video data of the determined combination of resolutions to the distribution server; an upload request receiving unit that accepts an upload request from the distribution server after streaming transmission; and an upload transmission unit that transmits video data of the multiple resolutions in the video data recording unit for the time period and resolution specified by the upload request to the distribution server.

[0017] The distribution server of the present invention is characterized by comprising a streaming receiving unit that receives video data of one or more resolutions that is streamed from a photographer's terminal; a streaming distribution unit that streams at least a portion of the received video data to a viewer's terminal; an archive storage unit that records the received video data for a specified time period and resolution; an upload request unit that, after streaming distribution, detects time periods that are missing for each of a plurality of resolutions from the video data in the archive recording unit and sends an upload request to the photographer's terminal requesting that video data of the detected time period and resolution be sent; and an upload video receiving unit that receives the video data that is uploaded from the photographer's terminal and records it in the archive recording unit.

[0018] The distribution server of the present invention is characterized by further comprising an archive transmission unit that measures the transmission rate of the communication line between the viewer terminal, determines the maximum resolution that can be streamed within the range of the measured transmission rate, and transmits video data of the determined resolution stored in the archive recording unit to the viewer terminal.

[0019] The effects obtained by the embodiment are briefly described below.

[0020] The distribution server of this embodiment can receive, distribute, and record archived video data of multiple resolutions when streaming content is distributed from a distributor terminal connected via a communication line. This eliminates the need for the conversion process from high-quality video to low-quality video that was previously performed by distribution servers, enabling streaming transmission with little time lag and recording archived video data of multiple resolutions with little load on the server.

[0021] FIG. 1 is a schematic diagram showing an example of the device configuration of a distribution server according to the first embodiment and a streaming content distribution system using the same. FIG. 2 is a schematic diagram showing an example of the configuration of video image data recorded in the video data recording unit 22. FIG. 3 is a diagram showing the relationship between the transmission rate and the frame, GOP, and resolution flag data of the video data. FIG. 4 is a schematic diagram showing an example of the configuration of video data transmitted from the streaming transmission unit 25. FIG. 5 is a schematic diagram showing an example of the configuration of video data recorded in the archive recording unit 13 at the end of streaming distribution. FIG. 6 is an operational flowchart of the photographer terminal 20 during streaming distribution. FIG. 7 is an operational flowchart of the distribution server 10 during streaming distribution and after streaming ends. FIG. 8 is an operational flowchart of the photographer terminal 20 when transmitting archived video data. FIG. 9 is an operational flowchart of the distribution server 10 when accumulating archived video data.

[0022] 1. Overview of the Embodiments First, an overview of the representative embodiments disclosed in the present application will be described. Reference numerals in parentheses in the drawings used in the overview of the representative embodiments merely illustrate components included in the concept of the components to which they are attached.

[0023] [1] <Streaming Distribution System with Archive Recording Unit> A representative embodiment of the present invention is a streaming distribution system including a photographer terminal (20) and a distribution server (10) connected to each other via a communication line, and comprising the following components: a photographer terminal (20) comprising: a video shooting unit (21) that generates video data from a shot video; a video data recording unit (22) that records the video data at multiple resolutions for each predetermined time period; a measurement unit (23) that measures the transmission rate of a communication line for streaming transmission to the distribution server (10); a streaming transmission determination unit (24) that determines, for each predetermined time period, from the multiple resolutions, a combination of resolutions that can be streamed within the measured transmission rate; a streaming transmission unit (25) that streams video data of the determined combination of resolutions to the distribution server (10); an upload request receiving unit (26) that accepts an upload request from the distribution server (10) after streaming transmission; and an upload transmission unit that uploads and transmits video data of the time period and resolution specified in the upload request from the video data recording unit (22) to the distribution server (10) in accordance with the upload request. The distribution server (10) comprising: A streaming receiving unit (11) receives video data streamed from a photographer's terminal (20); a streaming distribution unit (12) streams at least a portion of the received video data to a viewer terminal (30); an archive recording unit (13) records the received video data for each time period and resolution; an upload request unit (14) detects, after streaming distribution, time periods that are missing for each of a plurality of resolutions from the video data in the archive recording unit (13) and transmits an upload request to the photographer's terminal (20) requesting transmission of video data for the detected time period and resolution; and an upload video receiving unit (15) receives video data streamed from the photographer's terminal (20) and records it in the archive recording unit (13).

[0024] Here, the photographer terminal (20) refers to a terminal device having an imaging device. Specifically, this includes devices that integrate an imaging device with a terminal device, such as smartphones, mobile phones, and laptops, as well as digital cameras with network communication functions and personal computer terminals to which digital cameras are connected. Furthermore, the communication line is a means that connects the photographer terminal (20) and the distribution server (10) so that they can communicate with each other, such as the Internet or a mobile phone network, and the communication may be wired or wireless.

[0025] This means that when streaming video from multiple distributors connected via communication lines, no conversion processing is required on the distribution server, which reduces the time lag in the distribution from the distribution server to the viewer terminal during streaming, and also reduces the load on the CPU and storage device of the distribution server, making it possible to realize a distribution server that records archive video data at multiple resolutions at low cost.

[0026] [2] <Photographer Terminal (20)> A photographer terminal (20) comprising the following components: a video imaging unit (21) that generates video data from a captured video, a video data recording unit (22) that records the video data at multiple resolutions for each predetermined time period, a measurement unit (23) that measures the transmission rate of a communication line for streaming transmission to a distribution server (10), a streaming transmission determination unit (24) that determines a combination of resolutions that can be streamed from the multiple resolutions for each predetermined time period within the measured transmission rate, a streaming transmission unit (25) that streams video data of the determined combination of resolutions to the distribution server (10), an upload request receiving unit (26) that receives an upload request from the distribution server (10) after streaming transmission, and an upload transmission unit that transmits video data of the time period and resolution specified by the upload request from the video data recording unit (22) to the distribution server (10).

[0027] This means that when streaming video from multiple distributors connected via communication lines, no conversion processing is required on the distribution server, which reduces the time lag in the distribution from the distribution server to the viewer terminal during streaming, and also reduces the load on the CPU and storage device of the distribution server, making it possible to realize a distribution server that records archive video data at multiple resolutions at low cost.

[0028] This means that when streaming video from multiple distributors connected via communication lines, archive video data in multiple resolutions can be recorded on the server without performing conversion processing on the server, with less load on the server and at low cost.

[0029] [3] <Distribution Server (10)> A distribution server (10) comprising the following components: a streaming receiving unit (11) that receives video data at one or more resolutions streamed from a videographer terminal (20), a streaming distribution unit (12) that streams at least a portion of the received video data to a viewer terminal (30), an archive storage unit that records the received video data for each predetermined time period and resolution, an upload request unit (14) that, after streaming distribution, detects missing time periods for each of multiple resolutions from the video data in the archive recording unit (13) and transmits an upload request to the videographer terminal (20) to request transmission of video data for the detected time period and resolution, and an upload video receiving unit (15) that receives the video data uploaded from the videographer terminal (20) and records it in the archive recording unit (13).

[0030] This means that when streaming video from multiple distributors connected via communication lines, archive video data in multiple resolutions can be recorded on the server without performing conversion processing on the server, with less load on the server and at low cost.

[0031] [4] <Distribution to viewer terminal (30)> The distribution server (10) described in [1] to [3] further comprises the following configuration: An archive transmission unit (16) that measures the transmission rate of the communication line between the server and the viewer terminal (30), determines the maximum resolution that can be streamed within the range of the measured transmission rate, and transmits video data of the determined resolution stored in the archive recording unit (13) to the viewer terminal (30).

[0032] This makes it possible to provide a distribution server that allows a viewer terminal to view archived video data stored in the distribution server at an optimal resolution depending on the status of the communication line between the distribution server and the viewer terminal. 2. Details of the embodiment The embodiment will be described in further detail below.

[0033] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of a streaming distribution system including a distribution server according to a first embodiment.

[0034] As shown in FIG. 1 , the streaming distribution system 1 comprises a photographer terminal 20, a distribution server 10, and a viewer terminal 30, all connected via a communication line 40. In practice, multiple photographer terminals 20 and multiple viewer terminals 30 are connected to one distribution server 10. However, for the sake of convenience, the following description will be given assuming that one distribution server 10 and one viewer terminal 30 are connected to the distribution server 10. The communication line 40 may be, for example, the Internet or a mobile phone network, and the communication may be wired or wireless. <Configuration of the Distribution Server> The distribution server 10 shown in FIG. 1 comprises a streaming receiving unit 11, a streaming distribution unit 12, an archive recording unit 13, an upload request unit 14, an upload video receiving unit 15, and an archive transmitting unit 16. More specifically, the distribution server 10 may be a server in which the streaming receiving unit 11, the streaming distribution unit 12, the upload request unit 14, and the upload video receiving unit 15 have interfaces for connection to a communication line such as the Internet, and is preferably implemented using a cloud server. In this case, it is preferable that the required functions of each functional unit such as the streaming receiving unit 11 are realized by running a program (software) on a computer that constitutes the cloud server.

[0035] The streaming receiving unit 11 receives streaming distribution video data from a plurality of photographer terminals 20 via a communication line 40. The streaming receiving unit 11 is configured to be able to simultaneously and independently receive streaming distribution video from a plurality of photographer terminals 20.

[0036] The streaming distribution unit 12 distributes the streaming distribution video data received by the streaming receiving unit 11 to the viewer terminal 30 via the communication line 40. The streaming distribution unit 12 includes a buffer unit (not shown), temporarily records the video data in the buffer unit at predetermined intervals for a length of several seconds to several tens of seconds, and distributes the temporarily recorded video data in the buffer unit to the viewer terminal 30 as needed. The streaming distribution video data is encoded by the photographer terminal 20 using an existing image compression method. Image compression formats include MPEG2, MPEG4, Motion JPEG, etc., but MPEG2 is used here. The streaming distribution video data consists of video data of multiple resolutions and includes resolution flag data indicating the resolution of the included video data. The streaming distribution unit 12 measures the transmission rate of the communication line between the viewer terminal 30 using a measurement unit (not shown), selects a resolution appropriate for the transmission rate from the resolutions included in the streaming distribution video data received from the photographer terminal 20, and performs streaming distribution to the viewer terminal 30.

[0037] The archive recording unit 13 records the streaming distribution video data received by the streaming receiving unit 11 on a recording medium. The streaming distribution video data received by the streaming receiving unit 11 is distributed in real time to the viewer terminal 30 via the streaming distribution unit 12, and the recording of the streaming distribution video data by the archive recording unit 13 is performed simultaneously with the streaming distribution. Since the streaming distribution video data includes video data of one or more resolutions, it is recorded for each resolution and saved in sections at predetermined time intervals. Alternatively, it may be saved as a single file with additional data added as needed. The video data recorded in the archive recording unit includes the resolution flag data described above.

[0038] The upload request unit 14 determines which video data of a time period and resolution has not yet been transmitted from the photographer terminal 20 among the video data recorded in the archive recording unit 13, and transmits an upload request to the photographer terminal 20 requesting transmission of the video data of that time period and resolution. To make this determination, the resolution flag data recorded in the archive recording unit 13 is referenced. The method of determination using the resolution flag data will be described later.

[0039] The upload video receiving unit 15 records the video data uploaded and transmitted from the photographer terminal 20 in response to the upload request in the archive recording unit 13. By merging the video data with the video data already recorded in the archive recording unit 13, video data for all time periods and all predetermined resolutions is recorded in the archive recording unit 13.

[0040] The archive transmitter 16 distributes video data from the archive recorder 13 to the viewer terminal 30 in response to a request from the viewer terminal 30. This distribution occurs after streaming distribution, and provides an archive of the streaming distribution video data via so-called on-demand distribution. At this time, the archive recorder 13 stores video data at multiple resolutions for each time period, and the archive transmitter 16 selects and transmits video data with the optimal time period and resolution based on the communication rate of the communication line between the viewer terminal 30 and the archive recorder 13. <Configuration of the Photographer Terminal> The photographer terminal 20 shown in FIG. 1 includes a video capture unit 21, a video data recording unit 22, a measurement unit 23, a streaming transmission determination unit 24, a streaming transmission unit 25, an upload request receiving unit 26, and an upload transmission unit 27. More specifically, the photographer terminal 20 may be a terminal equipped with a camera in the video capture unit 21, and suitable devices include a smartphone, a tablet, a digital camera with communication functions, and a laptop computer. In this case, it is preferable that the required functions of each functional unit such as the video imaging unit 21 are realized by running a program (software) on a terminal such as a smartphone.

[0041] The video capture unit 21 includes a lens and an image sensor, captures video images, and converts the captured video images into digital data. The video capture unit 21 also includes an MPEG encoder (not shown), which compresses and encodes the digital video data to generate video image data. The MPEG encoder outputs video image data at multiple resolutions. Specifically, the multiple resolutions are, for example, three levels of resolution: low-quality SD (854 x 480 pixels), standard-quality HD (1280 x 720 pixels), and high-quality Full HD (1980 x 1080 pixels). The number of resolution levels may be smaller or larger, and the resolution is not limited to the above standards and may be another standard or any other resolution. The video capture unit 21 transmits the generated video image data to the video data recording unit 22 and the streaming transmission unit 25.

[0042] The video data recording unit 22 records the video data transmitted from the video imaging unit 21 on a recording medium. As described above, the video data is generated at multiple resolutions, recorded for each resolution, and saved in sections at predetermined time intervals. Alternatively, the video data may be saved as a single file with additional data added as needed.

[0043] The measurement unit 23 measures the transmission rate of the communication line 40 between the distribution server 10. The measurement of the transmission rate is calculated, for example, based on an ACK signal transmitted from the distribution server 10 at predetermined time intervals. The ACK signal is an acknowledgment that the video data has been correctly received from the distribution server 10, and can be calculated, for example, based on information such as the difference between the time the video data was transmitted from the photographer terminal 20 and the time the ACK signal was received.

[0044] The streaming transmission determination unit 24 determines a combination of resolutions of the video data to be transmitted to the distribution server 10 based on the transmission rate determined by the measurement unit 23. The streaming transmission determination unit 24 generates resolution flag data based on the determined combination of resolutions and transmits it to the streaming transmission unit 25. Details of the resolution flag data will be described later.

[0045] The streaming transmission unit 25 transmits to the distribution server 10, from among the moving image data received from the moving image capturing unit 21 based on the resolution flag data determined by the streaming transmission determination unit 24, moving image data of the specified resolution with the resolution flag data attached thereto.

[0046] The upload request receiving unit 26 receives an upload request sent from the distribution server 10 after streaming distribution. The upload request is data that indicates the time period and resolution of video data required for archiving the streaming distribution video on the distribution server 10 and that is requested to be transmitted to the photographer terminal 20. Details will be described later.

[0047] In response to an upload request received by the upload request receiving unit 26, the upload sending unit 27 transmits video data of a specified time period and resolution from the video data stored in the video data recording unit 22 to the distribution server 10. <Video Data Stored on the Photographer's Terminal> Figure 2 shows the structure of video data stored in the video data recording unit 22 of the photographer's terminal 20. The horizontal direction in Figure 2 represents the time axis, and lines a through c represent the structure of video data at each resolution. The lines a through c represent full HD, HD, and SD video data, respectively. Video data at each resolution is composed of consecutive frames, and the frame number indicates the number of frames on the time axis from the beginning of the video data. Frames with the same frame number in video data at each resolution represent the same image but at different resolutions. The multiple frames surrounded by dashed lines in Figure 2 represent a Group of Pictures (GOP), which is the unit of inter-frame compression in MPEG-2. As shown in Figure 2, a GOP consists of I frames, P frames, and B frames. I frames are compressed and encoded independently of other frames (intra-frame encoding) using only the pixel data within the frame. P frames are frames that are compression-encoded using inter-frame prediction, which exploits the correlation with frames that exist earlier in time. B frames are frames that are compression-encoded using inter-frame prediction, which exploits the correlation between frames that exist earlier and later in time. In this embodiment, six frames constitute a GOP, and the transmission resolution is switched in response to changes in the transmission state. The GOP unit is not limited to six frames; any number of frames, such as 15 frames, may be used as long as the GOP contains at least one I-frame at the beginning. The GOP numbers in Figure 2 indicate the number of GOPs in each video on the time axis from the beginning of the video data of the GOP. <Resolution Flag Data> Figure 3 shows the structure of the transmission rate measured by the measurement unit 23 and the resolution flag data determined by the streaming transmission determination unit 24. The horizontal axis indicates elapsed time, and in this embodiment, the elapsed time is shown in units of frames or GOPs. The measurement unit 23 measures the transmission rate in predetermined time intervals, and the resolution flag data is determined at intervals of one GOP or a multiple of a GOP. That is, the change in the combination of resolutions of the moving image data to be transmitted by the streaming transmission determination unit 24 is performed in units of an integral multiple of a GOP.The transmission rate in FIG. 3 represents the change in the transmission rate measured by the measurement unit 23, where a represents the frame number, b represents the GOP number given according to the elapsed time, and c represents the resolution flag data.

[0048] The example in Figure 3 shows the transmission rate changing from state A to state D, C, B, and A. State A is the best state, then deteriorates to state D, and recovers from state C to state B. In state A, video data of all resolutions can be transmitted. State B is the worst state, in which only video data of the lowest resolution, SD, can be transmitted. State C is an improvement over state B, in which video data of SD and HD resolutions can be transmitted, and state D is an even better state, in which video data of full HD and HD resolutions can be transmitted.

[0049] The resolution flag data is expressed as a three-digit binary number, and indicates the transmission status (full HD, HD, SD) from the left digit to the right digit. A 1 in each digit indicates that video data of the corresponding resolution can be transmitted. Therefore, in state A, the resolution flag data is 111, in state B it is 001, in state C it is 011, and in state D it is 110. The resolution flag data is associated with the GOP number and transmitted to the streaming transmitter 25 in combinations of b and c in Figure 3. <Streaming Transmission Data> Figure 4 shows the structure of video data transmitted from the streaming transmitter 25. The streaming transmitter 25 transmits video data to the distribution server 10 based on the resolution flag data received from the streaming transmission determination unit 24. Here, an example of streaming transmission video data determined based on the resolution flag data shown in Figure 3 is shown. Since the resolution flag data for GOP number 1 is 111, video data of all resolutions can be transmitted, and the streaming transmitter 25 includes video data of all resolutions of the video data received from the video capture unit 21 in the streaming transmission data. At this time, the resolution flag data and GOP number are also transmitted. Similarly, for states B, C, and D, the streaming transmission data includes a combination of video data with the resolution specified in the resolution flag data, and the streaming transmission data is configured together with the resolution flag data and GOP number. FIG. 5 shows the configuration of video image data stored in the archive recording unit 13 of the distribution server 10. The video image data is received by the streaming receiving unit 11 and recorded in the archive recording unit 13, and has the same configuration as FIG. 4. <Operation> The overall operation of the distribution server 10 and the photographer terminal 20 will be described with reference to FIGS. 6 to 9. FIGS. 6 and 8 are operational flowcharts of the photographer terminal 20, with FIG. 6 showing the operation during streaming distribution and FIG. 8 showing the operation after streaming distribution. FIGS. 7 and 9 are operational flowcharts of the distribution server 10, with FIG. 7 showing the operation during streaming distribution and FIG. 9 showing the operation after streaming distribution.

[0050] When a user instructs the photographer terminal 20 to perform streaming distribution, the photographer terminal 20 notifies the distribution server 10, and the distribution server 10 notifies the photographer terminal 20 that it is ready to start streaming distribution. The user then presses the capture button to begin shooting (S10). When shooting begins, the video capture unit 21 converts the captured video into digital data. The video capture unit 21 then compresses and encodes the digital data of the video using an MPEG encoder to generate video data of multiple resolutions, and sequentially transmits the generated video data to the video data recording unit 22 and the streaming transmission unit 25 (S11). The multiple resolutions specifically refer to three levels of resolution: low-quality SD (854 x 480 pixels), standard-quality HD (1280 x 720 pixels), and high-quality Full HD (1980 x 1080 pixels).

[0051] The moving image data recording unit 22 records the moving image data transmitted from the moving image capturing unit 21 (S12). The structure of the moving image data recorded in the moving image data recording unit 22 is as explained above with reference to Fig. 2, and the moving image data is saved for each resolution, and the moving image data transmitted from the moving image data recording unit 22 is recorded as needed during streaming distribution.

[0052] The measurement unit 23 measures the transmission rate of the communication line 40 between the distribution server 10 and the streaming transmission determination unit 24, and notifies the measured transmission rate (S13). The measurement of the transmission rate is calculated based on an ACK signal transmitted from the distribution server 10 at predetermined time intervals.

[0053] The streaming transmission determination unit 24 determines a combination of resolutions of video data that can be streamed to the distribution server 10 based on the transmission rate notified by the measurement unit 23, and transmits resolution flag data representing the combination to the streaming transmission unit 25 (S14). The specific configuration of the resolution flag data is as described above with reference to FIG. 3.

[0054] The streaming transmission unit 25 selects a combination of resolutions to transmit the moving image data received from the moving image capturing unit 21 based on the resolution flag data received from the streaming transmission determination unit 24, and transmits the selected combination to the distribution server 10 (S15). The configuration of the moving image data to be transmitted is as described above with reference to Fig. 4. The current combination of resolutions is maintained until new resolution flag data is received, and the combination is changed when there is a change in the resolution flag data.

[0055] The photographer terminal 20 determines whether there is an instruction from the user to complete the image capture (S16). If there is an instruction to complete, the photographer terminal 20 stops capturing the moving images, transmits an instruction to the distribution server 10 to end the streaming distribution, and ends the streaming distribution (S17). If there is no instruction to complete, the photographer terminal 20 continues capturing the moving images and streaming distribution, and returns to S11 to continue the operation.

[0056] Although FIG. 6 is expressed as a flowchart to facilitate understanding, the steps are not necessarily executed sequentially but are executed simultaneously in parallel.

[0057] The distribution server 10 receives the video data transmitted from the photographer terminal 20 in S15 at the streaming receiving unit 11 (S20). The video data is sequentially transmitted in units of multiple GOPs from the photographer terminal 20 during streaming distribution. The streaming receiving unit 11 receives the video data in units of multiple GOPs and sends it to the streaming distribution unit 12 and the archive recording unit 13.

[0058] The streaming distribution unit 12 streams the video data received from the streaming receiving unit 11 to multiple viewer terminals 30 via a communication line (S21). The streaming distribution unit 12 has a buffer unit (not shown), temporarily records the video data in the buffer unit at predetermined time intervals so that the data has a length of several seconds to several tens of seconds, and distributes the video data temporarily recorded in the buffer unit to the viewer terminals 30 as needed. The streaming distribution video data consists of video data at multiple resolutions and includes resolution flag data indicating which resolutions of video data are included. The streaming distribution unit 12 measures the transmission rate of the communication line between the viewer terminals 30 and performs streaming distribution at a resolution appropriate for the transmission rate from among the multiple resolutions of the streaming distribution video data.

[0059] The archive recording unit 13 records the video data received from the streaming receiving unit 11 in the storage means (S22). The structure of the recorded video data is as shown in FIG. 5. The video data is recorded for each resolution and stored in association with a GOP number and a resolution flag. The recording is added to the video data already recorded in units of multiple GOPs transmitted from the streaming receiving unit 11. A file may be created and recorded for each unit of multiple GOPs.

[0060] The streaming receiving unit 11 determines whether or not it has received a notification of the end of streaming from the photographer terminal 20, and if so, stops receiving the streaming and proceeds to the process of S24 (S23).If it has not received the notification of the end of streaming, it returns to the process of S20.

[0061] The upload request unit 14 references the video data recorded in the archive recording unit 13, and creates and transmits an upload request to the photographer terminal 20 to upload video data of a predetermined resolution and time period (S24).

[0062] The method and structure of an upload request will be described with reference to FIG. 5 . FIG. 5 shows video data stored in the archive storage unit 13. Video data is stored as files for each resolution, from the start to the end of streaming. Each video data file is organized into a single file for multiple GOPs or a single file for the entire streaming time. However, in this embodiment, regardless of the file storage unit, the frames and GOPs that make up the video data are described as the division units of the video data. Video data at each resolution has missing GOP-by-GOP portions. For example, in full HD video data, it can be seen that data is missing in GOP numbers 2 and 3. Similarly, in HD, it can be seen that data is missing in GOP number 2, and in SD, it can be seen that data is missing in GOP number 4. This can be determined by referencing the resolution flag for each GOP. Specifically, in the example of FIG. 5 , the resolution flag data for GOP numbers 1 to 4 are 111, 001, 011, and 110, respectively, and can be determined by referencing the GOP number and the 0 digits in the resolution flag data.

[0063] When generating an upload request, it is sufficient to request the data for the referenced 0, so data with the 0 and 1 of the resolution flag data can be generated. Specifically, the upload request for the video data in Figure 5 will be 000, 110, 100, and 001 for GOP numbers 1 to 4, respectively. This shows that upload is requested for GOP numbers 2 and 3 for full HD, GOP number 2 for HD, and GOP number 4 for SD. Therefore, the upload request is data that associates the GOP number with this three-digit code. The upload request may be in any format other than the above, as long as it can indicate the missing resolution for each GOP.

[0064] In the photographer terminal 20, the upload request receiving unit 26 receives the upload request sent from the distribution server 10 and sends the upload request to the upload sending unit (S31).

[0065] The upload transmission unit 27 reads the video data from the video data storage unit 22 in accordance with the received upload request (S32). In this embodiment, the upload request is as described above, and in accordance with this, data of GOP numbers 2 and 3 for full HD, GOP number 2 for HD, and GOP number 4 for SD is read.

[0066] The upload transmission unit 27 transmits the video image data read from the video data storage unit 22 to the distribution server 10 (S33). Because real-time transmission is not required, a protocol other than streaming, such as FTP, is suitable. FTP reduces communication overhead and allows for retransmission or other processing if an error occurs during transmission. For security reasons, encrypted SFTP may be used, or other known protocols may be used.

[0067] The upload video receiving unit 15 of the distribution server 10 receives the video data transmitted from the upload transmitting unit 27 (S41). The upload video receiving unit 15 merges the video data received in S41 with the video data in the archive storage unit 13 (S42). As a result, the archive storage unit 13 records all the GOP number data for all resolutions of video data.

[0068] The streaming distribution unit 12 transmits the video data recorded in the archive recording unit 13 to the viewer terminal 30 in response to a request from the viewer terminal 30 (S43). This is so-called on-demand distribution, and the viewer terminal 30 can request viewing at any time after the end of the streaming distribution. The streaming distribution unit 12 measures the transmission rate of the communication line with the viewer terminal 30, and transmits video data with the optimal resolution from the video data recorded in the archive recording unit 13.

[0069] According to this embodiment, when streaming content is distributed from the photographer terminal 20, conversion processing from high-definition video to low-definition video is not performed in the distribution server 10, so streaming distribution can be performed with little time lag, and even when storing archive video data of multiple resolutions, the load on the distribution server 10 can be reduced. Furthermore, even if the condition of the communication line between the distribution server 10 and the photographer terminal 20 is poor, video data of multiple resolutions can be stored after the fact.

[0070] REFERENCE SIGNS LIST 1 Streaming distribution system 10 Distribution server 11 Streaming receiving unit 12 Streaming distribution unit 13 Archive recording unit 14 Upload request unit 15 Upload video receiving unit 16 Archive transmitting unit 20 Photographer terminal 21 Video shooting unit 22 Video data recording unit 23 Measurement unit 24 Streaming transmission determining unit 25 Streaming transmitting unit 26 Upload request receiving unit 27 Upload transmitting unit 30 Viewer terminal

Claims

1. A streaming distribution system including a photographer terminal and a distribution server connected to each other via a communication line, wherein the photographer terminal comprises: a video capture unit that generates video data from a captured video; a video data recording unit that records the video data at a plurality of resolutions for each predetermined time period; a measurement unit that measures a transmission rate of a communication line for streaming transmission to the distribution server; a streaming transmission determination unit that determines a combination of resolutions that can be streamed within the measured transmission rate range from the plurality of resolutions for each predetermined time period; a streaming transmission unit that streams video data of the determined combination of resolutions to the distribution server; an upload request receiving unit that accepts an upload request from the distribution server after the streaming transmission; and an upload transmission unit that uploads video data of the time period and resolution specified in the upload request from the video data recording unit to the distribution server in accordance with the upload request; and the distribution server comprises: a streaming receiving unit that receives the video data streamed from the photographer terminal; a streaming distribution unit that streams at least a portion of the received video data to a viewer terminal; an archive recording unit that records the received video data for each time period and resolution; an upload request unit that detects, after the streaming distribution, time periods that are missing from the video data in the archive recording unit for each of the plurality of resolutions, and transmits an upload request to the photographer terminal to request transmission of the video data for the detected time periods and resolutions; and an upload video receiving unit that receives the video data uploaded from the photographer terminal and records it in the archive recording unit.

2. A photographer terminal comprising: a video imaging unit that generates video data from captured video; a video data recording unit that records the video data at multiple resolutions for each predetermined time period; a measurement unit that measures the transmission rate of a communication line for streaming transmission to a distribution server; a streaming transmission determination unit that determines a combination of resolutions that can be streamed from the multiple resolutions for each predetermined time period within the range of the measured transmission rate; a streaming transmission unit that streams video data of the determined combination of resolutions to the distribution server; an upload request receiving unit that accepts an upload request from the distribution server after the streaming transmission; and an upload transmission unit that transmits video data of the multiple resolutions from the video data recording unit to the distribution server, the video data of the time period and resolution specified by the upload request.

3. A distribution server comprising: a streaming receiving unit that receives video data of one or more resolutions streamed from a photographer's terminal; a streaming distribution unit that streams at least a portion of the received video data to a viewer's terminal; an archive storage unit that records the received video data for a predetermined time period and resolution; an upload request unit that, after the streaming distribution, detects time periods that are missing for each of the multiple resolutions from the video data in the archive recording unit and sends an upload request to the photographer's terminal to request the transmission of video data of the detected time period and resolution; and an upload video receiving unit that receives the video data uploaded from the photographer's terminal and records it in the archive recording unit.

4. The distribution server according to claim 1 or 3, further comprising an archive transmission unit which measures the transmission rate of the communication line between the viewer terminal, determines the maximum resolution that can be streamed within the range of the measured transmission rate, and transmits video data of the determined resolution stored in the archive recording unit to the viewer terminal.

Images