Bitrate selection device, bitrate selection method and program

Abstract

This bit rate selection device has: a selection unit configured so as to select the image bit rate for each combination from multiple levels of values in a manner such that the sum of the image bit rate for each combination between a plurality of first terminals and a plurality of second terminals is the smallest possible under the condition that the experience quality pertaining to the image of each user of said plurality of second terminals which receive images transmitted from each of said plurality of first terminals pertaining to online real time communication via a distribution device is equal to or greater than a threshold; and an instruction unit configured so as to transmit an encoding instruction at one or more bit rates selected by the selection unit for a combination pertaining to said first terminals. As a result, the possibility that an excessive bit rate is selected when encoding an image is reduced.

Description

[Technical Field]

[0001] The present invention relates to a bit rate selection device, a bit rate selection method, and a program. [Background technology]

[0002] In recent years, the use of online real-time communication services such as web conferencing has increased as teleworking has become more common. Technologies such as WebRTC are used to enable such services. WebRTC is a technology standardized by the World Wide Web Consortium and the Internet Engineering Task Force.

[0003] By utilizing such technologies, it is now possible to realize online real-time communication services. However, in order to provide such services to end users continuously, it is necessary to improve the user's quality of experience (QoE) while reducing the operational costs (network equipment costs, etc.) involved in providing the services.

[0004] WebRTC has proposed three architectures for realizing multi-person conferences: (1) P2P, which connects participating terminals in a full mesh, (2) Multi-point Control Unit, in which each client sends video data to a server, which transcodes the data and distributes it to participating clients, and (3) SFU (Selective Forwarding Unit), in which each client sends video data to a server, and the server distributes the video data as is to participating clients.

[0005] The SFU method delivers video of the same quality to all participating clients, so this method has the problem that if there is a participant with poor network conditions, the overall quality will be reduced to accommodate that participant.

[0006] To address this issue, the Simulcast method has been proposed. In Simulcast, the source client sends video at multiple levels of quality (bit rate, resolution, frame rate) to the server. The server selects the quality that best suits each client's network conditions and distributes the video at the selected quality. For example, the source client encodes video at high quality (1Mbps / 720p / 30fps), medium quality (480kbps / 480p / 30fps), and low quality (128kbps / 180p / 15fps) and sends it to the distribution server. The distribution server distributes high quality to destination clients with sufficiently wide download bandwidth and low quality to clients with narrow bandwidth.

[0007] The quality levels in Simulcast can be set manually, or alternatively, the selectable quality (bit rate, resolution, frame rate) can be changed appropriately depending on the network conditions of the receiving client (Non-Patent Document 1).

[0008] On the other hand, a control method has been proposed for WebRTC that takes into consideration not only quality improvement but also costs. This control maintains QoE by controlling the video bitrate of each terminal so that the target QoE set by the service provider is achieved, while reducing the amount of data transfer, thereby reducing the operating costs of the service provider (Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] S.Petrangeli, "Dynamic video bitrate adaptation for WebRTC-based remote teaching applications", IEEE, 2018 [Non-patent document 2] T. Kimura, T. Kimura, A. Matsumoto and J. Okamoto, "BANQUET: Balancing Quality of Experience and Traffic Volume in Adaptive Video Streaming"", IEEE CNSM 2019, Feb 2020 [Non-patent document 3] Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport, Recommendation ITU-T P.1203, 2017, [online], Internet<URL:https: / / www.itu.int / rec / T-REC-P.1203> Summary of the Invention [Problem to be solved by the invention]

[0010] With existing Simulcast, quality levels (video bitrate, resolution, frame rate) are set according to the knowledge of the person configuring the settings and the network conditions. This means that there is a risk that encoding will be set at an excessively high bitrate. In such cases, if the network conditions are very good, an excessive bitrate may be selected, and a high bitrate may be selected even though there is little improvement in QoE, which poses a risk of increasing the operational costs for the network and server equipment required by the service provider.

[0011] To address the issue of increasing data volumes as described above, a video bitrate control technology (Non-Patent Document 2) has been proposed to reduce the amount of data transferred, but this technology is targeted at video distribution services and does not take into account the Simulcast web conferencing service. Given these circumstances, a method is needed to appropriately control the multiple bitrate patterns uploaded by each terminal in order to suppress excessive quality in the Simulcast method.

[0012] The present invention has been made in view of the above-mentioned points, and has as its object to reduce the possibility that an excessive bit rate is selected when encoding video. [Means for solving the problem]

[0013] In order to solve the above problem, the bit rate selection device has a selection unit configured to select the bit rate of the video for each combination of the multiple first terminals and the multiple second terminals from multiple levels of values so that the sum of the bit rates of the video for each combination of the multiple first terminals and the multiple second terminals is minimized, under the constraint that the quality of experience of the video for each user of the multiple second terminals that receives the video transmitted from each of the multiple first terminals via a distribution device for online real-time communication is above a threshold, and an instruction unit configured to send to each of the first terminals an instruction to encode at one or more bit rates selected by the selection unit for the combination related to the first terminal. [Effects of the Invention]

[0014] This reduces the possibility that an excessive bit rate will be selected when encoding video. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of the configuration of an online real-time communication system according to a first embodiment. [Figure 2]2 is a diagram illustrating an example of a hardware configuration of a control server 10 according to the first embodiment. FIG. [Figure 3] 2 is a diagram illustrating an example of a functional configuration of a client terminal 20 and a control server 10 according to the first embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0016] In this embodiment, the upload bitrate is controlled for each source and destination based on quality information collected from clients, client information, available upload bandwidth, and available download bandwidth information, so that the estimated QoE (Quality of Experience) from T1 seconds ago to T2 seconds into the future approaches the target QoE. This makes it possible to reduce data traffic while maintaining the required QoE.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of an online real-time communication system in a first embodiment. In Fig. 1, a plurality of client terminals 20 are connected to a distribution server 30 and a control server 10 via a network such as the Internet.

[0018] Each of the multiple client terminals 20 is a communication terminal such as a PC (Personal Computer), smartphone, or tablet terminal used by a user participating in online real-time communication such as a web conference. The client terminals 20 transmit various logs to the control server 10, encode and transmit data related to media (video, audio, etc.) (hereinafter referred to as "media data") in accordance with instructions from the control server 10, and receive and play media data from other participants. Each client terminal 20 transmits, for example, media data of its own user. Each client terminal 20 also receives media data from other terminals and displays a screen including the respective media data.

[0019] The control server 10 is one or more computers that instruct each client terminal 20 on the upload bit rate.

[0020] The distribution server 30 is one or more computers that distribute media data transmitted (uploaded) from each client terminal 20 to other client terminals 20. When distributing to other client terminals 20, the distribution server 30 distributes at an optimal bit rate according to the download bandwidth of the other client terminals 20. That is, the distribution server 30 distributes media data at a bit rate according to the download bandwidth of each client terminal 20 by using WebRTC Simulcast.

[0021] Fig. 2 is a diagram showing an example of the hardware configuration of the control server 10 in the first embodiment. The control server 10 in Fig. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, and an interface device 105, which are all interconnected via a bus B.

[0022] A program that realizes processing in the control server 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0023] When an instruction to start a program is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The CPU 104 executes functions related to the control server 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0024] The client terminal 20 and the distribution server 30 may also have the same hardware configuration as that shown in Fig. 2. However, the client terminal 20 has a display device and speaker for outputting media data, a camera and microphone for inputting media data, etc.

[0025] FIG. 3 is a diagram illustrating an example of the functional configuration of the client terminal 20 and the control server 10 according to the first embodiment.

[0026] 3, the client terminal 20 includes a client log transmission unit 21, an upload bit rate control unit 22, a transmission data encoding unit 23, and a reception data decoding unit 24. Each of these units is realized by a process executed by a CPU of the client terminal 20 by one or more programs installed in the client terminal 20.

[0027] The control server 10 includes a client log collection unit 11, a target QoE input unit 12, an upload bit rate selection unit 13, and an upload bit rate instruction unit 14. Each of these units is realized by a process executed by the CPU 104 of one or more programs installed in the control server 10.

[0028] The processing procedure executed in this embodiment will be described with reference to Figure 3. Unless otherwise specified, the following processing is executed during online real-time communication such as a Web conference (hereinafter simply referred to as a "Web conference"). The following processing is executed for each of the multiple client terminals 20 participating in the Web conference.

[0029] The client log sending unit 21 of each client terminal 20 periodically (for example, at one-second intervals) collects a log (hereinafter referred to as a "client log") from the client terminal 20 after the start of the Web conference, which includes information necessary for quality estimation (information at that interval) and information necessary for upload bit rate estimation (information at that interval), and sends the collected client log to the control server 10.

[0030] The information necessary for quality estimation includes, for example, information about the quality of each received video (bit rate, playback stop information (playback stop time, number of times, interval, etc.), frame rate, resolution), and information about the client terminal 20 (information that can identify the type of the terminal being used (own terminal), display size (display area)). Each received video is synonymous with each other client terminal 20 (video from each other). For example, if A, B, and C are holding a conference, each received video at A's client terminal 20 refers to each video of B and C. The display size is collected for each received video because it is assumed that the display size (i.e., the size on the screen at which other participants are displayed) may differ for each received video. For example, if A, B, and C are holding a conference, this embodiment allows for a situation in which B's display size and C's display size differ on A's client terminal 20.

[0031] The information necessary for upload bit rate estimation is, for example, the available upload bandwidth and available download bandwidth of each user (each client terminal 20).

[0032] The client log collection unit 11 of the control server 10 receives the client log transferred from the client log transmission unit 21. Note that if the information included in the client log can be collected on the control server 10 side, it may be collected on the control server 10 side.

[0033] The control server 10 basically executes the following process in response to receiving a client log: However, the following process does not have to be executed every time a client log is received.

[0034] The upload bit rate selection unit 13 of the control server 10 calculates the upload bit rate for each source and destination based on formula (1) so as to minimize the amount of data to be transferred (the sum of the bit rates for each source and destination).

[0035]

number

[0036] Furthermore, time t is a preset timing for changing the upload bit rate, and the interval is the same as the collection cycle of the client log, for example.

[0037] Here, in order to achieve the target QoE, the upload bit rate selection unit 13 performs the calculation of equation (1) so as to satisfy the following three constraints, Constraints 1 to 3.

[0038] (constraint 1) Constraint 1 is expressed by the following equation (2).

[0039]

number

[0040] (constraint 2) Constraint 2 is expressed by the following equation (3).

[0041]

number

[0042] (constraint 3) Constraint 3 is expressed by the following equation (4).

[0043]

number

[0044] Equations (2) to (4) represent constraints on the optimization problem represented by equation (1).

[0045] In the calculation of equation (1), the upload bit rate selector 13 selects an upload bit rate from a group of bit rates (L) of multiple stages. The group of bit rates (L) is given in advance. The minimum QoE (hereinafter referred to as "QoE") that the service provider must guarantee is lower The maximum QoE (hereinafter referred to as "QoE") that we want to achieve from upper The bitrate group (L) is set so that the QoE is distributed according to the number of selectable bitrates (N) between the QoE intervals. The formula for calculating the bitrate when the QoE is set at regular intervals using the QoE estimation model is shown in Equation (5).

[0046]

number

[0047] QoE of each user in Eq. (2) u is calculated from the video transmitted by other users. Specifically, QoE uis the QoE value estimated for the section from T1 seconds before to T2 seconds after the current time using the following equation (6).

[0048]

number

[0049] Furthermore, f() on the outermost side of the right side of equation (6) represents a formula for estimating long-term QoE. In this embodiment, control is assumed to bring the QoE value for a certain period (for example, 1 minute when T1=30 seconds and T2=30 seconds) closer to the target QoE. In this case, as in Non-Patent Document 3, the QoE value is calculated every t (for example, every 1 second) within this period, and these are integrated to estimate the QoE value for the certain period. The QoE value for every t (for example, 1 second) is calculated within f() in equation (6). At this time, since each client terminal 20 displays videos of multiple other participants, f QoE The weighted average of the estimated values by the display size (=DS in equation (6)) is calculated, and the results are integrated by f().

[0050] Specifically, the calculation of equation (6) is performed as follows. (Step 1) The upload bit rate selection unit 13 uses the content of f() in equation (6) to calculate the QoE value for each u for each t from the client log at each t in the interval from T1 seconds before the current time. (Step 2) The upload bit rate selection unit 13 uses the content of f() in equation (6) to calculate the QoE value for each u for each t in the section T2 seconds after the current time. i,u (t)) is selected from L. For example, BR is selected so that the bit rate decreases from a relatively high bit rate to a relatively low bit rate as t progresses. i,u The value of (t) may be selected such that there is a rise in BR from relatively low bit rates to relatively high bit rates. i,u The value of (t) may be selected. The bit rate may be selected from L based on other rules. i,u (t), RES i,u (t), DS i,u (t)) uses the latest value of the client log, and playback stop information (STOP i,u (t)) can be set to 0. Also, some client applications may determine the frame rate and resolution depending on the selected bit rate. In such cases, the resolution and frame rate may be calculated according to the correspondence formula for bit rate, frame rate, and resolution used by the client application. (Step 3) The upload bitrate selector 13 integrates the QoE value for each t calculated in step 1 and the QoE value for each t calculated in step 2 for each u using f() in equation (6), and calculates the QoE u Calculate (estimate)

[0051] Note that the QoE calculated in step 3 u By calculating Equation (1) within a range that satisfies the constraint that exceeds the target QoE (Equation (2)) and also satisfies the constraints of Equations (3) and (4), the upload bitrate selector 13 selects BR i,u Select (t).

[0052] In this case, the ULBW in Equation (3) and Equation (4) i (t), DLBW u (t) is the time when the available upload bandwidth included in the client log from the client log transmission unit 21 continues for T2 seconds. i (t), DLBW u The latest client log value can be substituted for (t). i (t), DLBW u (t) may be estimated.

[0053] The upload bit rate instruction unit 14 receives from the upload bit rate selection unit 13 the upload bit rate (BR i,u By transmitting the upload bit rate (t)) to the client terminal 20 associated with each i, the client terminal 20 is instructed to encode at that upload bit rate. At this time, all upload bit rate values associated with i (i.e., one or more) of the upload bit rates for each source (i) x destination (u) are transmitted to a certain source terminal i.

[0054] The client upload bit rate control unit 22 of the client terminal 20 sets all the upload bit rates received from the upload bit rate instruction unit 14 of the control server 10 in the transmission data encoding unit 23 .

[0055] The transmission data encoding unit 23 encodes the media data based on the set upload bit rate, and transmits (uploads) the encoded media data to the distribution server 30.

[0056] The distribution server 30 distributes media data of one or more stages of upload bit rates uploaded from each client terminal 20 to client terminals 20 other than the client terminal 20. At this time, the distribution server 30 distributes the media data of one or more stages of upload bit rates uploaded from each client terminal 20 to client terminals 20 other than the client terminal 20.i,u The calculation result (selection result) of (t) does not necessarily need to be known. Basically, in Simulcast, the distribution server 30 determines an appropriate rate according to the download bandwidth of the client terminal 20 and distributes the data. i,u Since the download bandwidth information is used at the time of calculation of (t), the bit rate selected by the distribution server 30 is BR i,u This is because it is believed to naturally match the calculation result of (t).

[0057] Alternatively, the control server 10 or the client terminal 20 may transmit a BR to the distribution server 30. i,u The calculation result (selection result) of (t) may be transmitted.

[0058] As described above, according to the first embodiment, the video bit rate for each combination of source and destination is selected so as to minimize the amount of data transfer within a range that achieves the target QoE. Therefore, it is possible to reduce the possibility that an excessive bit rate is selected when encoding the video. As a result, it is possible to reduce the costs incurred by the service provider's facilities, for example.

[0059] Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as those in the first embodiment.

[0060] In the second embodiment, it is assumed that the upload bit rate is limited to a predetermined number R due to the network or client load of the source terminal. In this case, it is necessary to determine the limited bit rate number (R) for each source terminal.

[0061] In the second embodiment, the upload bit rate BR of each sender i in Equation (1) is i (t) number n(BR i (t)) is constrained to be less than or equal to R. However, BR i(t) means the types (list) of upload bit rates from an arbitrary source user i to all destinations. Also, the notation n(X) means the number of X. This constraint limits the number of upload bit rates for each source i to R, and each destination user u selects one bit rate from the R number of bit rates for the video from each source i. For example, if the number of selectable bit rates set in advance is N, the upload bit rate selector 13 calculates the upload bit rate for each source and destination based on the following equation (7-1) instead of equation (1) so as to minimize the amount of data transferred (the sum of the bit rates for each source and destination). In this case, the constraints of equations (2) to (6) are also valid.

[0062]

number

[0063] In each of the above embodiments, the control server 10 is an example of a bit rate selection device. The client terminal 20 is an example of a first terminal and a second terminal. The upload bit rate selection unit 13 is an example of a selection unit. The upload bit rate instruction unit 14 is an example of an instruction unit. The distribution server 30 is an example of a distribution device.

[0064] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

[0065] This application claims priority based on International Patent Application No. PCT / JP2022 / 019985, filed May 11, 2022, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0066] 10 Control Server 11 Client log collection unit 12 Target QoE input section 13 Upload bitrate selection section 14 Upload bitrate indicator 20 Client Terminals 21 Client log transmission unit 22 Upload Bitrate Control Unit 23 Transmission data encoding unit 24 Received data decoding unit 30 Distribution Server 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 CPU 105 Interface Device B Bus

Claims

1. a selection unit configured to select a bit rate of the video for each combination of a plurality of first terminals and a plurality of second terminals from a plurality of levels of values so that a sum of bit rates of the video for each combination of the plurality of first terminals and the plurality of second terminals is minimized under the constraint that a quality of experience related to the video of each user of the plurality of second terminals that receives the video transmitted from each of the plurality of first terminals via a distribution device is equal to or greater than a threshold; an instruction unit configured to transmit, to each of the first terminals, an instruction to encode at one or more bit rates selected by the selection unit for the combination associated with the first terminal; A bit rate selection device comprising:

2. The selection unit is further configured to select a bit rate of the video for each combination under the constraint that a total of bit rates excluding overlaps among the bit rates for each combination is equal to or less than an available bandwidth for uploading from the plurality of first terminals to the distribution device.

2. The bit rate selection device according to claim 1.

3. the selection unit is further configured to select a bit rate of the video for each of the combinations under the constraint that the sum of the bit rates for each of the combinations is equal to or less than an available bandwidth for download from the distribution device to the plurality of second terminals.

3. The bit rate selection device according to claim 1 or 2.

4. The selection unit is further configured to select a bit rate for the video for each of the combinations under the restriction that the number of bit rates that each of the first terminals can transmit for the video is equal to or less than a predetermined number.

2. The bit rate selection device according to claim 1.

5. a selection step of selecting a bit rate of the video for each combination of a plurality of first terminals and a plurality of second terminals from a plurality of levels of values so that a sum of the bit rates of the video for each combination of the plurality of first terminals and the plurality of second terminals is minimized under the constraint that the quality of experience of the video for each user of the plurality of second terminals receiving the video transmitted from each of the plurality of first terminals via a distribution device is equal to or greater than a threshold; an instruction step of transmitting, to each of the first terminals, an instruction to encode at one or more bit rates selected in the selection step for the combination related to the first terminal; A bit rate selection method characterized by being executed by a computer.

6. The selection step further includes selecting a bit rate of the video for each combination under the constraint that a total of bit rates excluding overlaps among the bit rates for each combination is equal to or less than an available bandwidth for uploading from the plurality of first terminals to the distribution device.

6. The bit rate selection method according to claim 5.

7. The selection step further includes selecting a bit rate of the video for each of the combinations under the constraint that the sum of the bit rates for each of the combinations is equal to or less than an available bandwidth for download from the distribution device to the plurality of second terminals.

7. The bit rate selection method according to claim 5 or 6.

8. a selection step of selecting a bit rate of the video for each combination of a plurality of first terminals and a plurality of second terminals from a plurality of levels of values so that a sum of the bit rates of the video for each combination of the plurality of first terminals and the plurality of second terminals is minimized under the constraint that the quality of experience of the video for each user of the plurality of second terminals receiving the video transmitted from each of the plurality of first terminals via a distribution device is equal to or greater than a threshold; an instruction step of transmitting, to each of the first terminals, an instruction to encode at one or more bit rates selected in the selection step for the combination related to the first terminal; A program characterized by causing a computer to execute the above.

Images