Video bit rate control device, video monitoring system, video bit rate control method and program

Abstract

This video bitrate control device makes it possible to perform a control suitable for remotely monitoring the bitrate of a video by comprising: a throughout estimation unit which is configured to calculate an estimated throughput in time series pertaining to the transmission of the video through a network from a moving body, which films the video, to a monitoring device for monitoring the video and an error distribution of the estimated throughput; and a video bitrate determination unit which is configured to determine the bitrate of the video by solving a prescribed optimization problem on the basis of the estimated throughput in time series and the error distribution of the estimated throughput over a period from a distance at which an object can be recognized on the basis of the video to a distance at which the moving body can stop without colliding with the object.

Description

[Technical Field]

[0001] The present invention relates to a video bit rate control device, a video monitoring system, a video bit rate control method, and a program. [Background technology]

[0002] There are high hopes for autonomous driving to alleviate traffic congestion, improve logistics efficiency, and reduce environmental impact, and many studies and demonstration experiments are being conducted.

[0003] Autonomous driving is divided into levels according to the degree of driver involvement. Levels 1 and 2 require driver supervision. Levels 3 and above require system supervision. At level 3, the system performs the driving task, but the driver is required to respond appropriately to requests for system intervention. Levels 4 and 5 will see the realization of fully automated driving, with no driver intervention required.

[0004] At levels 3 and above where the system performs the driving task, and especially at levels 4 and 5 where driver intervention is not required, measures such as remote monitoring are being considered to ensure safety. To achieve remote monitoring, video data needs to be transmitted from the monitored vehicle to the monitor with a quality that allows the remote monitor to recognize hazards.

[0005] However, communication quality is prone to fluctuation, especially in moving environments such as automobiles, and sufficient communication quality cannot always be ensured. If sufficient quality cannot be ensured, video quality will deteriorate, and there will be many times when the monitor cannot recognize dangerous objects, which poses a risk.

[0006] To address the above risks, it is necessary to control the video bit rate.

[0007] There are two possible methods for controlling the video bit rate.

[0008] The first is a method of controlling the video bit rate according to communication quality. Controlling the video bit rate according to communication quality has been widely studied in the fields of video distribution services and web conferencing services. Non-Patent Document 1 proposes a method of changing the video bit rate according to jitter and packet loss rate. Specifically, when the jitter or packet loss rate increases, it is assumed that congestion has occurred on the communication path, and the video bit rate is reduced to avoid stopping the playback of the video service.

[0009] The second point is to deliver the video at a fixed video bit rate that is of a quality that allows observers to recognize dangerous objects. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] S. Holmer, "A Google Congestion Control Algorithm for Real-Time Communication," "draft-ietf-rmcat-gcc-02," 2016, [online], Internet<https: / / datatracker.ietf.org / doc / html / draft-ietf-rmcat-gcc-02> Summary of the Invention [Problem to be solved by the invention]

[0011] The first method of controlling the video bitrate is the conventional method of controlling the video bitrate according to the communication quality. This method is effective in improving the quality of video viewing and conference participation in video distribution services and web conferencing services. However, when this technology is used in video surveillance, in a low-throughput environment, the quality is reduced according to the throughput situation to prevent playback from stopping, which may result in the quality being reduced to the point where the monitor cannot recognize dangerous objects. As a result, there is a high possibility of dangerous situations occurring in low-throughput environments.

[0012] The second method, which delivers images at a constant bit rate at a quality that allows the monitor to recognize dangerous objects, means that the quality is kept at the very limit of what the monitor can recognize. This requires a great deal of concentration to recognize dangerous objects, which places a burden on the monitor. This burden on the monitor leads to a decrease in concentration, which is thought to increase the risk of increased risk during monitoring. There is a problem that...

[0013] The present invention has been made in view of the above points, and an object of the present invention is to enable control of the video bit rate suitable for remote monitoring. [Means for solving the problem]

[0014] In order to solve the above problem, the video bit rate control device has a throughput estimation unit configured to calculate a time-series estimated throughput and an error distribution of the estimated throughput related to the transmission of the video via a network from a mobile body that captures the video to a monitoring device that monitors the video, and a video bit rate determination unit configured to determine the bit rate of the video by solving a predetermined optimization problem based on the time-series estimated throughput and the error distribution of the estimated throughput for a period from a distance at which an object can be recognized based on the video to a distance at which the mobile body can stop without colliding with the object. [Effects of the Invention]

[0015] It is possible to control the video bit rate in a manner suitable for remote monitoring. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a video monitoring system 1 according to an embodiment of the present invention. [Figure 2] 2 is a diagram illustrating an example of a hardware configuration of a control server 10 according to an embodiment of the present invention. FIG. [Figure 3] 10 is a flowchart illustrating an example of a processing procedure executed by the control server 10. DETAILED DESCRIPTION OF THE INVENTION

[0017] To solve the above problems, it is necessary to develop a method to deliver video at a quality (Quality of Experience (QoE)) that exceeds the quality at which a monitor can recognize dangerous objects in low-throughput environments, while delivering video at as high a quality (Quality of Experience) as possible in environments with sufficiently high throughput.

[0018] In response to the first problem regarding the control method (hereinafter referred to as "Problem 1"), if the quality (perceived quality) that the observer can recognize exceeds the quality that the observer can recognize even once between the distance at which the observer can recognize an object based on the video and the distance at which the moving body can stop after starting to decelerate without colliding with the object (hereinafter, the distance at which the moving body can stop after starting to decelerate without colliding with the object is referred to as the "stopping limit distance"), it is considered that the observer will notice the dangerous object and be able to make an emergency stop, etc. Therefore, in this embodiment, estimated time-series throughput information and estimated error distribution are utilized, and the video bit rate is instructed so that the quality (perceived quality) exceeds a quality threshold that the observer can recognize the object, which is set in advance, between the distance at which the observer can recognize the object and the stopping limit distance, as much as possible once.

[0019] In addition, to address the second problem regarding the control method (hereinafter referred to as "Problem 2"), in this embodiment, estimated time-series throughput information and estimated error distribution are utilized, and a video bit rate that provides the highest quality (quality experienced by the observer) is indicated, taking into account the probability that video cannot be delivered due to packet loss.

[0020] Specifically, an optimization problem is solved to maximize the quality index that takes into account the probability that video delivery will not be possible due to packet loss between the distance at which the observer can recognize an object and the stopping limit distance. In this case, a constraint is imposed so that the probability that the quality will never exceed a predetermined quality threshold for recognizing an object between the distance at which the observer can recognize an object and the stopping limit distance is equal to or less than a threshold.

[0021] The stopping limit distance is, for example, the distance from the limit position where braking can be applied without colliding with the object. In other words, the stopping limit distance is the minimum distance traveled (travel distance) required from the start of stopping (deceleration to stop) until the vehicle stops.

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the functional configuration of a video monitoring system 1 according to an embodiment of the present invention. As shown in Fig. 1, the video monitoring system 1 includes a client 20, a monitoring center 30, and a control server 10. These can communicate with each other via a network such as the Internet.

[0023] The client 20 is a moving object whose operation (behavior) is monitored using video. For example, the client 20 may be an autonomously driven vehicle. In FIG. 1, the client 20 includes a video transmission unit 21 and a client log acquisition unit 22. These units are realized by processes that the client 20 executes under the control of a program installed on the client 20.

[0024] The video transmission unit 21 encodes the video (hereinafter referred to as "monitoring video") captured by the client 20 in order to monitor the client 20 according to the video bit rate instructed by the control server 10 and transmits the encoded video to the monitoring center 30.

[0025] The client log acquisition unit 22 acquires a log including information on the location of the client 20, and transmits the log (hereinafter referred to as the "client log") to the control server 10. More specifically, the client log acquisition unit 22 acquires information that can identify the position of the client 20 at each time (in time series), the moving speed (e.g., vehicle speed) and direction of travel of the client 20, etc., which are necessary for estimating the throughput and determining the video bit rate, and periodically transmits the client log including the information to the control server 10. Note that the acquisition interval and the transmission interval of the client log may be different. The transmission interval may be set longer than the acquisition interval so that client logs for multiple times are transmitted at once, thereby reducing the transmission frequency of the client log. Note that in this embodiment, "time" refers to the timing of each acquisition interval of the client log (and the playback log, described later).

[0026] The transmission interval of the client log is synchronized within the system to match the control interval by the control server 10. The transmission interval may be made changeable taking into consideration the load on the client 20 and the control server 10. Each client log contains information (position, movement speed, direction of travel, etc.) from the immediately preceding transmission interval.

[0027] The monitoring center 30 is one or more computers that assist a monitor in monitoring the client 20 using the monitoring video from the client 20. In FIG. 1, the monitoring center 30 has a display control unit 31, a playback log acquisition unit 32, etc. These units are realized by processes that the monitoring center 30 executes in response to a program installed in the monitoring center 30. Note that the monitor is a person in charge of monitoring the client 20 by referring to the monitoring video. However, the monitor may be replaced by a computer.

[0028] The display control unit 31 decodes the surveillance video received from the client 20 and displays (plays) it on the surveillance monitor.

[0029] The playback log acquisition unit 32 acquires a log related to the playback of the surveillance video and transmits the log (hereinafter referred to as the "playback log") to the control server 10. More specifically, the playback log acquisition unit 32 collects information necessary for estimating the quality of the surveillance video at each time and periodically transmits a playback log including the information to the control server 10. The information necessary for quality estimation is information related to the quality of each received video (video bit rate, playback stop information, frame rate, resolution). The transmission interval to the control server 10 is synchronized within the system so as to match the control interval. Taking the load on the control server 10 into consideration, the transmission interval may be changeable (i.e., playback logs related to multiple times may be transmitted at one time). Each playback log includes information related to the quality of the received video at each immediately preceding time (acquisition interval) (video bit rate, playback stop information, frame rate, resolution).

[0030] The control server 10 is one or more computers that control the bit rate of the surveillance video. In Fig. 1, the control server 10 includes a log collection unit 11, a parameter setting unit 12, a throughput estimation unit 13, a video bit rate determination unit 14, and a video bit rate instruction unit 15.

[0031] The log collection unit 11 collects the playback logs sent from the monitoring center 30 .

[0032] The parameter setting unit 12 receives parameters for determining the bit rate of the monitoring video (hereinafter referred to as the "video bit rate") from the user, and sets the parameters in the video bit rate determination unit 14. In this embodiment, the parameters are the following three: (1) The quality threshold at which an object can be recognized QoE ) (2) Tolerance of probability below the quality threshold prob ) (3) The distance (X) from when the observer recognizes the object to the stopping limit distance when the moving speed (e.g., vehicle speed) of the client 20 is a reference value (e.g., 10 km / h) The values ​​of the above parameters only need to be set once before the throughput estimation unit 13, the video bit rate determination unit 14, and the video bit rate instruction unit 15 execute the processes. However, the values ​​of the above parameters may be changed after the processes start.

[0033] The throughput estimation unit 13 estimates the throughput based on the client log sent from the client 20 .

[0034] The video bit rate determination unit 14 determines the video bit rate to be instructed to the client 20 .

[0035] The video bit rate instruction unit 15 instructs the client 20 to encode the surveillance video at the video bit rate.

[0036] 2 is a diagram showing an example of the hardware configuration of the control server 10 according to the embodiment of the present invention. The control server 10 in FIG. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0037] A program that realizes processing in the control server 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0038] When an instruction to start a program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the control server 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0039] The client 20 and the monitoring center 30 may also have a hardware configuration as shown in FIG.

[0040] The log collection unit 11, parameter setting unit 12, throughput estimation unit 13, video bit rate determination unit 14, and video bit rate instruction unit 15 in Figure 1 are realized by processing in which one or more programs installed in the control server 10 are executed by the processor 104.

[0041] The following describes the processing procedure executed by the control server 10. Fig. 3 is a flowchart for explaining an example of the processing procedure executed by the control server 10.

[0042] When the throughput estimator 13 receives the client log from the client 20 (Yes in S101) and the log collector 11 receives the playback log from the monitoring center 30 (Yes in S102), steps S103 and onwards are executed.

[0043] As described above, the client log and the playback log are transmitted periodically (for example, every second), and the timing of each transmission is synchronized. Therefore, step S103 and subsequent steps are performed periodically.

[0044] In step S103, the throughput estimation unit 13 uses the client log (location information, direction of travel, and speed at each time) received from the client 20 to calculate a future time-series (for each time) estimated throughput (throughput at a location where the client 20 (vehicle) is likely to arrive at each time in the future) and information on the error distribution of the estimated throughput, taking into account the expected arrival locations of the client 20 (vehicle) from the current time onwards. The throughput estimation unit 13 inputs the calculation results to the video bitrate determination unit 14.

[0045] The throughput estimation unit 13 uses a technique for estimating throughput from information such as location and driving conditions. This technique is described in detail in, for example, "Arvind Narayanan et al., "Lumos5G: Mapping and Predicting Commercial mmWave 5G Throughput," In Proceedings of the ACM Internet Measurement Conference (IMC '20), https: / / dl.acm.org / doi / 10.1145 / 3419394.3423629." In this embodiment, the client log includes the location, moving speed, and traveling direction of the client 20 because it is assumed that the technique described in this document is used as a throughput estimation technique. When a technique for estimating throughput based on other parameters is used, the client log only needs to include information required by the technique.

[0046] Next, the video bit rate determination unit 14 determines an optimal command bit rate (video bit rate to be instructed to the client 20) by solving an optimization problem that minimizes the objective function of the following equation (1) (S104).

[0047]

number

[0048] The estimated throughput (est_th) and the error distribution of the estimated throughput (Err_th_dist) are substituted with values ​​calculated by the throughput estimator 13.

[0049] Equation (1) consists of two elements. The first element represents a normalized quality value, which means that the higher the quality achieved by the specified bitrate, the greater the gain. For example, the quality value is estimated using a model (hereinafter referred to as the "video quality estimation model") that estimates video quality from video bitrate, resolution, and frame rate, as described in "K. Yamagishi and T. Hayashi, "Parametric Quality-Estimation Model for Adaptive-Bitrate Streaming Services," IEEE Transactions on Multimedia, vol. 19, no. 7, pp. 1545-1557, 2017, https: / / ieeexplore.ieee.org / document / 7857103." Since the quality value ranges from 1 to 5, 1 is subtracted and then divided by 4 to normalize it to a value between 0 and 1.

[0050] The second factor is the probability that surveillance video delivered at the specified bit rate will be transmitted without loss (without packet loss for the video) taking into account the estimated throughput and estimated error distribution (i.e., the probability that no packet loss will occur), and is a factor that takes into account the degradation of quality due to packet loss.

[0051] Specifically, the calculation method of the second element when the error distribution is a normal distribution will be described. In the case of a normal distribution, the video bit rate determination unit 14 calculates the mean (μ) and variance (σ) of the estimated throughput. 2 ) is input as information on the error distribution. The probability that the bit rate will be less than the specified bit rate (x) during the search from the mean and variance is

[0052]

number

[0053] Therefore, the video bit rate determination unit 14:

[0054]

number

[0055] Multiplying the first and second elements of equation (1) (multiplying the first element by the second element) yields an index that achieves the highest possible quality of delivery, taking into account the estimated throughput and error distribution. This index is subtracted from 1 to apply it to the minimization problem. Solving this minimization problem addresses issue 2.

[0056] In this embodiment, when solving the minimization problem, the probability that the quality threshold cannot be achieved within the period (time length) T from when the observer recognizes the object to the stopping limit distance is constant (thre prob ) is given as a constraint such that: The constraint is defined by the following equation (2).

[0057]

number

[0058] If the moving speed of the client 20 is equal to the reference value (here, taking 10 km / h as an example) on the premise of X, then X can be calculated by dividing X by the moving speed. When the moving speed of the client 20 is not 10 km / h, since the stopping limit distance changes, X increases or decreases. That is, if the moving speed is faster than 10 km / h, the stopping limit distance becomes longer, so X becomes shorter; if the moving speed is slower than 10 km / h, the stopping limit distance becomes shorter, so X becomes longer. Therefore, when the moving speed of the client 20 is not 10 km / h, T can be obtained by dividing the corrected value of X based on the difference between the stopping limit distance y at the moving speed included in the client log (the moving speed of the client 20) and the stopping limit distance x at the moving speed of 10 km / h on which X is premised, by the moving speed. The correction of X can be done by X - (x - y). For example, if the moving speed of the client 20 is 40 km / h, and the stopping limit distance y in that case is 20 m, and the stopping limit distance x when x = 10 km / h is 5 m, then the difference (x - y) is 20 - 5 = 15 m. Therefore, it is possible to obtain T by dividing the value obtained by subtracting 15 m from X by 40 km / h. Note that the stopping limit distance at a certain moving speed can be calculated if the friction coefficient of the road surface and the reaction time are set. Or, the stopping limit distance may be set in a table format or the like in advance for each moving speed.

[0059] The left side of Equation (2) is the calculation formula for the probability that the quality threshold thre QoE cannot be achieved even once from T0 to T seconds. Specifically, the video bitrate determination unit 14 calculates the probability that the quality threshold thre QoE cannot be achieved for each time (for each acquisition interval of the client log and the playback log), and multiplies them. The calculation method for the probability for each time is as follows. Here, t is the current time (the time when the indicated bitrate is determined), and T' is an arbitrary time in the period from T0 to T0 + T. ·T ' <t: Calculate the realized quality from the playback log of the monitoring center 30 (for example, it can be calculated by inputting the playback log into the above video quality model), and if the value is the quality threshold thre QoEIf it exceeds, it is set to 0; if it does not exceed, it is set to 1. T ' ≧t: Using the estimated throughput and estimated error distribution at each time, calculate the probability that the estimated throughput will be lower than the specified bit rate. For example, if the error distribution is a normal distribution,

[0060]

number

[0061] The right side of equation (2) is the quality threshold thre QoE The tolerance threshold for the probability of falling below prob (a threshold indicating the lower limit at which the probability is acceptable), and the value set by the parameter setting unit 12 is used.

[0062] A specific example of processing performed by the video bit rate determination unit 14 will be described. (Process 1) Check the past quality values ​​from the current time t and continuously check the target quality threshold value for the period up to time t. QoE Search for the period in which the time was less than tT. The smallest time (start time) of that period is set as T0. However, T0 > tT. (Process 2) Solve the minimization problem so as to minimize equation (1) for the period from T0 to T seconds and satisfy the constraint of equation (2). (Process 3) From the video bit rate array (the (time-series) video bit rates for each time in period T) calculated by solving the minimization problem, the value (video bit rate) of the element corresponding to the current time is obtained, and this value is sent to video bit rate instruction unit 15 as the instruction bit rate. Note that which element in this array corresponds to the current time differs depending on which time is designated as T0. Therefore, this element can be identified based on the elapsed time from T0 to the current time.

[0063] In (Process 2), the bit rate is determined from T0 to the current time t and is confirmed to be below the quality threshold. Therefore, the optimization section should be set from the current time onwards (from the current time (t) to T0+T). By adjusting the section given to the constraint (Equation (2)) in this way, a similar optimization can be achieved.

[0064] Next, the video bit rate instruction unit 15 transmits the instruction bit rate received from the video bit rate determination unit 14 to the client 20, thereby instructing the client 20 to apply the instruction bit rate (S105).

[0065] The video transmission unit 21 of the client 20 encodes the monitoring video in accordance with the video bit rate instructed by the control server 10 and transmits the encoded monitoring video to the monitoring center 30 .

[0066] The display control unit 31 of the monitoring center 30 decodes the monitoring video received from the client 20 and displays the decoded monitoring video on a monitoring monitor.

[0067] As described above, according to this embodiment, it is possible to control the video bit rate in a manner suitable for remote monitoring. For example, in a remote vehicle monitoring service, by controlling the video bit rate so as to deliver high-quality video while increasing the probability that a remote monitor will detect a dangerous object from the video, it is possible to reduce the risk of accidents and the burden on the monitor, thereby providing a safe service.

[0068] In this embodiment, the control server 10 is an example of a video bit rate control device. QoE is an example of the first threshold. prob is an example of a second threshold. The monitoring center 30 is an example of a monitoring device.

[0069] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0070] 1. Video surveillance system 10 Control Server 11 Log collection section 12 Parameter setting section 13 Throughput Estimation Unit 14 Video bit rate determination unit 15 Video bit rate indicator 20 clients 21 Video transmission unit 22 Client log acquisition unit 30 Monitoring Center 31 Display control unit 32 Playback log acquisition unit 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 processors 105 Interface Device B Bus

Claims

1. a throughput estimation unit configured to calculate a time-series estimated throughput and an error distribution of the estimated throughput regarding transmission of video from a mobile object that captures video to a monitoring device that monitors the video via a network; a video bit rate determination unit configured to determine a bit rate of the video by solving a predetermined optimization problem based on the time-series estimated throughput and an error distribution of the estimated throughput for a period from a distance at which an object can be recognized based on the video to a distance at which the moving body can stop without colliding with the object; and A video bit rate control device comprising:

2. the video bit rate determination unit is configured to solve the optimization problem under a constraint that a probability that the video quality does not achieve a first threshold is equal to or less than a second threshold.

2. The video bit rate control device according to claim 1.

3. The video bit rate determination unit is configured to solve an optimization problem aimed at maximizing a quality index that takes into account a probability that no packet loss will occur for the video.

3. The video bit rate control device according to claim 1 or 2.

4. The optimization problem includes a first element that provides a gain as the quality achieved by the designated bit rate increases, and a second element that indicates a probability that the video at the designated bit rate will be transmitted without loss and is multiplied by the first element.

4. The video bit rate control device according to claim 3.

5. In the optimization problem, the quality of the video received by the monitoring device is applied to a timing before the current time within the period, and the estimated throughput and the error distribution are applied to a timing after the current time.

3. The video bit rate control device according to claim 1 or 2.

6. A video monitoring system including a moving object that captures video, a monitoring device for capturing the video, and a video bit rate control device, The video bit rate control device comprises: a throughput estimation unit configured to calculate a time-series estimated throughput and an error distribution of the estimated throughput regarding the transmission of the video from the mobile object to the monitoring device via a network; a video bit rate determination unit configured to determine a bit rate of the video by solving a predetermined optimization problem based on the time-series estimated throughput and an error distribution of the estimated throughput for a period from a distance at which an object can be recognized based on the video to a distance at which the moving body can stop without colliding with the object; and a video bit rate instruction unit that instructs the moving body to encode the video at the video bit rate determined by the video bit rate determination unit; A video surveillance system comprising:

7. a throughput estimation procedure for calculating a time-series estimated throughput and an error distribution of the estimated throughput relating to transmission of video from a mobile object that captures video to a monitoring device that monitors the video via a network; a video bit rate determination step of determining a bit rate of the video by solving a predetermined optimization problem based on the time-series estimated throughput and an error distribution of the estimated throughput for a period from a distance at which an object can be recognized based on the video to a distance at which the moving body can stop without colliding with the object; A video bit rate control method characterized by being executed by a computer.

8. a throughput estimation procedure for calculating a time-series estimated throughput and an error distribution of the estimated throughput relating to transmission of video from a mobile object that captures video to a monitoring device that monitors the video via a network; a video bit rate determination step of determining a bit rate of the video by solving a predetermined optimization problem based on the time-series estimated throughput and an error distribution of the estimated throughput for a period from a distance at which an object can be recognized based on the video to a distance at which the moving body can stop without colliding with the object; A program characterized by causing a computer to execute the above.

Images