Method for streaming real-time encoded video

Abstract

A method of streaming real-time encoded video is disclosed, the method (300) for streaming real-time encoded video to one or more clients and comprising: detecting (S302) a plurality of video stream requests; obtaining (S304) a maximum bitrate value that a return video stream is allowed to have; calculating (S306) a quality of experience, QoE, value for the requests; determining (S308) a number of return video streams lower than the plurality of video stream requests and having bitrates such that each video stream request can be matched with a return video stream, wherein the bitrates of the return video streams are determined such that a cumulative difference between the QoE value of each video stream request and the calculated return QoE of the matched return video stream is minimized; obtaining (S312) the return video streams; and streaming (S314) the return video streams. A streaming controller (20) for controlling the streaming of real-time encoded video is also disclosed.

Description

Technical Field

[0001] The present invention relates to methods and systems for streaming real-time encoded video to one or more clients, and more specifically, to methods and systems for optimizing the quality of experience (QoE) for each client. Background Technology

[0002] The widespread adoption of real-time communication technologies has led to an increased demand for efficient and scalable video streaming solutions. Some peer-to-peer technologies, such as WebRTC (Web Real-Time Communication), offer low latency but struggle to scale with the growing number of peers. WebRTC enables real-time communication, including voice, video, and data sharing, directly within a web browser. When multiple peers connect to a single camera, the varying capabilities and fluctuating bandwidth of each peer present challenges in delivering optimal video quality.

[0003] Current approaches to addressing these limitations involve generating a single stream with the lowest possible common quality, sacrificing overall quality, or generating multiple streams at a fixed bit rate, resulting in suboptimal quality for some peers. Furthermore, cameras often face constraints on the number of streams they can generate simultaneously.

[0004] Clearly, current solutions need to be improved to achieve efficient and scalable video streaming. Summary of the Invention

[0005] The object of this invention is to provide improved overall video quality across multiple video streams being streamed to one or more clients. Another object of this invention is to provide a method for determining the characteristics of video to be encoded and streamed in order to optimize the video quality of the streamed video as perceived by the user. Yet another object of this invention is to provide a method for determining characteristics that is flexible based on preconditions that are suitable for modification, for example, when adding, removing, or changing requests for video streams.

[0006] This invention is defined by the appended claims.

[0007] According to a first aspect of the invention, a method according to claim 1 is provided. The novel concept of the invention is to consider a global perspective of quality loss, rather than determining the video quality of a video stream based on a single video request. Furthermore, it considers the user's perspective of quality rather than the stringent technical characteristics of the video stream by examining the loss of Quality of Experience (QoE). Generally, Quality of Experience (QoE) refers to the overall satisfaction and perceived quality of a user's interaction with a service, system, or product. In the context of video streaming, QoE includes factors such as video resolution, compression level, frame rate, latency, and buffering time, which collectively affect the viewer's experience.

[0008] Specifically, QoE is metric (or measured) to determine the quality of the video stream and to determine the configuration of the video stream such that the minimum global loss of QoE is provided across all video streams. This approach can provide an improved overall visual experience and increased user satisfaction by enhancing video quality.

[0009] In some embodiments, the maximum bitrate value is obtained from the video stream request itself. In other embodiments, the maximum bitrate value is determined based on the available bandwidth for transmitting the return video stream. The available bandwidth can be calculated using a congestion control algorithm.

[0010] In some embodiments, the number of returned video streams is calculated based on the number of video stream requests out of multiple video stream requests. This allows the number of returned video streams to be adjusted based on changing demand, thereby reducing processing costs.

[0011] In some embodiments, the number of returned video streams is calculated based on a predefined formula that is a function of the number of video stream requests. The predefined formula may be a logarithmic function. This provides a scalable solution for managing multiple video stream requests with low computational complexity.

[0012] In some embodiments, the method takes into account the optimal power consumption of the encoder that encodes the returned video stream. This allows the system to strike a balance between minimizing QoE loss and power consumption, thereby reducing energy costs and heat in the device including the encoder while maintaining acceptable video quality.

[0013] In some embodiments, the requested QoE value and the returned QoE value are calculated using one or more of the following measures or metrics: SSIM, PSNR, VMAF, and UQI. Different metrics can be made compatible for comparison by applying a normalization algorithm.

[0014] Encoding characteristics can be determined from the video streaming request, predetermined encoding characteristics, or a combination of both. Predetermined encoding characteristics can depend on the video scene of the requested video stream. Specifically, predetermined encoding characteristics can specify different encoding characteristics based on the content of the video (e.g., the type of scene depicted in the video). Non-limiting examples of scene types include outdoor, indoor, traffic, panoramic, or 360-degree view. Alternatively or additionally, predetermined encoding characteristics can depend on the type of camera used to capture the requested video stream, such as a wearable camera, a still camera, a camera with mobility capabilities (e.g., a PTZ camera), a modular camera, or a panoramic camera. The QoE value for each video streaming request is calculated based on the requested encoding characteristics, or based on predetermined encoding characteristics, or a combination of both, whichever is valid for the video streaming request.

[0015] According to a second aspect, the invention is defined in claim 11, which defines a computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method according to the first aspect or any embodiment thereof.

[0016] According to a third aspect, the invention is defined in claim 12, which defines a streaming controller for controlling the streaming of real-time encoded video to one or more clients. The streaming controller includes circuitry adapted to perform the method according to the first aspect or any embodiment thereof.

[0017] According to a fourth aspect, the invention is defined in claim 13, which defines a camera system including the above-described streaming controller.

[0018] The further scope of applicability of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific embodiments indicate preferred embodiments of the invention, they are given by way of illustration only, as various variations and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description. Attached Figure Description

[0019] The invention will now be described in more detail by way of example and with reference to the accompanying schematic drawings, in which:

[0020] Figure 1a and Figure 1b The illustration shows a camera device suitable for streaming real-time encoded video according to existing technology.

[0021] Figure 2 The illustration shows a camera device suitable for streaming real-time encoded video according to an embodiment.

[0022] Figure 3 This is a flowchart of a method for streaming real-time encoded video according to an embodiment.

[0023] Figure 4 The diagram illustrates the relationship between bitrate and QoE for multiple video streams, and...

[0024] Figure 5 The illustration shows a camera device and a relay device suitable for streaming real-time encoded video according to an embodiment. Detailed Implementation

[0025] A common setup for delivering video to a client is streaming video with real-time encoding. In this context, "real-time encoding" means that the video is encoded and streamed directly to a receiver of the stream, which can be a recorder, a video management system (VMS), or a server, etc. In contrast, streaming video that is recorded (stored) (encoded before storage) does not meet the definition of real-time encoded video in the context of this application. Non-limiting examples of streaming protocols that can be used for streaming video with real-time encoding include WebRTC, RTSP, Warp (Live Media Port over QUIC), Media over QUIC, HLS (HTTP Live Streaming), and DASH (Dynamic Adaptive Streaming over HTTP). The network technology used for streaming video with real-time encoding can be wired or wireless, such as a Wi-Fi network or a cellular-based network (e.g., 5G).

[0026] Figure 1a The illustration shows a camera device 10, according to the prior art, suitable for performing real-time encoded video streaming. The camera device 10 includes a video encoder 12 configured to encode video data (captured by the image sensor of the camera device 10) into three video streams S with different fixed bit rates H (high), M (medium), and L (low). H S M S L For example, streams can differ in resolution, FPS, and / or compression settings that result in different bitrates. Bitrate is typically an indicator of how much the encoded video data has been degraded compared to the original captured video data—for example, reduced resolution, reduced FPS, or compression to a certain extent. In other words, the video stream S... M Data and video stream S L The data does not exhibit degradation compared to the standard data, but it is comparable to video stream S. H The data shows more downgrades compared to the previous data.

[0027] There are three clients 14a to 14c, each wishing to receive a real-time encoded video stream from camera device 10. In the context of this application, a client means a receiver of real-time encoded video data. Clients may have different purposes for receiving the real-time encoded video stream, such as storage (video recording), display for an operator, or video processing (e.g., performing video analysis such as object detection, tracking, or event detection). Figure 1a In this scenario, each client 14a to 14c can choose to receive a predefined video stream S via, for example, a request. H S M and S LOne of them. Alternatively, camera device 10 can be configured to send a video stream with the highest possible bit rate that each client can receive, based on an unspecified request for video from clients 14a to 14c. Figure 1a Configuration issues can lead to varying degrees of perceived video quality degradation for different clients (14a through 14c), depending on how well the requested or expected video corresponds to a predefined video stream. This quality is known as Quality of Experience (QoE). Generally, QoE refers to the overall satisfaction and perceived quality of a user's interaction with a service, system, or product. QoE is related to the measurement of Quality of Service (QoS), which is a measure indicating the quality of actual data and / or the transport network. In contrast, QoE metrics are designed to quantify the user's subjective experience. In other words, QoS metrics can, but are not designed, to indicate the quality of the user experience, while QoE metrics are constructed to indicate the quality of the user experience. Figure 1a Under this setting, clients 14a to 14c will therefore experience different losses in QoE, depending on how well the predefined video streams match their requests or expectations.

[0028] Figure 1a The existing configuration of the camera device 10 is limited in terms of flexibility to adapt to the needs of different clients 14a to 14c, especially in scenarios where the needs may change over time. Figure 1b A camera device 10 with enhanced flexibility in this respect is illustrated. In this scenario, clients 14a to 14d are allowed to make specific video stream requests R1 to R4 that define the requested video stream characteristics and bit rate. The encoder 12 is then instructed to encode the video stream accordingly, and the camera device 10 can provide adapted video streams S1 to S3 in response to the video stream requests R1 to R4. Therefore, clients 14a to 14c that receive the returned video stream at the requested bit rate are preferably satisfied because the QoE of the requested video stream is equal to the QoE of the returned video stream. However, the encoder 12 in the camera device 10 has limited encoding capacity. If the number of video stream requests R1 to R4 with different characteristics and bit rates exceeds the available encoding capacity, there will be one or more clients 14d that need to receive video according to another video stream request. As a result, the total loss relative to QoE (i.e., the reduction from the requested QoE to the returned QoE) becomes unevenly distributed among the clients, and factors such as the timing of the video requests may have an unreasonably high impact on the QoE loss of individual clients.

[0029] Figure 2The illustration depicts a camera device 10, adapted for streaming real-time encoded video streams to multiple clients 14a to 14d, according to an embodiment designed to improve upon prior art solutions, particularly with respect to the QoE of clients 14a to 14d. Clients 14a to 14d send multiple video stream requests to camera 10 to request video streams. Video stream requests may include simple requests with the basic meaning of "I want to receive a live video stream," or they may include more detailed and complex requests containing desired encoding characteristics and / or desired or maximum allowed bitrates of the requested video stream. It should be noted that a single client can send multiple video stream requests (one for each desired video stream), meaning that multiple video stream requests can originate from a single client. In other words, the number of clients is one or more, while in this embodiment, the number of video stream requests is multiple.

[0030] Camera device 10 includes encoder 12 and streaming controller 20. Similar to the prior art examples disclosed above, encoder 12 is a conventional video encoder adapted to encode video data according to one or more established video encoding formats. Non-limiting examples of video encoding formats (codecs) include H.264 (MPEG-4, AVC), H.265 (HEVC), H.266 (VVC), and AV1. Streaming controller 20 is part of camera device 10 in this implementation and is configured to control the configuration of the video stream being encoded in real time based on information about clients 14a to 14d. Streaming controller 20 is adapted to send encoding instructions to encoder 12 and receive encoded return video stream 24 from encoder. Streaming controller 20 is further adapted to instruct the transmission of the video stream at a level of detail, that is, which encoded return video stream 24 should be sent to which clients 14a to 14d. The streaming controller 20 is also adapted to receive and process information about clients 14a to 14d, particularly information included in video streaming requests and bandwidth estimates from the transmission channel between camera 10 and clients 14a to 14d. For the purpose of performing its functions and processing data, the streaming controller 20 includes circuitry that may be in the form of hardware, software, or a combination thereof.

[0031] The streaming controller 20 is configured to determine the configuration of the video streams to be returned to clients 14a to 14d based on the principle of minimizing the overall (global) QoE loss for all clients 14a to 14d. In other words, the returned video streams are configured such that the total reduction in QoE accumulated for all requested (desired, anticipated) video streams for all clients 14a to 14d is minimized for a given number of returned video streams. By using a control mechanism with a global perspective that ensures optimal QoE for clients 14a to 14d, the risk of a minority of clients suffering severe QoE degradation is reduced. Even if a larger number of clients 14a to 14d may experience minor QoE degradation compared to existing technologies, the principle based on the global perspective offsets the unbalanced quality degradation of clients 14a to 14d. However, the significant loss for a minority of clients can have a much greater impact on user experience than the minor loss for a majority of clients, which is the innovative implementation of this invention.

[0032] To gain a deeper understanding of the solution details, the streaming controller 20 is configured to perform actions such as those described by... Figure 3 The illustrated method 300.

[0033] The initial step in method 300 is to detect multiple video stream requests from clients 14a to 14d in step S302. In this example, there are several clients 14a to 14d; however, other embodiments may include a single client generating multiple video stream requests. As mentioned above, the video stream requests can be simple or complex, and a combination of simple and complex requests is feasible.

[0034] Next, based on the video stream request, S304 returns the maximum bit rate value that the video stream is allowed to have. The maximum bit rate value can mean the maximum (estimated) bit rate value that ensures the return video stream is transmitted with minimal risk of packet loss or with a specific maximum (estimated) degree of packet retransmission required. The maximum bit rate value can be defined by clients 14a to 14d or by streaming controller 20 or other modules in camera 10.

[0035] In one embodiment, the maximum bitrate value for each returned video stream may be included (for some or all of the video stream requests) in the video stream request and thus obtained from the video stream request. In this embodiment, the clients 14a to 14d sending the video stream request may determine the maximum bitrate value. In another embodiment, the maximum bitrate value is estimated by the streaming controller 20 or another module of the camera device 10. For example, a so-called congestion control algorithm may be performed to determine the available bandwidth for data transmission from the camera device 10 to one of the clients 14a to 14d. As another option, the maximum bitrate value may be known in advance for a particular client or transport channel. Therefore, the maximum bitrate value may be retrieved as a stored value in a memory communicatively connected to the streaming controller 20. These are known examples of determining the maximum bitrate of a returned video stream and are utilized in conventional video streaming techniques, and are considered to be available to those skilled in the art, without the need to disclose further details herein.

[0036] Once the maximum bitrate value of S304 is obtained, the QoE value of the request (S306) is calculated for each video stream request. The QoE value of the request refers to the QoE value (based on the selected metric) of the video stream corresponding to the video stream request (i.e., the video stream that the client sending the video stream request expects or anticipates returning). Similar to a data quality indicator or QoS value, the QoE value of a video stream indicates the degree of quality degradation (based on the user's perceived quality) caused by processing the video data captured by the image sensor for the purpose of generating the video stream to be transmitted to the client. This processing involves, for example, image processing and encoding (including compression) in an image processing pipeline. The QoE value can also be calculated based on estimated or measured distortion caused during transmission to the client. In the context of video streaming and in this application, the calculation of the QoE value may include factors such as video resolution, frame rate, GOP length, codec, bitrate variation, latency, and buffering time, which collectively affect the viewer's experience.

[0037] Well-established QoE metrics exist designed to indicate quality values ​​that reflect a user's quality experience of a video. The disclosed method 300 can be performed based on any of these known metrics, provided the calculated QoE values ​​are comparable. Non-limiting examples of metrics that can be used include the Structural Similarity Index (SSIM) measure, Peak Signal-to-Noise Ratio (PSNR), Video Multi-Method Estimation Fusion (VMAF), and Universal Quality Index (UQI). To ensure that different metrics are comparable, normalization algorithms can be utilized. For example, a first QoE value is calculated for a first video stream having a first FPS value generated according to a first metric, and a second QoE value is calculated for a second video stream having a second FPS value generated according to a second metric. The first metric is calculated based on FPS, while the second metric is not. To enable comparison of the first and second QoE values, the second QoE value can be normalized by multiplying it by a factor corresponding to the ratio between the first and second FPS values. Similar reasoning can be applied to other coding characteristics, such as resolution. A QoE value not calculated based on resolution can be compared with a QoE value calculated by multiplying a first QoE value by a factor corresponding to the ratio between the resolutions of the relevant video image streams.

[0038] The requested QoE value can be calculated (S306) based on the encoding characteristics (if available) included in the corresponding video stream request. Typically, encoding characteristics include encoding options or encoding settings that define how the video data is encoded by encoder 12. Non-limiting examples of encoding characteristics include FPS, resolution, GOP structure, and compression (e.g., QP settings). Requested encoding characteristics may include, for example, the requested resolution and / or the requested FPS. For requests that do not provide this information, predetermined encoding characteristics can be used to calculate the requested QoE value. Predetermined encoding characteristics can be constructed as an optimal encoding pattern defined for the receiving client. The meaning of the optimal encoding pattern can differ between clients, depending on what the video stream is used for or what it represents. For example, one set of encoding characteristics may provide an optimal encoding pattern for video storage, while another set may provide an optimal encoding pattern for processing via video analysis. Furthermore, one set of encoding characteristics may provide an optimal encoding pattern for traffic video (i.e., video capturing traffic scenes), while another set may provide an optimal encoding pattern for video of an indoor environment. The type of camera can also affect the predetermined encoding characteristics, for example, a still camera compared to a wearable camera (or body-worn camera).

[0039] It should be noted that the requested QoE value can be calculated based on a combination of information obtained from the corresponding video stream request (such as the requested FPS) and predetermined encoding characteristics (e.g., the resolution that provides the optimal encoding mode for the panorama).

[0040] Next, the streaming controller 20 performs optimization processing S307, the purpose of which is to determine the number of returned video streams and the bit rate of the returned video streams. Optimization processing S307 includes a first action of determining the number of returned video streams in S308, which is lower than the number of detected streaming requests. Optimization processing S307 also includes a second action of determining the bit rate of the returned video streams in S310. The first and second actions can be performed consecutively (one after another) or as a combined action, wherein the first and second actions are performed simultaneously and may depend on each other. It should be noted that the terms "first" and "second" do not indicate the order in which the actions are performed, but rather mean that the second action can be performed before the first action.

[0041] The purpose of optimization process S307 is to determine the number and bitrate of the returned video streams such that, for the determined number of returned video streams, the cumulative difference between the QoE value requested for each video stream and the calculated returned QoE of the matching returned video streams is minimized. This difference is called the global QoE loss. Further reference will follow. Figure 4 To explain the optimization process S307, Figure 4 The illustration shows the relationship between bitrate and QoE for multiple video streams that are candidates to be identified as return video streams, according to one embodiment. This relationship depends on which QoE metric is used and any prerequisites for calculating the QoE of a particular video stream request. The prerequisites for the QoE value of a particular video stream are the same as those explained above in conjunction with the calculation of the QoE of a request for a video stream request. For example, when a video is a candidate for return as part of a particular video stream request, the requested FPS or resolution can form prerequisites for calculating the QoE value of the video stream. As previously mentioned, predetermined coding characteristics determined based on the video scene or camera type can form prerequisites for calculating the QoE of candidate video streams. Therefore, prerequisites based on one or more parameters for calculating the QoE value can be valid for all candidate video streams or a subset (i.e., one or more candidate video streams, but not all candidate video streams).

[0042] Therefore, the streaming controller 20 determines the QoE value of each candidate video stream that can be generated as a return video stream to match the video stream request. These are in Figure 4The graph represents vertical lines representing different bit rates, and their vertical lengths indicate their corresponding QoE values. Based on this information, the streaming controller 20 determines which combination of candidate video streams results in the minimum overall reduction (i.e., the minimum QoE loss) for the cumulative QoE across all returned video streams, compared to video stream requests that match the candidate video streams. The reduction (or loss) is caused by the fact that the number of returned video streams is less than the number of video stream requests, and therefore, one or more video stream requests need to be matched with returned video streams with a QoE lower than the requested QoE. In the illustrated example, for a requested video stream that needs to be matched with a returned video stream with a lower QoE (and lower bit rate), the reduction in QoE is marked by downward arrows Δ1, Δ2, Δ3 along the QoE axis. The streaming controller 20 determines a combination of returned video streams B1, B2, B3 whose cumulative reduction (i.e., corresponding to the cumulative reduction of downward arrows Δ1, Δ2, Δ3) is minimized for the determined number of returned video streams (three in this case).

[0043] The streaming controller 20 solves this optimization problem using conventional mathematical methods. The task of the streaming controller 20 is to determine the number of returned video streams and their bitrates. The number of returned video streams can be set to a fixed value before the bitrate value is determined, or it can be a parameter that the streaming controller 20 is allowed to vary as part of solving the optimization problem. Another aspect considered by the streaming controller 20 in solving the optimization problem is that all video stream requests should be able to match the returned video streams. Depending on the characteristics of the system to which the streaming controller 20 is part, this aspect can imply different requirements for the optimization process S307. If the returned video streams are to be ready for direct delivery and use by clients 14a to 14d, then there must exist a returned video stream with a bitrate that matches the video stream request containing the lowest bitrate; otherwise, the video stream request cannot be matched. However, if the surrounding system contains a component that can process or reconfigure the return video stream en route to a client with lower bitrate requirements, it is possible to avoid generating a return video stream that meets the minimum requirements and instead rely on, for example, a transcoding process that can generate a return video stream with a lower bitrate during transmission to a specific client. An example of such a component is a server that receives video from several cameras and provides the video stream to one or more clients. The server has the capability to perform video transcoding on the received video stream (in this context, the return video stream). The server could, for example, be part of a video conferencing system. When performing optimization processing S307, the streaming controller 20 needs to be aware of the system's transcoding capabilities.

[0044] A deeper understanding of the details of determining the number of video streams returned by the S308 is possible, either before or in parallel with determining the bitrate of the returned video streams. In other words, when determining the bitrate, there can be a fixed number of returned video streams to optimize the bitrate, or multiple numbers of returned video streams can be estimated to determine the optimal combination of the number of S308 returned video streams and their bitrate.

[0045] In one embodiment, the number of returned video streams is determined S308 as the maximum number of possible video streams that encoder 12 can generate in parallel. The maximum number of possible video streams may be specific to a certain resolution; for example, encoder 12 may generate a 1080p video stream, a 720p video stream, and a 360p video stream in parallel. Streaming controller 20 may be arranged to selectively perform optimization processing S307 if the number of detected video stream requests (optionally specific to a certain resolution) exceeds the maximum number of video streams encoder 12 can generate simultaneously. In another embodiment, the preferred power consumption of encoder 12 is considered when determining the number of returned video streams. In this case, encoder 12 is limited to a threshold number of encoded video streams to avoid exceeding the preferred power consumption, even if encoder 12 can generate more video streams. In yet another embodiment, the number of returned video streams is calculated based on the number of detected video stream requests (the number of video stream requests). A predefined calculation formula can be used for this purpose. In one embodiment, the calculation formula defines a logarithmic function to calculate the number of returned video streams from the number of detected video stream requests. In other words, the number of returned video streams is calculated as the logarithm of the number of video stream requests. A suitable example of a logarithmic function is In this example, the number of requested video streams is N. request =3 will result in N video streams being returned. return =2, is the smallest integer greater than or equal to log2(3), and N request =6 will result in N return =3, is the smallest integer greater than or equal to log2(6). For different video stream request volumes, a favorable balance between processing utilization and user experience can be achieved by calculating the number of returned video streams using a predefined formula.

[0046] In optimization process S307, different numbers of candidate return video streams can be considered compared to the number of return video streams determined independently of their bitrates. Different numbers of return video streams can be considered in optimization process S307 to find the optimal set of return video streams and their bitrates. For example, streaming controller 20 can be configured to select a higher number of return video streams (assuming encoder 12 can generate them) when the cumulative QoE loss can be reduced to a certain extent compared to the optimal subset of lower numbers of return video streams. The power consumption of encoder 12 can be a parameter on which the optimization of the number of return video streams is based. In other words, higher power consumption as a result of a higher number of return video streams is only reasonable if the potential reduction in cumulative QoE loss compared to a lower number of return video streams is sufficiently high.

[0047] exist Figure 4 In the example, the streaming controller 20 concludes in its optimization process S307 that the combination of the three video streams is optimal, and the set of bit rates B1, B2, and B3 of the three video streams results in the lowest cumulative QoE reduction (or QoE decrease) = Δ1 + Δ2 + Δ3. It can be noted that there may exist a candidate set of a higher number of video streams that would produce a lower cumulative QoE loss; however, these candidate sets have been excluded by the streaming controller 20 according to the constraints in the optimization process S307 (e.g., based on power consumption).

[0048] Next, the streaming controller 20 obtains the number of return video streams in S312 based on the determined number of return video streams and the bit rate they should have. Therefore, the streaming controller 20 sends instruction 22 to the encoder 12 to perform encoding of the return video streams. Instruction 22 may also include any constraints on one or more of the return video streams (e.g., one or more of the return video streams should use the FPS value or resolution for encoding). In one embodiment, the encoder 12 encodes the return video streams 24 and returns them to the streaming controller 20. Next, in response to multiple video stream requests, the encoded return video streams 24 are transmitted to the client. When sending the instruction to the encoder 12, the streaming controller 20 has already determined how the return video streams 24 match the multiple video stream requests, as this is a result of the optimization process S307. In another embodiment, the streaming controller 20 does not process the actual return video streams 24, but only ensures that they are streamed (transmitted) to the client according to the match. Information about which return video stream matches which video stream request can be included in the instruction 22 sent to the encoder 12. In other words, the returned video stream 24 can be sent from the encoder 12 to the transmitter (not shown) for streaming to the client without going through the streaming controller 20.

[0049] Method 300, executed by streaming controller 20, is a continuous process. This means that when changes are detected in multiple video streaming requests (such as new video streaming requests, changed video streaming requests, or if a client disconnects (meaning its video streaming request disappears from the perspective of streaming controller 20)), method 300 can be triggered to update the number and bitrate of the returned video streams in the encoded real-time video stream, in order to optimize the user experience for currently connected clients. Method 300 can also be triggered when the current bandwidth of one or more returned video streams being streamed in the returned video stream from camera 10 exceeds a threshold (i.e., the bandwidth used for the transmission channel increases or decreases beyond a threshold). The bandwidth of one or more returned video streams being streamed in the returned video stream can be periodically measured, estimated, or otherwise obtained by camera 10. Specifically, the bandwidth can be provided by one or more clients 14a to 14d. In other words, the streaming controller 20 may be adapted to obtain the current bandwidth for transmitting video to one or more clients 14a to 14d, and to trigger the execution of method 300 for transmission to at least one of the one or more clients when the current bandwidth deviates from a reference bandwidth value by more than a threshold. The reference bandwidth value may be the bandwidth valid for the relevant transmission channel at the last execution of method 300.

[0050] Figure 5The illustration shows an embodiment where the streaming controller 20 is part of a relay device 50 connected to a camera device 10 and a client 58. The relay device 50 may be, for example, a cloud server that distributes received video streams to one or more clients, or a recorder that functions as a relay for video streams between one or more cameras and one or more clients.

[0051] In this embodiment, the streaming controller 20 detects multiple video stream requests from client 58 in S302. Based on the same concept and considerations as described above, the following steps are performed: obtaining the maximum bitrate value of the returned video stream in S304, calculating the QoE value of the request in S306, determining the number of returned videos in S308, and determining the bitrate of the returned videos in S310 (through the optimization process described above). For the step of obtaining multiple returned videos in S312, the streaming controller 20 sends instruction 52 to camera device 10. Camera device 10 receives the instruction and uses its encoder 12 to generate a returned video stream 54 according to the received instruction 52, and sends the returned video stream 54 to relay device 50. The streaming controller 20 in relay device 50 then streams the returned video stream 54 to client 58 in S314, meaning that the streaming controller 20 ensures that the returned video stream is used to respond to each video stream request according to the match determined during the optimization process. As described in the previous embodiment, the returned video stream 54 may or may not pass through the streaming controller 20 on its way to client 58.

[0052] The relay device 50 may include circuitry for reconfiguring the returned video stream with respect to encoding characteristics. For example, the returned video stream 54 may be transcoded (decoded and re-encoded) to obtain a video stream with a different resolution, FPS, or encoding format. As previously mentioned, the streaming controller 20 is aware of this reconfiguration capability and takes it into account when performing optimization processing S307.

[0053] Entering into the technical configuration of the streaming controller 20, it can be noted that the functionality of the elements disclosed herein (e.g., the streaming controller 20) can be configured in a hardware implementation, a software implementation, or a combination of both. Functionality can be implemented using circuitry or processing circuitry configured or programmed to perform that functionality, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application-Specific Integrated Circuits), conventional circuitry, and / or combinations thereof. Processors are considered processing circuitry or circuitry because they include transistors and other circuitry. The circuitry of the streaming controller 20 can be in the form of a processor, such as a microprocessor, associated with computer code instructions stored on a (non-transitory) computer-readable medium, such as non-volatile memory, causing the streaming controller 20 to perform (a portion of) any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical discs, etc. In the software case, the streaming controller 20 can therefore correspond to a portion of computer code instructions stored on a computer-readable medium, which, when executed by the processor of the streaming controller 20, causes the streaming controller 20 to perform the described functionality. It should be understood that a combination of hardware and software implementations is possible. As an example, some functions of the streaming controller 20 can be implemented in hardware, while others can be implemented in software.

[0054] The embodiments described herein should be understood as illustrative examples of this disclosure. Further embodiments of this disclosure are contemplated. For example, the proposed method is applicable to different system architectures that may include a front-end server and a cloud server, or a combination of both. Furthermore, the system architecture may include communication based on various network technologies including wired networks, wireless networks, and cellular networks.

[0055] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. Furthermore, equivalent schemes and modifications not described above may be employed without departing from the scope of this disclosure as defined by the appended claims.

Claims

1. A method for streaming real-time encoded video to one or more clients, the method comprising: Detect (S302) multiple video stream requests from one or more clients (14a to 14d, 58), For each video stream request, obtain (S304) the maximum bitrate value that the video stream (24, 54) is allowed to have. Based on the maximum bitrate for the video stream request, calculate the Quality of Experience (QoE) value for each video stream request (S306). Determine (S308, S310) the number of returned video streams (24, 54) with bitrates (B1, B2, B3) lower than the plurality of video stream requests, such that each of the video stream requests can be matched with a returned video stream (24, 54) having a bitrate equal to or lower than the maximum bitrate value of the video stream request, wherein the bitrates (B1, B2, B3) of the returned video streams (24, 54) are determined such that, for the determined number of returned video streams (24, 54), the cumulative difference between the QoE value of each video stream request and the calculated returned QoE of the matched returned video stream is minimized. Based on the determined bitrates (B1, B2, B3), the number of returned video streams (24, 54) is obtained (S312). The returned video stream (24, 54) is streamed (S314) to the one or more clients (14a to 14d, 58) such that each video stream request is responded to using a matching returned video stream (24, 54).

2. The method according to claim 1, wherein, The action of obtaining (S304) the maximum bit rate value that the returned video stream (24, 54) is allowed to have includes: obtaining the maximum bit rate value of the request included in the video stream request.

3. The method according to claim 1, wherein, The action of obtaining (S304) the maximum bit rate value that the returned video stream (24, 54) is allowed to have includes: determining the available bandwidth for transmitting the returned video stream.

4. The method according to claim 1, wherein, The action of determining (S308) the number of returned video streams (24, 54) includes: calculating the number based on the number of video stream requests among the plurality of video stream requests.

5. The method according to claim 4, wherein, The number of returned video streams (24, 54) is calculated (S308) as the logarithm of the number of video stream requests.

6. The method according to claim 1, wherein, The number of returned video streams (24, 54) is determined based on the preferred power consumption of the encoder (12) that encodes the returned video streams (S308).

7. The method according to claim 1, wherein, The requested QoE value and the returned QoE value are calculated as one of the following metrics: SSIM, PSNR, VMAF, UQ1.

8. The method according to claim 1, wherein, The video stream request includes one or more encoding characteristics of the request, and wherein the QoE value of the request for each video stream request is calculated based on the encoding characteristics of the request (S306).

9. The method according to claim 1, wherein, The QoE value of each video stream request is calculated based on predetermined encoding characteristics when the video stream request does not include the encoding characteristics of the request (S306).

10. The method according to claim 9, wherein, The predetermined encoding characteristics are determined based on the video scene of the requested video stream or based on the type of camera (10) used to capture the requested video stream.

11. A computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

12. A streaming controller for controlling the streaming of real-time encoded video to one or more clients (14a to 14d, 58), the streaming controller (20) comprising circuitry adapted to perform the method according to any one of claims 1 to 10.

13. The streaming controller according to claim 12, wherein, The circuit is further adapted to obtain the current bandwidth for transmitting video to the one or more clients (14a to 14d, 58), and to trigger the execution of the method according to any one of claims 1 to 10 when the current bandwidth deviates from a threshold for transmitting to at least one of the one or more clients (14a to 14d, 58).

14. A camera system comprising a streaming controller (20) according to claim 12.

15. The camera system according to claim 14, wherein, The streaming controller (20) is included in the relay device (50) of the camera system.

Images