Adaptive playing method based on scalable video coding and related device

Abstract

The invention discloses a self-adaptive playing method based on scalable video coding and a related device. The method comprises the steps that an acquisition terminal acquires YUV video data; preprocessing the YUV video data to generate a plurality of different processed YUV video data; performing scalable video coding on each processed YUV video data to obtain a scalable video comprising a plurality of levels, and transmitting the scalable video to a transfer server, and the transfer server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or the equipment performance of each receiving terminal and transmits the sub-layer code stream to the corresponding receiving terminal for playing. According to the invention, the technical problem that the existing SHVC coding technology can generate different layers of video data only by inputting one piece of YUV video data, and the coding efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of video coding technology, and in particular to an adaptive playback method based on scalable video coding and related devices. Background Art

[0002] With the rapid development of the Internet, people's demand for high-quality video transmission is also growing. As a new form of Internet, visual networking is gradually emerging. Visual networking uses Internet transmission technology to transmit multimedia video and audio data, and can realize a variety of application scenarios such as high-definition video live broadcast and playback, solving the challenges faced by large-scale video transmission, such as high bandwidth costs, complex networks, and low quality experience of high-definition video communication.

[0003] Video coding is involved in the video transmission process. SHVC (Scalable High-Efficiency Video Coding) is a scalable extension of the high-efficiency video coding standard. The basic idea of ​​SHVC coding is to output a multi-layer code stream after encoding the information of the video sequence once. The output code stream includes a base layer (Base Layer, BL) and several enhancement layers (Enhancement Layer, EL). The base layer refers to the basic coding layer of the video, which uses a low frame rate sequence and can be decoded on all receiving terminals; the enhancement layer is based on the base layer and adds some additional sub-streams, which can provide higher resolution, bit rate, frame rate and other advanced functions, but requires higher transmission bandwidth and decoding capabilities to support. However, SHVC coding technology can only generate different layers of video data by inputting a YUV video data. SHVC coding technology first encodes the YUV video data to obtain the base layer, and then encodes the base layer to obtain the enhancement layer to obtain coded videos at different levels, which has the problem of low coding efficiency. Summary of the invention

[0004] The present application provides an adaptive playback method and related devices based on scalable video coding, which are used to improve the technical problem that the existing SHVC coding technology can only generate different layers of video data by inputting a YUV video data, and has low coding efficiency.

[0005] In view of this, a first aspect of the present application provides an adaptive playback method based on scalable video coding, which is applied to a capture terminal, and the method includes:

[0006] Collect YUV video data;

[0007] Preprocessing the YUV video data to generate a plurality of different processed YUV video data;

[0008] Scalable video encoding is performed on each of the processed YUV video data to obtain a scalable video including multiple layers, and the scalable video is transmitted to a transit server. The transit server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

[0009] Optionally, the preprocessing of the YUV video data to generate a plurality of different processed YUV video data includes:

[0010] Downsampling and / or privacy masking of regions of interest are performed on the YUV video data to generate a plurality of different processed YUV video data.

[0011] Optionally, performing scalable video encoding on each of the processed YUV video data to obtain scalable videos including multiple levels includes:

[0012] The SHVC encoder is used to perform scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels; wherein the sublayers of the scalable video include a time layer, a space layer, a quality layer, a hybrid coding layer, and an area of ​​interest privacy protection layer.

[0013] A second aspect of the present application provides a collection terminal, including:

[0014] An acquisition unit, used for acquiring YUV video data;

[0015] A preprocessing unit, used for preprocessing the YUV video data to generate a plurality of different processed YUV video data;

[0016] An encoding unit, configured to perform scalable video encoding on each of the processed YUV video data to obtain scalable videos including multiple levels;

[0017] The transmission unit is used to transmit the scalable video to the transfer server, and the transfer server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

[0018] Optionally, the preprocessing unit is specifically used to downsample the YUV video data and / or perform privacy masking processing on the region of interest to generate a plurality of different processed YUV video data.

[0019] Optionally, the encoding unit is specifically used to use an SHVC encoder to perform scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels; wherein the sublayers of the scalable video include a time layer, a spatial layer, a quality layer, a hybrid coding layer, and an area of ​​interest privacy protection layer.

[0020] A third aspect of the present application provides an adaptive playback system based on scalable video coding, comprising: a collection terminal, a transfer server, and a receiving terminal;

[0021] The acquisition terminal is used to acquire YUV video data; pre-process the YUV video data to generate a plurality of different processed YUV video data; perform scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels, and transmit the scalable video to the transfer server;

[0022] The transfer server is used to extract the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmit it to the corresponding receiving terminal for playback.

[0023] Optionally, the transit server is an edge box or a streaming media forwarding server.

[0024] A fourth aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0025] The memory is used to store program code and transmit the program code to the processor;

[0026] The processor is used to execute any one of the adaptive playback methods based on scalable video coding according to the instructions in the program code of the first aspect.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, it implements the adaptive playback method based on scalable video coding described in any one of the first aspects.

[0028] It can be seen from the above technical solutions that this application has the following advantages:

[0029] The adaptive playback method based on scalable video coding provided by the present application performs pre-processing on an input YUV video data to generate multiple YUV video data, and then performs scalable video coding on each YUV video data to obtain coded videos of multiple levels, thereby improving the coding efficiency and improving the technical problem that the existing SHVC coding technology can only generate different layers of video data by inputting one YUV video data, and has low coding efficiency; moreover, according to the changes in the network bandwidth and device performance of the receiving terminal, the coded video of the corresponding sub-layer can be selected for transmission, and the quality and bit rate of the video can be automatically adjusted to ensure the smoothness and clarity of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A flowchart of an adaptive playback method based on scalable video coding provided in an embodiment of the present application;

[0032] Figure 2 Another flowchart of an adaptive playback method based on scalable video coding provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of a collection terminal provided in an embodiment of the present application;

[0034] Figure 4 A structural diagram of an adaptive playback system based on scalable video coding provided in an embodiment of the present application

[0035] Figure 5 Another structural schematic diagram of an adaptive playback system based on scalable video coding provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] For easier understanding, please refer to Figure 1 In a first aspect, the present application provides an adaptive playback method based on scalable video coding, which is applied to a capture terminal, and the method includes:

[0038] Step 110: Collect YUV video data.

[0039] The acquisition terminal in this application can be a camera that supports encoding function and is used to acquire YUV video data.

[0040] Step 120: pre-process the YUV video data to generate a plurality of different processed YUV video data.

[0041] The processed YUV video data includes the original YUV video data. The acquisition terminal can downsample the acquired YUV video data to obtain a plurality of YUV video data of different resolutions; the acquisition terminal can also perform privacy masking processing on the YUV video data in the region of interest to obtain YUV video data with privacy masking on the region of interest. When performing privacy masking processing on the region of interest on the YUV video data, the privacy mask can be applied as a cover to the region of interest of the image frame of the YUV video data, and the privacy mask can be a black rectangular block, or other polygonal shapes or other colors.

[0042] Step 130: Perform scalable video encoding on each processed YUV video data to obtain a scalable video including multiple layers, and transmit the scalable video to the transit server. The transit server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

[0043] The acquisition terminal takes the processed YUV video data as input, and uses the SHVC encoder to perform scalable video encoding on each processed YUV video data, so as to obtain a scalable video with different time layers, space layers, quality layers, hybrid coding layers and privacy protection layers of the region of interest. The present application generates a plurality of YUV video data of different resolutions by pre-processing the input YUV video data, and then performs SHVC encoding on the YUV video data of each resolution, and can use multi-threaded parallel SHVC encoding on the YUV video data of each resolution; when performing SHVC encoding on the YUV video data of each resolution, only the basic layer can be encoded to generate a scalable video including multiple levels, which saves the encoding time of the basic layer when encoding the enhancement layer, and improves the encoding efficiency. In addition, the present application can also perform SHVC encoding on the YUV video data with privacy masking in the region of interest, and obtain the privacy protection layer of the region of interest, thus realizing the layered video function of privacy masking in the region of interest.

[0044] The SHVC encoder uses the SHVC encoding library for scalable video encoding. The SHVC encoding library implements SHVC scalable video encoding based on x265 and refers to the implementation principle of SHM. SHM is the official reference model of H.265 / SHVC. It is constantly updated with the development of the SHVC project. It provides a series of executable files for SHVC encoding and data analysis. x265 is an open source free software and function library for encoding movies that meet the High Efficiency Video Coding (HEVC / H.265) standard. When using the SHVC encoding library for scalable video encoding, configuration parameters are used. It can set different time layers, spatial layers, quality layers, hybrid coding layers, and privacy protection layers of interest areas according to specific needs to obtain the desired scalable video. For example, if you want to use the SHVC encoding library to perform scalable video encoding on the original YUV video data of three resolutions to generate a scalable video with 7 layers, including 1 base layer (layer0), 1 temporal layer (layer1), 1 spatial layer (layer4), 1 quality layer (layer3), 2 hybrid coding layers (layer2, layer5), and 2 privacy protection layers of the region of interest (layer6, layer7), you can set the output file name to shvc.265; the three original YUV video data use a unified frame rate of 25f / s, so set the maximum frame rate to 25; the test segment duration is 4s, 25f / s×4 s=100f, so the maximum number of coded frames is set to 100 frames; if it is designed as n time layers, then GOP contains at most 2n-1 frames, so GOPSize=2; the basic layer coding mode uniformly adopts the HEVC mode including SEI; the layered mode adopts the medium granularity, assigned 1; the key frame coding adopts the medium granularity mode accordingly, assigned 1; and the non-key frame adopts the mode combining motion estimation and motion compensation, assigned 2. After configuring these parameters, generate the configuration file; use the configuration file to call the SHVC encoding library to perform scalable video encoding on the original YUV video data of three resolutions, and finally obtain the expected scalable video containing 7 levels. The specific layering of the video is as follows Figure 2 shown.

[0045] After the acquisition terminal performs scalable video encoding on the YUV video data, the network transmission protocol is used to transmit the encoded video data to the transit server (such as an edge box or a streaming media forwarding server). Specifically, it can be uploaded to the edge box or the streaming media forwarding server through RFC6190 encapsulation as the basic means of video negotiation encoding and transmission within the video system. The edge box or the streaming media forwarding server extracts the corresponding sub-layer code stream according to the needs of different receiving terminals and transmits it to each receiving terminal for playback. The transit server can select the sub-layer code stream with the corresponding resolution, quality or bit rate according to the network status and / or device performance of the receiving terminal and transmit it to the corresponding receiving terminal. For a receiving terminal with limited bandwidth, a low-resolution sub-layer video can be selected to be transmitted to the receiving terminal.

[0046] The present application can divide video data into multiple layers, each of which can be encoded and decoded as needed, thereby realizing video transmission and playback of different resolutions and different bit rates; the present application can select the encoded video of the corresponding sub-layer for transmission according to the changes in the network bandwidth and device performance of the receiving terminal, and can automatically adjust the quality and bit rate of the video to ensure the smoothness and clarity of the video, and can provide the best video quality under different network conditions; and the improved scalable video encoding method proposed in the present application compresses the network bandwidth, has more flexible storage management, and eliminates the secondary encoding step.

[0047] The adaptive playback method based on scalable video coding in this application can be applied to remote monitoring. The monitoring system can transmit low-resolution videos for monitoring when the bandwidth is limited, and save high-resolution videos for reference when the local or network conditions permit; it can also be applied to multi-view monitoring. The video streams of different monitoring cameras can be retrieved on demand at the central end to obtain videos of different qualities to meet the needs of real-time monitoring and post-event evidence collection; it can also be applied to multi-level storage. During the storage and transmission process, the low-resolution version can be selected for storage according to demand, thereby saving storage space and transmission costs. The improved scalable video coding method provided in this application enables network resources to be used more efficiently by providing video streams of different resolutions and qualities, reducing unnecessary high bandwidth consumption; during peak hours, by adaptively adjusting the quality of the video stream, the network load can be effectively reduced, network congestion can be avoided, and the overall network service quality can be improved.

[0048] Please refer to Figure 3 The second aspect of the present application provides a collection terminal, including:

[0049] An acquisition unit, used for acquiring YUV video data;

[0050] A preprocessing unit, used for preprocessing the YUV video data to generate a plurality of different processed YUV video data;

[0051] An encoding unit, used for performing scalable video encoding on each processed YUV video data to obtain scalable videos including multiple levels;

[0052] The transmission unit is used to transmit the scalable video to the transit server, which extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

[0053] As a further improvement, the preprocessing unit is specifically used to perform downsampling and / or privacy masking processing on the YUV video data to generate a plurality of different processed YUV video data.

[0054] As a further improvement, the encoding unit is specifically used to use the SHVC encoder to perform scalable video encoding on each processed YUV video data respectively, so as to obtain a scalable video including multiple levels; wherein the sublayers of the scalable video include a time layer, a spatial layer, a quality layer, a hybrid coding layer, and an area of ​​interest privacy protection layer.

[0055] The acquisition terminal provided in the present application can realize scalable video coding based on video acquisition, perform pre-processing on an input YUV video data to generate multiple YUV video data, and then perform scalable video coding on each YUV video data to obtain a coded video including multiple levels, thereby improving the coding efficiency.

[0056] Please refer to Figure 4 , the third aspect of the present application provides an adaptive playback system based on scalable video coding, including: a collection terminal, a transfer server and a receiving terminal;

[0057] The acquisition terminal is used to acquire YUV video data; pre-process the YUV video data to generate a plurality of different processed YUV video data; perform scalable video encoding on each processed YUV video data to obtain a scalable video including multiple levels, and transmit the scalable video to the transfer server;

[0058] The transfer server is used to extract the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmit it to the corresponding receiving terminal for playback.

[0059] Among them, the transit server is an edge box or a streaming media forwarding server.

[0060] Please refer to Figure 5An embodiment of an adaptive playback system based on scalable video coding is provided. After a high-definition camera supporting SHVC coding collects YUV video data, it performs downsampling and privacy masking of the region of interest (ROI), and then performs SHVC encoding on the YUV video data of each resolution and the YUV video data with privacy masking of the region of interest to obtain a scalable video including multiple layers, and then uses a network transmission protocol to transmit the encoded scalable video to an edge box. The edge box selects the corresponding sub-layer code stream according to the network bandwidth and device performance of the receiving terminal to transmit it to the corresponding receiving terminal for playback. The existing system requires the transit server to support transcoding and add privacy coding functions to support multi-terminal playback, while the transit server in this system can meet the terminal playback requirements without supporting transcoding, adding privacy coding and other functions.

[0061] The adaptive playback system based on scalable video coding provided by the present application is that the acquisition terminal can perform pre-processing on an input YUV video data to generate multiple YUV video data, and then perform scalable video coding on each YUV video data to obtain coded videos of multiple levels, thereby improving the coding efficiency and improving the technical problem that the existing SHVC coding technology can only generate different layers of video data by inputting one YUV video data, and has low coding efficiency; the transit server in the system can also select the coded video of the corresponding sub-layer for transmission according to the changes in the network bandwidth and device performance of the receiving terminal, and can automatically adjust the quality and bit rate of the video to ensure the smoothness and clarity of the video, and can provide the best video quality under different network conditions.

[0062] A fourth aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0063] The memory is used to store the program code and transmit the program code to the processor;

[0064] The processor is used to execute the adaptive playback method based on scalable video coding of the first aspect according to the instructions in the program code.

[0065] In a fifth aspect, the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, the adaptive playback method based on scalable video coding in the first aspect is implemented.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0069] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0070] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0073] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An adaptive playback method based on scalable video coding, characterized in that: Applied to a collection terminal, the method includes: Collect YUV video data; Preprocessing the YUV video data to generate a plurality of different processed YUV video data; Scalable video encoding is performed on each of the processed YUV video data to obtain a scalable video including multiple layers, and the scalable video is transmitted to a transit server. The transit server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

2. The adaptive playback method based on scalable video coding according to claim 1, characterized in that: The preprocessing of the YUV video data to generate a plurality of different processed YUV video data includes: Downsampling and / or privacy masking of regions of interest are performed on the YUV video data to generate a plurality of different processed YUV video data.

3. The adaptive playback method based on scalable video coding according to claim 1, characterized in that: The step of performing scalable video encoding on each of the processed YUV video data to obtain scalable videos including multiple levels includes: The SHVC encoder is used to perform scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels; wherein the sublayers of the scalable video include a time layer, a space layer, a quality layer, a hybrid coding layer, and an area of ​​interest privacy protection layer.

4. A collection terminal, characterized in that: include: An acquisition unit, used for acquiring YUV video data; A preprocessing unit, used for preprocessing the YUV video data to generate a plurality of different processed YUV video data; An encoding unit, configured to perform scalable video encoding on each of the processed YUV video data to obtain scalable videos including multiple levels; The transmission unit is used to transmit the scalable video to the transfer server, and the transfer server extracts the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmits it to the corresponding receiving terminal for playback.

5. The acquisition terminal according to claim 4, characterized in that: The pre-processing unit is specifically used to perform down-sampling and / or privacy masking processing on the YUV video data to generate a plurality of different processed YUV video data.

6. The acquisition terminal according to claim 4, characterized in that: The encoding unit is specifically used to use an SHVC encoder to perform scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels; wherein the sublayers of the scalable video include a time layer, a space layer, a quality layer, a hybrid coding layer, and an area of ​​interest privacy protection layer.

7. An adaptive playback system based on scalable video coding, characterized in that: include: Collection terminal, transfer service terminal and receiving terminal; The acquisition terminal is used to acquire YUV video data; Preprocessing the YUV video data to generate a plurality of different processed YUV video data; performing scalable video encoding on each of the processed YUV video data to obtain a scalable video including multiple levels, and transmitting the scalable video to the transfer server; The transfer server is used to extract the corresponding sub-layer code stream from the scalable video according to the network bandwidth and / or device performance of each receiving terminal and transmit it to the corresponding receiving terminal for playback.

8. The adaptive playback system based on scalable video coding according to claim 7, characterized in that: The transfer server is an edge box or a streaming media forwarding server.

9. An electronic device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the adaptive playback method based on scalable video coding according to any one of claims 1-3 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and when the program codes are executed by a processor, the adaptive playback method based on scalable video coding described in any one of claims 1 to 3 is implemented.

Images