Video data processing method and apparatus, and coder, network device and loader
By receiving the target transmission rate of the network device and the target terminal and adjusting the video encoding parameters, the problem of mismatch between the transmission rate and the video quality in the video streaming media is solved, and the synchronous improvement of video quality and bandwidth usage is achieved.
Patent Information
- Application Number
- PCT/CN2024/093317
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-28
AI Technical Summary
In the prior art, there is a problem that the transmission rate and the video quality do not match in video streaming media playback, resulting in a bandwidth step effect and the video quality and the transmission rate increase in synchronously.
By receiving the target transmission rate between the network device and the target terminal, the encoding parameters of the video to be played, including the encoding parameters of the base layer and the enhancement layer, and the video code stream is adjusted to match the transmission rate and video quality.
It realizes the matching of transmission rate and video quality during video streaming, ensuring synchronous improvement of video quality and bandwidth usage.
Smart Images

Figure CN2024093317_28112024_PF_FP_ABST
Abstract
Description
Video data processing method, device, encoder, network equipment and loader
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to Chinese patent application No. 202310575417.X filed in China on May 19, 2023, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the field of wireless communication technology, and in particular to a video data processing method, apparatus, encoder, network equipment, and loader. Background Art
[0004] Related video encoding technologies suffer from a significant step effect: as transmission rates increase, video quality and user experience do not improve simultaneously. Instead, they only improve when the rate reaches the next resolution threshold. For example, a 2Mbps bandwidth allows for 480p video, while a 10Mbps bandwidth allows for 1080p video. However, within the range [2-10), even if the bandwidth increases, video quality does not improve synchronously because the bandwidth threshold for the next video quality level is not reached.
[0005] Figure 1 illustrates the bandwidth step effect. As data rates increase, video resolution doesn't gradually increase. Instead, it increases in steps after reaching the minimum transmission rate. Similarly, when the transmission rate drops below the minimum rate, even if the transmission rate is significantly higher than the minimum rate required for the next level of transmission, only the next level of video can be played. Therefore, the related art of video streaming suffers from a mismatch between transmission rate and video quality.
[0006] Summary of the Invention
[0007] The purpose of the present disclosure is to provide a video data processing method, apparatus, encoder, network device and loader to solve the problem of mismatch between transmission rate and video quality in video streaming media playback in the related art.
[0008] To achieve the above objectives, an embodiment of the present disclosure provides a video data processing method, applied to an encoder, comprising:
[0009] receiving a target transmission rate sent by a network device, where the target transmission rate is a transmission rate between the network device and a target terminal;
[0010] Determining a first parameter of the video to be played according to the target transmission rate, the first parameter including: a first encoding parameter of a base layer, or the first parameter including: a first encoding parameter of the base layer and a first number and a second encoding parameter of an enhancement layer;
[0011] Encode the video to be played according to the first parameter to obtain an output video stream including at least one network abstraction layer unit NALU;
[0012] Send the network abstraction layer unit to the loader.
[0013] Furthermore, the method for determining the first coding parameter of the base layer includes:
[0014] Determine a preset ratio of the target transmission rate as a first transmission rate of the base layer;
[0015] The first encoding parameter is determined according to the first transmission rate.
[0016] Furthermore, the method for determining the first number of enhancement layers and the second coding parameter includes:
[0017] determining a first number of enhancement layers according to the target transmission rate and the first transmission rate;
[0018] The second encoding parameter is determined according to the target transmission rate, the first transmission rate, and the first number.
[0019] Furthermore, the first encoding parameter and the second encoding parameter include:
[0020] Resolution, frame rate, and encoding quality;
[0021] The highest level of the enhancement layer corresponds to the highest resolution of the video to be played.
[0022] Furthermore, the network abstraction layer includes at least one group of pictures GoP.
[0023] To achieve the above objectives, embodiments of the present disclosure provide a video data processing method, applied to a network device, comprising:
[0024] Get the status data of the target terminal;
[0025] determining a target transmission rate between the network device and the target terminal based on the status data and the load data of the network device;
[0026] Sending the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played;
[0027] Sending a transport block size TBS allocated for the video to be played to the loader;
[0028] Receives the Network Abstraction Layer Unit (NALU) sent by the loader.
[0029] Furthermore, determining the target transmission rate according to the status data and the load data of the network device includes:
[0030] determining an average transmission rate from the network device to the target terminal based on the status data and the load data of the network device;
[0031] The target transmission rate is determined according to the average transmission rate.
[0032] Furthermore, the sending of the transport block size TBS allocated for the video to be played to the loader includes:
[0033] Acquiring a channel condition of the target terminal, the channel condition including: channel state information and a channel quality indicator;
[0034] According to the channel condition of the target terminal and the channel resource utilization rate of the network device, a transport block size TBS allocated to each group of pictures GoP corresponding to the video to be played is determined.
[0035] Furthermore, the status data includes:
[0036] The transmission bandwidth, channel quality parameters and the highest supported modulation mode of the target terminal;
[0037] The load data includes:
[0038] The number of access terminals, the number of activated terminals, and the air interface resource utilization rate of the network device.
[0039] To achieve the above objectives, an embodiment of the present disclosure provides a video data processing method, applied to a loader, comprising:
[0040] Receive a transmission block size TBS allocated for the video to be played, sent by a network device;
[0041] Receiving at least one network abstraction layer unit NALU sent by the encoder;
[0042] The network abstraction layer unit NALU is sent to the network device according to the transport block size TBS.
[0043] Furthermore, the sending the network abstraction layer unit NALU to the network device according to the transport block size TBS includes:
[0044] Determining whether to discard a network abstraction layer unit NALU according to a transport block size TBS allocated by the network device for the video to be played;
[0045] If it is determined that the network abstraction layer unit NALU needs to be discarded, discard at least one target network abstraction layer unit NALU in the network abstraction layer unit;
[0046] The remaining network abstraction layer unit NALUs after discarding the target network abstraction layer unit NALU in the network abstraction layer unit NALU are sent to the network device.
[0047] Furthermore, the transport block size TBS allocated for the video to be played includes:
[0048] The transport block size TBS allocated to each group of pictures corresponding to the video to be played.
[0049] Furthermore, the determining whether it is necessary to discard a network abstraction layer unit NALU according to a transport block size TBS allocated by the network device to each picture group includes:
[0050] Determine the association relationship between each of the network abstraction layer units NALU;
[0051] And according to the association relationship and the transport block size TBS allocated by the network device to each picture group, it is determined whether the network abstraction layer unit needs to be discarded.
[0052] Furthermore, the method for determining the target network abstraction layer unit NALU includes:
[0053] Determine the target network abstraction layer unit according to the second parameter of the network abstraction layer unit NALU;
[0054] The second parameter includes at least one of the following: a level of a corresponding enhancement layer, a degree of dependence on other network abstraction layer units, a corresponding region of interest, and a number of historical discarded times.
[0055] To achieve the above objectives, an embodiment of the present disclosure provides a video data processing device, applied to an encoder, comprising:
[0056] A first receiving module, configured to receive a target transmission rate sent by a network device;
[0057] A first determining module is configured to determine a first parameter of a to-be-played video according to the target transmission rate, wherein the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of enhancement layers;
[0058] an encoding module, configured to encode the video to be played according to the first parameter to obtain an output video stream including at least one network abstraction layer unit NALU;
[0059] The first sending module is configured to send the network abstraction layer unit to a loader.
[0060] To achieve the above objectives, an embodiment of the present disclosure provides a video data processing apparatus, applied to a network device, comprising:
[0061] An acquisition module is used to obtain the status data of the target terminal;
[0062] a second determining module, configured to determine a target transmission rate based on the state data and the load data of the network device;
[0063] A second sending module, configured to send the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played;
[0064] A third sending module is used to send a transport block size TBS allocated for the video to be played to the loader;
[0065] The second receiving module is used to receive the network abstraction layer unit NALU sent by the loader.
[0066] To achieve the above-mentioned objectives, an embodiment of the present disclosure provides a video data processing device, applied to a loader, comprising:
[0067] A third receiving module is configured to receive a transmission block size TBS allocated for the video to be played, sent by a network device;
[0068] A fourth receiving module, configured to receive at least one network abstraction layer unit NALU sent by the encoder;
[0069] The fourth sending module is configured to send the network abstraction layer unit NALU to the network device according to the transport block size TBS.
[0070] To achieve the above object, an embodiment of the present disclosure provides an encoder, comprising a processor and a transceiver, wherein:
[0071] The transceiver is used to receive the target transmission rate sent by the network device;
[0072] The processor is configured to determine a first parameter of a to-be-played video according to the target transmission rate, where the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of an enhancement layer;
[0073] Encode the video to be played according to the first parameter to obtain an output video stream including at least one network abstraction layer unit NALU;
[0074] The transceiver is further configured to send the network abstraction layer unit to a loader.
[0075] To achieve the above object, an embodiment of the present disclosure provides a network device, including a processor and a transceiver, wherein:
[0076] The transceiver is used to obtain status data of the target terminal;
[0077] The processor is configured to determine a target transmission rate based on the status data and load data of the network device;
[0078] The transceiver is further configured to send the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played;
[0079] Sending a transport block size TBS allocated for the video to be played to the loader;
[0080] Receives the Network Abstraction Layer Unit (NALU) sent by the loader.
[0081] To achieve the above-mentioned object, an embodiment of the present disclosure provides a loader, comprising a processor and a transceiver, wherein:
[0082] The transceiver is used to receive a transmission block size TBS allocated for a video to be played, which is sent by a network device;
[0083] Receiving at least one network abstraction layer unit NALU sent by the encoder;
[0084] The network abstraction layer unit NALU is sent to the network device according to the transport block size TBS.
[0085] To achieve the above-mentioned objectives, an embodiment of the present disclosure provides a terminal device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the video data processing method as described above is implemented.
[0086] To achieve the above objectives, an embodiment of the present disclosure provides a readable storage medium having a program or instruction stored thereon, which implements the steps in the above-mentioned video data processing method when executed by a processor.
[0087] The beneficial effects of the above technical solution disclosed in the present invention are as follows:
[0088] In the video data processing method of an embodiment of the present disclosure, the encoder determines, based on the received target transmission rate between the network device and the target terminal, first parameters of the video to be played, including first encoding parameters of the base layer, or first encoding parameters of the base layer and first number and second encoding parameters of the enhancement layers, and determines, based on the first parameters, a video code stream including at least one network abstraction layer unit. The solution of the present disclosure determines the number and encoding parameters of the enhancement layers based on the transmission rate between the network device and the terminal, and can adjust the number and encoding parameters of the enhancement layers as the transmission rate changes, thereby ensuring that the transmission rate matches the video quality during the playback of the video streaming media. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] FIG1 is a schematic diagram of the bandwidth step effect;
[0090] FIG2 is a schematic diagram of the structure of scalable video coding;
[0091] FIG3 is a schematic diagram of a process of scalable video coding;
[0092] FIG4 is a schematic diagram of the structure of a network abstraction layer unit;
[0093] FIG5 is a second schematic diagram of the structure of the network abstraction layer unit;
[0094] FIG6 is a flowchart of a video data processing method applied to an encoder according to an embodiment of the present disclosure;
[0095] FIG7 is a flowchart of a video data processing method applied to a network device according to an embodiment of the present disclosure;
[0096] FIG8 is a flow chart of a method for processing video data applied to a loader according to an embodiment of the present disclosure;
[0097] FIG9 is a logic diagram of video data processing according to an embodiment of the present disclosure;
[0098] FIG10 is a logic diagram of video data processing according to another embodiment of the present disclosure;
[0099] FIG11 is a schematic structural diagram of a video data processing device applied to an encoder according to an embodiment of the present disclosure;
[0100] FIG12 is a schematic structural diagram of a video data processing apparatus applied to a network device according to an embodiment of the present disclosure;
[0101] FIG13 is a schematic structural diagram of a video data processing device applied to a loader according to an embodiment of the present disclosure;
[0102] FIG14 is a schematic structural diagram of an encoder according to an embodiment of the present disclosure;
[0103] FIG15 is a schematic diagram of the structure of a network device according to an embodiment of the present disclosure;
[0104] FIG16 is a schematic structural diagram of a loader according to an embodiment of the present disclosure;
[0105] FIG17 is a schematic structural diagram of a terminal device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0106] In order to make the technical problems, technical solutions and advantages to be solved by the present disclosure clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0107] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present disclosure. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout this specification do not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0108] In the various embodiments of the present disclosure, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.
[0109] Additionally, the terms "system" and "network" are often used interchangeably herein.
[0110] In the embodiments provided in the present disclosure, it should be understood that "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.
[0111] It should be noted that the encoder of the embodiment of the present disclosure is a radio access network (RAN) encoder based on scalable video coding (SVC) and scalable high efficiency video coding (SHVC), and the encoder can communicate with a RAN side device (ie, a network device); a NALU loader that can selectively load the network abstraction layer unit (NALU) output by the encoder to the RAN and can communicate with the network device.
[0112] As shown in Figure 2, Scalable Video Coding (SVC) is a technology that splits a video stream into multiple spatial, quality, and temporal layers. It is an extension of the H.264 video codec standard used by most video devices today. Video devices use SVC to transmit and receive multi-layer video streams consisting of a small base layer and multiple optional layers (enhancement layers) that increase resolution, frame rate, and quality.
[0113] As shown in Figure 3, SVC adapts to channel variations by splitting a high-quality video stream into multiple self-decoding streams. These streams are divided into an independently decodable base layer (BL) and an enhancement layer (EL) that relies on the BL and underlying decoding layers. Since H.264 Advanced Video Coding (AVC), video coding has been divided into a two-layer structure: the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The VCL encodes the video into a longer raw stream, which is then segmented into shorter streams based on image slices. The NAL encapsulates the segmented streams into Network Abstraction Layer Units (NALUs) for transmission over the network.
[0114] NALU can be considered as the smallest video coding unit transmitted on the network, and its structure is shown in Figures 4 and 5. NALU is divided into a header (NAL Header) and a network raw byte sequence payload (Raw Byte Sequence Payload, RBSP), of which RBSP is divided into three types: a VCL NAL unit containing a picture parameter set (PPS), a program stream (PS) NAL unit containing a sequence parameter set (SPS), and a SEI NAL unit containing supplementary enhancement information (SEI). Among them, a VCL NAL unit generally carries the coded data corresponding to a slice.
[0115] As shown in FIG6 , a video data processing method according to an embodiment of the present disclosure, applied to an encoder, includes the following steps:
[0116] Step 601: Receive a target transmission rate sent by a network device, where the target transmission rate is the transmission rate between the network device and a target terminal.
[0117] Step 602: Determine a first parameter of the video to be played based on the target transmission rate, where the first parameter includes: a first encoding parameter of the base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number of enhancement layers and a second encoding parameter of each enhancement layer.
[0118] Step 603: Encode the video to be played according to the first parameter to obtain an output video code stream including at least one network abstraction layer unit NALU.
[0119] Step 604: Send the network abstraction layer unit to the loader.
[0120] Optionally, each of the network abstraction layer units corresponds to only a slice in one image.
[0121] Optionally, the average bit rate of the video to be played is determined according to the target transmission rate, and then the maximum resolution of the video to be played is determined.
[0122] In the video data processing method of the embodiment of the present disclosure, the highest resolution of the video to be played is slightly higher than the bit rate corresponding to the target transmission rate.
[0123] It should be noted that the first parameter is updated as the target transmission rate changes.
[0124] In one embodiment of the present disclosure, the encoder can determine the average bitrate of the video to be played based on the target transmission rate sent by the network device, thereby determining the maximum resolution. The base layer is a required option for encoding the video to be played, and the enhancement layer is an optional option for the video to be played. For example, if the transmission rate is too low, only the base layer data is encoded and transmitted. Otherwise, the maximum resolution is the resolution corresponding to the highest enhancement layer, and the average bitrate of the corresponding video encoding at the highest level is slightly greater than the target transmission rate.
[0125] In the video data processing method of an embodiment of the present disclosure, the encoder determines, based on the received target transmission rate between the network device and the target terminal, first parameters of the video to be played, including first encoding parameters of the base layer, or first encoding parameters of the base layer and first number and second encoding parameters of the enhancement layers, and determines, based on the first parameters, a video code stream including at least one network abstraction layer unit. The solution of the present disclosure determines the number and encoding parameters of the enhancement layers based on the transmission rate between the network device and the terminal, and can adjust the number and encoding parameters of the enhancement layers as the transmission rate changes, thereby ensuring that the transmission rate matches the video quality during the playback of the video streaming media.
[0126] Optionally, a method for determining the first coding parameter of the base layer includes:
[0127] Determine a preset ratio of the target transmission rate as a first transmission rate of the base layer;
[0128] The first encoding parameter is determined according to the first transmission rate.
[0129] Optionally, the preset ratio ranges from 10% to 50%.
[0130] Optionally, when the target transmission rate is too low, only the base layer data is encoded and transmitted.
[0131] In one embodiment of the present disclosure, the target transmission rate of a preset ratio is used as the transmission rate of the base layer, and the coding parameters of the base layer are determined based on this; then the number and coding parameters of the enhancement layers are determined according to the coding parameters of the base layer.
[0132] Optionally, the enhancement layer may be selected as an integer multiple, such as 240p, 480p, 960p, 1920p or 270p, 540p, 1080p, 2160p or 360p, 720p, 1440p.
[0133] Optionally, the method for determining the first number of enhancement layers and the second coding parameter includes:
[0134] determining a first number of enhancement layers according to the target transmission rate and the first transmission rate;
[0135] The second encoding parameter is determined according to the target transmission rate, the first transmission rate, and the first number.
[0136] The step of determining the first number of enhancement layers according to the target transmission rate and the first transmission rate includes:
[0137] Determine the average bit rate of the video to be played according to the target transmission rate;
[0138] determining a maximum resolution according to the average bit rate;
[0139] The first number is determined according to the highest resolution, the first transmission rate, and a first preset common ratio.
[0140] For example, if the highest resolution is 1920p, the minimum resolution corresponding to the first transmission rate is 240p, and the first preset common ratio is 2, then according to the first preset common ratio, the resolution needs to be reduced three times from 1920p to 240p, and the number of enhancement layers is 3;
[0141] If the highest resolution is 960p, the minimum resolution corresponding to the first transmission rate is 240p, and the first preset common ratio is 2, then according to the first preset common ratio, the resolution needs to be reduced 2 times from 960p to 240p, and the number of enhancement layers is 2.
[0142] Determining the second encoding parameter according to the target transmission rate, the first transmission rate, and the first quantity includes:
[0143] Determine the average bit rate and average frame rate of the video to be played according to the target transmission rate;
[0144] Determine the highest resolution according to the average bit rate, and determine the highest frame rate according to the average frame rate;
[0145] Determining a resolution in the second encoding parameter according to the highest resolution and a second preset common ratio;
[0146] The frame rate in the second encoding parameter is determined according to the highest frame rate, the first transmission rate, and a third preset common ratio.
[0147] Exemplarily, if the highest resolution is 1920p, the second preset common ratio is 2, and the minimum resolution corresponding to the first transmission rate is 240p, then according to the second preset common ratio, the resolution in the second encoding parameter can be 1920p, 960p, 480p, or 240p;
[0148] If the highest resolution is 1920p, the second preset common ratio is 2, and the minimum resolution corresponding to the first transmission rate is 480p, then according to the second preset common ratio, the resolution in the second encoding parameter can be 1920p, 960p, or 480p;
[0149] If the maximum frame rate is 120 Hz, the third preset common ratio is 2, and the minimum frame rate corresponding to the first transmission rate is 30p, then the frame rate in the second encoding parameter can be 120 Hz, 60 Hz, or 30 Hz;
[0150] If the highest frame rate is 120 Hz, the third preset common ratio is 2, and the minimum frame rate corresponding to the first transmission rate is 60p, then the frame rate in the second encoding parameter can be 120 Hz or 60 Hz.
[0151] Optionally, the first encoding parameter and the second encoding parameter include:
[0152] Resolution, frame rate, and encoding quality;
[0153] The highest level of the enhancement layer corresponds to the highest resolution of the video to be played.
[0154] Optionally, determining a first transmission rate, resolution, frame rate, and encoding quality of the base layer according to the target transmission rate;
[0155] The resolution, frame rate, and encoding quality of each enhancement layer are determined according to the target transmission rate, the first transmission rate, and the first number.
[0156] Optionally, the encoder starts encoding after determining the first encoding parameter of the base layer, the number of enhancement layers, and the second encoding parameter;
[0157] The network abstraction layer units output by the encoder are separated in units of group of pictures.
[0158] Optionally, the network abstraction layer includes at least one group of pictures (Group of Pictures, GoP).
[0159] It should be noted that the frames and slices within the same GoP are related, but those between different GoPs can be considered unrelated.
[0160] The video data processing method of the embodiment of the present disclosure determines the encoding parameters of the base layer through the target transmission rate, thereby determining the number and encoding parameters of the enhancement layers. It can adjust the number and encoding parameters of the enhancement layers as the transmission rate changes, thereby ensuring that the transmission rate and video quality match during the playback of video streaming media.
[0161] As shown in FIG7 , a video data processing method according to an embodiment of the present disclosure, applied to a network device, includes the following steps:
[0162] Step 701, obtaining status data of the target terminal;
[0163] Step 702: determining a target transmission rate between the network device and the target terminal based on the status data and the load data of the network device;
[0164] Step 703: Send the target transmission rate to the encoder, where the target transmission rate is used by the encoder to determine a first parameter of the video to be played.
[0165] Step 704: Sending the transport block size (TBS) allocated for the video to be played to the loader;
[0166] Step 705: Receive the network abstraction layer unit NALU sent by the loader.
[0167] In one embodiment of the present disclosure, the network device determines the expected average transmission rate based on the resource size allocated to the corresponding video service that can be provided to the target terminal in the future period, so that the encoder determines the encoding parameters according to the target transmission rate.
[0168] In one embodiment of the present disclosure, the baseband processing unit (BBU) of the network device determines the transport block size TBS allocated to each picture group based on its own current channel resource utilization and the channel conditions of the target terminal (channel state information (CSI) and channel quality indicator (CQI)).
[0169] The baseband processing unit BBU of the network device sends the loaded NALU. If the NALU corresponding to the GoP can be fully loaded in a scheduled transmission block, no additional operation is required; if multiple scheduling is required and some NALUs cannot be received correctly, the BBU needs to notify the NALU loader of the NALU information that cannot be accepted, and the NALU removes all NALUs associated with the NALUs that failed to send.
[0170] Optionally, the transport block size TBS may be the sum of transport block sizes sent to the target terminal in multiple times.
[0171] The video data processing method of the embodiment of the present disclosure can determine the target transmission rate between the network device and the target terminal by obtaining the status data of the target terminal and combining it with the load data of the network device; by sending the transmission block size TBS allocated to the picture group GoP to the loader, the loader can send the network abstraction layer unit to the network device according to the transmission block size, so that the network device sends the network abstraction layer unit to the target terminal for video playback.
[0172] Optionally, obtaining status data of the target terminal includes:
[0173] The transmission bandwidth, channel quality parameters, and the highest supported modulation mode of the target terminal are obtained.
[0174] Optionally, the method for acquiring the load data of the network device includes:
[0175] Determine the terminals accessing the network device, and determine the number of terminals accessing the network device, the number of activated terminals accessing the network device, and the air interface resource utilization of each terminal.
[0176] Optionally, determining the target transmission rate according to the status data and the load data of the network device includes:
[0177] determining an average transmission rate from the network device to the target terminal based on the status data and the load data of the network device;
[0178] The target transmission rate is determined according to the average transmission rate.
[0179] Optionally, the target transmission rate is equal to or slightly greater than the average transmission rate.
[0180] Optionally, the sending a transport block size TBS allocated for the video to be played to the loader includes:
[0181] Acquiring a channel condition of the target terminal, the channel condition including: channel state information and a channel quality indicator;
[0182] According to the channel condition of the target terminal and the channel resource utilization rate of the network device, a transport block size TBS allocated to each group of pictures GoP corresponding to the video to be played is determined.
[0183] Optionally, the status data includes:
[0184] The transmission bandwidth, channel quality parameters and the highest supported modulation mode of the target terminal;
[0185] The load data includes:
[0186] The number of access terminals, the number of activated terminals, and the air interface resource utilization rate of the network device.
[0187] Optionally, the network device is mainly responsible for assisting the encoder in determining the number of enhancement layers and encoding parameters of each enhancement layer, and also needs to provide a TBS size expected to be allocated to each GoP;
[0188] For the loader, the network device also needs to ensure that the TB size indicated by the corresponding Digital Copyright Identifier (DCI) in the Physical Downlink Control Channel (PDCCH) is as close as possible to the GoP TBS;
[0189] After the loader completes the loading of the NALU, the network device is responsible for sending the corresponding NALU to the target terminal.
[0190] As shown in FIG8 , a video data processing method according to an embodiment of the present disclosure, applied to a loader, includes the following steps:
[0191] Step 801: receiving a transmission block size TBS allocated for a video to be played, sent by a network device;
[0192] Step 802: Receive at least one network abstraction layer unit (NALU) sent by an encoder;
[0193] Step 803: Send the network abstraction layer unit NALU to the network device according to the transport block size TBS.
[0194] Optionally, the loader loads the group of pictures according to the transport block size TBS.
[0195] In one embodiment of the present disclosure, the baseband processing unit (BBU) of the network device determines the transport block size TBS allocated to each picture group based on its own current channel resource utilization and the channel conditions of the target terminal (channel state information (CSI) and channel quality indicator (CQI)).
[0196] Optionally, the sending the network abstraction layer unit NALU to the network device according to the transport block size TBS includes:
[0197] Determining whether to discard a network abstraction layer unit NALU according to a transport block size TBS allocated by the network device for the video to be played;
[0198] If it is determined that the network abstraction layer unit NALU needs to be discarded, discard at least one target network abstraction layer unit NALU in the network abstraction layer unit;
[0199] The remaining network abstraction layer unit NALUs after discarding the target network abstraction layer unit NALU in the network abstraction layer unit NALU are sent to the network device.
[0200] Optionally, the transport block size TBS allocated for the video to be played includes:
[0201] The transport block size TBS allocated to each group of pictures GoP corresponding to the video to be played.
[0202] In one embodiment of the present disclosure, if the GoP TBS size is smaller than the total NALU size, a portion of the NALU is discarded to prioritize the base layer.
[0203] Optionally, the method further includes:
[0204] receiving a first network abstraction layer unit sent by a network device indicating that the unit fails to be sent to the target terminal;
[0205] Determine and discard a second network abstraction layer unit based on the first network abstraction layer unit;
[0206] The second network abstraction layer unit is a network abstraction layer unit that is dependent on the first network abstraction layer unit and is not yet sent to the network device.
[0207] Optionally, the determining whether to discard a network abstraction layer unit NALU according to a transport block size TBS allocated by the network device to each picture group includes:
[0208] Determine the association relationship between each of the network abstraction layer units NALU;
[0209] And according to the association relationship and the transport block size TBS allocated by the network device to each picture group, it is determined whether the network abstraction layer unit needs to be discarded.
[0210] Optionally, determining the association relationship between each of the network abstraction layer units NALU can be understood as follows: since one Slice corresponds to only one NALU, the association relationship between the NALUs is equivalent to the relationship between the corresponding Slices. The Slice association information can be shown in Table 1:
[0211] Table 1
[0212] In one embodiment of the present disclosure, since the Slice and its related information are also contained in the NALU itself, the loader can parse the NALU and obtain the Slice association information. If the encoder supports it, the Slice association information can also be explicitly passed to the loader to facilitate it to determine whether it needs to be discarded. After obtaining the Slice association information, the NALU loader loads it according to the TBS of the corresponding GoP indicated by the BBU. If the indicated GoP TBS is larger than the size of all NALUs of the corresponding GoP, the NALU is not discarded.
[0213] Optionally, the method for determining the target network abstraction layer unit NALU includes:
[0214] Determine the target network abstraction layer unit according to the second parameter of the network abstraction layer unit NALU;
[0215] The second parameter includes at least one of the following: a level of a corresponding enhancement layer, a degree of dependence on other network abstraction layer units, a corresponding region of interest, and a number of historical discarded times.
[0216] Optionally, the target network abstraction layer unit selection strategy is:
[0217] The enhancement layer with a higher level is discarded first;
[0218] Slices that are more dependent on other Slices (NALU) are discarded first;
[0219] Non-regions of interest (ROI) are discarded first. In the absence of the ROI, the central area and the three-division point are considered as the ROI;
[0220] Previously, the corresponding position of the Slice (NALU) that has been discarded is discarded first.
[0221] The video data processing method of the embodiment of the present disclosure determines the target network abstraction layer unit to be discarded in the network abstraction layer unit according to the second parameter of the network abstraction layer unit, thereby adapting to the change of bandwidth and the discarding strategy.
[0222] The logic diagram of the video data processing method according to an embodiment of the present disclosure is shown in FIG9 :
[0223] The network device (RAN) determines a target transmission rate based on the acquired target terminal status data and the load data of the network device; and sends the target transmission rate to the encoder (Codec);
[0224] The encoder determines, according to the target transmission rate, a first encoding parameter of a base layer corresponding to the video to be played, or the first encoding parameter and the number of enhancement layers and a second encoding parameter;
[0225] A loader (NALU Loader) receives a transport block size allocated to a picture group sent by the network device; receives at least one network abstraction layer unit sent by an encoder, and loads the network abstraction layer unit according to the transport block size and the at least one network abstraction layer unit; and when the size of the transport block is smaller than the total size of the network abstraction layer units, selects a target network abstraction layer unit to be discarded according to a network abstraction layer unit selection strategy, sends the network abstraction layer unit that has discarded the target network abstraction layer unit to the network device; and forwards the network abstraction layer unit to the target terminal through the network device.
[0226] In the case that the target terminal cannot correctly receive all of the network abstraction layer units, the network device sends information (Slice identifier) of the first network abstraction layer unit that was not correctly received to the loader. The loader discards the first network abstraction layer unit that was not correctly received and sends the network abstraction layer unit excluding the target network abstraction layer unit and the first network abstraction layer unit to the network device again.
[0227] The process of the video data processing method according to an embodiment of the present disclosure is shown in FIG10 :
[0228] The encoder Codes encodes according to the target transmission rate, determines the base layer and enhancement layer, as well as the resolution of the base layer (e.g., 720p), the number and resolution of the enhancement layers (e.g., two enhancement layers, with resolutions of 1080p and 2160p respectively), encapsulates the base layer or the base layer and enhancement layer through the network abstraction layer, and sends the encapsulated data to the loader.
[0229] The loader receives the transport block size of the picture group sent by the BBU of the network device, and receives the network abstraction layer unit sent by the encoder, and loads the network abstraction layer unit; wherein, when the channel quality (determined according to the target transmission rate) is good, all the data (all the network abstraction layer units) are loaded; when the channel quality is average, part of the enhancement layer (the network abstraction layer unit corresponding to the enhancement layer) is selectively discarded; when the channel quality is poor, only the data of the base layer (the network abstraction layer unit of the base layer) is loaded.
[0230] As shown in FIG11 , a video data processing apparatus 1100 according to an embodiment of the present disclosure, applied to an encoder, includes:
[0231] The first receiving module 1101 is configured to receive a target transmission rate sent by a network device;
[0232] A first determining module 1102 is configured to determine a first parameter of a video to be played according to the target transmission rate, where the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of enhancement layers;
[0233] The encoding module 1103 is configured to encode the video to be played according to the first parameter to obtain an output video stream including at least one network abstraction layer unit NALU;
[0234] The first sending module 1104 is configured to send the network abstraction layer unit to a loader.
[0235] As shown in FIG12 , a video data processing apparatus 1200 according to an embodiment of the present disclosure is applied to a network device and includes:
[0236] An acquisition module 1201 is used to acquire status data of a target terminal;
[0237] A second determining module 1202 is configured to determine a target transmission rate based on the state data and the load data of the network device;
[0238] A second sending module 1203 is configured to send the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played;
[0239] The third sending module 1204 is configured to send a transport block size TBS allocated for the video to be played to the loader;
[0240] The second receiving module 1205 is configured to receive the network abstraction layer unit NALU sent by the loader.
[0241] As shown in FIG13 , a video data processing device 1300 according to an embodiment of the present disclosure, applied to a loader, includes:
[0242] The third receiving module 1301 is configured to receive a transport block size TBS allocated for a video to be played, sent by a network device;
[0243] The fourth receiving module 1302 is configured to receive at least one network abstraction layer unit NALU sent by the encoder;
[0244] The fourth sending module 1303 is configured to send the network abstraction layer unit NALU to the network device according to the transport block size TBS.
[0245] As shown in FIG14 , an encoder 1400 according to an embodiment of the present disclosure includes a processor 1410 and a transceiver 1420 , wherein:
[0246] The transceiver is used to receive the target transmission rate sent by the network device;
[0247] The processor is configured to determine a first parameter of a to-be-played video according to the target transmission rate, where the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of an enhancement layer;
[0248] Encode the video to be played according to the first parameter to obtain an output video stream including at least one network abstraction layer unit NALU;
[0249] The transceiver is further configured to send the network abstraction layer unit to a loader.
[0250] As shown in FIG15 , a network device 1500 according to an embodiment of the present disclosure includes a processor 1510 and a transceiver 1520 , wherein:
[0251] The transceiver is used to obtain status data of the target terminal;
[0252] The processor is configured to determine a target transmission rate based on the status data and load data of the network device;
[0253] The transceiver is further configured to send the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played;
[0254] Sending a transport block size TBS allocated for the video to be played to the loader;
[0255] Receives the Network Abstraction Layer Unit (NALU) sent by the loader.
[0256] As shown in FIG16 , a loader 1600 according to an embodiment of the present disclosure includes a transceiver 1620 , wherein:
[0257] The transceiver is used to receive a transmission block size TBS allocated for a video to be played, which is sent by a network device;
[0258] Receiving at least one network abstraction layer unit NALU sent by the encoder;
[0259] The network abstraction layer unit NALU is sent to the network device according to the transport block size TBS.
[0260] A terminal device according to another embodiment of the present disclosure, as shown in FIG17 , includes a transceiver 1710, a processor 1700, a memory 1720, and a program or instruction stored in the memory 1720 and executable on the processor 1700; when the processor 1700 executes the program or instruction, the above-mentioned method for processing video data is implemented.
[0261] The transceiver 1710 is configured to receive and send data under the control of the processor 1700 .
[0262] In FIG17 , the bus architecture may include any number of interconnected buses and bridges, specifically various circuits of one or more processors represented by processor 1700 and memory represented by memory 1720, linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1710 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 1730 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.
[0263] The processor 1700 is responsible for managing the bus architecture and general processing, and the memory 1720 can store data used by the processor 1700 when performing operations.
[0264] A readable storage medium of an embodiment of the present disclosure stores a program or instruction thereon. When the program or instruction is executed by a processor, the steps in the video data processing method described above are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0265] The processor is the processor in the terminal device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.
[0266] It should be further noted that the terminals described in this specification include but are not limited to smartphones, tablet computers, etc., and many functional components described are referred to as modules in order to more particularly emphasize the independence of their implementation methods.
[0267] In the disclosed embodiments, modules can be implemented in software so that they can be executed by various types of processors. For example, an identified executable code module can include one or more physical or logical blocks of computer instructions, for example, which can be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different locations, which, when logically combined together, constitute the module and achieve the specified purpose of the module.
[0268] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed in the middle of different programs, and distributed across a plurality of memory devices.Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form.Described operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.
[0269] When a module can be implemented using software, taking into account the current state of hardware technology, those skilled in the art can build corresponding hardware circuits to implement the corresponding functions of the module that can be implemented using software, without considering the cost. The hardware circuits include conventional very large scale integration (VLSI) circuits or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. The module can also be implemented using programmable hardware devices, such as field programmable gate arrays, programmable array logic, programmable logic devices, etc.
[0270] The above exemplary embodiments are described with reference to the accompanying drawings. Many different forms and embodiments are possible without departing from the spirit and teachings of this disclosure. Therefore, this disclosure should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be complete and thorough and will convey the scope of this disclosure to those skilled in the art. In the drawings, component sizes and relative sizes may be exaggerated for clarity. The terminology used herein is for purposes of describing specific exemplary embodiments only and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to encompass such plural forms. It will be further understood that the terms "comprising" and / or "including," when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, elements, and / or groups thereof. Unless otherwise indicated, when stated, a range of values includes the upper and lower limits of that range and any subranges therebetween.
[0271] The above is a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the present disclosure.
Claims
1. A video data processing method, applied to an encoder, the method comprising: Receiving a target transmission rate sent by a network device, wherein the target transmission rate is a transmission rate between the network device and a target terminal; Determine a first parameter of the video to be played according to the target transmission rate, the first parameter comprising: a first encoding parameter of a base layer, or the first parameter comprising: a first encoding parameter of the base layer and a first number of enhancement layers and a second encoding parameter of each enhancement layer; According to the first parameter, the video to be played is encoded to obtain an output video code stream including at least one network abstraction layer unit NALU; The network abstraction layer unit is sent to the loader.
2. The video data processing method according to claim 1, wherein: The method for determining the first coding parameter of the base layer includes: Determine a preset ratio of the target transmission rate as a first transmission rate of the base layer; The first encoding parameter is determined according to the first transmission rate.
3. The video data processing method according to claim 2, wherein: The method for determining the first number and the second coding parameter of the enhancement layer includes: Determining a first number of enhancement layers according to the target transmission rate and the first transmission rate; The second encoding parameter is determined according to the target transmission rate, the first transmission rate, and the first number.
4. The video data processing method according to claim 1 or 2, wherein: The first encoding parameter and the second encoding parameter include: Resolution, frame rate, and encoding quality; The highest level of the enhancement layer corresponds to the highest resolution of the video to be played.
5. The video data processing method according to claim 1, wherein: The network abstraction layer comprises at least one group of pictures GoP.
6. A video data processing method, applied to a network device, the method comprising: Obtaining status data of the target terminal; Determining a target transmission rate between the network device and the target terminal according to the state data and the load data of the network device; Sending the target transmission rate to an encoder, wherein the target transmission rate is used by the encoder to determine a first parameter of a video to be played; Sending to the loader a transport block size TBS allocated for the video to be played; Receives the Network Abstraction Layer Unit (NALU) sent by the loader.
7. The video data processing method according to claim 6, wherein: The determining the target transmission rate according to the state data and the load data of the network device includes: Determining an average transmission rate from the network device to the target terminal according to the state data and the load data of the network device; The target transmission rate is determined according to the average transmission rate.
8. The video data processing method according to claim 6, wherein: The sending to the loader a transport block size TBS allocated for the video to be played includes: Acquire a channel condition of the target terminal, the channel condition including: channel state information and a channel quality indicator; According to the channel condition of the target terminal and the channel resource utilization rate of the network device, a transmission block size TBS allocated to each group of pictures GoP corresponding to the video to be played is determined.
9. The video data processing method according to claim 6, wherein: The status data includes: The transmission bandwidth, channel quality parameters and the highest modulation mode supported by the target terminal; The load data includes: The number of access terminals, the number of activated terminals and the air interface resource utilization rate of the network device.
10. A video data processing method, applied to a loader, the method comprising: Receive a transmission block size TBS allocated for the video to be played sent by a network device; Receiving at least one network abstraction layer unit NALU sent by the encoder; The network abstraction layer unit NALU is sent to the network device according to the transport block size TBS.
11. The video data processing method according to claim 10, wherein: The sending the network abstraction layer unit NALU to the network device according to the transmission block size TBS includes: Determine whether to discard a network abstraction layer unit NALU according to a transmission block size TBS allocated by the network device for the video to be played; If it is determined that the network abstraction layer unit NALU needs to be discarded, discard at least one target network abstraction layer unit NALU in the network abstraction layer unit; The remaining network abstraction layer unit NALUs after discarding the target network abstraction layer unit NALU in the network abstraction layer unit NALU are sent to the network device.
12. The video data processing method according to claim 10 or 11, wherein: The transmission block size TBS allocated for the video to be played includes: The transport block size TBS allocated to each picture group corresponding to the video to be played.
13. The video data processing method according to claim 11, wherein: The determining whether to discard a network abstraction layer unit NALU according to a transport block size TBS allocated by the network device to each picture group includes: Determine the association relationship between each of the network abstraction layer units NALU; And according to the association relationship and the transmission block size TBS allocated by the network device to each picture group, it is determined whether the network abstraction layer unit needs to be discarded.
14. The video data processing method according to claim 11, wherein: The method for determining the target network abstraction layer unit NALU includes: Determine the target network abstraction layer unit according to the second parameter of the network abstraction layer unit NALU; The second parameter includes at least one of the following: a level of a corresponding enhancement layer, a degree of dependence on other network abstraction layer units, a corresponding region of interest, and a number of historical discards.
15. A video data processing device, applied to an encoder, the device comprising: A first receiving module, used for receiving a target transmission rate sent by a network device; A first determination module is used to determine a first parameter of the video to be played according to the target transmission rate, wherein the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of an enhancement layer; An encoding module, configured to encode the video to be played according to the first parameter to obtain an output video code stream including at least one network abstraction layer unit NALU; The first sending module is used to send the network abstraction layer unit to the loader.
16. A video data processing device, applied to a network device, comprising: An acquisition module, used to acquire the status data of the target terminal; A second determination module, configured to determine a target transmission rate according to the state data and the load data of the network device; A second sending module, used to send the target transmission rate to an encoder, where the target transmission rate is used by the encoder to determine a first parameter of a video to be played; A third sending module, used for sending a transport block size TBS allocated for the video to be played to the loader; The second receiving module is used to receive the network abstraction layer unit NALU sent by the loader.
17. A video data processing device, applied to a loader, the device comprising: A third receiving module is used to receive a transmission block size TBS allocated for the video to be played and sent by a network device; A fourth receiving module, configured to receive at least one network abstraction layer unit NALU sent by the encoder; The fourth sending module is used to send the network abstraction layer unit NALU to the network device according to the transmission block size TBS.
18. An encoder comprising: transceivers and processors; The transceiver is used to receive the target transmission rate sent by the network device; The processor is used to determine a first parameter of the video to be played according to the target transmission rate, where the first parameter includes: a first encoding parameter of a base layer, or the first parameter includes: a first encoding parameter of the base layer and a first number and a second encoding parameter of an enhancement layer; According to the first parameter, the video to be played is encoded to obtain an output video code stream including at least one network abstraction layer unit NALU; The transceiver is further configured to send the network abstraction layer unit to a loader.
19. A network device comprising: transceivers and processors; The transceiver is used to obtain status data of the target terminal; The processor is used to determine a target transmission rate according to the state data and the load data of the network device; The transceiver is also used to send the target transmission rate to the encoder, and the target transmission rate is used by the encoder to determine the first parameter of the video to be played; Sending to the loader a transport block size TBS allocated for the video to be played; Receives the Network Abstraction Layer Unit (NALU) sent by the loader.
20. A loader, comprising: Transceiver; The transceiver is used to receive a transmission block size TBS allocated for a video to be played and sent by a network device; Receiving at least one network abstraction layer unit NALU sent by the encoder; The network abstraction layer unit NALU is sent to the network device according to the transport block size TBS.
21. A terminal device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; wherein, when the processor executes the program or instruction, the video data processing method as described in any one of claims 1 to 5 is implemented; or, the video data processing method as described in any one of claims 6 to 9 is implemented; or, the video data processing method as described in any one of claims 10 to 14 is implemented.
22. A readable storage medium having a program or instruction stored thereon, wherein: When the program or instruction is executed by the processor, it implements the steps in the video data processing method as described in any one of claims 1 to 5; or, implements the steps in the video data processing method as described in any one of claims 6 to 9; or, implements the steps in the video data processing method as described in any one of claims 10 to 14.
23. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the video data processing method as described in any one of claims 1 to 5; or, implement the steps in the video data processing method as described in any one of claims 6 to 9; or, implement the steps in the video data processing method as described in any one of claims 10 to 14.
Citation Information
Patent Citations
Scaled code stream processing method, scaled code stream processing device and scaled code stream encoder
CN103716633A
Video transmission method and device
CN113395522A
Data layered transmission method, device and system
CN115150367A
Bit-rate control for video coding using object-of-interest data
US20140355671A1
Scalable video encoding rate adaptation based on perceived quality
US20150281709A1
Cited By
Generative video transmission method and device
CN122069256A
