Video information processing method, system, device and storage medium

By performing high sampling ratio downsampling and pre-analysis on the original video frames, the quantization parameters of the video encoder are optimized, which solves the problem of excessively long pre-analysis of the video encoder and improves the video encoding speed and live broadcast fluency.

CN117478881BActive Publication Date: 2024-11-29TAOBAO CHINA SOFTWARE
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202311467208.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-06
Publication Date
2024-11-29
Estimated Expiration
2043-11-06

AI Technical Summary

Technical Problem

The existing video encoders take a long time to pre-analyze the process in real-time processing, becoming a bottleneck in bandwidth cost and playback fluency in high-definition live broadcasts.

Method used

By downsampling the original video frame with a higher sampling ratio, the number of pixels in the downsampled image is reduced, and pre-analyzed based on the downsampled image is performed, encoding complexity and quantization parameter offset are determined, thereby optimizing the quantization parameters of the video frame. , to improve video encoding speed.

Benefits of technology

It reduces the complexity and time-consuming of the video frame pre-analysis process, improves the video encoding speed, and improves the fluency and user experience of live videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117478881B_ABST
    Figure CN117478881B_ABST
Patent Text Reader

Abstract

The present application provides a video information processing method, system, device and storage medium, the method comprising: obtaining an original video frame to be processed, wherein the original video frame is divided into a plurality of processing units; downsampling the original video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling; determining the encoding complexity of the original video frame and the quantization parameter offset corresponding to each of the plurality of processing units according to the downsampled image; determining the frame-level quantization parameter of the original video frame according to the encoding complexity and the current bandwidth resources; determining the target quantization parameter of the corresponding encoding block according to the quantization parameter offset and the frame-level quantization parameter. The present application reduces the complexity of the video frame pre-analysis process, improves the speed of the pre-analysis process, and thereby improves the video encoding speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a video information processing method, system, device and storage medium. Background Art

[0002] Video coding refers to the method of converting the original video format file into another video format file through compression technology. Video coding technology mainly achieves video compression by eliminating redundancy in video signals.

[0003] With the development of Internet technology, high-definition live broadcasting has become more and more popular, but the problem is higher bandwidth costs and worse playback fluency. In order to solve these challenges, more efficient and advanced video encoding standards have gradually been applied to live broadcasting services, but the computational complexity has also doubled. In order to meet real-time processing requirements, the complexity of the video encoder needs to be optimized.

[0004] Commonly used video encoders often include a pre-analysis module, which quantitatively analyzes the spatiotemporal dependencies between video blocks before encoding to optimally decide subsequent encoding parameters, and can further improve compression efficiency based on the encoding standard. However, the time consumption of the existing pre-analysis process accounts for a high proportion of the total encoding time, which has become a bottleneck for the development of real-time encoding. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a video information processing method, system, device and storage medium, which realizes the prediction and calculation of pre-analysis after downsampling the original video frame with a higher sampling ratio, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and further improving the video encoding speed.

[0006] In a first aspect, an embodiment of the present application provides a video information processing method, comprising: obtaining an original video frame to be processed, wherein the original video frame is divided into multiple processing units; downsampling the original video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling; determining, based on the downsampled image, the coding complexity of the original video frame and the quantization parameter offset corresponding to each processing unit in the multiple processing units; determining, based on the coding complexity and current bandwidth resources, a frame-level quantization parameter of the original video frame; and determining, based on the quantization parameter offset and the frame-level quantization parameter, a target quantization parameter of the corresponding coding block.

[0007] In one embodiment, based on the downsampled image, the quantization parameter offset corresponding to each of the multiple processing units of the original video frame is determined, including: performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units; based on the motion vectors, respectively calculating the propagation costs corresponding to the multiple processing units, and determining the quantization parameter offsets corresponding to the multiple processing units based on the propagation costs.

[0008] In one embodiment, determining the coding complexity of the original video frame based on the downsampled image includes: performing inter-frame prediction processing on the multiple processing units based on the downsampled image to obtain inter-frame coding costs of the multiple processing units; performing intra-frame prediction processing on the multiple processing units based on the downsampled image to obtain intra-frame coding costs of the multiple processing units; determining target coding costs applicable to the multiple processing units based on the inter-frame coding costs and the intra-frame coding costs, and the coding complexity includes the target coding costs of the multiple processing units.

[0009] In one embodiment, determining the target coding costs applicable to the multiple processing units based on the inter-frame coding cost and the intra-frame coding cost includes: for the current processing unit, determining the minimum value of the inter-frame coding cost and the intra-frame coding cost as the target coding cost of the current processing unit.

[0010] In one embodiment, determining the frame-level quantization parameter of the original video frame based on the coding complexity and the current bandwidth resources includes: determining a transmission bit rate that matches the original video frame based on the target coding cost of the multiple processing units and the current bandwidth resources; and determining the frame-level quantization parameter of the original video frame based on the transmission bit rate.

[0011] In one embodiment, determining the target quantization parameter of the corresponding coding block based on the quantization parameter offset and the frame-level quantization parameter includes: adjusting the quantization parameters of the corresponding coding blocks based on the frame-level quantization parameters according to the quantization parameter offsets corresponding to the processing units, and determining the target quantization parameter of the corresponding coding block.

[0012] In one embodiment, the preset sampling ratio is 4:1; downsampling the original video frame according to the preset sampling ratio to obtain the downsampled image includes: downsampling the original video frame by 4:1 in the width direction and the height direction respectively to obtain the downsampled image.

[0013] In one embodiment, the processing unit has a width of 32 pixels and a height of 32 pixels.

[0014] In one embodiment, after determining the target quantization parameter of the corresponding coding block, the method further includes: encoding the original video frame according to the target quantization parameters corresponding to multiple coding blocks in the original video frame to obtain an encoded video frame.

[0015] In a second aspect, an embodiment of the present application provides a video information processing method, the method comprising: receiving a live video frame to be processed in response to a live broadcast request; dividing the live video frame into multiple processing units; downsampling the live video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling; determining, based on the downsampled image, the encoding complexity of the live video frame and the quantization parameter offset corresponding to each processing unit in the multiple processing units; determining, based on the encoding complexity and current bandwidth resources, the frame-level quantization parameter of the live video frame; determining, based on the quantization parameter offset and the frame-level quantization parameter, a target quantization parameter of the corresponding coding block; encoding the live video frame according to the target quantization parameter corresponding to multiple coding blocks in the live video frame to obtain an encoded video frame; and transmitting the encoded video frame to a receiving terminal.

[0016] In a third aspect, an embodiment of the present application provides a video information processing device, including:

[0017] An acquisition module, used for acquiring an original video frame to be processed, wherein the original video frame is divided into a plurality of processing units;

[0018] A sampling module, configured to downsample the original video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than a sampling ratio corresponding to half sampling;

[0019] A first determination module, configured to determine, according to the downsampled image, the coding complexity of the original video frame and a quantization parameter offset corresponding to each processing unit in the plurality of processing units;

[0020] A second determination module, configured to determine a frame-level quantization parameter of the original video frame according to the coding complexity and current bandwidth resources;

[0021] The third determination module is used to determine a target quantization parameter of a corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter.

[0022] In one embodiment, the first determination module is used to perform inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units; based on the motion vectors, the propagation costs corresponding to the multiple processing units are respectively calculated, and the quantization parameter offsets corresponding to the multiple processing units are determined based on the propagation costs.

[0023] In one embodiment, the first determination module is used to perform inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain inter-frame coding costs of the multiple processing units; perform intra-frame prediction processing on the multiple processing units according to the downsampled image to obtain intra-frame coding costs of the multiple processing units; determine target coding costs applicable to the multiple processing units based on the inter-frame coding costs and the intra-frame coding costs, and the coding complexity includes the target coding costs of the multiple processing units.

[0024] In one embodiment, the first determination module is specifically configured to determine, for a current processing unit, a minimum value between the inter-frame coding cost and the intra-frame coding cost as a target coding cost of the current processing unit.

[0025] In one embodiment, the second determination module is used to determine a transmission bit rate that matches the original video frame according to the target encoding costs of the multiple processing units and the current bandwidth resources; and determine a frame-level quantization parameter of the original video frame according to the transmission bit rate.

[0026] In one embodiment, the third determination module is used to adjust the quantization parameters of the corresponding coding blocks based on the frame-level quantization parameters according to the quantization parameter offsets corresponding to the processing units, and determine the target quantization parameters of the corresponding coding blocks.

[0027] In one embodiment, the preset sampling ratio is 4:1; the sampling module is used to perform 4:1 downsampling on the original video frame in the width direction and the height direction respectively to obtain the downsampled image.

[0028] In one embodiment, the processing unit has a width of 32 pixels and a height of 32 pixels.

[0029] In one embodiment, it also includes: an encoding module, which is used to encode the original video frame according to the target quantization parameters corresponding to multiple encoding blocks in the original video frame after determining the target quantization parameter of the corresponding encoding block, to obtain an encoded video frame.

[0030] In a fourth aspect, an embodiment of the present application provides a video information processing system, comprising: a pre-analysis module, a bit rate control module and an encoder; wherein: the pre-analysis module is used to obtain an original video frame to be processed, and the original video frame is divided into multiple processing units; the original video frame is downsampled according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling; according to the downsampled image, the coding complexity of the original video frame and the quantization parameter offset corresponding to each processing unit in the multiple processing units are determined; according to the coding complexity and the current bandwidth resources, the frame-level quantization parameter of the original video frame is determined; the bit rate control module is connected to the pre-analysis module, and is used to determine the target quantization parameter of the corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter; the encoder is respectively connected to the pre-analysis module and the bit rate control module, and is used to encode the original video frame according to the target quantization parameters corresponding to multiple coding blocks in the original video frame to obtain an encoded video frame.

[0031] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0032] at least one processor; and

[0033] a memory communicatively coupled to the at least one processor;

[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0035] In a sixth aspect, an embodiment of the present application provides a cloud device, including:

[0036] at least one processor; and

[0037] a memory communicatively coupled to the at least one processor;

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of the above aspects.

[0039] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the above aspects is implemented.

[0040] In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0041] The video information processing method, system, device and storage medium provided by the embodiments of the present application, for the original video frame to be processed, firstly downsamples according to a sampling ratio greater than 2:1 to reduce the number of pixels contained in the downsampled image, then performs pre-analysis based on the downsampled image, and outputs the coding complexity of the original video frame and the quantization parameter offset of each processing unit in the pre-analysis, further determines the frame-level quantization parameter of the original video frame according to the coding complexity and bandwidth resources, and then combines the quantization parameter offset of each processing unit and the frame-level quantization parameter of the original video frame to determine the target quantization parameter of the corresponding coding block in the encoding stage. In this way, downsampling with a higher sampling ratio is adopted and then the prediction and calculation of the pre-analysis are performed to reduce the number of pixels contained in the downsampled image, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0043] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of an application scenario of a video information processing solution provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of an application scenario of live video information processing provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a video information processing system provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the principle of a video information processing system provided in an embodiment of the present application;

[0048] Figure 6 A flowchart of a video information processing method provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a pre-analysis process provided in an embodiment of the present application;

[0050] Figure 8 A flowchart of a video information processing method provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of a video information processing device provided in an embodiment of the present application;

[0052] Fig.10 A schematic diagram of the structure of a cloud device provided in an embodiment of the present application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0055] The term "and / or" in this article is used to describe the association relationship of associated objects, specifically indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0057] In order to more clearly describe the technical content of the present application, the terms involved in the embodiments of the present application are first interpreted as follows:

[0058] Video frame: a basic unit constituting a video. A video is a frame sequence consisting of multiple video frames. A video frame is also called an image frame. When a video frame is encoded, the encoded video frame is called an encoded frame.

[0059] Video frames can include I-frames, B-frames, and P-frames. I-frames are intra-frame coded frames, which are independent frames that carry all information and can be decoded independently without referring to other image frames. P-frames are inter-frame coded frames, which are also called inter-frame prediction frames or forward prediction frames. They can use intra-frame prediction coding and inter-frame prediction coding, and can use forward reference prediction coding. B-frames are also inter-frame coded frames. B uses intra-frame prediction coding and inter-frame prediction coding, and can use forward, backward, and bidirectional reference prediction coding.

[0060] H.265: High Efficiency Video Coding, high-efficiency video coding.

[0061] CU: Coding Unit, a block unit used for encoding, also known as a coding block, with variable size. Large CU blocks can be recursively divided into smaller CU blocks from top to bottom according to the quadtree. For example, the CU blocks of H.265 can be 64x64, 32x32, 16x16, and 8x8. Taking the 32×32 CU block as an example, it means that the corresponding CU block is 32 pixels wide and 32 pixels high, that is, 32×32 pixels. For simplicity, 32×32 pixels are abbreviated as 32×32 in the embodiment of the present application, and the description of other sizes is similar.

[0062] SATD: Sum of Absolute Transformed Difference, the absolute value sum of the coefficients after Hadamard transform of the prediction residual, can be used to estimate coding distortion or measure coding complexity.

[0063] MV: motion vector, motion vector.

[0064] QP: Quantization Parameter, quantization parameter.

[0065] Cu-tree: Adjust the QP value according to the degree to which the current block is referenced by subsequent frames. If the current block is referenced more by subsequent frames, its importance is higher, and the QP should be reduced to improve the encoding quality of the block. Otherwise, the QP of the block should be increased.

[0066] Intra: intra-frame prediction.

[0067] Inter: Inter-frame prediction.

[0068] YUV is a color encoding method. "Y" represents brightness, that is, the grayscale value, and "U" and "V" represent chromaticity.

[0069] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1A processor is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11, and the instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment to achieve the prediction and calculation of pre-analysis after downsampling the original video frame with a higher sampling ratio, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed.

[0070] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0071] Figure 2 Schematic diagram of an application scenario system 200 of a video information processing solution provided in an embodiment of the present application. Figure 2 As shown, the system includes: a server 210 and a terminal 220, wherein:

[0072] The server 210 may be a data platform that provides video information processing services, such as an e-commerce live broadcast platform. In actual scenarios, an e-commerce live broadcast platform may have multiple servers 210. Figure 2 In the figure, one server 210 is taken as an example.

[0073] The terminal 220 may be a computer, mobile phone, tablet or other device used by the user to log in to the e-commerce live broadcast platform. There may also be multiple terminals 220. Figure 2 Two terminals 220 are taken as an example for illustration.

[0074] The terminal 220 and the server 210 can transmit information via the Internet, so that the terminal 220 can access the data on the server 210. The terminal 220 and / or the server 210 can be implemented by the electronic device 1.

[0075] The video information processing solution of the embodiment of the present application can be deployed on the server 210, can also be deployed on the terminal 220, or can be partially deployed on the server 210 and partially deployed on the terminal 220. In actual scenarios, it can be selected based on actual needs, and this embodiment does not limit it.

[0076] When the video information processing solution is fully or partially deployed on the server 210 , a calling interface may be opened to the terminal 220 to provide algorithm support to the terminal 220 .

[0077] The method provided in the embodiment of the present application can be implemented by executing the corresponding software code by the electronic device 1, and can be implemented by exchanging data with the server. The electronic device 1 can be a local terminal device. When the method is run on the server, the method can be implemented and executed based on the cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0078] In a possible implementation, the method provided by the embodiment of the present invention provides a graphical user interface through a terminal device, wherein the terminal device can be the local terminal device mentioned above, or can be the client device in the cloud interaction system mentioned above.

[0079] The video information processing method of the embodiment of the present application can be applied to any scenario requiring human-computer interaction.

[0080] Video coding refers to the method of converting the original video format file into another video format file through compression technology. Video coding technology mainly achieves video compression by eliminating redundancy in video signals.

[0081] With the development of Internet technology, high-definition live broadcasting has become more and more popular, but the problem is higher bandwidth costs and worse playback fluency. In order to solve these challenges, more efficient and advanced video encoding standards have gradually been applied to live broadcasting services, but the computational complexity has also doubled. In order to meet real-time processing requirements, the complexity of the video encoder needs to be optimized.

[0082] Commonly used video encoders often include a pre-analysis module, which quantitatively analyzes the spatiotemporal dependencies between video blocks before encoding to optimally decide subsequent encoding parameters, and can further improve compression efficiency based on the encoding standard. However, the time consumption of the existing pre-analysis process accounts for a high proportion of the total encoding time, which has become a bottleneck for the development of real-time encoding.

[0083] like Figure 3 As shown, it is a schematic diagram of an application scenario of live video information processing provided by an embodiment of the present application. Taking an e-commerce scenario as an example, in an e-commerce live broadcast platform, a live broadcast user can shoot relevant scenes through a live broadcast terminal 221 to obtain a live video. Assuming that the video information processing method provided by an embodiment of the present application is deployed on the terminal, the live video can be encoded by the video information processing method provided by an embodiment of the present application to obtain a live video stream, and then the live video stream is sent to a server 210 of the e-commerce live broadcast platform. The live broadcast server 210 can forward the live video stream to a viewing terminal 222. The viewing terminal 222 can be one or more. After receiving the live video stream forwarded by the live broadcast server 210, the viewing terminal 222 can use a corresponding decoding method to decode the live video stream to obtain the live video, thereby facilitating users to watch live broadcasts.

[0084] In this process, the relevant video encoding pre-analysis scheme is: the pre-analysis is performed at 1 / 2 resolution of the video frame (half-sampled image, that is, a downsampled image with a sampling ratio of 2:1), and 16x16 blocks (at the original resolution) are selected as the basic processing unit, and mapped to the downsampled image is 8x8 blocks as the basic processing unit.

[0085] The disadvantages of the above scheme are: the downsampling ratio is low, the size of the basic processing unit is too small, the total number of processing units of the video frame is too large, and the computational complexity of the pre-analysis is high.

[0086] In order to solve the above problems, the embodiment of the present application provides a video information processing solution. For the original video frame to be processed, firstly, downsampling is performed according to a sampling ratio greater than 2:1 to reduce the number of pixels contained in the downsampled image, and then pre-analysis is performed based on the downsampled image. The pre-analysis outputs the coding complexity of the original video frame and the quantization parameter offset of each processing unit, and further determines the frame-level quantization parameter of the original video frame according to the coding complexity and bandwidth resources. Then, the target quantization parameter of the corresponding coding block is determined by combining the quantization parameter offset of each processing unit and the frame-level quantization parameter of the original video frame. In this way, downsampling with a higher sampling ratio is adopted and then the prediction and calculation of pre-analysis are performed to reduce the number of pixels contained in the downsampled image, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and thus improving the video encoding speed.

[0087] like Figure 4 FIG. 4 is a schematic diagram of a video information processing system 400 provided in an embodiment of the present application, the system comprising: a pre-analysis module, a bit rate control module and an encoder.

[0088] The pre-analysis module is used to obtain an original video frame to be processed, where the original video frame is divided into multiple processing units. The original video frame is downsampled according to a preset sampling ratio to obtain a downsampled image, where the preset sampling ratio is greater than the sampling ratio corresponding to half sampling. According to the downsampled image, the coding complexity of the original video frame and the quantization parameter offset corresponding to each processing unit in the multiple processing units are determined. According to the coding complexity and the current bandwidth resources, the frame-level quantization parameter of the original video frame is determined.

[0089] The bit rate control module is connected to the pre-analysis module and is used to determine the target quantization parameter of the corresponding processing unit according to the quantization parameter offset and the frame-level quantization parameter.

[0090] The encoder is connected to the pre-analysis module and the bit rate control module respectively, and is used to encode the original video frame according to the target quantization parameters corresponding to the multiple processing units to obtain the encoded video frame.

[0091] In one embodiment, if Figure 5 As shown, assuming that the original video frame is divided into multiple processing units of size 32x32, and the preset sampling ratio is 4:1, when the original video frame to be processed (assuming it is a YUV frame) is input into the pre-analysis module, the original video frame is first downsampled by 4:1, and the pre-analysis module performs a pre-analysis process based on the 4:1 downsampling block of the 32x32 processing unit, such as intra / inter prediction, and decides the encoder parameters in a direct or indirect way. On the one hand, the pre-analysis module outputs the encoding complexity of the encoded frame (which can be represented by the Frame satdcost at the frame level) to the bit rate control module, and the bit rate control module calculates the Frame QP (frame-level QP) of the original video frame according to the network bandwidth and buffer (buffer) buffering status as the basic QP of the frame. On the other hand, the pre-analysis module outputs the frame type (Frame type) of the current original video frame, whether to insert an I frame, the QP offset of each processing unit, etc. to directly control the encoder. The FrameQP and the QP offset of each processing unit can be combined to determine the target QP of each processing unit. The encoder encodes each processing unit based on the target QP and outputs the encoded bit stream (Bit-stream). In this embodiment, the intra / inter prediction of the pre-analysis module is performed on a 4:1 downsampled image, and the parameters required for various encodings are calculated based on the original 32x32 processing unit (8x8 block after downsampling).

[0092] Compared with the half sampling method in the aforementioned related art, a larger downsampling ratio is adopted: the pre-analysis is performed on a 4:1 downsampled image, and the number of downsampled pixels is reduced by 75%.

[0093] Using a larger processing unit size: Using a 32x32 processing unit as the basic processing unit, the total number of processing units is reduced by 75% compared to a 16x16-based processing unit.

[0094] Compared with the 2:1 downsampling solution based on 16x16 processing units, the basic processing unit after downsampling remains unchanged at 8x8, but the total processing units are reduced by 75%, the processing complexity can be reduced by 75%, and the processing efficiency is increased by 4 times.

[0095] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0096] Please see Figure 6 , which is a video information processing method of an embodiment of the present application, the method can be Figure 1The electronic device 1 shown is used to perform and can be applied to Figure 2-Figure 5 In the application scenario of video information processing shown in , the original video frame is downsampled with a higher sampling ratio and then the pre-analysis prediction and calculation are performed, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed. This embodiment takes the terminal 220 as an execution end as an example, and the method includes the following steps:

[0097] Step 601: obtaining an original video frame to be processed, where the original video frame is divided into a plurality of processing units.

[0098] In this step, the original video frame may be a live video captured by the live broadcast terminal 221 during the live broadcast process, or may be a video captured by the call terminal during an online video call. The original video frame may be a video frame sequence including multiple video frames, and each video frame may be processed separately during video processing. The processing unit refers to the smallest basic processing unit of a video frame, which may be used in the pre-analysis stage or in the encoding stage. For example, the processing unit may be the same size as the coding block (CU block) in the encoding stage, or may be of a different size.

[0099] In one embodiment, step 601 may specifically include: in response to a video processing request, obtaining an original video frame to be processed.

[0100] In this embodiment, the acquisition of the original video frame can be actively triggered by the user, for example, it can be a real-time video stream captured by the live broadcast device after the user triggers a live broadcast request in a live broadcast scene, thereby improving the interactive experience.

[0101] On the other hand, the original video frame may also be an existing video stream actively uploaded by the user.

[0102] Step 602: down-sample the original video frame according to a preset sampling ratio to obtain a down-sampled image, wherein the preset sampling ratio is greater than a sampling ratio corresponding to half sampling.

[0103] In this step, the sampling ratio may refer to the pixel ratio between the original video frame and the downsampled image obtained after downsampling. For example, taking a 32×32 processing unit as a basic processing unit, the downsampled image mapped to a sampling ratio of 4:1 is an 8×8 block. In the embodiment of the present application, the preset sampling ratio is greater than the sampling ratio corresponding to half sampling, that is, the preset sampling ratio is greater than 2:1, for example, the preset sampling ratio may be 4:1 or 8:1. Since the related art adopts a half-sampling method, the number of pixels of the downsampled image is large, and the pre-analysis process is complex and time-consuming. Therefore, using a preset sampling ratio greater than 2:1 to downsample the original video frame can reduce the number of pixels in the downsampled image, thereby reducing the amount of data calculation for pre-analysis and improving the speed of pre-analysis when pre-analyzing based on the downsampled image.

[0104] In one embodiment, the preset sampling ratio is 4: 1. Step 602 may specifically include: performing 4: 1 downsampling on the original video frame in the width direction and the height direction respectively to obtain a downsampled image.

[0105] In this embodiment, the input original YUV frame can be downsampled by 4:1 in the horizontal and vertical directions, that is, the original video frame can be downsampled by 4:1 in the width and height directions, and the number of pixels in the obtained downsampled image is reduced to 1 / 16 of the original video frame, which is a significant reduction in the number of pixels. Inter prediction and intra prediction are both performed on the low-resolution YUV video frame of the downsampled image, which greatly reduces the amount of data calculation in the pre-analysis process and improves the pre-analysis speed.

[0106] In an optional embodiment, assuming that the original video frame is an original YUV frame, the pre-analysis downsampling may be performed on the brightness Y of the original YUV frame, because the brightness already contains most of the key information, and UV may not be processed to save computing resources.

[0107] In one embodiment, the size of the processing unit may be larger than 16x16. For example, when the preset sampling ratio is 4:1, the size of the processing unit may be: 32 pixels in width and 32 pixels in height.

[0108] In this embodiment, the pre-analysis can select a 32x32 processing unit (at the original resolution) as the basic processing unit, and mapped to the downsampling map, the intra or inter prediction is based on an 8x8 block. Thus, compared with the use of 16x16 blocks as the basic processing unit in the related art, the embodiment of the present application uses a larger block size, and the total number of processing units can be reduced by 75%, which greatly reduces the amount of data for video frame pre-analysis and improves the pre-analysis speed.

[0109] In one embodiment, the preset sampling ratio and the size of the processing unit may also be combined as follows:

[0110] The preset sampling ratio is 4:1, 64x64 basic processing unit, and 16x16 blocks after downsampling.

[0111] The preset sampling ratio is 8:1, the basic processing unit is 64x64, and after downsampling it is 8x8 block.

[0112] The preset sampling ratio is 8:1, 128x128 basic processing unit, and 16x16 block after downsampling.

[0113] In actual scenarios, the preset sampling ratio and the size information of the multiple processing units can be determined based on the preset complexity parameters of the original video frame, and a suitable solution can be selected according to actual needs.

[0114] Step 603: Determine the coding complexity of the original video frame and the quantization parameter offset corresponding to each processing unit in the plurality of processing units according to the downsampled image.

[0115] In this step, the pre-analysis module can predict the coding complexity of the original video frame based on the downsampled image. The coding complexity can characterize the coding consumption or coding cost of the original video frame. The coding complexity can be expressed by Framesatdcost (frame-level loss cost). The QP offset (i.e., quantization parameter offset) of each processing unit is used to characterize the offset of the corresponding processing unit relative to the basic QP of the original video frame, and is used to adjust the QP parameter of the corresponding processing unit. The QP parameter adjustment is mainly to adjust the details of the image, and ultimately achieve the effect of adjusting the picture quality. The QP value is inversely proportional to the bit rate. The smaller the QP value, the higher the picture quality. On the contrary, the larger the QP value, the lower the picture quality. In actual scenes, the image information of different processing units in a frame of image may be different, so the corresponding image complexity is also different. Some processing units may be referenced by subsequent video frames, so the image quality requirements are higher. The target QP of such processing units should be reduced, and some unreferenced processing units, if the image information contained is also relatively small, can increase the corresponding target QP.

[0116] Pre-analyze each processing unit of the original video frame according to the downsampled image to determine the QP offset (i.e., quantization parameter offset) of each processing unit, which is used to fine-tune the image quality of each processing unit.

[0117] In one embodiment, determining the quantization parameter offset corresponding to each of the multiple processing units of the original video frame in step 603 according to the downsampled image may specifically include: performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units. According to the motion vectors, respectively calculating the propagation costs corresponding to the multiple processing units, and determining the quantization parameter offsets corresponding to the multiple processing units according to the propagation costs.

[0118] In this embodiment, inter-frame prediction (inter prediction) is to use time correlation to explore the correlation between adjacent video frames, and predict the current video frame by using the video frames that have been reconstructed in time, that is, to predict the current video frame by using other video frames that have been reconstructed before the current video frame. The reference frame can be determined by using other processing units that have been encoded and reconstructed around the current processing unit, and the reference frame can be used to perform predictive encoding on the current processing unit through motion estimation (ME), so as to eliminate the temporal redundant information of the video.

[0119] like Figure 7As shown, it is a schematic diagram of a pre-analysis process provided by an embodiment of the present application. After the original YUV video frame is downsampled by 4:1, a low-resolution luminance frame is obtained. Assuming that the processing unit size of the original video frame is divided into 32x32, it is mapped to the downsampled image, and the basic processing unit is an 8x8 downsampled block (i.e., based on an 8x8 block size). After the low-resolution motion vector MV of the 8x8 downsampled block is obtained through inter-frame prediction, the propagation cost is calculated based on the MV, and the CU-tree is calculated based on the propagation cost, combined with the inter-frame cost and the intra-frame cost, to determine the CU-tree QP offset corresponding to multiple processing units. CU-tree adjusts the QP value of the current processing unit by estimating how much information the current processing unit will be referenced by future frames in inter-frame prediction. If the current processing unit contributes more information to subsequent frames, the encoding quality of the area should be improved and the QP should be reduced. On the contrary, the QP value of the area is increased.

[0120] In one embodiment, variance calculation can be performed based on the original 32x32 pixel brightness block of the original YUV video frame, and spatial domain adaptive quantization processing can be further performed based on the variance calculation result to obtain the basic QP offset of the original 32x32 pixel brightness block. For each processing unit, the corresponding CU-tree QP offset is added to the basic QP offset to obtain the final QP offset output of the corresponding processing unit, so as to improve the accuracy of the final QP offset of the processing unit.

[0121] That is to say, the calculation of CU-tree is based on 32x32 processing units relative to the image of the original video frame, and the QP offset is also based on 32x32 processing units as the basic processing unit. Therefore, compared with the 2:1 downsampling solution in the related art, the total number of processing units is reduced to 1 / 4 of the original, which greatly reduces the amount of data calculation in the pre-analysis process and improves the pre-analysis speed.

[0122] In one embodiment, determining the coding complexity of the original video frame according to the downsampled image in step 603 may specifically include: performing inter-frame prediction processing on multiple processing units according to the downsampled image to obtain inter-frame coding costs of the multiple processing units. Performing intra-frame prediction processing on multiple processing units according to the downsampled image to obtain intra-frame coding costs of the multiple processing units. Determining target coding costs applicable to the multiple processing units according to the inter-frame coding costs and the intra-frame coding costs, the coding complexity includes the target coding costs of the multiple processing units.

[0123] In this embodiment, if Figure 7As shown in the figure, through inter-frame prediction, not only the motion vector MV of the 8x8 down-sampled block can be obtained, but also the inter-frame coding cost of each processing unit can be obtained. The inter-frame coding cost can be represented by Inter SATD (inter-frame SATD). According to the result of inter-frame prediction, the inter-frame prediction satd cost calculation can be performed on each processing unit to obtain the inter-frame coding cost (i.e. Figure 7 inter-frame cost in ).

[0124] On the other hand, intra-frame prediction processing can be performed on multiple processing units based on the downsampled image to obtain the intra-frame coding costs of multiple processing units. Intra-frame prediction uses spatial correlation to explore the correlation within the video frame. The encoding process is performed on the video frame block by block, and the encoding end will also reconstruct each processing unit block by block. Intra-frame prediction can directly use the encoded reconstructed blocks adjacent to the current processing unit in the video frame for prediction. Since the video frame sequence images have a strong spatial correlation, for example, a desktop with the same texture is contained in an image frame. Among the multiple processing units into which the video frame is divided, the processing units related to the desktop may display the same or similar information. For these processing units with high spatial correlation, intra-frame prediction coding technology can be used for encoding.

[0125] like Figure 7 As shown in FIG. 1 , after the original YUV video frame is downsampled by 4:1, a low-resolution luminance frame is obtained. Assuming that the processing unit size of the original video frame is 32x32 and mapped to the downsampled image, the basic processing unit is an 8x8 downsampled block (i.e., based on an 8x8 block size). By performing intra-frame prediction on each 8x8 downsampled block, and then calculating the intra-frame prediction satd cost for each processing unit according to the intra-frame prediction result, the intra-frame prediction satd cost is used to represent the intra-frame coding cost (i.e., Figure 7 Intra-frame cost in ).

[0126] In this way, in order to accurately predict the coding complexity of the current video frame, it is necessary to select the most appropriate coding method for the current video frame. Through inter-frame prediction and intra-frame prediction, the target coding cost applicable to each processing unit can be evaluated in time and space respectively, and then the target coding cost of each processing unit is used to characterize the coding complexity of the current video frame to improve the accuracy of the result.

[0127] In one embodiment, in step 603, target coding costs applicable to multiple processing units are determined based on inter-frame coding costs and intra-frame coding costs, which may specifically include: for the current processing unit, the minimum value of the inter-frame coding cost and the intra-frame coding cost is determined as the target coding cost of the current processing unit.

[0128] In this embodiment, in order to accurately predict the coding complexity of the current video frame, it is necessary to select the most appropriate coding method for the current video frame. The coding cost of each processing unit can be evaluated in time and space through inter-frame prediction and intra-frame prediction. For the current processing unit, the minimum value of the corresponding inter-frame coding cost and intra-frame coding cost is determined as the target coding cost of the current processing unit. In this way, the coding cost of the current frame video is ensured to be the minimum, the coding complexity of the current video frame is reduced, and the subsequent coding speed is improved.

[0129] Step 604: Determine a frame-level quantization parameter of the original video frame according to the coding complexity and the current bandwidth resources.

[0130] In this step, the current bandwidth resources determine the available transmission bit rate resources for the original video frame. The coding complexity of the original video frame can be used to determine the actual demand for the transmission bit rate resources of the original video frame. In actual scenarios, the bit rate control module can be used to comprehensively consider the current bandwidth resources and the specific coding complexity of the original video frame to determine the frame-level quantization parameter Frame QP of the original video frame as the basic quantization parameter of the current original video frame, thereby making full use of the existing transmission resources.

[0131] In one embodiment, step 604 may specifically include: determining a transmission bit rate that matches the original video frame according to target encoding costs of the multiple processing units and current bandwidth resources, and determining a frame-level quantization parameter of the original video frame according to the transmission bit rate.

[0132] In this embodiment, the bit rate of the current frame can be determined based on the target encoding cost of each processing unit in the current video frame and the currently allowed bandwidth resources, and then the frame-level QP can be determined. Figure 7 As shown in , the whole frame satd cost calculation can be performed based on the inter-frame prediction satd cost calculation result of each processing unit of the current video frame and the intra-frame prediction satd cost calculation result of the corresponding processing unit to obtain the Frame satdcost (frame-level coding cost) corresponding to the current video frame, which is used to characterize the coding complexity of the current video frame. Compared with the 2:1 downsampling solution in the related art, because the total number of processing units is reduced to 1 / 4 of the original, the total Frame satdcost will also be reduced to about 1 / 4 of the original, and the Frame satdcost needs to be enlarged by 4 times to ensure the accuracy of the result.

[0133] Determine the transmission bit rate that matches the original video frame based on the Frame satdcost and the current bandwidth resources. In actual scenarios, the QP parameter is used to quantize the video frame, which will cause distortion of the video image. It also directly determines the size of the residual data, which greatly affects the size of the final encoded bit stream. When the transmission bit rate is determined, the greater the complexity of the current video frame, the smaller the Frame QP (frame-level quantization parameter) is. Therefore, the transmission bit rate of the current video frame can be determined first, and then the Frame QP of the current video frame can be determined based on the transmission bit rate.

[0134] Step 605: Determine a target quantization parameter of a corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter.

[0135] In this step, the coding block refers to the coding unit used by the encoder to encode the image. The video encoding process can be encoded in Coding Tree Unit (CTU). Therefore, during the encoding process, the video frames in the video sequence can be divided into CTUs of uniform size. Furthermore, the CTU can be divided into smaller coding blocks (CU) according to the quadtree structure, so as to better adjust the different spatial characteristic coding methods (inter-frame or intra-frame), and decide whether to use intra-frame prediction coding (using spatial correlation) or inter-frame prediction coding (using temporal correlation) at the CU level.

[0136] The target QP of the corresponding coding block is fine-tuned based on the QP offset of each processing unit and the frame-level QP of the current video frame, thereby determining the final quantization parameter of the current video frame and improving the accuracy of the quantization parameter.

[0137] In one embodiment, step 605 may specifically include: adjusting the quantization parameters of the corresponding coding blocks based on the frame-level quantization parameters according to the quantization parameter offsets corresponding to the processing units, and determining the target quantization parameters of the corresponding coding blocks.

[0138] In this embodiment, the QP parameter adjustment is mainly used to adjust the details of the image, and finally to adjust the picture quality. The QP value is inversely proportional to the bit rate. The smaller the QP value, the higher the picture quality. On the contrary, the larger the QP value, the lower the picture quality. In actual scenes, the image information of different CU blocks in a frame of image may be different. For example, a frame of image contains objects with different textures, and the image complexity of the corresponding CU blocks is also different. Some CU blocks may be referenced by subsequent video frames, so the image quality requirements are higher. Such CU blocks can be based on the QP offset of the corresponding processing unit determined in the above steps. The corresponding target QP is reduced on the basis of the Frame QP of the current video frame, thereby ensuring the image quality of the CU block. For some unreferenced CU blocks, if the image information contained is also relatively small, the corresponding target QP is increased on the basis of the Frame QP of the current video frame according to the QP offset of the corresponding processing unit determined in the above steps, so as to reduce the resources occupied by the CU block and improve the coding efficiency of the current video frame.

[0139] In this embodiment, the processing unit corresponding to the CU block may refer to one or more processing units contained in the CU block. In actual scenarios, the coding block in the encoder may be a quadtree variable-size CU block, ranging in size from 64x64 to 8x8. In this application, the size of the processing unit in the pre-analysis stage may be fixed, for example, the size of the processing unit in the pre-analysis stage is 32x32, and the calculation of the QP offset is also based on 32x32 as the basic processing unit. Therefore, the size of the basic processing unit in the pre-analysis stage may be the same as or different from the size of the basic coding unit (CU block) in the encoding stage. Taking the processing unit size of 32x32 in the pre-analysis stage as an example, in the encoder, since the size of the coding block (CU block) may not be fixed, the encoder directly uses the QPoffset (QP offset) corresponding to the 32x32 processing unit to which the coding block belongs to adjust the QP value of the coding block for coding blocks less than or equal to 32x32. For a larger coding block with a size greater than 32x32, for example, a larger coding block may include multiple sub-blocks of size 32x32, the encoder can take an average of the QP offsets of all 32x32 sub-blocks included in the larger coding block, and use the average value as the QP offset of the larger coding block to adjust the QP value of the larger coding block.

[0140] Step 606: Encode the original video frame according to the target quantization parameters corresponding to the multiple coding blocks in the original video frame to obtain an encoded video frame.

[0141] In this step, the video encoding process can be encoded in units of coding tree units CTU. Therefore, during the encoding process, the original video frame in the original video sequence can be divided into CTUs of uniform size. Further, the CTU can be divided into multiple smaller coding blocks (CU blocks) according to the quadtree structure. According to the target QP of each CU block fine-tuned in step 605, the current video frame is encoded and processed to obtain the encoded video frame, and the encoded video frame is transmitted in the network. Taking the e-commerce live broadcast scene as an example, after the user initiates the live broadcast request, the live broadcast device shoots the original video stream of the target scene, and after processing by the embodiment of the present application, the encoded video stream is obtained, and the encoded video stream is transmitted to the viewing terminal. In this way, the downsampling with a higher sampling ratio is adopted, and then the pre-analysis prediction and calculation are performed, thereby reducing the complexity of the video frame pre-analysis process, increasing the speed of the pre-analysis process, and then increasing the video encoding speed, thereby increasing the fluency of the live video and improving the user experience.

[0142] The above-mentioned video information processing method, for the original video frame to be processed, first performs downsampling according to a sampling ratio greater than 2:1 to reduce the number of pixels contained in the downsampled image, and then performs pre-analysis based on the downsampled image, and the pre-analysis outputs the coding complexity of the original video frame and the quantization parameter offset of each processing unit, and further determines the frame-level quantization parameter of the original video frame according to the coding complexity and bandwidth resources, and then combines the quantization parameter offset of each processing unit and the frame-level quantization parameter of the original video frame to determine the target quantization parameter of the corresponding coding block. In this way, after downsampling with a higher sampling ratio, the prediction and calculation of the pre-analysis are performed to reduce the number of pixels contained in the downsampled image, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed.

[0143] Taking 4:1 downsampling and using 32x32 processing units as basic processing units as an example, the solution of the embodiment of the present application has at least the following advantages: compared with the 2:1 downsampling solution based on 16x16 blocks in the related art, the basic processing unit after downsampling remains unchanged at 8x8, but the total processing units are reduced by 75%, the processing complexity can be reduced by 75%, and the processing efficiency is improved by 4 times.

[0144] Please see Figure 8 , which is a video information processing method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2-Figure 5In the application scenario of video information processing shown in , the original video frame is downsampled with a higher sampling ratio and then the pre-analysis prediction and calculation are performed, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed. This embodiment takes the terminal 220 as an execution end as an example. Compared with the above embodiment, this embodiment takes the live broadcast scene as an example. The method includes the following steps:

[0145] Step 801: In response to a live broadcast request, receive a live broadcast video frame to be processed.

[0146] In this step, in a live broadcast scenario, a user can trigger a live broadcast request through a live broadcast device, such as logging into an e-commerce live broadcast platform through a mobile phone and triggering a live broadcast request. In response to the live broadcast request, the live broadcast device captures a live video stream in a specific scenario.

[0147] Step 802: Divide the live video frame into multiple processing units.

[0148] In this step, before encoding and analyzing the live video stream, each live video frame is first divided into multiple processing units to facilitate the refined encoding and analysis of the live video frame in blocks to improve efficiency. The processing unit division method can refer to the description of the above embodiment.

[0149] Step 803: downsample the live video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling. For details, refer to the description of step 602 in the above embodiment.

[0150] Step 804: Determine the encoding complexity of the live video frame and the quantization parameter offset corresponding to each processing unit in the plurality of processing units according to the downsampled image. For details, refer to the description of step 603 in the above embodiment.

[0151] Step 805: Determine the frame-level quantization parameter of the live video frame according to the coding complexity and the current bandwidth resources. For details, refer to the description of step 604 in the above embodiment.

[0152] Step 806: Determine the target quantization parameter of the corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter. For details, refer to the description of step 605 in the above embodiment.

[0153] Step 807: According to the target quantization parameters corresponding to the multiple coding blocks in the live video frame, the live video frame is coded to obtain a coded video frame. For details, refer to the description of step 606 in the above embodiment.

[0154] Step 808: Transmit the encoded video frame to a receiving terminal.

[0155] In this step, in a live broadcast scenario, if a viewing user wants to watch the live video, the encoded video frame can be transmitted to the viewing user's receiving terminal without transmitting the original live video stream, thus saving transmission resources.

[0156] For details of each step of the above method, please refer to the relevant description of the above-mentioned embodiments, which will not be repeated here.

[0157] Please see Fig. 9 , which is a video information processing device 900 of an embodiment of the present application, the device can be applied to Figure 1 The electronic device 1 shown can be applied to Figure 2-Figure 5 In the application scenario of video information processing shown in , the original video frame is downsampled with a higher sampling ratio and then the pre-analysis prediction and calculation are performed, thereby reducing the complexity of the video frame pre-analysis process, improving the speed of the pre-analysis process, and then improving the video encoding speed. The device includes: an acquisition module 901, a sampling module 902, a first determination module 903, a second determination module 904 and a third determination module 905, and the functional principles of each module are as follows:

[0158] The acquisition module 901 is used to acquire the original video frame to be processed, where the original video frame is divided into multiple processing units.

[0159] The sampling module 902 is used to downsample the original video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling.

[0160] The first determination module 903 is used to determine the coding complexity of the original video frame and the quantization parameter offset corresponding to each processing unit in the multiple processing units according to the downsampled image.

[0161] The second determination module 904 is used to determine the frame-level quantization parameter of the original video frame according to the coding complexity and the current bandwidth resources.

[0162] The third determination module 905 is used to determine the target quantization parameter of the corresponding coding block according to the quantization parameter offset and the frame level quantization parameter.

[0163] In one embodiment, the first determination module 903 is used to perform inter-frame prediction processing on multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units, calculate propagation costs corresponding to the multiple processing units according to the motion vectors, and determine quantization parameter offsets corresponding to the multiple processing units according to the propagation costs.

[0164] In one embodiment, the first determination module 903 is used to perform inter-frame prediction processing on multiple processing units according to the downsampled image to obtain inter-frame coding costs of the multiple processing units. Perform intra-frame prediction processing on multiple processing units according to the downsampled image to obtain intra-frame coding costs of the multiple processing units. According to the inter-frame coding cost and the intra-frame coding cost, determine the target coding cost applicable to the multiple processing units, and the coding complexity includes the target coding cost of the multiple processing units.

[0165] In one embodiment, the first determining module 903 is specifically configured to determine, for a current processing unit, a minimum value between an inter-frame coding cost and an intra-frame coding cost as a target coding cost of the current processing unit.

[0166] In one embodiment, the second determination module 904 is used to determine a transmission bit rate that matches the original video frame according to target encoding costs of the multiple processing units and current bandwidth resources, and to determine a frame-level quantization parameter of the original video frame according to the transmission bit rate.

[0167] In one embodiment, the third determination module 905 is used to adjust the quantization parameters of the corresponding coding blocks based on the frame-level quantization parameters according to the quantization parameter offsets corresponding to the processing units, and determine the target quantization parameters of the corresponding coding blocks.

[0168] In one embodiment, the preset sampling ratio is 4: 1. The sampling module 902 is used to perform 4: 1 downsampling on the original video frame in the width direction and the height direction respectively to obtain a downsampled image.

[0169] In one embodiment, the processing unit has a width of 32 pixels and a height of 32 pixels.

[0170] In one embodiment, it further includes: an encoding module for encoding the original video frame according to the target quantization parameters corresponding to multiple encoding blocks in the original video frame after determining the target quantization parameter corresponding to the encoding block to obtain an encoded video frame.

[0171] For a detailed description of the above-mentioned video information processing device 900, please refer to the description of the relevant method steps in the above-mentioned embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0172] Fig.10 The following is a schematic diagram of a cloud device 100 provided in an exemplary embodiment of the present application. The cloud device 100 can be used to run the method provided in any of the above embodiments. Fig.10 As shown, the cloud device 100 may include: a memory 1004 and at least one processor 1005, Fig.10 A processor is taken as an example.

[0173] The memory 1004 is used to store computer programs and can be configured to store various other data to support operations on the cloud device 100. The memory 1004 can be an object storage service (OSS).

[0174] Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0175] The processor 1005 is coupled to the memory 1004 and is used to execute the computer program in the memory 1004 to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0176] Furthermore, if Fig.10 The cloud device also includes: a firewall 1001, a load balancer 1002, a communication component 1006, a power supply component 1003 and other components. Fig.10 Only some components are shown schematically, which does not mean that the cloud device only includes Fig.10 Components shown.

[0177] In one embodiment, the above Fig.10 The communication component 1006 in is configured to facilitate wired or wireless communication between the device where the communication component 1006 is located and other devices. The device where the communication component 1006 is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, LTE (Long Term Evolution, Long Term Evolution, referred to as LTE), 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 1006 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1006 also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (Radio Frequency Identification, referred to as RFID) technology, Infrared Data Association (Infrared Data Association, referred to as IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (bluetooth, referred to as BT) technology and other technologies to achieve.

[0178] In one embodiment, the above Fig.10 The power supply component 1003 provides power to various components of the device where the power supply component 1003 is located. The power supply component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0179] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0180] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0182] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0183] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM (Random Access Memory) memory, and may also include non-volatile storage NVM (Nonvolatile memory, NVM for short), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0184] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special computer.

[0185] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0187] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0189] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0190] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video information processing method, characterized in that: The method comprises: Acquire an original video frame to be processed, wherein the original video frame is divided into a plurality of processing units; the width of the processing unit is greater than 16 pixels and the height is greater than 16 pixels; Downsampling the original video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than a sampling ratio corresponding to half sampling; Determining, according to the downsampled image, a coding complexity of the original video frame and a quantization parameter offset corresponding to each of the multiple processing units, wherein the coding complexity includes target coding costs of the multiple processing units; Determining a transmission bit rate matching the original video frame according to target encoding costs of the multiple processing units and current bandwidth resources; Determine a frame-level quantization parameter of the original video frame according to the transmission bit rate; Determining a target quantization parameter of a corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter; Wherein, determining, according to the downsampled image, a quantization parameter offset corresponding to each processing unit in the plurality of processing units of the original video frame comprises: Performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units; According to the motion vectors, propagation costs corresponding to the multiple processing units are calculated respectively, and quantization parameter offsets corresponding to the multiple processing units are determined according to the propagation costs.

2. The method according to claim 1, characterized in that The step of determining the encoding complexity of the original video frame according to the downsampled image includes: Performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain inter-frame coding costs of the multiple processing units; Performing intra-frame prediction processing on the multiple processing units according to the downsampled image to obtain intra-frame coding costs of the multiple processing units; According to the inter-frame coding cost and the intra-frame coding cost, target coding costs applicable to the multiple processing units are determined.

3. The method according to claim 2, characterized in that The determining, according to the inter-frame coding cost and the intra-frame coding cost, target coding costs applicable to the multiple processing units comprises: For a current processing unit, a minimum value between the inter-frame coding cost and the intra-frame coding cost is determined as a target coding cost of the current processing unit.

4. The method according to claim 1, characterized in that The determining, according to the quantization parameter offset and the frame-level quantization parameter, a target quantization parameter of a corresponding coding block includes: According to the quantization parameter offsets corresponding to the respective processing units, the quantization parameters of the corresponding coding blocks are adjusted respectively on the basis of the frame-level quantization parameters to determine the target quantization parameters of the corresponding coding blocks.

5. The method according to claim 1, characterized in that The preset sampling ratio is 4:1; and downsampling the original video frame according to the preset sampling ratio to obtain a downsampled image includes: The original video frame is downsampled by 4:1 in the width direction and the height direction respectively to obtain the downsampled image.

6. The method according to claim 5, characterized in that The processing unit has a width of 32 pixels and a height of 32 pixels.

7. The method according to claim 1, characterized in that After determining the target quantization parameter of the corresponding coding block, the method further includes: The original video frame is encoded according to the target quantization parameters corresponding to the multiple encoding blocks in the original video frame to obtain an encoded video frame.

8. A video information processing method, characterized in that: The method comprises: In response to the live broadcast request, receiving a live broadcast video frame to be processed; Divide the live video frame into a plurality of processing units; the width of the processing unit is greater than 16 pixels and the height is greater than 16 pixels; Downsampling the live video frame according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than a sampling ratio corresponding to half sampling; Determine, according to the downsampled image, the coding complexity of the live video frame and the quantization parameter offset corresponding to each of the multiple processing units, wherein the coding complexity includes target coding costs of the multiple processing units; Determining a transmission bit rate matching the live video frame according to target encoding costs of the multiple processing units and current bandwidth resources; Determine a frame-level quantization parameter of the live video frame according to the transmission bit rate; Determining a target quantization parameter of a corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter; According to the target quantization parameters corresponding to the plurality of coding blocks in the live video frame, encoding the live video frame to obtain an encoded video frame; Transmitting the encoded video frame to a receiving terminal; Wherein, determining, according to the downsampled image, a quantization parameter offset corresponding to each processing unit in the multiple processing units of the live video frame includes: Performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units; According to the motion vectors, propagation costs corresponding to the multiple processing units are calculated respectively, and quantization parameter offsets corresponding to the multiple processing units are determined according to the propagation costs.

9. A video information processing system, characterized in that: include: Pre-analysis module, bit rate control module and encoder; wherein: The pre-analysis module is used to obtain an original video frame to be processed, wherein the original video frame is divided into a plurality of processing units; the width of the processing unit is greater than 16 pixels and the height is greater than 16 pixels; the original video frame is downsampled according to a preset sampling ratio to obtain a downsampled image, wherein the preset sampling ratio is greater than the sampling ratio corresponding to half sampling; according to the downsampled image, the coding complexity of the original video frame and the quantization parameter offset corresponding to each of the plurality of processing units are determined, wherein the coding complexity includes the target coding costs of the plurality of processing units; according to the target coding costs of the plurality of processing units and current bandwidth resources, a transmission bit rate matching the original video frame is determined; and according to the transmission bit rate, a frame-level quantization parameter of the original video frame is determined; The bit rate control module is connected to the pre-analysis module and is used to determine a target quantization parameter of a corresponding coding block according to the quantization parameter offset and the frame-level quantization parameter; The encoder is connected to the pre-analysis module and the bit rate control module respectively, and is used to encode the original video frame according to the target quantization parameters corresponding to the multiple coding blocks in the original video frame to obtain an encoded video frame; The pre-analysis module is used to determine the quantization parameter offset corresponding to each processing unit in the multiple processing units of the original video frame according to the downsampled image, specifically for: Performing inter-frame prediction processing on the multiple processing units according to the downsampled image to obtain motion vectors of the multiple processing units; According to the motion vectors, propagation costs corresponding to the multiple processing units are calculated respectively, and quantization parameter offsets corresponding to the multiple processing units are determined according to the propagation costs.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1 to 8.

11. A cloud device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Code rate control method and device for sport video

    CN105898306A

  • Video coding method and device, video decoding method and device, computer equipment and memory medium

    CN108924553A

  • Image encoding method and device

    CN108989804A

  • Video coding method, device and equipment and storage medium

    CN110049321A

  • Video coding rate control method and device, equipment and storage medium

    CN110324622A