Video encoding method and apparatus

Abstract

The present application provides a video encoding method and apparatus. The video encoding method comprises: acquiring a first slice solution for a first video frame; determining encoding performance prediction values of a number N of second slices obtained when a second video frame adopts the first slice solution; determining a second slice solution for the second video frame according to the encoding performance prediction values of the N second slices; and dividing the second video frame into N third slices according to the second slice solution, and encoding each of the N third slices. Embodiments of the present application, by means of determining predicted encoding performance of a current frame and a second frame, determine a second slice solution suitable for the second video frame and more balanced in encoding performance, significantly improving the encoding performance of the video frame and the whole video.

Description

Video encoding method and device

[0001] This application claims priority to Russian patent application No. RU2023115400 filed with the Russian Federal Intellectual Property Office on June 13, 2023, entitled “Slice balancing”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of video coding and decoding technology, and in particular to a method and device for video coding. Background Art

[0003] Cloud multimedia services have experienced rapid growth in recent years, including live streaming, real-time communication (RTC), video conferencing, and cloud desktops. These services are also pursuing a higher user experience or lower latency. As a core technology for media services, video encoding and decoding technologies require optimization in encoding latency, encoding speed, and encoding efficiency to build technological competitiveness. Typically, to increase the encoding speed of a video encoder, each frame of the video needs to be encoded in parallel. Therefore, each video frame can be segmented into slices, and then the resulting slices can be encoded in parallel, thereby increasing the encoding speed of each video frame.

[0004] In current video coding standards, the slicing scheme for slicing video frames does not consider video content, but rather divides the slices equally. This results in uneven encoding performance, such as encoding latency and encoding speed, across multiple slices of a video frame, degrading the encoding performance of the frame and, ultimately, the entire video.

[0005] Summary of the Invention

[0006] The present application provides a method and device for video encoding, which determines the predicted encoding performance of the current video frame, i.e., the second video frame, and determines a second slicing scheme suitable for the second video frame and with more balanced encoding performance, thereby significantly improving the encoding performance of the entire video frame and even the video.

[0007] In a first aspect, a method for video encoding is provided, the method comprising: obtaining a first slicing scheme for a first video frame, the first slicing scheme being used to divide the first video frame into N first slices, where N is a positive integer greater than or equal to 2; determining encoding performance prediction values ​​of N second slices obtained when a second video frame adopts the first slicing scheme, the N second slices corresponding one-to-one to the N first slices, each second slice being located at the same position in the second video frame as the corresponding first slice is located in the first video frame, and the area of ​​each second slice being the same as the area of ​​the corresponding first slice; determining a second slicing scheme for the second video frame based on the encoding performance prediction values ​​of the N second slices, the second slicing scheme being used to divide the second video frame into N third slices; dividing the second video frame into the N third slices based on the second slicing scheme; and encoding the N third slices respectively to obtain the encoded second video frame.

[0008] According to the technical solution provided in the present application, the encoding performance of each slice of the current video frame, i.e., the second video frame, is predicted based on the first slicing scheme of the encoded first video frame, so as to determine a second slicing scheme that is suitable for the second video frame and has more balanced encoding performance, thereby significantly improving the encoding performance of the entire second video frame and even the entire video.

[0009] The second video frame can be the frame following the first video frame, or any frame arranged after the first video frame in the video frame sequence. In other words, the first video frame is any encoded video frame before the second video frame. In the embodiment of the present application, the second video frame is preferably the frame following the first video frame. In this way, the slicing scheme determination or adjustment process is smoother, thereby improving the encoding performance of the entire video.

[0010] In combination with the first aspect, in certain implementations of the first aspect, determining the second slicing scheme of the second video frame based on the coding performance prediction values ​​of the N second slices includes: adjusting the areas of some or all of the second slices in the N second slices based on the coding performance prediction values ​​of the N second slices to obtain the second slicing scheme.

[0011] According to the technical solution provided by this application, the second slicing scheme corresponding to the second video frame can be different from the first slicing scheme of the first video frame, thereby realizing a dynamic slicing process for different video frames. The slicing scheme corresponding to each video frame can achieve balanced encoding performance of video frame slices, thereby improving the encoding performance of the entire video.

[0012] In combination with the first aspect, in certain implementations of the first aspect, adjusting the area of ​​some or all of the N second slices includes: determining a fourth slice from the N second slices, wherein, when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fourth slice is the smallest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fourth slice is the largest; and reducing the area of ​​the fourth slice.

[0013] According to the technical solution provided in the present application, the overall encoding performance of the entire video frame is improved by reducing the area of ​​the fourth slice with the worst encoding performance.

[0014] In combination with the first aspect, in some implementations of the first aspect, reducing the area of ​​the fourth slice includes: moving a slice boundary between the fourth slice and at least one adjacent second slice by K slice rows toward the fourth slice.

[0015] According to the technical solution provided in the present application, by adjusting the slice boundary between the fourth slice and the adjacent second slice, the area of ​​the fourth slice can be quickly adjusted without recalculating the adjusted area, thereby improving the coding performance.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: determining a fifth slice from the N second slices, wherein, when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fifth slice is the largest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fifth slice is the smallest; reducing the area of ​​the fourth slice includes: moving one or more slice boundaries between the fourth slice and the fifth slice by K slice rows toward the direction of the fourth slice.

[0017] According to the technical solution provided in the present application, by adjusting the slice boundary between the fourth slice and the fifth slice while keeping the area of ​​the other second slices unchanged, the encoding performance of the fourth slice can be improved and the encoding performance of the fifth slice can be reduced while the encoding performance of the other second slices remains basically unchanged, thereby obtaining N third slices and second slices with more balanced encoding performance, thereby improving the encoding performance of the second video frame.

[0018] In combination with the first aspect, in some implementations of the first aspect, the value of K is 1.

[0019] According to the technical solution provided in this application, when the value of K is 1, the adjustment amplitude of the slicing scheme of different video frames in the same sequence can be reduced, thereby avoiding frequent adjustment of the slicing scheme or excessive adjustment amplitude.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the value of K is positively correlated with the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices.

[0021] According to the technical solution provided by this application, the greater the difference between the maximum and minimum values ​​in the coding performance prediction values ​​of the N second slices, the larger the value of K. In this way, the coding performance gap between the two second slices corresponding to the maximum and minimum values ​​in the coding performance prediction values ​​can be quickly narrowed.

[0022] In combination with the first aspect, in certain implementations of the first aspect, before determining the second slicing scheme of the second video frame, the method also includes: determining whether the difference between the maximum and minimum values ​​of the encoding performance prediction values ​​of the N second slices reaches a target difference threshold.

[0023] According to the technical solution provided in this application, it is possible to determine whether the coding performance of N second slices is balanced. When the difference between the maximum and minimum values ​​of the predicted coding performance values ​​of the N second slices reaches a target difference threshold, it indicates that the coding performance of the N second slices is not balanced, and further indicates that the first slicing scheme needs to be adjusted to determine a more balanced second slicing scheme. Otherwise, the first slicing scheme is reused.

[0024] In combination with the first aspect, in certain implementations of the first aspect, determining the coding performance prediction values ​​of the N second slices obtained when the second video frame adopts the first slicing scheme includes: determining the coding performance prediction value of the i-th second slice among the N second slices corresponding to the i-th first slice based on the target prediction model and the coding parameters and / or coding performance values ​​of the i-th first slice among the N first slices, where i is a positive integer less than or equal to N.

[0025] According to the technical solution provided in the present application, by performing numerical time delay training fitting on the coding parameters and / or coding performance values ​​of N first slices, a prediction model that can accurately predict the coding performance prediction values ​​of N second slices is obtained. This prediction model can be used to guide the slice segmentation process of the video frame.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the coding performance prediction value is a coding delay prediction value, the target prediction model is a target prediction formula, and according to the target prediction model and the coding parameters and / or coding performance values ​​of the i-th first slice in the N first slices, the coding performance prediction value of the i-th second slice in the N second slices corresponding to the i-th first slice is determined, including: obtaining the number of bytes after encoding the i-th first slice and the number of macroblocks in the i-th first slice; and determining the coding delay prediction value of the i-th second slice corresponding to the i-th first slice according to the target prediction formula, the number of bytes after encoding the i-th first slice, and the number of macroblocks in the i-th first slice.

[0027] In a second aspect, a method for video decoding is provided, the method comprising: obtaining an encoded second video frame, the second video frame comprising N encoded third slices, the N third slices being obtained by slicing the second video frame using a second slicing scheme, the second slicing scheme being determined by coding performance prediction values ​​of the N second slices, the N second slices being obtained by the second video frame using a first slicing scheme, the first slicing scheme being used to divide the first video frame into N first slices; decoding the encoded N third slices to obtain N video frame slices; and splicing the N video frame slices to obtain the decoded second video frame.

[0028] In an embodiment of the present application, the first video frame may be any video frame arranged before the second video frame in a video frame sequence, or in other words, any encoded video frame before the second video frame.

[0029] In combination with the second aspect, in certain implementations of the second aspect, the areas of the N third slices in the second slicing scheme are determined by adjusting areas of some or all of the N second slices.

[0030] According to a third aspect, a video encoding device is provided, which includes: an acquisition module for acquiring a first slicing scheme for a first video frame, the first slicing scheme being used to divide the first video frame into N first slices, where N is a positive integer greater than or equal to 2; a processing module for determining encoding performance prediction values ​​of N second slices obtained when the second video frame adopts the first slicing scheme, the N second slices corresponding one-to-one to the N first slices, each second slice being located at the same position in the second video frame as the corresponding first slice is located in the first video frame, and the area of ​​each second slice being the same as the area of ​​the corresponding first slice, and determining a second slicing scheme for the second video frame based on the encoding performance prediction values ​​of the N second slices, the second slicing scheme being used to divide the second video frame into N third slices; and an encoding module for dividing the second video frame into the N third slices based on the second slicing scheme, and encoding the N third slices respectively to obtain the encoded second video frame.

[0031] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically used to: adjust the areas of some or all of the N second slices according to the coding performance prediction values ​​of the N second slices to obtain the second slicing scheme.

[0032] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically used to: determine a fourth slice from the N second slices, wherein, when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fourth slice is the smallest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fourth slice is the largest; and reduce the area of ​​the fourth slice.

[0033] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically configured to: move a slice boundary between the fourth slice and at least one adjacent second slice by K slice rows in a direction closer to the fourth slice.

[0034] In combination with the third aspect, in certain implementations of the third aspect, the processing module is further used to: determine a fifth slice from the N second slices, wherein, when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fifth slice is the largest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fifth slice is the smallest; the processing module is specifically used to: move one or more slice boundaries between the fourth slice and the fifth slice by K slice rows toward the direction of the fourth slice.

[0035] In combination with the third aspect, in some implementations of the third aspect, the value of K is 1.

[0036] In combination with the third aspect, in certain implementations of the third aspect, the value of K is positively correlated with the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices.

[0037] In combination with the third aspect, in certain implementations of the third aspect, before determining the second slicing scheme of the second video frame, the processing module is also used to: determine whether the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices reaches the target difference threshold.

[0038] In combination with the third aspect, in certain implementations of the third aspect, the processing module is specifically used to: determine the coding performance prediction value of the i-th second slice among the N second slices corresponding to the i-th first slice based on the target prediction model and the coding parameters and / or coding performance values ​​of the i-th first slice among the N first slices, where i is a positive integer less than or equal to N.

[0039] In combination with the third aspect, in certain implementations of the third aspect, the coding performance prediction value is a coding delay prediction value, the target prediction model is a target prediction formula, and the processing module is specifically used to: obtain the number of bytes after encoding the i-th first slice and the number of macroblocks in the i-th first slice; determine the coding delay prediction value of the i-th second slice corresponding to the i-th first slice based on the target prediction formula, the number of bytes after encoding the i-th first slice, and the number of macroblocks in the i-th first slice.

[0040] In a fourth aspect, a device for video decoding is provided, which includes: an acquisition module for acquiring an encoded second video frame, the second video frame including N encoded third slices, the N third slices being obtained by slicing the second video frame through a second slicing scheme, the second slicing scheme being determined by encoding performance prediction values ​​of the N second slices, the N second slices being obtained by the second video frame using a first slicing scheme, the first slicing scheme being used to divide the first video frame into N first slices; a decoding module for decoding the encoded N third slices to obtain N video frame slices; and splicing the N video frame slices to obtain the decoded second video frame.

[0041] In combination with the fourth aspect, in certain implementations of the fourth aspect, the areas of the N third slices in the second slicing scheme are determined by adjusting areas of part or all of the N second slices.

[0042] In a fifth aspect, a video encoder is provided, comprising a memory and a processor, wherein the processor calls a program code stored in the memory to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0043] In a sixth aspect, a video decoder is provided, comprising a memory and a processor, wherein the processor calls a program code stored in the memory to execute the method in the second aspect or any possible implementation manner of the second aspect.

[0044] In the seventh aspect, a computing device is provided, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, so that the computing device executes the method in the first aspect or any possible implementation of the first aspect, or the computing device executes the method in the second aspect or any possible implementation of the second aspect.

[0045] In an eighth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call and run the instructions from the memory, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect, or the computing device cluster executes the method in the second aspect or any possible implementation of the second aspect.

[0046] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0047] In the ninth aspect, a chip is provided, which obtains instructions and executes the instructions to implement the method in the above-mentioned first aspect or any possible implementation of the first aspect, or to implement the method in the second aspect or any possible implementation of the second aspect.

[0048] Optionally, as an implementation method, the chip includes a processor and a data interface, and the processor reads instructions stored in the memory through the data interface to execute the method in the above-mentioned first aspect or any possible implementation method of the first aspect, or executes the method in the second aspect or any possible implementation method of the second aspect.

[0049] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the above-mentioned first aspect or any possible implementation method of the first aspect, or to execute the method in the second aspect or any possible implementation method of the second aspect.

[0050] In the tenth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device or a computing device cluster, the computing device or the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation of the first aspect, or executes the method in the second aspect or any possible implementation of the second aspect.

[0051] In the eleventh aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer instructions are executed by a computing device or a computing device cluster, the computing device or the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation of the first aspect, or executes the method in the second aspect or any possible implementation of the second aspect.

[0052] By way of example, these computer-readable storage media include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0053] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1 is a schematic diagram of the architecture of a video encoding and decoding system provided in an embodiment of the present application.

[0055] FIG2 is a schematic diagram of a video composition structure provided in an embodiment of the present application.

[0056] FIG3 is a schematic diagram of a video slicing solution provided in an embodiment of the present application.

[0057] FIG4 is a schematic flowchart of a video encoding method provided in an embodiment of the present application.

[0058] FIG5 is a schematic flowchart of a video decoding method provided in an embodiment of the present application.

[0059] FIG6 is a schematic diagram of a slicing scheme determined according to an embodiment of the present application.

[0060] FIG7 is a schematic diagram of a fitting result of the encoding delay of a macroblock and the number of encoded bytes of the macroblock provided in an embodiment of the present application.

[0061] FIG8 is a schematic flowchart of a slicing scheme determined according to an embodiment of the present application.

[0062] FIG9 is a schematic diagram of achieving frame-by-frame changes of multiple slices by adjusting slice boundaries, provided by an embodiment of the present application.

[0063] FIG10 is a schematic diagram of experimental results of a video comparison experiment provided in an embodiment of the present application.

[0064] FIG11 is a schematic diagram of another experimental result of a video comparison experiment provided in an embodiment of the present application.

[0065] FIG12 is a schematic diagram of another experimental result of a video comparison experiment provided in an embodiment of the present application.

[0066] FIG13 is a schematic diagram of experimental results of another video comparison experiment provided in an embodiment of the present application.

[0067] FIG14 is a schematic block diagram of a video encoding device provided in an embodiment of the present application.

[0068] FIG15 is a schematic block diagram of a video decoding device provided in an embodiment of the present application.

[0069] FIG16 is a schematic structural block diagram of a computing device provided in an embodiment of the present application.

[0070] FIG17 is a schematic structural block diagram of a computing device cluster provided in an embodiment of the present application.

[0071] FIG18 is a schematic structural block diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0073] In order to facilitate understanding of the technical solution of this application, the professional terms mentioned in this application are briefly explained below.

[0074] 1. Video encoding and decoding technology

[0075] Video coding technology is used to compress raw video, thereby reducing the transmission and storage costs of video data. The video coding process involves sampling frames or image data from a video stream, encoding the frames or image data, and then compressing them into smaller data packets for transmission. Video coding primarily compresses redundant information in video data, such as spatial redundancy caused by strong correlations between adjacent pixels in the same video frame, temporal redundancy caused by strong correlations between adjacent frames, and visual redundancy in video information that is imperceptible to the human eye. Currently, the main video coding standards include H.264 (advanced video coding, AVC), H.265 (high efficiency video coding, HEVC), and H.266 (versatile video coding, VVC).

[0076] Correspondingly, video decoding technology is used to decode or decompress encoded video. For example, the video decoding process can involve the receiving end of a video player opening the received encoded video data packets, decoding the video packets, and reconstructing them into multiple frames, and finally arranging the multiple frames in sequence, frame by frame, to form a video stream.

[0077] Figure 1 is a schematic diagram of the architecture of a video encoding and decoding system provided in an embodiment of the present application. The video encoding and decoding system provided in an embodiment of the present application may include an acquisition terminal 110, a transmission unit 150, and a playback terminal 120. The acquisition terminal 110 may include an acquisition unit 130 and an encoding unit 140, and the playback terminal 120 may include a stream receiving unit 160, a decoding unit 170, and a display unit 180.

[0078] The acquisition unit 130 is used to acquire images at a preset frame rate, including but not limited to acquiring images from a camera, acquiring images from a graphics card, acquiring images from a graphics engine, etc.

[0079] The encoding unit 140 is used to perform video encoding, encoding each video frame according to the format specified in the video coding standard, such as H.264, H.265, etc. mentioned above. This application does not limit the specific encoding format. In an embodiment of the present application, the encoding unit 140 is used to parallel encode the video frames divided into N slices and output the encoded video stream, where N is a positive integer greater than or equal to 2. The encoding unit 140 may include a software encoder or a hardware encoder.

[0080] The transmission unit 150 is used to transmit the video stream output by the acquisition end 110 and distribute it to the playback end 120 .

[0081] The stream receiving unit 160 is used to download and cache the video stream, and download the video stream from the transmission unit 150 to the local cache.

[0082] The decoding unit 170 is used to decode the video stream, that is, to decode the video frame containing the encoded N slices, reassemble the N slices to obtain a video frame picture or image, and then transmit it to the display unit 180. The decoding unit 170 may include a software decoder or a hardware decoder.

[0083] The display unit 180 is used to display images, and displays the images sent by the decoding unit 170 on a display screen.

[0084] It is worth noting that Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, the acquisition end 110 may also include a flow unit, which is not limited in this application.

[0085] 2. Sequence, frame, slice, macroblock

[0086] Figure 2 shows the composition structure of the video in the video encoding process provided by an embodiment of the present application. Among them, the video within a period of time is generated into a sequence (sequence) including multiple video frames (frame) by the acquisition unit 130. Each video frame in the sequence is further divided into multiple slices (slices), and each slice is an independent coding unit. Each slice includes multiple macroblocks (macroblocks), and each macroblock can be composed of 4×4 to 128×128 pixels (pixels). In other embodiments of the present application, each macroblock can also be composed of an integer number of blocks (blocks), and each block can be composed of multiple pixels (pixels). For example, in H.264, a block can be a luminance pixel block containing 16×16 pixels and two color pixel blocks containing 8×8 pixels. The above division process is described in detail below.

[0087] In a video over a period of time, the differences in pixels, brightness, and color temperature between adjacent frames are typically minimal. Therefore, the first frame of the period can be fully encoded, and the encoding of the next frame only records the differences in pixel, brightness, and color temperature compared to the first frame. A set of video images with minimal changes in video content over this period is called a sequence. In other words, a sequence consists of a segment of video frame data with identical or similar features. Video frame types include I (intra-coded) frames, which represent key frames; P (predictive-coded) frames, which record the difference from the previous frame; and B (bidirectionally predicted) frames, which record the difference from the preceding and following frames. For example, the first frame in a sequence can be an I frame, and the remaining frames can be P frames or B frames.

[0088] In the current video standards such as H.264, H.265, and H.266, each video frame can be divided into one or more slices, and each slice can be independently encoded and decoded by a separate processor, without relying on other slices in the same video frame. As shown in Figure 2, the video frame is divided into 4 slices, and the embodiments of the present application do not limit the number of slices. In the embodiments of the present application, the areas of the multiple slices (that is, the number of macroblocks contained) can be the same area that is evenly divided, or they can be different areas, that is, the number of macroblocks contained is different. In other embodiments of the present application, the shape of the slice can be a rectangle as shown in Figure 2, or it can be an irregular shape, that is, the slice boundary between adjacent slices spans multiple macroblock rows. For example, a macroblock row is divided into two parts by the slice boundary, the first part is the last row of macroblock rows of the i-th slice, and the second part is the first row of macroblock rows of the i+1-th slice.

[0089] The advantages of slicing video frames are that they reduce error propagation, reduce the size of transmitted data packets, and enable parallel encoding and decoding of multiple slices. In the case of parallel encoding and decoding, after the video frame is sliced, the overall encoding performance of the video frame depends on the slice with the worst encoding performance. If the encoding delay is used as a consideration for encoding performance, Figure 3 shows a schematic diagram of a non-slicing encoding method and a slicing encoding method. Among them, R0 to R15 are slice rows, which are rows composed of the most basic video encoding units in the embodiment of the present application. For example, if the most basic video encoding unit is a single macroblock, the slice row can be a macroblock row composed of a whole row of macroblocks; if the most basic video encoding unit is a single pixel, the slice row can be a pixel row composed of a whole row of pixels. This application does not limit the type and number of slice rows used for encoding in the slice, and the 16 slice rows shown in Figure 3 are only examples.

[0090] According to the encoder's core design, encoding each video coding unit (VCU) in a slice row requires relying on previously encoded VCUs for prediction. Generally, within a slice, macroblocks or pixels are encoded in scan order. For example, if the VCUs are pixels, each pixel in the first slice row relies on the previously encoded pixel for prediction, while the first pixel in the second slice row relies on the previously encoded pixel in the first row for prediction. Therefore, as shown in Figure 3, the encoding of each slice row starts slightly later than the previously encoded slice row.

[0091] As shown in Figure 3(a), if a video frame is not segmented, the total encoding delay of the video frame depends on the last slice row, or the slice row that completes the encoding process last. That is, the total encoding delay of the video frame is the encoding completion time t1 of slice row R15 shown in Figure 3(a). As shown in Figure 3(b), if a video frame is segmented, each slice is encoded independently. The total encoding delay of the video frame depends on the encoding completion time t2 of the slice that completes the encoding process last, that is, the encoding completion time t2 of slice row R7 shown in Figure 3(b). This is because, as shown in Figure 3(b), R4, R8, and R12, which originally needed to wait for the previously encoded slice row, can start encoding together with R0, thereby reducing the total encoding delay of the video. Since t2 is less than t1, the total encoding delay of the video frame in Figure 3(b) is less than that in Figure 3(a). Therefore, segmenting the video frame improves overall encoding performance.

[0092] Most traditional slicing methods currently use a fixed, evenly spaced slicing scheme, meaning that each frame uses the same slicing scheme and each slice has the same area. For example, each video frame in a sequence is evenly segmented into four slices, as shown in Figure 3(b).

[0093] However, with the expansion of applications in multiple real-time fields such as video conferencing, online education, remote terminals, game interaction, e-commerce interactive live broadcast, and telemedicine, the traditional fixed and average slice segmentation method is not suitable for the real-time video encoding process in the above fields. The reason is that there is an imbalance in the video content in the above fields, that is, part of the video frame content changes less, the texture is smooth, the video complexity is low, and the encoding speed is fast, while the other part changes drastically, the texture is complex, the video complexity is high, and the encoding speed is slow, which leads to uneven encoding performance of each slice in the same video frame. For example, in the video content of video conferencing or interactive live broadcast, the video content containing portraits changes more drastically than the video content containing background walls. As shown in (b) of Figure 3, if the average slice segmentation method is used, the encoding delay of the second slice containing the portrait is much longer than that of the fourth slice containing the background wall, resulting in the fourth slice being encoded first, but it needs to wait for the second slice to complete encoding and cannot start encoding the next frame in advance.

[0094] In addition, the video used for encoding may include multiple sequences, and the adjacent video frames in each sequence or the video frames contained in different sequences have different change trends. If a fixed slicing segmentation scheme is used for all of them, the encoding performance of some video frames will be much lower than that of other video frames. For example, the first slice of the i-th video frame contains a drastically changing portrait, and the second slice contains a slowly changing background wall, while the first slice of the m-th video frame contains a slowly changing background wall, and the second slice contains a drastically changing portrait. If the m-th video frame uses the same slicing segmentation method as the i-th video frame, the encoding performance of the two video frames will be unbalanced, that is, the encoding performance of the m-th video frame will be much lower than that of the i-th video frame.

[0095] In order to solve the problem of uneven coding performance between slices in the above-mentioned video frame and between video frames, the present application proposes a video coding method 400. Figure 4 shows a schematic flow chart of a video coding method 400 provided by an embodiment of the present application, wherein the method 400 includes steps 410 to 440. In the method 400, the coding performance of each slice of the current video frame is predicted based on the slicing scheme of the first video frame, thereby determining the slicing scheme corresponding to the current video frame. The slicing scheme corresponding to the current frame may be different from the slicing scheme of the first video frame, thereby realizing a dynamic slicing process of different video frames. The device for implementing the method 400 may be the encoding unit 140 shown in Figure 1.

[0096] Step 410: Obtain a first slicing scheme for a first video frame.

[0097] Among them, the first video frame is a video frame that has been encoded. The first slicing scheme is used to divide the first video frame into N first slices, where N is a positive integer greater than or equal to 2. The areas of the N first slices can be the same. For example, when the first video frame is a key frame I frame in a sequence, that is, when the first video frame is difficult to refer to the previous frame or the previous frames with large differences for predictive encoding, the embodiment of the present application can adopt an average slicing segmentation method for the first video frame. The areas of the N first slices can also be different. For example, when the first video frame is a P frame or a B frame, the areas of the N first slices correspond to the actual video content changes of the first video frame, that is, the more complex the changes in the included video content, the smaller the area of ​​the first slice.

[0098] Step 420: Determine the coding performance prediction values ​​of the N second slices obtained when the second video frame adopts the first slicing scheme.

[0099] The second video frame is the current video frame, i.e., the video frame to be encoded. The second video frame can be the frame following the first video frame, or any frame arranged after the first video frame in the video frame sequence. In other words, the first video frame is any encoded video frame before the second video frame. In an embodiment of the present application, the second video frame is preferably the frame following the first video frame. This allows for a smoother process for determining or adjusting the slicing scheme, thereby improving the encoding performance of the entire video.

[0100] Coding performance can be any one or more of coding delay, coding speed, coding complexity, coding efficiency, and adaptive coding control. For example, in scenarios with high latency requirements, such as live streaming, video conferencing, and cloud gaming, coding performance can be coding delay or coding speed. That is, higher coding performance means lower coding delay or faster coding speed. In scenarios with high video quality and latency requirements, such as telemedicine, coding performance can be coding efficiency or compression ratio. That is, higher coding performance means higher coding efficiency or higher compression ratio.

[0101] The predictive coding performance is the predictive coding performance of N second slices obtained by using the first slicing scheme for the second video frame. In the embodiment of the present application, the first slicing scheme is adopted because, according to the above description of the sequence, if the second video frame and the first video frame are in the same sequence or even adjacent video frames in the same sequence, the difference in pixels, brightness, and color temperature between the two video frames is small, and thus the difference in the slicing schemes of the two video frames is also small.

[0102] The N second slices are the slices determined when the first slicing scheme is used for the second video frame. These N second slices correspond one-to-one to the N first slices, each second slice is located at the same position in the second video frame as the corresponding first slice is located in the first video frame, and the area of ​​each second slice is the same as the area of ​​the corresponding first slice. It is worth noting that these N second slices are not obtained by actually segmenting the second video frame, but are reference slices determined according to the slicing scheme of the first slicing scheme for predicting coding performance.

[0103] In the embodiment of the present application, the method for determining the coding performance prediction value of each second slice can be based on an empirical formula, a fitting formula, or a prediction model, etc. The method for predicting coding performance is described in detail below and is not repeated here.

[0104] Step 430: Determine a second slicing scheme for the second video frame according to the coding performance prediction values ​​of the N second slices.

[0105] According to the coding performance prediction values ​​of each second slice in the second video frame obtained in step 420, it can be determined whether there is a problem of uneven coding performance among the N second slices. If there is an imbalance problem, that is, the predicted coding performance of some second slices is much lower than the predicted coding performance of other second slices, then the embodiment of the present application can adjust the first slicing scheme, that is, reduce the area of ​​the part of the second slices, and finally obtain a second slicing scheme with balanced coding performance. The specific adjustment method will be introduced below and will not be described in detail here. In other embodiments of the present application, if the predicted coding performance of each slice is relatively balanced or the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices is small, then the second slicing scheme can directly reuse the first slicing scheme.

[0106] Step 440: Divide the second video frame into N third slices according to the second slicing scheme, and encode the N third slices respectively.

[0107] If the second slicing scheme is adjusted relative to the first slicing scheme, the area of ​​the N third slices is different from the area of ​​the N second slices, and the encoding performance of the N third slices is more balanced. If the second slicing scheme is the same as the first slicing scheme, the area of ​​the N third slices is the same as the area of ​​the N second slices. By encoding the N third slices in parallel, the encoding speed is accelerated, and if a coding error occurs in one or more third slices, the error can be limited to the one or more third slices and will not spread to other third slices, thereby reducing the area of ​​the screen artifact during decoding.

[0108] The embodiments of the present application determine the predicted coding performance of the current video frame, i.e., the second video frame, and determine a second slicing scheme that is suitable for the second video frame and has more balanced coding performance, thereby significantly improving the coding performance of the entire video frame and even the video.

[0109] Method 400 can be used in any real-time video encoding scenario, such as the aforementioned video conferencing, online education, remote terminals, interactive gaming, e-commerce live streaming, and telemedicine. For example, in a real-time interactive cloud gaming scenario, game content is rendered and generated on a cloud server, and players perform game operations on the client. The client uploads the operating instructions to the cloud, and the cloud generates game content based on the user's operating instructions. The game content is then encoded and transmitted to the client via the network. For another example, in live broadcasts and video conferencing scenarios, the host or anchor needs to interact with multiple parties in real time via video. In these scenarios, high encoding speed and low encoding latency are both important.

[0110] The method provided in this application can be used in various real-time video applications, including but not limited to video conferencing applications, Internet phone applications, and streaming media applications. Related technical products may include codecs, video conferencing software, streaming media servers, etc. The method provided in this application can be implemented through hardware devices or software products, or through a software platform provided as a service or a software development kit (SDK) supporting cloud services.

[0111] Correspondingly, FIG5 shows a schematic flow chart of a video decoding method 500 provided in an embodiment of the present application, wherein the method 500 includes steps 510 to 530. The apparatus implementing the method 500 may be the decoding unit 170 shown in FIG1 .

[0112] Step 510: Obtain a second encoded video frame, wherein the second video frame includes the N third slices obtained through the parallel encoding described above.

[0113] Step 520: Decode the N encoded third slices in the second video frame to obtain N video frame slices.

[0114] Step 530: splice the N video frame slices to obtain the decoded second video frame.

[0115] The N video slices may be spliced ​​by calculating the position of each slice in the video frame according to the position field first_mb_in_slice of the first macroblock of each slice, and splicing the N video slices one by one according to the order of the positions.

[0116] The following will introduce the implementation process of method 400 or step 420 and step 430 in conjunction with specific embodiments.

[0117] FIG6 shows a schematic block diagram of a slicing scheme determined according to an embodiment of the present application.

[0118] As shown in FIG6(a), a first video frame is segmented into N first slices according to a first slicing scheme. In step 420, as shown in FIG6(b), the second video frame is segmented using the first slicing scheme to obtain N second slices. The dotted lines in FIG6(b) indicate that the N second slices in the figure are not actual segmentations of the second video frame, but rather serve as reference slices for simulating the segmentation of the second video frame when predicting coding performance. In other words, the N second slices serve only as reference slices in the prediction process and are not slices obtained during the actual segmentation process.

[0119] Using empirical formulas, fitting formulas, or prediction models, the predicted coding performance of the N second slices is ultimately obtained, such as the coding delay shown in Figure 6. The coding delay of second slice #2 is higher than that of second slice #1 and significantly higher than that of second slice #N. The following describes in detail the technical solution for determining the predicted coding performance values ​​in embodiments of the present application.

[0120] Optionally, in an embodiment of the present application, a prediction model can be used to predict coding performance. The prediction model can be an artificial neural network model or a decision tree model such as random forest, XGBoost, etc. When training the prediction model, the input of the prediction model can be the relevant parameters of each slice in the k1-th frame video frame, such as the coding performance of each slice, such as the coding delay of each slice, the number of bytes after encoding, and any one or more of the coding parameters such as the coding bit rate, the number of macroblocks or pixels included, and the coding type. The output of the prediction model can be the actual coding performance of each slice obtained by slicing the k2-th frame video frame according to the slicing scheme of the k1-th frame. In this way, when using the prediction model for prediction, the coding performance prediction values ​​of the N second slices can be determined based on the values ​​of the relevant parameters of the N first slices.

[0121] Optionally, in an embodiment of the present application, the trained fitting formula can be used to predict the coding performance of the N second slices. Taking the coding delay shown in FIG6(b) as a consideration of coding performance, in order to accurately predict the dependent variable, that is, the coding delay of each second slice, it is necessary to select a suitable independent variable.

[0122] For example, an example of a fitting formula is given below. Through training and fitting of numerical experiments, the embodiment of the present application obtains that the encoding delay of each macroblock is logarithmically related to the number of encoded bytes, wherein the number of encoded bytes of the macroblock is related to the amount of motion and details of the image area in the macroblock. Figure 7 shows a schematic diagram of the fitting results of the encoding delay and the number of encoded bytes of each macroblock at 720p and 4K video resolutions, which includes scattered point values ​​and fitting formulas. According to Figure 7, it can be seen that the fitting results of the fitting formula are relatively small in most cases. In fact, other cases with large errors are mostly when scene changes and sequence switches occur in the video, that is, when the current frame is a key frame I frame.

[0123] The two fitting formulas in FIG7 can be expressed by the following formula 1: macroblock_time = a × ln (macroblock_size) + b (Formula 1)

[0124] Where macroblock_size is the number of bytes after macroblock encoding (bytes), and macroblock_time is the macroblock encoding delay (ms). The values ​​of a and b depend not only on information such as video resolution, but also on the platform performing video encoding, such as different types or models of video encoders or different devices equipped with the same type or model of video encoder.

[0125] Formula 1 indicates that the encoding time of a macroblock is related to the number of bytes after the macroblock is encoded. Moreover, since the degree of variation between video frames in the same sequence is small, the number of bytes after the macroblock of the first video frame is encoded is substantially the same as the number of bytes after the macroblock at the same position in the second video frame is encoded. Therefore, when predicting the encoding delay of each second slice, the embodiments of the present application can reuse the number of bytes after the macroblock of each first slice is encoded as the predicted number of bytes for the macroblock at the same position in the corresponding second slice.

[0126] Finally, the predicted coding delay of the second slice is shown in Formula 2:

[0127] Among them, pre_slice_time is the coding delay prediction value of the second slice (ms), macroblock_size i is the predicted coded bytes of the i-th macroblock in the second slice, that is, the actual coded bytes of the i-th macroblock in the first slice corresponding to the second slice, where the second slice includes a total of n macroblocks.

[0128] In other embodiments of the present application, to save computational time during the encoding process, the number of coded bytes of multiple macroblocks in the second slice can be considered approximately the same. That is, the predicted number of coded bytes for each macroblock is the ratio of the predicted total number of coded bytes of the second slice to the number of macroblocks in the second slice, or in other words, the ratio of the total number of coded bytes of the first slice corresponding to the second slice to the number of macroblocks. Formula 2 can then be equivalently converted to Formula 3 as shown below:

[0129] Wherein, slice_size is the predicted total number of coded bytes of the second slice, that is, the total number of coded bytes of the corresponding first slice, and num_of_macroblocks is the number of macroblocks in the second slice.

[0130] Optionally, in other embodiments of the present application, Formula 3 can be trained or fitted based on the historical data used to train the prediction model described above, ultimately obtaining specific values ​​of a and b. Step 420 can predict the coding delay prediction value of the second slice using Formula 3 including the specific values ​​of a and b.

[0131] In other embodiments of the present application, if it is only necessary to compare the order of the predicted coding delays between the second slices without predicting specific values, a and b can be directly constant values, for example, a is 1 and b is 0. In this case, Formula 3 can be converted to Formula 4 as shown below:

[0132] Using Equation 4, the relationship between the predicted coding delays of the second slices can be determined. For example, the relationship between the coding delays of the second slices shown in Figure 6(b) is: second slice #2 > second slice #1 > second slice #N. When subsequently determining the second slicing scheme, the areas of the N second slices, or the areas of the N first slices corresponding to the N second slices, can be adjusted only for the second slices ranked higher or lower in the size relationship.

[0133] In step 430, according to the coding delay prediction values ​​t1 to t N , determine a second slicing scheme. The second slicing scheme may be determined by adjusting the first slicing scheme, such as the second slicing scheme shown in FIG6(c), or may be a reuse of the first slicing scheme.

[0134] Optionally, in an embodiment of the present application, before determining the second slicing scheme, method 400 may further include determining whether the predicted coding delays of the N second slices are balanced, such as whether the difference between the maximum and minimum values ​​of the predicted coding delays of the N second slices is greater than a target difference threshold. For example, determine whether the difference between the minimum and maximum values ​​of the predicted coding delays of the N second slices is greater than 20% of the maximum value, or whether the difference between the maximum and minimum values ​​is greater than 100ms. When it is determined that the predicted coding delays of the N second slices are not balanced, it is determined that the first slicing scheme needs to be adjusted to determine the second slicing scheme. In this way, the frequent adjustment of the slicing scheme during the encoding process of continuous video frames can be reduced.

[0135] An embodiment of adjusting the first slicing scheme to obtain the second slicing scheme will be specifically described below.

[0136] FIG8 is a schematic flow chart of a second slicing scheme according to an embodiment of the present application, wherein the dotted lines represent optional steps. In the case where the coding performance prediction value is positively correlated with the coding performance, the second slice with the largest coding performance prediction value has the best coding performance; in the case where the coding performance prediction value is negatively correlated with the coding performance, the second slice with the smallest coding performance prediction value has the worst coding performance.

[0137] If the coding delay is used as a consideration for coding performance, when it is determined that the second video frame is not a key frame I frame and that the predicted coding performance (i.e., coding delay) of the N second slices is not balanced, the N second slices are traversed. First, the second slice with the worst coding performance, i.e., the highest coding delay, is determined, such as the second slice #2 shown in (b) of Figure 6, and the area of ​​the second slice #2 is adjusted, for example, by reducing the number of macroblocks or reducing the number of slice rows. Preferably, in order to obtain higher adjustment efficiency, the embodiment of the present application adjusts the area of ​​the second slice by reducing K slice rows. As shown in (c) of Figure 6, the third slice #2 has 1 fewer slice rows than the second slice #2. In an embodiment of the present application, the value of K is related to the difference between the maximum and minimum values ​​of the coding performance prediction value, for example, the larger the difference, the larger the value of K. In other embodiments of the present application, in order to reduce the amplitude of the slicing scheme adjustment to avoid frequent adjustments, the value of K can be 1.

[0138] Optionally, the macroblocks or slice rows reduced in the second slice #2 may be evenly divided into other second slices, so that one or more macroblocks are added to part or all of the other second slices.

[0139] Optionally, as shown in FIG8 , embodiments of the present application may further determine a second slice with optimal coding performance, i.e., a minimum coding delay, i.e., second slice #N shown in FIG6( b ), and divide the reduced macroblocks or slice rows in second slice #2 into second slice #N, while the areas of the other second slices may remain unchanged. As shown in FIG6( c ), third slice #N has one more slice row than second slice #N.

[0140] Finally, as shown in (c) in Figure 6, the second slicing scheme is determined after adjusting the first slicing scheme, and the second video frame is sliced ​​and divided according to the second slicing scheme to obtain N third slices. In this way, compared with the encoding performance of the N first slices, the encoding performance of the N third slices is more balanced, and the total encoding delay shown in (c) in Figure 6 is lower than the total encoding delay shown in (b) in Figure 6. Similarly, if the encoding performance of the N third slices in the second video frame still does not meet the balanced requirements, the present application determines the third slicing scheme when encoding the third video frame. As shown in (d) in Figure 6, the fourth slice #1 and the fourth slice #2 in the third video frame are reduced by 1 slice row compared to the third slice #1 and the third slice #2, and the fourth slice #N has an additional slice row compared to the third slice #N.

[0141] Optionally, the method for adding or removing slice rows can be to directly adjust the slice boundaries between the slices. Figure 9 shows a schematic diagram of a frame-by-frame change in a slicing scheme. As shown in Figure 9 (a), video frame A is divided into slices #A to #D. Figures 9 (b), (c), and (d) respectively illustrate different slice boundary adjustment methods.

[0142] For example, in combination with Formula 4, for a 1024×1920 video, assuming that the video frame is divided into 4 slices as shown in Figure 9, and each macroblock contains 64×64 pixels, then each video frame contains a total of 480 macroblocks. Table 1 shows a data table for encoding performance prediction corresponding to video frame A. The second and third columns of Table 1 show the number of encoded bytes and the number of macroblocks contained in each slice in video frame A, and the fourth column of Table 1 shows the predicted encoding delay of video frame B using the slicing scheme of video frame A calculated based on Formula 4. The number of macroblocks can be obtained based on the slice_segment_address field in the video analyzer, and the number of encoded bytes of the slice can be obtained through the slice size in the video analysis tool.

[0143] Table 1

[0144] According to the predicted coding delay in the fourth column of Table 1 and the process of adjusting the slice area in Figure 8, it is necessary to reduce slice #A by 1 slice row and increase slice #B by 1 row. As shown in (b) of Figure 9, by moving the slice boundary AB between slice #A and slice #B in the direction of slice #A by 1 row, the slicing scheme of video frame B is obtained. In other words, if the two slices that need to be adjusted are adjacent slices, the slice boundary between the two slices can be moved by 1 slice row in the direction of the slice with the worst coding performance. After encoding video frame B, the data table of coding performance prediction corresponding to video frame B can be obtained as shown in Table 2, where the fourth column of Table 2 is the predicted coding delay when video frame C adopts the slicing scheme of video frame B.

[0145] Table 2

[0146] According to the predicted coding delay shown in the fourth column of Table 2, the difference between the maximum predicted coding delay of 360ms and the minimum predicted coding delay of 309ms does not reach 20% of the maximum value of 360ms, or 100ms. Therefore, as shown in Figure 9(c), the slicing scheme of video frame C is not adjusted based on the slicing scheme of video frame B. After encoding video frame C, the coding performance prediction data table corresponding to video frame C is obtained, as shown in Table 3. The fourth column of Table 3 shows the predicted coding delay when video frame D adopts the slicing scheme of video frame C.

[0147] Table 3

[0148] Based on the predicted coding delay in the fourth column of Table 3 and the process for adjusting the slice area in Figure 8, it is necessary to reduce slice #A by one slice row and increase slice #D by one slice row. As shown in (d) of Figure 9, by moving all slice boundaries between slice #A and slice #D (slice boundaries AB, BC, and CD) by one row toward slice #A, the slicing scheme for video frame D is obtained. In other words, if the two slices to be adjusted are not adjacent slices, all slice boundaries between the two slices can be moved by one slice row toward the slice with the worst coding performance. This achieves smooth, row-by-row movement of slice boundaries between consecutive frames.

[0149] This application determines the predicted coding performance of the current video frame and determines the slicing scheme corresponding to the current video frame, so that the coding performance of each slice of the current video frame is more balanced after slicing, and the overall coding performance of the current video frame is improved.

[0150] The following will introduce the beneficial effects of the embodiments of the present application in combination with specific experimental results.

[0151] Tables 4 and 5 show experimental results for videos of different types or resolutions provided by an embodiment of the present application.

[0152] Table 4 shows the encoding speed (frames per second, FPS) and encoding quality (quality) of 18 videos using the video encoding method of this application, for both live show and live streaming sales scenarios. The resolutions of these 18 videos range from 480×720 to 1080×1920. As shown in Table 4, in both live show and live streaming sales scenarios, the average encoding speed of the 18 videos increased by 6%, with the largest increase of 19%, while the average encoding quality remained almost unchanged.

[0153] Table 4

[0154] Table 5 shows the encoding speed (frames per second, FPS) and encoding quality of 30 videos in a gaming scenario using the video encoding method of this application. The resolutions of these 30 videos are categorized as less than 2K, 2K, and 4K. As shown in Table 5, in the gaming scenario, the average encoding speed of the 30 videos increased by 3% to 4%, with the largest increase of 10%, while the average encoding quality remained almost unchanged.

[0155] Table 5

[0156] Figure 10 is a schematic diagram of the experimental results of a video comparison experiment provided by an embodiment of the present application. Figure 10 shows the encoding delay of each slice in multiple video frames of the experimental video, wherein the experimental results of Figure 10 are obtained by applying a video encoding method including a traditional equal-slicing and fixed slicing scheme and a video encoding method including the dynamic slicing scheme of the present application to the video frames in the same video.

[0157] As shown in (a) of Figure 10, when the traditional evenly divided and fixed slicing scheme is adopted, the encoding delays of the four slices are not balanced. For example, the encoding delays of slices #A and #C are about 5s to 15s, the encoding delay of slice #B is about 15s to 25s, and the encoding delay of slice #D is concentrated in the range of 5s to 10s. As shown in (b) of Figure 10, when the dynamic slicing scheme in the present application is adopted, the encoding delays of slices #A to #D are relatively balanced, and the encoding delays are all concentrated in the range of 10s to 15s. Therefore, the dynamic slicing method of the present application can balance the encoding delays of each slice and significantly reduce the total encoding delay of the video frame, that is, the encoding delay of the slice that completes the encoding process last.

[0158] Figure 11 is a schematic diagram of another experimental result of the above-mentioned video comparison experiment. Figure 11 shows the number of encoded bytes for each slice in multiple video frames, wherein the experimental results of Figure 11 were obtained by applying a video encoding method including a traditional evenly divided and fixed slicing scheme and a video encoding method including the dynamic slicing scheme of the present application to the video frames in the same experimental video in Figure 10.

[0159] As shown in (a) of Figure 11, when the traditional evenly divided and fixed slicing scheme is adopted, the number of coded bytes of the four slices is not balanced, the number of coded bytes of slice #D is much smaller than that of slice #A and slice #B, and the change or fluctuation of the number of coded bytes of slice #A is more drastic. As shown in (b) of Figure 11, when the dynamic slicing scheme in the present application is adopted, the number of coded bytes of slices #A to slice #C is mainly concentrated in the range of 2000 bytes to 4000 bytes, and the number of coded bytes of slice D is mainly concentrated in the range of 1000 bytes to 2000 bytes, and except for some fluctuation values, the degree of change in the number of coded bytes of slice #A gradually decreases. Therefore, the dynamic slicing scheme of the present application achieves a balance in the number of coded bytes of each slice.

[0160] Figure 12 is a diagram showing another experimental result of the above video comparison experiment. Figure 12 shows the ratio of the maximum encoding delay to the minimum encoding delay and the ratio of the maximum encoding byte count to the minimum encoding byte count of each slice in each video frame of the experimental video.

[0161] As shown in (a) of Figure 12, when the traditional evenly divided and fixed slicing scheme is adopted, the maximum value of the ratio of the maximum coding delay to the minimum coding delay is about 7, and the average value is about 3.4. When the dynamic slicing scheme of the present application is adopted, the ratio of the maximum coding delay to the minimum coding delay of each video frame is mostly between 1 and 2, with the maximum value of the ratio being about 3 and the average value being 1.7. It can be seen that when the scheme of the present application is adopted, the maximum and average values ​​of the ratio of the maximum coding delay to the minimum coding delay are both reduced by 2 times.

[0162] As shown in (b) of Figure 12, when the traditional evenly divided and fixed slicing scheme is adopted, the maximum value of the ratio of the maximum number of coded bytes to the minimum number of coded bytes is about 180, and the average value is about 60. When the dynamic slicing scheme of the present application is adopted, the ratio of the maximum number of coded bytes to the minimum number of coded bytes of each video frame is mostly between 5 and 10, and the maximum value of the ratio is about 30. It can be seen that when the scheme of the present application is adopted, the ratio of the maximum number of coded bytes to the minimum number of coded bytes decreases more significantly, that is, the distribution of the number of coded bytes of each slice in the same video frame is more balanced.

[0163] Figure 13 is a schematic diagram of the experimental results of another video comparison experiment. The video comparison experiment compares the total encoding delay (i.e., the sum of the encoding delays of all video frames) of multiple videos with a resolution of less than 2K and a resolution of 4K respectively. As shown in (a) in Figure 13, compared with the traditional video encoding method, when the video encoding method of the present application is adopted, the total encoding delay of 5 videos with a resolution of less than 2K is reduced by 11.2% at most and 2.4% at least. As shown in (b) in Figure 13, when the solution of the present application is adopted, the total encoding delay of 3 videos with a resolution of 4K is reduced by 10.8% at most and 9.3% at least. It can be seen that the technical solution of the present application can significantly reduce the total encoding delay of the video.

[0164] The above describes the video encoding and decoding method according to the embodiment of the present application. The following describes the apparatus and device according to the embodiment of the present application in conjunction with Figures 14 to 18 respectively.

[0165] 14 is a structural diagram of a video encoding apparatus 1400 according to an embodiment of the present application. The video encoding apparatus 1400 includes an acquisition module 1410 , a processing module 1420 , and an encoding module 1430 .

[0166] The acquisition module 1410 is configured to acquire a first slicing scheme for the first video frame and execute step 410 shown in FIG. 4 .

[0167] The processing module 1420 is used to determine the coding performance prediction values ​​of N second slices obtained when the second video frame adopts the first slicing scheme, and determine the second slicing scheme of the second video frame based on the coding performance prediction values ​​of the N second slices.

[0168] The encoding module 1430 is configured to divide the second video frame into N third slices according to the second slicing scheme, and encode the N third slices respectively.

[0169] 15 is a structural diagram of a video decoding device 1500 according to an embodiment of the present application. The video decoding device 1500 includes an acquisition module 1510 and a decoding module 1520 .

[0170] The acquisition module 1510 is configured to acquire the encoded second video frame, where the second video frame includes the encoded N third slices.

[0171] The decoding module 1520 is configured to decode the encoded N third slices to obtain N video frame slices, and splice the N video frame slices to obtain a decoded second video frame.

[0172] The above modules can be implemented in software or hardware. For example, the implementation of processing module 1420 will be described below using processing module 1420 as an example. Similarly, the implementation of acquisition modules 1410 and 1510, decoding module 1520, and encoding module 1430 can refer to the implementation of processing module 1420.

[0173] As an example of a software functional unit, the processing module 1420 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the processing module 1420 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0174] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0175] As an example of a hardware functional unit, processing module 1420 may include at least one computing device, such as a server. Alternatively, processing module 1420 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0176] The multiple computing devices included in processing module 1420 can be distributed in the same region or in different regions. The multiple computing devices included in processing module 1420 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in processing module 1420 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices included in processing module 1420 can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0177] This application also provides a computing device 1600. As shown in Figure 16, computing device 1600 includes a bus 1602, a processor 1604, a memory 1606, and a communication interface 1608. Processor 1604, memory 1606, and communication interface 1608 communicate with each other via bus 1602. Computing device 1600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1600.

[0178] Bus 1602 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG16 shows only one line, but this does not imply a single bus or type of bus. Bus 1602 may include a path for transmitting information between various components of computing device 1600 (e.g., memory 1606, processor 1604, and communication interface 1608).

[0179] The processor 1604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0180] Memory 1606 may include volatile memory, such as random access memory (RAM). Memory 1606 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0181] Memory 1606 stores executable program code. Processor 1604 executes the executable program code to implement the functions of acquisition module 1410 and acquisition module 1510, processing module 1420, decoding module 1520, and encoding module 1430, thereby implementing the aforementioned video encoding method. In other words, memory 1606 stores instructions for executing the aforementioned video encoding method.

[0182] The communication interface 1608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1600 and other devices or a communication network.

[0183] FIG17 of an embodiment of the present application further provides a computing device cluster. The computing device cluster includes at least two computing devices. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0184] As shown in Figure 17, the computing device cluster includes at least two computing devices 1600. The memories 1606 in the multiple computing devices 1600 in the computing device cluster may store the same instructions for executing the above-mentioned video encoding method or video decoding method.

[0185] In some possible implementations, the memories 1606 of the multiple computing devices 1600 in the computing device cluster may also each store some instructions for executing the above-described video encoding method. In other words, a combination of one or more computing devices 1600 may jointly execute instructions for executing the above-described video encoding method.

[0186] It should be noted that the memory 1606 in different computing devices 1600 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1606 in different computing devices 1600 can implement the functions of one or more modules among the acquisition module, the processing module, and the encoding module.

[0187] In some possible implementations, multiple computing devices in a computing device cluster can be connected via a network. The network can be a wide area network or a local area network, etc. Figure 18 shows a possible implementation. As shown in Figure 18, two computing devices 1600A and 1600B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 1606 in the computing device 1600A stores instructions for the functions of the acquisition module. At the same time, the memory 1606 in the computing device 1600B stores instructions for executing the functions of the processing module and the encoding module.

[0188] It should be understood that the functionality of the computing device 1600A shown in FIG18 may also be implemented by multiple computing devices 1600. Similarly, the functionality of the computing device 1600B may also be implemented by multiple computing devices 1600.

[0189] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the above-mentioned video encoding method.

[0190] The present application also provides a computer program product including instructions. This computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When this computer program product is run on at least one computing device, it causes the at least one computing device to perform the above-described video encoding method.

[0191] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-described video encoding method.

[0192] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

[0194] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0196] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0197] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0198] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0199] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for video encoding, It is characterized in that The method comprises: Obtain a first slicing scheme for a first video frame, where the first slicing scheme is used to divide the first video frame into N first slices, where N is a positive integer greater than or equal to 2; Determine coding performance prediction values ​​of N second slices obtained when the second video frame adopts the first slicing scheme, where the N second slices correspond to the N first slices one-to-one, a position of each second slice in the second video frame is the same as a position of a corresponding first slice in the first video frame, and an area of ​​each second slice is the same as an area of ​​the corresponding first slice; Determining a second slicing scheme for the second video frame according to the coding performance prediction values ​​of the N second slices, where the second slicing scheme is used to divide the second video frame into N third slices; According to the second slicing scheme, the second video frame is divided into the N third slices; The N third slices are respectively encoded to obtain the encoded second video frame.

2. The method according to claim 1, It is characterized in that The determining, according to the coding performance prediction values ​​of the N second slices, a second slicing scheme of the second video frame comprises: According to the coding performance prediction values ​​of the N second slices, the areas of some or all of the N second slices are adjusted to obtain the second slicing scheme.

3. The method according to claim 2, It is characterized in that The adjusting the areas of some or all of the N second slices includes: Determine a fourth slice from the N second slices, wherein when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fourth slice is the smallest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fourth slice is the largest; The area of ​​the fourth slice is reduced.

4. The method according to claim 3, It is characterized in that The reducing the area of ​​the fourth slice comprises: A slice boundary between the fourth slice and at least one adjacent second slice is moved by K slice rows in a direction close to the fourth slice, where K is a positive integer.

5. The method according to claim 3, It is characterized in that The method further comprises: Determine a fifth slice from the N second slices, wherein when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fifth slice is the largest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fifth slice is the smallest; The reducing the area of ​​the fourth slice comprises: One or more slice boundaries between the fourth slice and the fifth slice are moved toward the fourth slice by K slice rows, where K is a positive integer.

6. The method according to claim 4 or 5, It is characterized in that The value of K is 1.

7. The method according to claim 4 or 5, It is characterized in that The value of K is positively correlated with the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices.

8. The method according to any one of claims 2 to 7, It is characterized in that Before determining the second slicing scheme of the second video frame, the method further includes: It is determined that a difference between a maximum value and a minimum value of the coding performance prediction values ​​of the N second slices reaches a target difference threshold.

9. The method according to any one of claims 1 to 8, It is characterized in that The determining of the coding performance prediction values ​​of N second slices obtained when the second video frame adopts the first slicing scheme includes: According to the target prediction model and the values ​​of the coding parameters and / or coding performance of the i-th first slice among the N first slices, determine the coding performance prediction value of the i-th second slice among the N second slices corresponding to the i-th first slice, where i is a positive integer less than or equal to N.

10. The method according to claim 9, It is characterized in that The coding performance prediction value is a coding delay prediction value, the target prediction model is a target prediction formula, and determining the coding performance prediction value of the i-th second slice among the N second slices corresponding to the i-th first slice according to the target prediction model and the coding parameter and / or coding performance value of the i-th first slice among the N first slices, includes: Obtaining the number of bytes after encoding the i-th first slice and the number of macroblocks in the i-th first slice; Determine a coding delay prediction value of the i-th second slice corresponding to the i-th first slice according to the target prediction formula, the number of bytes after encoding the i-th first slice, and the number of macroblocks in the i-th first slice.

11. A video encoding device, It is characterized in that The device comprises: An acquisition module, used to acquire a first slicing scheme of a first video frame, where the first slicing scheme is used to divide the first video frame into N first slices, where N is a positive integer greater than or equal to 2; a processing module, configured to determine coding performance prediction values ​​of N second slices obtained when the second video frame adopts the first slicing scheme, the N second slices correspond to the N first slices one-to-one, a position of each second slice in the second video frame is the same as a position of a corresponding first slice in the first video frame, and an area of ​​each second slice is the same as an area of ​​the corresponding first slice, and determine a second slicing scheme for the second video frame according to the first slicing scheme and the coding performance prediction values ​​of the N second slices, the second slicing scheme being used to divide the second video frame into N third slices; An encoding module is used to divide the second video frame into the N third slices according to the second slicing scheme, and encode the N third slices respectively to obtain the encoded second video frame.

12. The device according to claim 11, It is characterized in that The processing module is specifically configured to adjust the areas of some or all of the N second slices according to the coding performance prediction values ​​of the N second slices to obtain the second slicing scheme.

13. The device according to claim 12, It is characterized in that The processing module is specifically used for: Determine a fourth slice from the N second slices, wherein when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fourth slice is the smallest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fourth slice is the largest; The area of ​​the fourth slice is reduced.

14. The device according to claim 13, It is characterized in that The processing module is specifically used for: A slice boundary between the fourth slice and at least one adjacent second slice is moved by K slice rows in a direction close to the fourth slice.

15. The device according to claim 13, It is characterized in that The processing module is also used for: Determine a fifth slice from the N second slices, wherein when the coding performance prediction value is positively correlated with the coding performance, the coding performance prediction value of the fifth slice is the largest, and when the coding performance prediction value is negatively correlated with the coding performance, the coding performance prediction value of the fifth slice is the smallest; The processing module is specifically used for: One or more slice boundaries between the fourth slice and the fifth slice are moved toward the fourth slice by K slice rows.

16. The device according to claim 14 or 15, It is characterized in that The value of K is 1.

17. The device according to claim 14 or 15, It is characterized in that The value of K is positively correlated with the difference between the maximum and minimum values ​​of the coding performance prediction values ​​of the N second slices.

18. The device according to any one of claims 12 to 17, It is characterized in that Before determining the second slicing scheme of the second video frame, the processing module is further used for: It is determined that a difference between a maximum value and a minimum value of the coding performance prediction values ​​of the N second slices reaches a target difference threshold.

19. The device according to any one of claims 11 to 18, It is characterized in that The processing module is specifically used for: According to the target prediction model and the values ​​of the coding parameters and / or coding performance of the i-th first slice among the N first slices, determine the coding performance prediction value of the i-th second slice among the N second slices corresponding to the i-th first slice, where i is a positive integer less than or equal to N.

20. The device according to claim 19, It is characterized in that The coding performance prediction value is a coding delay prediction value, the target prediction model is a target prediction formula, and the processing module is specifically used for: Obtaining the number of bytes after encoding the i-th first slice and the number of macroblocks in the i-th first slice; Determine a coding delay prediction value of the i-th second slice corresponding to the i-th first slice according to the target prediction formula, the number of bytes after encoding the i-th first slice, and the number of macroblocks in the i-th first slice.

21. A computing device cluster, It is characterized in that include: at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device so that The computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, It is characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, It is characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images