Video coding method and related equipment

Abstract

The invention discloses a video coding method and related equipment. The method comprises the following steps: acquiring a video frame and information of the video frame; dividing the video frame to obtain prediction unit (PU) blocks; according to the information of the video frame, obtaining an initial position of motion search corresponding to each PU block; based on the initial position of the motion search corresponding to each PU block, motion search is carried out on the PU blocks in parallel; and coding the video frame based on a result after the motion search. Motion search can be carried out on PU blocks in parallel, so that coding time delay is reduced.

Description

[0001] This application claims the priority of a Chinese patent application with the application number 202410094522.6, titled "A Video Coding Method, Apparatus and System Based on Generative Technology", which was filed with the National Intellectual Property Administration on January 23, 2024, and the entire content of which is incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular, to a video coding method and related devices. Background Art

[0003] Since users' requirements for video service experience are getting higher and higher, on the one hand, it promotes the development of videos towards high resolution, such as 8K, 12K, 3D and higher frame rates, making video files larger and larger; on the other hand, it promotes the gradual transformation of video services from traditional one-way video playback to real-time interaction, such as Augmented Reality (AR), Virtual Reality (VR), interactive live broadcast, cloud games, multi-person virtual collaborative design, etc., making the latency requirements for videos lower and lower.

[0004] Facing the gradually increasing video files, generally, the video files will be encoded and then pushed to the terminal device through network streaming for decoding and display. In the current coding protocol, generally, serial prediction processing is performed on the prediction unit (PU), resulting in a large coding latency. Summary of the Invention

[0005] This application provides a video coding method and related devices, which can perform motion search on PU blocks in parallel, thereby reducing the coding latency.

[0006] In a first aspect, this application provides a video coding method, which includes: obtaining a video frame and information of the video frame; dividing the video frame to obtain prediction unit (PU) blocks; obtaining the starting positions of motion search respectively corresponding to each PU block according to the information of the video frame; based on the starting positions of motion search respectively corresponding to each PU block, performing motion search on the PU blocks in parallel; encoding the video frame based on the results after motion search. Based on the above technical solution, when performing video coding, starting from the starting positions of motion search respectively corresponding to each obtained PU block, motion search is performed on the PU blocks in parallel, thereby eliminating the dependency relationship between PU blocks during motion search, improving the coding parallelism, and reducing the coding latency.

[0007] In a possible implementation manner of the first aspect, the information of the video frame includes the motion vectors of each pixel included in each PU block, and obtaining the starting positions of motion search respectively corresponding to each PU block according to the information of the video frame includes:

[0008] According to the motion vectors of the pixels included in each PU block, the starting positions of the motion search corresponding to each PU block are respectively obtained.

[0009] Based on the above technical solution, by respectively obtaining the starting positions of the motion search corresponding to each PU block according to the motion vectors of the pixels included in each PU block, and using the starting positions of the motion search corresponding to each PU block as the starting points of the motion search, it is not only possible to help search for accurate motion vectors, but also to eliminate the dependency relationship between PU blocks during motion search, thereby achieving low-latency coding.

[0010] In a possible implementation manner of the first aspect, the starting position of the motion search corresponding to any PU block can be determined based on the motion vectors of all the pixels included in the PU block.

[0011] In a possible implementation manner of the first aspect, the starting position of the motion search corresponding to any PU block can be determined based on the motion vectors of some of the pixels included in the PU block.

[0012] In a possible implementation manner of the first aspect, the starting position of the motion search corresponding to any PU block can be determined based on the motion vector of any one of the pixels included in the PU block.

[0013] It should be understood that in actual applications, the starting position of the motion search corresponding to any PU block can be determined based on all, some, any one of the motion vectors or other methods included in the PU block, and the present application does not make any limitations.

[0014] In a possible implementation manner of the first aspect, according to the motion vectors of the pixels included in each PU block, respectively obtaining the starting positions of the motion search corresponding to each PU block includes:

[0015] According to the average value or median value of the motion vectors of all the pixels included in each PU block, respectively obtaining the starting positions of the motion search corresponding to each PU block.

[0016] In a possible implementation manner of the first aspect, according to the motion vectors of the pixels included in each PU block, respectively obtaining the starting positions of the motion search corresponding to each PU block includes:

[0017] Respectively obtaining the maximum value and the minimum value among the motion vectors of all the pixels included in each PU block;

[0018] According to the average value or median value of the motion vectors of all the pixels except the maximum value and the minimum value among the motion vectors of all the pixels included in each PU block, respectively obtaining the starting positions of the motion search corresponding to each PU block.

[0019] Based on the above technical solution, when calculating the starting position of the motion search corresponding to the PU block, the maximum and minimum values of the motion vectors of the pixels included in the PU block are excluded, and the average value or median value of the remaining pixel motion vectors included in the PU block is used as the starting position of the motion search corresponding to the PU block, so as to avoid the problem of large deviation of the overall data caused by large errors in individual samples.

[0020] In a possible implementation manner of the first aspect, the number of PU blocks that perform motion search in parallel at any size within the video frame refers to the number of all PU blocks included in the size.

[0021] Based on the above technical solution, using the number of all PU blocks included in any size as the number of PU blocks that perform motion search in parallel at that size can achieve the maximum coding parallelism and reduce the coding delay.

[0022] In a possible implementation manner of the first aspect, the method further includes:

[0023] Obtain the target coding delay of the video frame;

[0024] Determine the number of PU blocks that perform motion search in parallel according to the target coding delay of the video frame.

[0025] Based on the above technical solution, the number of PU blocks that perform motion search in parallel can be determined based on the configured target coding delay to support different application scenarios and delay requirements, and improve the flexibility of parallel coding and the user experience.

[0026] In a possible implementation manner of the first aspect, the information of the video frame includes the resolution of the video frame. Determining the number of PU blocks that perform motion search in parallel according to the target coding delay of the video frame includes:

[0027] Obtain the number of all PU blocks according to the resolution of the video frame;

[0028] Obtain the average delay of the motion search corresponding to a single PU block;

[0029] Calculate the number of PU blocks that perform motion search in parallel according to the number of all PU blocks, the average delay of the motion search corresponding to a single PU block, and the target coding delay of the video frame.

[0030] In a second aspect, the present application provides a video coding method, and the method includes:

[0031] Obtain video frames and information of the video frames; divide the video frames to obtain prediction unit (PU) blocks of several sizes; obtain the starting positions of motion search respectively corresponding to each PU block under each size according to the information of the video frames; based on the starting positions of motion search respectively corresponding to each PU block under each size, perform motion search on the PU blocks under each size in parallel; encode the video frames based on the results after motion search.

[0032] It should be understood that after determining the starting positions of motion search respectively corresponding to each PU block under each size, motion search can be performed on all or part of the PU blocks under this size respectively.

[0033] Based on the above technical solution, when performing video encoding, taking the starting positions of motion search respectively corresponding to each PU block under each obtained size as the starting points, perform motion search on the PU blocks under each size in parallel, thereby eliminating the dependency relationship between PU blocks during motion search, improving the encoding parallelism, and reducing the encoding delay.

[0034] In a possible implementation manner of the second aspect, the information of the video frames includes the motion vectors of each pixel included in each PU block. Obtaining the starting positions of motion search respectively corresponding to each PU block under each size according to the information of the video frames includes:

[0035] Respectively obtain the starting positions of motion search respectively corresponding to each PU block according to the motion vectors of the pixels included in each PU block.

[0036] Based on the above technical solution, respectively obtaining the starting positions of motion search respectively corresponding to each PU block according to the motion vectors of the pixels included in each PU block, and using the starting positions of motion search respectively corresponding to each PU block as the starting points of motion search can not only help to search for accurate motion vectors, but also eliminate the dependency relationship between PU blocks during motion search, thereby realizing low-delay encoding.

[0037] In a possible implementation manner of the second aspect, the starting position of motion search corresponding to any PU block can be determined based on the motion vectors of all pixels included in this PU block.

[0038] In a possible implementation manner of the second aspect, the starting position of motion search corresponding to any PU block can be determined based on the motion vectors of some pixels included in this PU block.

[0039] In a possible implementation manner of the second aspect, the starting position of motion search corresponding to any PU block can be determined based on the motion vector of any pixel included in this PU block.

[0040] It should be understood that in actual applications, the starting position of the motion search corresponding to any PU block can be determined based on all, part, any motion vector or other means included in the PU block, and the present application does not make any limitations.

[0041] In a possible implementation manner of the second aspect, according to the motion vectors of the pixels included in each PU block, the starting positions of the motion search respectively corresponding to each PU block are obtained, including:

[0042] According to the average value or median value of the motion vectors of all the pixels included in each PU block, the starting positions of the motion search respectively corresponding to each PU block are obtained.

[0043] In a possible implementation manner of the second aspect, according to the motion vectors of the pixels included in each PU block, the starting positions of the motion search respectively corresponding to each PU block are obtained, including:

[0044] Respectively obtain the maximum value and the minimum value in the motion vectors of all the pixels included in each PU block;

[0045] According to the average value or median value of the motion vectors of all the pixels except the maximum value and the minimum value in the motion vectors of all the pixels included in each PU block, the starting positions of the motion search respectively corresponding to each PU block are obtained.

[0046] Based on the above technical solution, when calculating the starting position of the motion search corresponding to the PU block, the maximum value and the minimum value of the motion vectors of the pixels included in the PU block are excluded, and the average value or median value of the remaining motion vectors of the pixels included in the PU block is used as the starting position of the motion search corresponding to the PU block, so as to avoid the problem of large deviation of the overall data caused by large errors of individual samples.

[0047] In a possible implementation manner of the second aspect, the number of PU blocks that perform motion search in parallel in any size refers to the number of all PU blocks included in the size.

[0048] Based on the above technical solution, using the number of all PU blocks included in any size as the number of PU blocks that perform motion search in parallel in the size can achieve the maximum coding parallelism and reduce the coding delay.

[0049] In a possible implementation manner of the second aspect, the method further includes:

[0050] Obtain the target coding delay of the video frame;

[0051] Determine the number of PU blocks that perform motion search in parallel in each size according to the target coding delay of the video frame.

[0052] Based on the above technical solution, the number of PU blocks that perform motion search in parallel at any size can be determined based on the configured target encoding delay, so as to support different application scenarios and delay requirements, and improve the flexibility of parallel encoding and the user experience.

[0053] In a possible implementation manner of the second aspect, the information of the video frame includes the resolution of the video frame. Determining the number of PU blocks that perform motion search in parallel at each size according to the target encoding delay of the video frame includes:

[0054] Obtain the number of all PU blocks included at each size according to the resolution of the video frame;

[0055] Obtain the average delay of the motion search corresponding to a single PU block at each size;

[0056] Calculate and obtain the number of PU blocks that perform motion search in parallel at each size respectively according to the number of all PU blocks included at each size, the average delay of the motion search corresponding to a single PU block at each size, and the target encoding delay of the video frame.

[0057] In a third aspect, the present application provides a video encoding method, and the method includes:

[0058] Obtain a video frame and the information of the video frame;

[0059] Divide the video frame to obtain prediction unit PU blocks;

[0060] Obtain the starting positions of the motion search respectively corresponding to each PU block according to the information of the video frame;

[0061] Obtain the target encoding delay of the video frame;

[0062] Determine the number of PU blocks that perform motion search in parallel according to the target encoding delay of the video frame;

[0063] According to the starting positions of the motion search respectively corresponding to each PU block, perform motion search on the above-mentioned number of PU blocks in parallel;

[0064] Encode the video frame based on the result after the motion search.

[0065] Based on the above technical solution, when performing video encoding, starting from the starting positions of the motion search respectively corresponding to each obtained PU block, perform motion search on the above-mentioned number of PU blocks in parallel, thereby eliminating the dependency relationship between PU blocks during motion search, improving the encoding parallelism, and reducing the encoding delay. In addition, determining the number of motion searches in parallel at each size based on the target encoding delay can meet the user's delay requirements for different application scenarios, and improve the user experience and the flexibility of parallel encoding.

[0066] In a possible implementation of the third aspect, the target encoding delay of the video frame is determined based on user configuration.

[0067] In a possible implementation of the third aspect, the information of the video frame includes the resolution of the video frame. The number of PU blocks for parallel motion search is determined according to the target encoding delay, including:

[0068] Obtain the number of all PU blocks according to the resolution of the video frame;

[0069] Obtain the average delay of the motion search corresponding to a single PU block;

[0070] Calculate the number of PU blocks for parallel motion search respectively according to the number of all PU blocks, the average delay of the motion search corresponding to a single PU block, and the target encoding delay of the video frame.

[0071] In a possible implementation of the third aspect, the information of the video frame includes the motion vectors of the pixels included in each PU block. The starting positions of the motion search corresponding to each PU block are obtained according to the information of the video frame, including:

[0072] Obtain the starting positions of the motion search corresponding to each PU block respectively according to the motion vectors of the pixels included in each PU block.

[0073] In a possible implementation of the third aspect, obtaining the starting positions of the motion search corresponding to each PU block respectively according to the motion vectors of the pixels included in each PU block includes:

[0074] Obtain the starting positions of the motion search corresponding to each PU block respectively according to the average value or median of the motion vectors of all the pixels included in each PU block.

[0075] In a fourth aspect, the present application provides a video encoding method, which includes:

[0076] Obtain a video frame and the information of the video frame;

[0077] Divide the video frame to obtain prediction unit (PU) blocks of several sizes;

[0078] Obtain the starting positions of the motion search corresponding to each PU block of each size according to the information of the video frame;

[0079] Obtain the target encoding delay of the video frame;

[0080] Determine the number of PU blocks for parallel motion search of each size according to the target encoding delay;

[0081] Perform motion search on the number of PU blocks of each size in parallel according to the starting positions of the motion search corresponding to each PU block of each size;

[0082] Encode video frames based on the results after motion search.

[0083] Based on the above technical solution, when performing video encoding, taking the starting positions of motion search corresponding to each PU block at each obtained size as the starting points, motion search is performed on the PU blocks at each size in parallel, thereby eliminating the dependency relationship between PU blocks during motion search, improving the encoding parallelism, and reducing the encoding delay. Additionally, determining the number of PU blocks for parallel motion search at each size based on the target encoding delay can meet the delay requirements of users for different application scenarios, improving the user experience and the flexibility of parallel encoding.

[0084] In a possible implementation manner of the fourth aspect, the information of the video frame includes the resolution of the video frame. Determining the number of PU blocks for parallel motion search at each size according to the target encoding delay of the video frame includes:

[0085] Obtain the number of all PU blocks included at each size according to the resolution of the video frame;

[0086] Obtain the average delay of motion search corresponding to a single PU block at each size;

[0087] Calculate the number of PU blocks for parallel motion search at each size respectively according to the number of all PU blocks included at each size, the average delay of motion search corresponding to a single PU block at each size, and the target encoding delay of the video frame.

[0088] In a possible implementation manner of the fourth aspect, the information of the video frame includes the motion vectors of each pixel included in each PU block. Obtaining the starting positions of motion search corresponding to each PU block at each size according to the information of the video frame includes:

[0089] Obtain the starting positions of motion search corresponding to each PU block respectively according to the motion vectors of the pixels included in each PU block.

[0090] In a possible implementation manner of the fourth aspect, obtaining the starting positions of motion search corresponding to each PU block respectively according to the motion vectors of the pixels included in each PU block includes:

[0091] Obtain the starting positions of motion search corresponding to each PU block respectively according to the average value or median of the motion vectors of all pixels included in each PU block.

[0092] In a fifth aspect, the present application provides a video encoding device. The beneficial effects can be referred to the description in the first aspect and will not be elaborated here. The device has the functions to implement the behaviors in the method examples of the first aspect above. These functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the device includes:

[0093] An acquisition module, configured to acquire a video frame and information of the video frame;

[0094] A processing module, configured to divide the video frame to obtain prediction unit (PU) blocks;

[0095] The processing module is further configured to obtain the starting positions of motion search respectively corresponding to each PU block according to the information of the video frame;

[0096] The processing module is further configured to perform motion search on the PU blocks in parallel based on the starting positions of motion search respectively corresponding to each PU block;

[0097] An encoding module, configured to encode the video frame based on the result after motion search.

[0098] In a possible implementation manner of the fifth aspect, the information of the video frame includes the motion vectors of each pixel included in each PU block, and the processing module is further configured to:

[0099] Obtain the starting positions of motion search respectively corresponding to each PU block according to the motion vectors of the pixels included in each PU block.

[0100] In a possible implementation manner of the fifth aspect, the processing module is further configured to:

[0101] Obtain the starting positions of motion search respectively corresponding to each PU block according to the average value or median value of the motion vectors of all pixels included in each PU block.

[0102] In a possible implementation manner of the fifth aspect, the number of PU blocks for performing motion search in parallel at any size within the video frame refers to the number of all PU blocks included in the size.

[0103] In a possible implementation manner of the fifth aspect, the device further includes:

[0104] The acquisition module is further configured to acquire the target encoding delay of the video frame;

[0105] The processing module is further configured to determine the number of PU blocks for performing motion search in parallel according to the target encoding delay of the video frame.

[0106] In a possible implementation manner of the fifth aspect, the information of the video frame includes the resolution of the video frame, and the processing module is further configured to:

[0107] Obtain the number of all PU blocks according to the resolution of the video frame;

[0108] Obtain the average latency of motion search corresponding to a single PU block;

[0109] Calculate the number of PU blocks for parallel motion search respectively according to the number of all PU blocks, the average latency of motion search corresponding to a single PU block, and the target encoding latency of the video frame.

[0110] In a sixth aspect, the present application provides a video coding apparatus. The beneficial effects can be referred to the description in the second aspect and will not be elaborated here. The apparatus has the functions to implement the behaviors in the method examples of the above second aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the apparatus includes:

[0111] An obtaining module, configured to obtain a video frame and information of the video frame;

[0112] A processing module, configured to divide the video frame to obtain prediction unit (PU) blocks;

[0113] The processing module is further configured to obtain the starting positions of motion search respectively corresponding to each PU block according to the information of the video frame;

[0114] The obtaining module is further configured to obtain the target encoding latency of the video frame;

[0115] The processing module is further configured to determine the number of PU blocks for parallel motion search according to the target encoding latency;

[0116] The processing module is further configured to perform motion search on the above-mentioned number of PU blocks in parallel according to the starting positions of motion search respectively corresponding to each PU block;

[0117] An encoding module, configured to encode the video frame based on the result after motion search.

[0118] In a possible implementation manner of the sixth aspect, the information of the video frame includes the resolution of the video frame, and the processing module is further configured to:

[0119] Obtain the number of all PU blocks according to the resolution of the video frame;

[0120] Obtain the average latency of motion search corresponding to a single PU block;

[0121] Calculate the number of PU blocks for parallel motion search according to the number of all PU blocks, the average latency of motion search corresponding to a single PU block, and the target encoding latency of the video frame.

[0122] In a possible implementation of the sixth aspect, the information of the video frame includes the motion vectors of the pixels included in each PU block, and the processing module is further configured to:

[0123] Obtain the starting positions of the motion search respectively corresponding to each PU block according to the motion vectors of the pixels included in each PU block.

[0124] In a possible implementation of the sixth aspect, the processing module is further configured to:

[0125] Obtain the starting positions of the motion search respectively corresponding to each PU block according to the average value or the median value of the motion vectors of all the pixels included in each PU block.

[0126] In a seventh aspect, the present application provides a computing device cluster, including at least one computing device, and each computing device includes a processor and a memory; the processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation manner of the first aspect or the method in the second aspect or any possible implementation manner of the second aspect.

[0127] In an eighth aspect, the present application provides a computer program product including instructions, and is characterized in that when the instructions are run by the computing device cluster, the computing device cluster executes the method in the first aspect or any possible implementation manner of the first aspect or the method in the second aspect or any possible implementation manner of the second aspect.

[0128] In a ninth aspect, the present application provides a chip system, and the chip system includes a processor for implementing the method in the first aspect or any possible implementation manner of the first aspect or the method in the second aspect or any possible implementation manner of the second aspect. In a possible design, the chip system further includes a memory for storing program instructions and / or data. The chip system may be composed of chips or may include chips and other discrete devices.

[0129] In a tenth aspect, the present application provides a computer-readable storage medium, and is characterized in that it includes computer program instructions, and when the computer program instructions are executed by the computing device cluster, the computing device cluster executes the method in the first aspect or any possible implementation manner of the first aspect or the method in the second aspect or any possible implementation manner of the second aspect.

[0130] The solutions of the second to tenth aspects above are used to implement or cooperate with the implementation of the method in the first aspect above or any possible implementation manner thereof, or the method in the second aspect above or any possible implementation manner of the second aspect. Therefore, they can achieve the same or corresponding beneficial effects as the first aspect, and will not be elaborated here. Description of the Drawings

[0131] Figure 1 It is a schematic diagram of the encoding and decoding process of generative video;

[0132] Figure 2 It is a schematic diagram of the inter-frame prediction process;

[0133] Figure 3 It is a schematic diagram of the division of HEVC video image coding units;

[0134] Figure 4 It is a system architecture diagram of the video coding method provided by the embodiment of the present application;

[0135] Figure 5 It is a schematic diagram of a process of the video coding method provided by the embodiment of the present application;

[0136] Figure 6 It is another schematic diagram of a process of the video coding method provided by the embodiment of the present application;

[0137] Figure 7 It is another system architecture diagram of the video coding method provided by the embodiment of the present application;

[0138] Figure 8 It is a schematic diagram of a structure of the video coding device provided by the embodiment of the present application;

[0139] Figure 9 It is another schematic diagram of a structure of the video coding device provided by the embodiment of the present application;

[0140] Figure 10a It is a schematic diagram of a structure of the computing device provided by the embodiment of the present application;

[0141] Figure 10b It is a schematic diagram of a structure of a computing device cluster provided by the embodiment of the present application;

[0142] Figure 11 It is another schematic diagram of a structure of a computing device cluster provided by the embodiment of the present application. Detailed Embodiments

[0143] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application. As those skilled in the art know, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0144] In the present application, the terms "a", "an" and other similar words do not intend to mean that there is only one thing, but rather that the relevant description is only directed to one of the things, and the thing may have one or more. In this document, the terms "comprising", "including" and other similar words are intended to represent a logical relationship, rather than a spatial structure relationship. For example, "A includes B" is intended to mean that B belongs to A logically, rather than that B is located inside A spatially. Additionally, the meanings of the terms "comprising", "including" and other similar words should be regarded as open-ended, rather than closed. For example, "A includes B" is intended to mean that B belongs to A, but B does not necessarily constitute all of A, and A may also include other elements such as C, D, E, etc.

[0145] In the present application, the terms "first", "second" and other similar words do not intend to imply any order, quantity or importance, but are only used to distinguish different elements. In this document, the terms "embodiment", "the present embodiment", "one embodiment", "a single embodiment" do not mean that the relevant description only applies to a specific embodiment, but rather that these descriptions may also apply to one or more other embodiments. Those skilled in the art should understand that in this document, any description made for a particular embodiment can be substituted, combined, or otherwise combined with the relevant descriptions in one or more other embodiments, and the new embodiments generated by such substitution, combination, or other combination are easily conceivable by those skilled in the art and belong to the scope of protection of the present application.

[0146] First, some terms in the embodiments of the present application will be explained to facilitate understanding by those skilled in the art.

[0147] (1) Video

[0148] Video is a sequence composed of a series of static single-frame images (also called single frames). Except for a few moments of scene switching, there is a strong temporal correlation between most consecutive image data. Video includes generative video.

[0149] In this application, video can refer to various technologies that capture, record, process, store, transmit, and reproduce a series of static images in the form of electrical signals. Video utilizes the principle of the human eye's visual persistence. By playing a series of pictures (also known as frames), it gives people the feeling of motion.

[0150] (2) Generative video

[0151] Generative video refers to a video different from traditional videos. Traditional videos are generated by video capture devices such as cameras, mobile phones with camera functions, video recorders, etc., by shooting scenes in the real world. In contrast to traditional videos, generative videos are not generated by video capture devices but are usually generated by computer software such as renderers, artificial intelligence (AI), etc., and are videos composed of corresponding continuous pictures generated according to user needs. The most common examples are videos in scenarios such as games, digital human live broadcasts, virtual concerts, VR conferences, and multi-person virtual collaborative design.

[0152] Since the video generation process in the generative video scenario has a large amount of computation, the video generation process in the generative video scenario is usually carried out on non-local devices, that is, video generation and video display are usually not carried out on the same device. Typically, video generation is carried out on cloud devices and then transmitted to the terminal device for display through the network. Since the generated video files are very large, they usually need to be video encoded (also known as video compression) by a video encoder and then pushed to the terminal device through network streaming for decoding and display. As Figure 1 shown, in the non-local scenario, the encoding and decoding process of generative video usually follows the following workflow.

[0153] 1. The video generation renderer generates video frames according to user requirements, forming the original video stream.

[0154] 2. The video encoder encodes the original video stream.

[0155] 3. The video encoder compresses the encoded video stream and transmits it to the terminal user device through the network.

[0156] 4. The terminal user device decodes the compressed encoded video stream through a decoder, and after decoding, displays the video frames through a display.

[0157] (3) Motion assessment

[0158] In a video sequence, there is a large amount of similar image content between adjacent frames. To make full use of the already encoded and reconstructed image content to predict the image content within the current frame and reduce the bit rate required for encoding the video sequence, the encoder usually performs motion assessment (Motion Estimation, ME).

[0159] During the process of motion assessment in video coding, the best matching block in the reference frame within a specific search range can be found through motion search (MS), that is, the best matching block for the current block. The searched best matching block can be used as the prediction block for the current block, and then the residual between the current block and the prediction block can be calculated, and the residual is quantized, transformed, and entropy encoded. Thus, through motion assessment, the difference between adjacent frames can be transmitted in compression coding without transmitting all the information of each frame, and the bit rate required for encoding the original video can be significantly reduced while ensuring a certain reconstructed video quality. Among them, motion search refers to the process of finding the best matching block for the current block in a search area of a certain size within the reference frame during motion assessment.

[0160] In one example, as Figure 2 shown are two consecutive frames in a video (the dashed boxes in the figure are for illustration only and not the content of the frames). In these two consecutive frames, only the position of object 1 has changed. Then, the pixels in the area of object 1 in the second frame (the dashed square area) can be predicted by the pixels in the area of object 1 in the previous frame (the dashed square area). The difference between the corresponding pixels of the two areas before and after is the residual, and adding the motion vector information of object 1 can perform predictive coding on the second frame. Obviously, when performing inter-frame prediction on the latter frame, a search needs to be carried out in the previous frame to find the matching area, and this search process is motion assessment. Since the motion of an object is not necessarily an integer multiple of pixels, in addition to performing integer-pixel search, sub-pixels are interpolated and sub-pixel motion search is also performed. And the position change of an object or pixel between two consecutive frames is the motion vector (MV).

[0161] (4) Video coding

[0162] Video coding refers to the method of converting a file in the original video format into another video format file through compression technology. A video is a sequence of continuous images composed of continuous frames. Since the similarity between consecutive frames is extremely high, in order to facilitate storage and transmission, the original video needs to be encoded and compressed to remove redundancy in the spatial and temporal dimensions. Video image data has strong correlation, that is, there is a large amount of redundant information, and the redundant information can be divided into spatial redundant information and temporal redundant information. Compression technology is to remove the redundant information in the data. Compression technology includes intra-frame image data compression technology, inter-frame image data compression technology, and entropy coding compression technology.

[0163] In a video, there is strong spatial correlation among pixels in adjacent ranges within a single image (also known as a single frame). There is strong temporal correlation among consecutive image data. Video coding mainly adopts intra-frame prediction and inter-frame prediction methods, using the already encoded pixels of an image to predict spatially and temporally adjacent pixels, thereby effectively compressing the redundancy of video image pixels in the time domain and spatial domain.

[0164] In this application, "video compression" and "video coding" can be interchanged in some scenarios.

[0165] It should be understood that the data volume of the video before compression is extremely large. Taking 1080P, which is widely used in current scenarios, as an example, each pixel contains luminance and chrominance, represented by 12 bits in total, and the frame rate is 60 (that is, 60 frames are output per second). In this case, the required network transmission bandwidth is approximately 1.39 Gbps. The bandwidth requirement for 4K video is even greater. It is very difficult to meet such a large Internet bandwidth requirement. A key technology in video stream applications is video coding, also known as video compression. The device that completes the video compression function is a video encoder. Similar to text compression, the purpose is to remove redundant data in the video data and reduce the data volume after encoding and compression.

[0166] (5) High Efficiency Video Coding

[0167] High Efficiency Video Coding (HEVC) is the next-generation video coding standard after H.264. Its core goal is to double the compression efficiency based on H.264, that is, to reduce the bit rate of the video stream by 50% while ensuring the same video image quality. HEVC adopts a block-based hybrid coding framework.

[0168] It should be noted that the image coding method provided in the embodiments of this application can be applied in various video coding standards (such as H.264, H.265, H.266, etc.), and no limitation is made here.

[0169] As the size of the video image increases, such as 8K and 12K resolutions, in order to better perform compression coding, the video coding protocol usually divides the image into many small blocks for compression coding, such as the 16x16 pixel-sized macroblock in Advanced Video Coding (AVC). To more efficiently compress different textures and motion change details in the video scene, High Efficiency Video Coding (HEVC) defines a Coding Tree Unit (CTU) with 64x64 pixels as the unit for the image. Each image in the video is divided into several non-overlapping CTUs. As Figure 3 shown, Figure 3The figure shows a schematic diagram of the division of HEVC video image coding units. Inside the CTU, a cyclic hierarchical structure based on a quadtree is adopted. For example, the top layer is a 64x64 pixel unit, which can be divided into a lower layer, divided into 4 coding units (CUs) of 32x32 pixels in size. Each 32x32 pixel size can be further divided downwards until the smallest 4x4 coding unit. The CU division of the versatile video coding (VVC) video coding standard can support up to 128x128 pixels at most, and the division is more flexible and rich.

[0170] Among them, when using HEVC for encoding, encoding is performed one by one according to the coding tree unit CTU unit. Each CTU can be divided into different coding units CUs, and each CU can be further divided into different prediction units PUs. During prediction, prediction is performed according to the size of the PU. During encoding, encoding is performed according to the size of the CU. The CTU is the largest coding unit of HEVC, the CU is the coding unit of HEVC, the CTU can be divided downwards into CUs for encoding, the PU is the prediction unit of HEVC, and the CU can be divided into different PUs for prediction.

[0171] As Figure 3 shown by the lines in (a) of Figure 3 video encoding is performed in units of CTUs in the order from left to right and from top to bottom. As

[0172] shown by the lines in (b) of

[0172] , inside the CTU, it is continuously divided down to the smallest 4x4 pixel unit, and the CUs inside the CTU are predicted and encoded in the Z-shaped scan order. The prediction information of the previous prediction block, such as the motion vector, etc., needs to be used as the input for the prediction of the next CU. Since people's requirements for video service experience are getting higher and higher, on the one hand, it promotes the development of video towards high resolution, such as 8K, 12K, 3D, and higher frame rates, making the original video files larger and larger; on the other hand, video services have shifted from traditional one-way video playback (that is, users cannot give feedback or response to the video generation end) to real-time interaction, such as AR / VR, interactive live broadcast, cloud games, multi-person virtual collaborative design, etc., requiring lower latency. Therefore, low-latency video coding, also known as low-latency video compression, is a key technology for video services.

[0173] It can be understood that in the current video codec protocols, such as Figure 3As shown in the figure, when the CTU internally performs predictive coding on the CUs in a Z-shaped scanning order, it is necessary to use the prediction information of the previous prediction block, such as the motion vector, etc., as the input for the next CU prediction. Therefore, adjacent CU blocks cannot be compressed and encoded in parallel simultaneously. Specifically, when the CU performs motion search based on the partitioned PUs, it is necessary to determine the starting point of the current PU's motion search based on the motion vectors of the PUs (in the lower left, left, upper left, upper, upper right) that have been encoded around. Therefore, there is a dependency in the encoding between the PUs and parallel computing cannot be performed (requiring that the surrounding PU blocks have been encoded).

[0174] To solve the above problems, the embodiments of the present application provide a video encoding method and related devices, which can perform motion search on the PU blocks in parallel, thereby reducing the encoding delay. Please refer to Figure 4 , Figure 4 which is a system architecture diagram of the video encoding method provided by the embodiments of the present application. As Figure 4 shown, the video generator (such as a renderer, an artificial intelligence generated content (AIGC) tool, etc.) outputs one or more ( Figure 4 the example in is to output 3) generated raw video streams and video-related information. The video encoder performs parallel motion search on the PUs based on the raw video streams, video-related information, and configuration information input by the user output by the video generator. After completing the video encoding, the encoded video stream is compressed and the compressed bitstream is output.

[0175] Among them, both the video generator and the video encoder can be composed of software functional modules or hardware devices.

[0176] Please refer to Figure 5 , Figure 5 which is a flowchart of the video encoding method provided by the embodiments of the present application. This method can be executed by an encoder, and the method includes the following steps.

[0177] S501. The encoder obtains a video frame and information of the video frame.

[0178] Video is a continuous sequence of images composed of continuous frames. The video generator outputs a video stream and related information corresponding to the video stream, and the encoder processes each frame of the video frames included in the video stream one by one.

[0179] In a possible implementation, the information of the video frame includes the motion vectors of each pixel.

[0180] In a possible implementation, the information of the video frame includes information such as the motion vectors of each pixel, whether the object is a new object that appears due to a scene switch, and the coordinate information of the pixels of the newly appeared object in the picture.

[0181] S502. The encoder divides the video frame to obtain prediction unit PU blocks.

[0182] PU is the prediction unit of HEVC. The encoder can divide the video frame into different PU blocks for prediction. The number of PU blocks corresponding to different sizes may be different.

[0183] It should be understood that the encoder divides the video frame to obtain PU blocks of all or part of the sizes contained in the video frame.

[0184] S503. The encoder obtains the start position of the motion search corresponding to each PU block according to the information of the video frame.

[0185] In order to get rid of the encoder's dependence on surrounding PU blocks when performing motion search based on PU blocks, the encoder calculates the starting position of the motion search corresponding to each PU block, removes the dependency between PU blocks when performing motion search, and thus realizes parallel motion search of PU blocks. It should be understood that each PU block refers to each PU block of each size in the video frame. Correspondingly, the encoder can obtain the starting position of the motion search corresponding to each PU block of each size according to the information of the video frame.

[0186] In a possible implementation, the information of the video frame includes the motion vector of each pixel included in each PU block, and the encoder obtains the starting position of the motion search corresponding to each PU block according to the motion vector of the pixel included in each PU block.

[0187] In a possible implementation manner, the starting position of the motion search corresponding to any PU block may be determined based on the motion vectors of all pixels included in the PU block.

[0188] In a possible implementation manner, the starting position of the motion search corresponding to any PU block may be determined based on the motion vectors of some pixels included in the PU block.

[0189] In a possible implementation manner, the starting position of the motion search corresponding to any PU block may be determined based on the motion vector of any pixel included in the PU block.

[0190] It should be understood that in actual applications, the starting position of the motion search corresponding to any PU block can be determined based on all, part, or any motion vector contained in the PU block or in other ways, and this application does not limit this.

[0191] In a possible implementation manner, the encoder obtains the start position of the motion search corresponding to each PU block according to the average value or median value of the motion vectors of all pixels included in each PU block.

[0192] In a possible implementation, the encoder respectively obtains the maximum and minimum values of the motion vectors of all pixels included in each PU block; the encoder respectively obtains the starting positions of motion search corresponding to each PU block according to the average value or median value of the motion vectors of all pixels except the maximum and minimum values of the motion vectors of all pixels included in each PU block.

[0193] It should be understood that the encoder uses the average value or median value of the motion vectors of the remaining pixels included in the PU block as the starting position of the motion search corresponding to the PU block, so as to avoid the problem of large deviation of the overall data caused by large errors of individual samples.

[0194] In a possible implementation, the encoder obtains the target coding delay of the video frame.

[0195] Each video stream corresponds to a different application scenario, and different scenario requirements may correspond to different target coding delays. For example, the target coding delay of interactive digital human live broadcast is T1, the target coding delay of VR / AR applications is T2, and the target coding delay of non-interactive live broadcast applications is T3. By obtaining the target coding delay of the video frame, the encoder can match the corresponding target coding delay for different application scenarios, so as to calculate the parallel computing resources adapted to the scenario and achieve the best cost performance.

[0196] In a possible implementation, the target coding delay of the video frame can be obtained based on user configuration. For example, the user sends delay information to the encoder based on their own or scenario requirements.

[0197] It should be understood that the target coding delay of the video frame can be configured based on various methods, and the encoder can also obtain the target coding delay of the video frame based on various methods, which is not limited here.

[0198] In a possible implementation, the encoder determines the number of PU blocks for parallel motion search at each size according to the target coding delay of the video frame.

[0199] After obtaining the target coding delay of the video frame, the encoder can determine the number of PU blocks for parallel motion search at each size based on the target coding delay of the video frame, so as to further improve the coding parallelism and reduce the coding delay.

[0200] In a possible implementation, the information of the video frame includes the resolution of the video frame. The encoder obtains the number of all PU blocks included in each size according to the resolution of the video frame; the encoder obtains the average latency of motion search corresponding to a single PU block in each size; the encoder calculates the number of PU blocks for parallel motion search in each size respectively according to the number of all PU blocks included in each size, the average latency of motion search corresponding to a single PU block in each size, and the target encoding latency of the video frame.

[0201] In a possible implementation, the encoder obtains the average latency of motion search corresponding to a single PU block in each size based on offline testing. For example, the encoder can obtain the average latency of motion search corresponding to a single PU block in each size by offline testing various scenarios, such as game scenarios, VR / AR scenarios, digital human live broadcast scenarios, etc.

[0202] Since in the software testing scenario, different software search algorithms may correspond to different search latencies, corresponding to different application scenarios, the encoder needs to re-obtain the average latency of motion search corresponding to the PU block. In another possible implementation, the encoder obtains the average latency of motion search corresponding to a single PU block in each size based on a hardware encoder. For example, the video encoder uses fixed-point integer pixels and sub-pixel motion search. Such as a common hardware encoder calculates the average latency of motion search corresponding to a single PU block in each size, that is, the average latency of motion search is a fixed value.

[0203] In a possible implementation, the number of PU blocks for parallel motion search in any size is determined based on the quotient of the product of the number of all PU blocks included in that size and the average latency of motion search corresponding to a single PU block in that size and the target encoding latency of the video frame.

[0204] It should be understood that if the final calculated result is not an integer, the encoder can finally determine the number of PU blocks for parallel motion search in that size based on various methods. For example, rounding, ceiling, floor, or using the remainder, etc.

[0205] In an example, the specific process for the encoder to calculate the number of PU blocks for parallel motion search in each size is as follows:

[0206] a) The encoder obtains the target encoding latency of the video frame.

[0207] b) The encoder calculates the number of PU in various sizes N_pu according to the video resolution. For example, the number of PU blocks N_pu in the 64x64 size of a 4K (3840x2160) video is 3840 * 2160 / 64 / 64 = 2025.

[0208] c) The encoder obtains the average latency \(T_{latency\_pu}\) of the motion search corresponding to each individual PU block for each size.

[0209] d) The encoder obtains the number \(T=(N_{pu}*T_{latency\_pu}) / \text{target latency}\) of PU blocks for which motion search is performed in parallel for each size based on \(N_{pu}\), \(T_{latency\_pu}\), and the target encoding latency.

[0210] It should be understood that the encoder can also determine the number of PU blocks for which motion search is performed in parallel for each size based on other means. In a possible implementation, the number of PU blocks for which motion search is performed in parallel for any size refers to the number of all PU blocks included in the size. Thus, the maximum coding parallelism can be achieved, and low-latency coding can be realized.

[0211] S504. The encoder performs motion search on the PU blocks in parallel based on the starting positions of the motion search corresponding to each PU block.

[0212] After the encoder determines the starting positions of the motion search corresponding to each PU block, it can perform motion search on each PU block in parallel. Specifically, the encoder can perform motion search on the PU blocks for each size based on the starting positions of the motion search corresponding to each PU block for each size.

[0213] Among them, the number of PU blocks for which motion search is performed in parallel for each size can be determined based on the implementation method described in S503, or can be determined based on other methods. For example, the number of PU blocks for which motion search is performed in parallel for each size can be the number of all PU blocks included in each size or a specified partial number, etc., which is not limited here.

[0214] During the process of the encoder performing parallel motion search, since the PU blocks for which motion search is performed in parallel may be in different search stages / states, such as initiating pixel reading operations, waiting for pixel return, calculating prediction value residuals, obtaining the next search position, etc. Therefore, in a possible implementation, the encoder monitors / obtains / records the states of the PU blocks that are performing motion search to schedule each of the PU blocks for which motion search is performed in parallel. For example, if the encoder monitors that the state of a certain PU block is waiting for pixel return (the process of the encoder obtaining the motion vector of the pixel from the cache takes time), the encoder can schedule another PU block to perform motion search during the time when the PU block is waiting for pixel return, so as to reduce the bubbles in the motion search process, thereby accelerating the time of parallel processing.

[0215] S505. The encoder encodes the video frame based on the results after motion search.

[0216] In a possible implementation, after completing parallel motion search, the encoder determines the partitioning method of the CU according to the rate-distortion optimized (RDO) algorithm, and performs operations such as transformation, quantization, deblocking filtering, and entropy coding on the selected unique partitioning method and the residual, and finally outputs the compressed video stream.

[0217] In one example, the encoder tries all CU partitioning methods, then performs motion evaluation, prediction calculation, discrete cosine transform (DCT), quantization, and entropy coding processes, and finally obtains the final partitioning method through rate-distortion calculation. When compressing in units of CTU, for each CU partitioning depth, namely the four partitioning methods of 64x64, 32x32, 16x16, and 8x8, motion evaluation is performed respectively to obtain motion vectors, and based on the motion vectors, inter-frame prediction, DCT, quantization, bit number evaluation entropy coding, and distortion calculation are performed, and then comparison is performed layer by layer according to the rate-distortion optimized (RDO) algorithm. For example, for a 16x16 pixel region, there are four partitioning methods: 4 8x8 CUs, 2 4x8 CUs, 2 8x4 CUs, or 1 16x16 CU. The starting positions of the motion search for each PU block corresponding to different sizes are determined for these four partitioning methods respectively, and the motion search is completed, and finally the partitioning method with the minimum rate distortion is selected for the subsequent encoding process.

[0218] In a possible implementation, for sub-pixel scenarios with simple motion, such as linear motion, the encoder can directly use the motion vector of the PU block calculated from the pixel motion vector provided by the video generator as the starting point of sub-pixel motion search, and skip the motion search process for whole pixels (such as animation scenes). Correspondingly, the video encoding method provided by the embodiments of the present application is mainly applied to whole pixel scenarios or scenarios combining whole pixels and sub-pixels.

[0219] Please refer to Figure 6 , Figure 6 which is another flowchart of the video encoding method provided by the embodiments of the present application. This method can be executed by an encoder, and this method is mainly applied to the system architecture as shown in Figure 7 . In a possible implementation, this method can be applied to the scenario of cloud services. In the scenario of cloud services, this method is executed by the cloud management platform, and the encoder can be located inside the cloud management platform and is used to determine and count the number of PU blocks for motion search based on the target coding delay. This method includes the following steps.

[0220] S601. The encoder obtains the video frame and the information of the video frame.

[0221] S602. The encoder divides the video frame to obtain prediction unit (PU) blocks.

[0222] S603. The encoder obtains the starting positions of motion search corresponding to each PU block according to the information of the video frame.

[0223] It should be understood that the specific content of the above implementation can refer to the descriptions of S501 to S503, which will not be elaborated here.

[0224] S604. The encoder obtains the target coding delay of the video frame.

[0225] In a possible implementation, the target coding delay of the video frame is determined based on user configuration. By obtaining the target coding delay of the video frame, the encoder calculates the number of motion searches according to the user's requirements, so as to improve the flexibility of configuration.

[0226] Each video stream corresponds to a different application scenario, and different scenario requirements may correspond to different target coding delays. For example, the target coding delay of interactive digital human live broadcast is T1, the target coding delay of VR / AR applications is T2, and the target coding delay of non-interactive live broadcast applications is T3. By obtaining the target coding delay of the video frame, the encoder can allocate matching target coding delays for different application scenarios, thereby calculating the appropriate parallel computing resources and achieving the best cost performance.

[0227] It should be understood that the execution order of S601 to S603 and S604 is not limited.

[0228] S605. The encoder determines the number of PU blocks for parallel motion search according to the target coding delay.

[0229] After obtaining the target coding delay, the encoder can determine the number of PU blocks for parallel motion search in each size based on the target coding delay, thereby further improving the coding parallelism and reducing the coding delay.

[0230] In a possible implementation, the information of the video frame includes the resolution of the video frame. The encoder obtains the number of all PU blocks included in each size according to the resolution of the video frame; the encoder obtains the average delay of motion search corresponding to a single PU block in each size; the encoder calculates the number of PU blocks for parallel motion search in each size respectively according to the number of all PU blocks included in each size, the average delay of motion search corresponding to a single PU block in each size, and the target coding delay of the video frame.

[0231] In a possible implementation, the encoder obtains the average latency of motion search corresponding to a single PU block at each size based on offline testing. For example, the encoder can obtain the average latency of motion search corresponding to a single PU block at each size by offline testing various scenarios, such as game scenarios, VR / AR scenarios, digital human live broadcast scenarios, etc.

[0232] Since in the software testing scenario, different software search algorithms may correspond to different search latencies, and corresponding to different application scenarios, the encoder needs to obtain different average latencies, which may increase the running latency. In another possible implementation, the encoder obtains the average latency of motion search corresponding to a single PU block at each size based on a hardware encoder. For example, the video encoder uses fixed-point integer pixels and sub-pixel motion search, and the average latency of motion search corresponding to a single PU block at each size is calculated by a common hardware encoder.

[0233] In a possible implementation, the number of PU blocks performing motion search in parallel at any size is determined based on the quotient of the product of the number of all PU blocks included in that size and the average latency of motion search corresponding to a single PU block at that size and the target encoding latency of the video frame.

[0234] It should be understood that if the final calculated result is not an integer, the number of PU blocks performing motion search in parallel at that size can be finally determined based on various methods. For example, rounding, ceiling, floor, or using the remainder, etc.

[0235] In an example, the specific process for the encoder to calculate the number of PU blocks performing motion search in parallel at each size is as follows:

[0236] a) The encoder obtains the target encoding latency of the video frame.

[0237] b) The encoder calculates the number of PU blocks N_pu of various sizes according to the video resolution. For example, the number of PU blocks N_pu in the 64x64 size of a 4K (3840x2160) video is 3840 * 2160 / 64 / 64 = 2025.

[0238] c) The encoder obtains the average latency T_latency_pu of motion search corresponding to each single PU block at each size.

[0239] d) The encoder obtains the number of PU blocks T performing motion search in parallel at each size according to N_pu, T_latency_pu, and the target encoding latency, where T = (N_pu * T_latency_pu) / target latency.

[0240] It should be understood that the encoder can determine the number of PU blocks that perform motion search in parallel at each size based on other means. In one possible implementation, the number of PU blocks that perform motion search in parallel at any size refers to the number of all PU blocks included in the size. Thus, the maximum coding parallelism can be achieved, and low-latency coding can be realized.

[0241] S606. The encoder performs motion search on this number of PU blocks in parallel according to the starting positions of the motion search respectively corresponding to each PU block.

[0242] S607. The encoder encodes the video frame based on the results after the motion search.

[0243] It should be understood that the specific content of the implementation manners of S606 to S607 can refer to the descriptions of S504 to S505, and will not be elaborated here.

[0244] Please refer to Figure 8 , Figure 8 , which is a schematic structural diagram of a video encoding device provided in an embodiment of the present application. The video encoding device 8000 includes:

[0245] An obtaining module 8001, configured to obtain a video frame and information of the video frame;

[0246] A processing module 8002, configured to divide the video frame to obtain prediction unit PU blocks;

[0247] The processing module 8002 is further configured to obtain the starting positions of the motion search respectively corresponding to each PU block according to the information of the video frame;

[0248] The processing module 8002 is further configured to perform motion search on the PU blocks in parallel based on the starting positions of the motion search respectively corresponding to each PU block;

[0249] An encoding module 8003, configured to encode the video frame based on the results after the motion search.

[0250] Taking the processing module 8002 as an example of a software functional unit, the processing module 8002 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the processing module 8002 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Usually, one region may include multiple AZs.

[0251] Similarly, the multiple hosts / virtual machines / containers for running the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC, and the interconnection between VPCs is achieved through the communication gateway.

[0252] Taking the processing module 8002 as an example of a hardware functional unit, the processing module 8002 may include at least one computing device, such as a server, etc. Or, the processing module 8002 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0253] The multiple computing devices included in the processing module 8002 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 8002 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module 8002 can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0254] In a possible implementation manner, the information of the video frame includes the motion vectors of the pixels included in each PU block, and the processing module 8002 is further configured to:

[0255] Obtain the starting positions of the motion search respectively corresponding to each PU block according to the motion vectors of the pixels included in each PU block.

[0256] In a possible implementation manner, the processing module 8002 is further configured to:

[0257] Obtain the starting positions of the motion search respectively corresponding to each PU block according to the average value or median of the motion vectors of all the pixels included in each PU block.

[0258] In a possible implementation manner, the number of PU blocks for parallel motion search at any size within the video frame refers to the number of all PU blocks included in the size.

[0259] In a possible implementation manner, the apparatus further includes:

[0260] An acquisition module 8001, further configured to acquire the target encoding delay of the video frame;

[0261] The processing module 8002 is further configured to determine the number of PU blocks for parallel motion search according to the target encoding delay of the video frame.

[0262] In a possible implementation manner, the information of the video frame includes the resolution of the video frame, and the processing module 8002 is further configured to:

[0263] Obtain the number of all PU blocks included according to the resolution of the video frame;

[0264] Obtain the average delay of the motion search corresponding to a single PU block;

[0265] Calculate and obtain the number of PU blocks for parallel motion search respectively according to the number of all PU blocks, the average delay of the motion search corresponding to a single PU block, and the target encoding delay of the video frame.

[0266] In this embodiment, the operations performed by each unit in the video encoding apparatus 8000 are similar to those described in the foregoing Figure 5 method embodiment shown, and can be used to implement the functions of the computing device in the foregoing method embodiment, and can also achieve the beneficial effects of the foregoing method embodiment, which will not be elaborated herein.

[0267] Please refer to Figure 9 , Figure 9 which is another structural schematic diagram of the video encoding apparatus provided in the embodiment of the present application. The video encoding apparatus 9000 includes:

[0268] An acquisition module 9001, configured to acquire a video frame and information of the video frame;

[0269] A processing module 9002, configured to divide the video frame to obtain prediction unit PU blocks;

[0270] The processing module 9002 is further configured to obtain the starting positions of motion search respectively corresponding to each PU block according to the information of the video frame;

[0271] The acquisition module 9001 is further configured to acquire the target coding delay of the video frame;

[0272] The processing module 9002 is further configured to determine the number of PU blocks for which motion search is performed in parallel according to the target coding delay;

[0273] The processing module 9002 is further configured to perform motion search on the number of PU blocks in parallel according to the starting positions of motion search respectively corresponding to each PU block;

[0274] An encoding module 9003, configured to encode the video frame based on the result after motion search.

[0275] As an example of a software functional unit, taking the processing module 9002 as an example, the processing module 9002 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the foregoing computing instance may be one or more. For example, the processing module 9002 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code may be distributed in the same region, or may be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ), or may be distributed in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Usually, one region may include multiple AZs.

[0276] Similarly, multiple hosts / virtual machines / containers for running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Usually, one VPC is set up within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, communication gateways need to be set up within each VPC, and the interconnection between VPCs is achieved through the communication gateways.

[0277] As an example of a hardware functional unit, the processing module 9002 may include at least one computing device, such as a server. Alternatively, the processing module 9002 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0278] The multiple computing devices included in the processing module 9002 can be distributed within the same region or across different regions. The multiple computing devices included in the processing module 9002 can be distributed within the same availability zone (AZ) or across different AZs. Similarly, the multiple computing devices included in the processing module 9002 can be distributed within the same VPC or across multiple VPCs. Among them, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0279] In a possible implementation, the information of the video frame includes the resolution of the video frame, and the processing module 9002 is further configured to:

[0280] Obtain the number of all prediction unit (PU) blocks based on the resolution of the video frame;

[0281] Obtain the average latency of motion search corresponding to a single PU block;

[0282] Calculate the number of PU blocks for parallel motion search based on the number of all PU blocks, the average latency of motion search corresponding to a single PU block, and the target encoding latency of the video frame.

[0283] In a possible implementation, the information of the video frame includes the motion vectors of the pixels included in each PU block, and the processing module 9002 is further configured to:

[0284] Obtain the starting positions of the motion search corresponding to each PU block respectively according to the motion vectors of the pixels included in each PU block.

[0285] In a possible implementation, the processing module 9002 is further configured to:

[0286] Obtain the starting positions of the motion search corresponding to each PU block respectively according to the average value or the median of the motion vectors of all the pixels included in each PU block.

[0287] In this embodiment, the operations performed by each unit in the video encoding device 9000 are similar to those described in the foregoing Figure 6 method embodiment, and can be used to implement the functions of the computing device in the foregoing method embodiment, and can also achieve the beneficial effects of the foregoing method embodiment, which will not be elaborated here.

[0288] The video encoding method provided by the embodiment of the present application can be executed by a computing device, which can be various terminal devices with functions such as shooting, video recording, screen recording, and screen mirroring, or a video encoder in the terminal device. The present application does not make any restrictions on this. Among them, the terminal device includes but is not limited to: mobile phones, computers, video surveillance devices, dash cams, etc. The video encoder can be a hardware encoder or a software program. The video encoding method can also be executed by a chip. The present application takes the execution by a computing device as an example for illustration. In different types of terminal devices, better encoding effects and more efficient encoding can be achieved according to the video encoding method provided by the present application.

[0289] Please refer to Figure 10a , Figure 10a which is a schematic structural diagram of a computing device provided by an embodiment of the present application. As Figure 10a shown, the computing device 1000 is implemented by a general bus architecture. As Figure 10a shown, the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other through the bus 1002, and can also communicate through other means such as wireless transmission. The memory 1006 is used to store instructions, and the processor 1004 is used to execute the instructions stored in the memory 1006. The memory 1006 stores program codes, and the processor 1004 can call the program codes stored in the memory 1006 to execute the video encoding method provided by the embodiment of the present application. It should be understood that the present application does not limit the number of processors and memories in the computing device 1000.

[0290] In a possible implementation, the execution Figure 5 and Figure 6 The encoder of the method shown is the computing device 1000 or is located inside the computing device 1000.

[0291] In a possible implementation, Figure 8 The video encoding device 8000 described and Figure 9 The video encoding device 9000 shown are located inside the computing device 1000.

[0292] The bus 1002 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10a only one line is shown in, but it does not mean that there is only one bus or one type of bus. The bus 1002 can include a path for transmitting information between various components of the computing device 1000 (for example, the memory 1006, the processor 1004, the communication interface 1008).

[0293] The processor 1004 is a general-purpose central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices (PLD), transistor logic devices, hardware components, or any combination thereof. The above PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0294] Optionally, the memory 1006 is a read-only memory (ROM) or other type of static storage device that can store static information and instructions. Alternatively, the memory 1006 is a random access memory (RAM) or other type of dynamic storage device that can store information and instructions. Alternatively, the memory 1006 is an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disc storage media, or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0295] Optionally, the memory 1006 exists independently and is connected to the processor 1004 via the bus 1002. Optionally, the memory 1006 and the processor 1004 are integrated together.

[0296] The communication interface 1008 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 1008 includes a wired communication interface. Optionally, the communication interface 1008 further includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof, etc.

[0297] It can be understood that the method steps in the embodiments of the present application can be implemented in hardware or in software instructions executable by the processor 1004. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium well-known in the art. Additionally, however, the scope of the devices described in the present application is not limited thereto, and the structure of the devices can be unrestricted by Figure 8 and Figure 9 the following. The device can be an independent device or can be a part of a larger device. For example, the device can be:

[0298] (1) An independent integrated circuit IC, or chip, or chip system or subsystem;

[0299] (2) A set having one or more ICs, optionally, the IC set may also include a storage component for storing data and / or instructions;

[0300] (3) A module that can be embedded in other devices;

[0301] (4) A receiver, a terminal, a smart terminal, a wireless device, a handheld device, a mobile unit, a vehicle-mounted device, an artificial intelligence device, a machine device, a home device, a medical device, an industrial device, etc.;

[0302] (5) Others and so on.

[0303] This application provides a computing device cluster. The computing device cluster includes at least one computing device 1000 as Figure 10a shown. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0304] Please refer to Figure 10b , Figure 10b which is a schematic structural diagram of the computing device cluster provided by the embodiments of this application. As Figure 10b shown, the computing device cluster includes at least one computing device 1000. Instructions for executing a video encoding method can be stored in the memory 1006 of one or more of the computing devices 1000 in the computing device cluster.

[0305] In some possible implementation manners, partial instructions for executing a video encoding method can also be stored separately in the memory 1006 of one or more of the computing devices 1000 in the computing device cluster. In other words, a combination of one or more of the computing devices 1000 can jointly execute the instructions for executing the video encoding method.

[0306] It should be noted that the memories 1006 in different computing devices 1000 in the computing device cluster can store different instructions, respectively for executing partial functions of the video encoding device. That is, the instructions stored in the memories 1006 in different computing devices 1000 can implement the functions of one or more of the acquisition module, the processing module, and the encoding module. Among them, the acquisition module, the processing module, and the encoding module can be the acquisition module 8001, the processing module 8002, and the encoding module 8003 as Figure 8 shown, or can be the acquisition module 9001, the processing module 9002, and the encoding module 9003 as Figure 9 shown.

[0307] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. Among them, the network can be a wide area network or a local area network, etc. Figure 11 A possible implementation is shown. Please refer to Figure 11 , Figure 11 which is another schematic structural diagram of the computing device cluster provided by the embodiments of the present application. As Figure 11 shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in the computing device 1000A stores instructions for executing the functions of the acquisition module. At the same time, the memory 1006 in the computing device 1000B stores instructions for executing the functions of the processing module and the verification module.

[0308] It should be understood that Figure 11 the functions of the computing device 1000A shown in

[0309] can also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B can also be completed by multiple computing devices 1000.

[0310] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute a video encoding method.

[0311] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0312] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical, or other forms.

[0313] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0314] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0315] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

Claims

1. A video encoding method, characterized in that, The method includes: Obtaining a video frame and information of the video frame; Dividing the video frame to obtain prediction unit (PU) blocks; Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame; Based on the starting positions of motion search respectively corresponding to each of the PU blocks, performing motion search on the PU blocks in parallel; Encoding the video frame based on the result after motion search.

2. The method according to claim 1, wherein The information of the video frame includes motion vectors of each pixel included in each of the PU blocks. Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame includes: Respectively obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the motion vectors of the pixels included in each of the PU blocks.

3. The method according to claim 2, characterized in that, Respectively obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the motion vectors of the pixels included in each of the PU blocks includes: Respectively obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the average value or median of the motion vectors of all the pixels included in each of the PU blocks.

4. The method according to any one of claims 1 to 3, characterized in that, The number of the PU blocks performing motion search in parallel at any size within the video frame refers to the number of all the PU blocks included in the size.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtaining the target encoding delay of the video frame; Determining the number of the PU blocks performing motion search in parallel according to the target encoding delay of the video frame.

6. The method according to claim 5, wherein The information of the video frame includes the resolution of the video frame. Determining the number of the PU blocks performing motion search in parallel according to the target encoding delay of the video frame includes: Obtaining the number of all the PU blocks according to the resolution of the video frame; Obtaining the average delay of motion search corresponding to a single PU block; Calculating the number of the PU blocks performing motion search in parallel according to the number of all the PU blocks, the average delay of motion search corresponding to a single PU block, and the target encoding delay of the video frame.

7. A video encoding method, characterized in that, The method includes: Obtaining a video frame and information of the video frame; Dividing the video frame to obtain prediction unit (PU) blocks; Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame; Obtaining the target encoding delay of the video frame; Determining the number of the PU blocks performing motion search in parallel according to the target encoding delay; Based on the starting positions of motion search respectively corresponding to each of the PU blocks, performing motion search on the number of the PU blocks in parallel; Encoding the video frame based on the result after motion search.

8. The method according to claim 7, wherein The information of the video frame includes the resolution of the video frame. Determining the number of the PU blocks performing motion search in parallel according to the target encoding delay includes: Obtaining the number of all the PU blocks according to the resolution of the video frame; Obtaining the average delay of motion search corresponding to a single PU block; Calculating the number of the PU blocks performing motion search in parallel according to the number of all the PU blocks, the average delay of motion search corresponding to a single PU block, and the target encoding delay of the video frame.

9. The method according to claim 7 or 8, characterized in that The information of the video frame includes the motion vectors of each pixel included in each of the PU blocks. Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame includes: Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the motion vectors of the pixels included in each of the PU blocks.

10. The method according to claim 9, wherein Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the motion vectors of the pixels included in each of the PU blocks includes: Obtaining the starting positions of motion search respectively corresponding to each of the PU blocks according to the average value or median value of the motion vectors of all the pixels included in each of the PU blocks.

11. A video encoding device, characterized in that, The apparatus includes: An acquisition module, configured to acquire a video frame and the information of the video frame; A processing module, configured to partition the video frame to obtain prediction unit (PU) blocks; The processing module is further configured to obtain the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame; The processing module is further configured to perform motion search on the PU blocks in parallel based on the starting positions of motion search respectively corresponding to each of the PU blocks; An encoding module, configured to encode the video frame based on the result after motion search.

12. The device according to claim 11, characterized in that, The information of the video frame includes the motion vectors of each pixel included in each of the PU blocks. The processing module is further configured to: Obtain the starting positions of motion search respectively corresponding to each of the PU blocks according to the motion vectors of the pixels included in each of the PU blocks.

13. The device according to claim 12, characterized in that, The processing module is further configured to: Obtain the starting positions of motion search respectively corresponding to each of the PU blocks according to the average value or median value of the motion vectors of all the pixels included in each of the PU blocks.

14. The device according to any one of claims 11 to 13, characterized in that, The number of the PU blocks that perform motion search in parallel at any size within the video frame refers to the number of all the PU blocks included in that size.

15. The device according to any one of claims 11 to 13, characterized in that The apparatus further includes: The acquisition module is further configured to acquire the target encoding delay of the video frame; The processing module is further configured to determine the number of the PU blocks that perform motion search in parallel according to the target encoding delay of the video frame.

16. The device according to claim 15, characterized in that, The information of the video frame includes the resolution of the video frame. The processing module is further configured to: Obtain the number of all the PU blocks according to the resolution of the video frame; Obtain the average delay of motion search corresponding to a single PU block; Calculate and obtain the number of the PU blocks that perform motion search in parallel respectively according to the number of all the PU blocks, the average delay of motion search corresponding to a single PU block, and the target encoding delay of the video frame.

17. A video encoding device, characterized in that, The apparatus includes: An acquisition module, configured to acquire a video frame and the information of the video frame; A processing module, configured to partition the video frame to obtain prediction unit (PU) blocks; The processing module is further configured to obtain the starting positions of motion search respectively corresponding to each of the PU blocks according to the information of the video frame; The acquisition module is further configured to acquire the target encoding delay of the video frame; The processing module is further configured to determine the number of the PU blocks that perform motion search in parallel according to the target encoding delay; The processing module is further configured to perform motion search on the PU blocks in parallel according to the starting positions of the motion search respectively corresponding to the PU blocks; The encoding module is configured to encode the video frame based on the result after the motion search.

18. A cluster of computing devices, characterized in that, It includes at least one computing device, and each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

19. A computer program product containing instructions, characterized in that, When the instructions are run by the computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

20. A computer-readable storage medium, characterized in that, It includes computer program instructions, and when the computer program instructions are executed by the computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

Images