Video processing method, apparatus and computer device, storage medium, and program product

HK40084591BActive Publication Date: 2026-09-11TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
HK42023073445
Authority / Receiving Office
HK · HK
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2026-09-11

AI Technical Summary

Technical Problem

In existing video coding technologies, the decision-making process for selecting reference frames is complex and inefficient, resulting in low video coding efficiency.

Method used

By setting the reference frame for inter-frame prediction of image blocks as a restricted reference frame, the reference frame selection process is simplified, ensuring that image blocks and their sub-blocks do not require additional reference frame selection during inter-frame prediction.

Benefits of technology

It simplifies the reference frame selection decision process in video coding and improves video coding efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a video processing method, device, computer equipment, storage medium and program product, which comprises: encoding an image block to be encoded in a video, dividing the image block according to a first division manner, and encoding a plurality of sub-blocks obtained by the division; in the encoding process, if the image block is subjected to translational inter-frame prediction, any sub-block of the plurality of sub-blocks is not further divided, and motion information when the plurality of sub-blocks are subjected to inter-frame prediction is consistent with motion information when the image block is subjected to inter-frame prediction, a reference frame used when the image block is subjected to translational inter-frame prediction is set as a limited reference frame of the image block; the limited reference frame is used as a reference frame for other sub-blocks obtained by dividing the image block according to a second division manner, and a further divided block of the other sub-blocks subjected to inter-frame prediction. By using the embodiments of the present application, the decision process of reference frame selection can be simplified, the selection efficiency of the reference frame can be accelerated, and the video encoding efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of video coding technology, specifically to a video processing method, a video processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of computer technology, video encoding technology has been widely used in various video-related business scenarios (such as video conferencing business scenarios, video-on-demand business scenarios, etc.). Video encoding technology refers to the technology of compressing video to reduce its size. Through video encoding technology, storage space can be effectively saved and transmission efficiency can be improved.

[0003] In current video coding technologies, videos can be encoded by dividing them into blocks. For a certain image block in a video, the sub-blocks obtained by dividing the image block through different methods all need to select a reference frame when performing inter-frame prediction. The decision-making process for selecting the reference frame is complex and inefficient, which leads to low video coding efficiency. Summary of the Invention

[0004] This application provides a video processing method, apparatus, computer equipment, storage medium, and program product, which can simplify the decision-making process for selecting reference frames, accelerate the selection efficiency of reference frames, and thus improve video coding efficiency.

[0005] On one hand, embodiments of this application provide a video processing method, which includes:

[0006] Identify the image blocks to be encoded in the video; these image blocks can be divided in multiple ways.

[0007] Encode the image block, and divide the image block according to the first division method among multiple division methods, and encode the multiple sub-blocks obtained from the division;

[0008] In the process of encoding an image block and multiple sub-blocks, if the image block is subjected to translational inter-frame prediction, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block is subjected to translational inter-frame prediction is obtained.

[0009] The reference frame used for inter-frame prediction of image blocks during translation is set as the limiting reference frame of the image block; the limiting reference frame is used as the reference frame for inter-frame prediction of other sub-blocks and other sub-blocks when the image block is divided into other sub-blocks according to the second division method among multiple division methods.

[0010] Accordingly, embodiments of this application provide a video processing apparatus, which includes:

[0011] The processing unit is used to determine the image blocks to be encoded in the video, and the image blocks can be divided according to multiple division methods;

[0012] The processing unit is also used to encode image blocks, divide image blocks according to a first division method among multiple division methods, and encode multiple sub-blocks obtained from the division.

[0013] The acquisition unit is used to acquire the reference frame used when the image block performs translational inter-frame prediction during the encoding process of the image block and multiple sub-blocks, if the image block performs translational inter-frame prediction, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction.

[0014] The processing unit is further configured to set the reference frame used when performing translational inter-frame prediction of the image block as a limiting reference frame for the image block; the limiting reference frame is used as a reference frame when the image block is divided into other sub-blocks according to the second division method among the multiple division methods, and other sub-blocks are further divided into blocks for inter-frame prediction.

[0015] In one implementation, the acquisition unit is configured to, if an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then when acquiring the reference frame used for the translational inter-frame prediction of the image block, specifically performs the following steps:

[0016] If an image block is subjected to inter-frame translation prediction, and the prediction residual signal of the image block satisfies the inter-frame association condition, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block is subjected to inter-frame translation prediction is obtained.

[0017] In one implementation, the processing unit is further configured to perform the following steps:

[0018] Acquire the prediction residual signal when performing inter-frame translation prediction on image patches;

[0019] The predicted residual signal is compressed.

[0020] If the compression result of the predicted residual signal is less than the residual threshold, then the predicted residual signal of the image block during inter-frame prediction satisfies the inter-frame correlation condition.

[0021] In one implementation, the prediction residual signal includes multiple signal components; the processing unit is configured to determine, if the compression result of the prediction residual signal is less than the residual threshold, that the prediction residual signal during the inter-frame prediction of the image patch satisfies the inter-frame correlation condition, specifically by performing the following steps:

[0022] If the compression result of the target signal component among multiple signal components is less than the residual threshold, then it is determined that the compression result of the predicted residual signal is less than the residual threshold.

[0023] In one implementation, the processing unit, when acquiring the prediction residual signal during inter-frame translation prediction of image blocks, specifically performs the following steps:

[0024] In the reference frame used for inter-frame prediction of image patch translation, the reference region associated with the image patch is determined;

[0025] Based on the pixel difference between the image patch and the reference region, the prediction residual signal is determined when the image patch is used for inter-frame prediction with translation.

[0026] In one implementation, the processing unit, when setting the reference frame used for inter-frame prediction of image blocks as the limiting reference frame for image blocks, specifically performs the following steps:

[0027] Obtain the reference frame list for the image patch; the reference frame list for the image patch includes one or more reference frames, and the reference frames used when performing inter-frame prediction for translation of the image patch are selected from the reference frame list of the image patch;

[0028] In the reference frame list of the image patch, the reference frame used for inter-frame prediction of the image patch during translation is set as the limiting reference frame of the image patch.

[0029] In one implementation, when the processing unit sets the reference frame used for inter-frame prediction of the image patch as the limiting reference frame of the image patch in the reference frame list, it specifically performs the following steps:

[0030] Set the reference frames used for inter-frame prediction of image blocks in the reference frame list to available reference frames;

[0031] Additionally, the reference frames in the reference frame list of the image patch, except for the reference frames used when performing inter-frame prediction for translation of the image patch, are set to unavailable reference frames.

[0032] In one implementation, when the processing unit sets the reference frame used for inter-frame prediction of the image patch as the limiting reference frame of the image patch in the reference frame list, it specifically performs the following steps:

[0033] The reference frames in the list of retained image patches are the reference frames used when performing translational inter-frame prediction for the image patches;

[0034] In addition, the list of reference frames to be removed from the image patch includes all reference frames except those used for inter-frame prediction of the image patch during translation.

[0035] In one implementation, when the processing unit obtains the reference frame list for the image patch, it specifically performs the following steps:

[0036] If an image block has a parent block, then the reference frame list of the parent block of the image block is determined as the reference frame list of the image block.

[0037] If an image block does not have a parent block, the reference frame list of the image frame to which the image block belongs in the video is determined as the reference frame list of the image block; the reference frame list of the image frame to which the image block belongs in the video is constructed according to the reference frame list construction rules.

[0038] In one implementation, the motion information includes a prediction mode, a reference frame selection mode, and a reference frame index;

[0039] The motion information of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction includes: the prediction mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction, and both are translation inter-frame predictions; the reference frame selection mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction; and the reference frame index of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction.

[0040] In one implementation, the first partitioning method is any one of multiple partitioning methods, and the second partitioning method is any other partitioning method besides the first partitioning method among the multiple partitioning methods;

[0041] The various division methods include at least: horizontal two-part division, vertical two-part division, four-part division, horizontal three-part division, and vertical three-part division; the first division method is the four-part division method.

[0042] Accordingly, embodiments of this application provide a computer device, which includes:

[0043] A processor is a tool for implementing computer programs.

[0044] A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the video processing method described above.

[0045] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned video processing method.

[0046] Accordingly, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned video processing method.

[0047] In this embodiment, when an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks obtained by dividing the image block according to the first division method among multiple division methods are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, the reference frame used when the image block is subjected to translational inter-frame prediction can be set as the restricted reference frame of the image block. By setting the restricted reference frame, the restricted reference frame can be used as the reference frame when the image block is divided into other sub-blocks according to the second division method among multiple division methods, and other sub-blocks and their further division blocks are subjected to inter-frame prediction. That is, after setting the restricted reference frame, when the image block is divided into other sub-blocks according to the second division method among multiple division methods, the other sub-blocks and their further division blocks can directly use the restricted reference frame as the reference frame used during inter-frame prediction without the need for reference frame selection. This simplifies the decision-making process for reference frame selection in video encoding, accelerates the selection efficiency of reference frames in video encoding, and thus improves video encoding efficiency. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating an image block not being divided and a division method provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of an image block segmentation result provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of an inter-frame prediction process provided in an embodiment of this application;

[0052] Figure 4 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0053] Figure 5 This is a flowchart illustrating another video processing method provided in an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] Video is a sequence of consecutive image frames. Due to the persistence of vision, when this sequence is played back at a certain rate, we perceive continuous motion in the video. Generally, consecutive image frames in a video have high similarity, and there is a large amount of redundant information both within each frame and between consecutive frames. Therefore, before storing or transmitting video, video encoding technology is often used to compress the video to reduce its size, thereby effectively saving storage space and improving transmission efficiency. The basic principles of video encoding technology are introduced below:

[0058] In video coding technology, video can be encoded by dividing it into blocks, which is a top-down recursive process. Specifically, image frames in a video can be initially divided into multiple blocks of the same size. These blocks may not be divided, or they may be further divided according to the division method. The sub-blocks obtained from the further division may not be divided, or they may continue to be divided according to the division method, until the size of the resulting blocks reaches the division limit. All blocks obtained in the division process can be called image blocks.

[0059] Here, "no division" means directly using the current image block as a CU (Coding Unit) without further division. CU refers to the basic unit of coding prediction, such as... Figure 1In (a) shown, the image block is not further divided. Dividing refers to further subdividing the current image block into sub-blocks. The division method can include at least one of the following: horizontal binary division, vertical binary division, quadruple division, horizontal tri-division, and vertical tri-division. Horizontal binary division refers to dividing the image block into two sub-blocks along the horizontal direction. The two sub-blocks can be the same size or different sizes, such as... Figure 1 In example (b), the image block is divided into two equal-sized sub-blocks horizontally; vertical binary division refers to dividing the image block into two sub-blocks vertically, and the two sub-blocks can be the same size or different sizes, such as... Figure 1 In example (c), the image patch is divided into two equal-sized sub-blocks along the vertical direction; the four-part division method refers to dividing the image patch into four equal-sized sub-blocks, such as... Figure 1 As shown in (d); the horizontal trisection method refers to dividing an image block into three sub-blocks along the horizontal direction. The three sub-blocks can be the same size or different sizes, such as... Figure 1 In diagram (e), the image block is divided into three sub-blocks horizontally. From top to bottom, the first and third sub-blocks are the same size, while the second sub-block is a different size. Vertical trisection refers to dividing the image block into three sub-blocks vertically. The three sub-blocks can be the same or different sizes, such as... Figure 1 In (f) shown, the image block is divided into three sub-blocks along the vertical direction. From left to right, the first and third sub-blocks are the same size, while the second sub-block is a different size from them.

[0060] For a specific image block, the video coding standard does not specify the exact form it should be divided into; any division that conforms to the specified method is considered compliant with the standard. Different division methods result in different coding efficiencies. To maximize efficiency, the encoder often tries all possible division methods and ultimately selects the one with the best performance. There are numerous combinations of different division methods, leading to extremely high complexity in the coding process. In other words, during video coding, the initially divided image block will ultimately be divided into several Control Units (CUs). Figure 2 This illustrates one possible partitioning form of the image block obtained from the initial partitioning. All possible CUs can be organized into a search tree; generally, encoders use a top-down recursive process to traverse this block partitioning tree to determine the final partitioning form of the current image block; similar to a preorder traversal, the encoder first tries not to partition, and then recursively considers various cases of continuing partitioning; finally, after traversing all possible partitioning forms, the encoder will use the method with the highest encoding efficiency.

[0061] If the current image block attempts not to be divided, the encoder needs to select a suitable prediction mode. Prediction modes can be broadly categorized into intra-frame prediction and inter-frame prediction. Intra-frame prediction is a technique that performs predictive coding in the spatial domain; when using intra-frame prediction, the encoding of the current image frame does not need to reference information from other image frames. Inter-frame prediction is a technique that performs predictive coding in the temporal domain; when using inter-frame prediction, the encoding of the current image frame needs to reference information from other image frames. Inter-frame prediction can be further divided into translational inter-frame prediction and affine inter-frame prediction, depending on the motion type. In translational inter-frame prediction, the motion vector of each pixel in the CU is the same; in affine inter-frame prediction, the motion vector of each CU may be different, making it suitable for scaling and rotational motions.

[0062] During inter-frame prediction, the encoder constructs a reference frame list for the current image frame. The construction of this reference frame list is related to the direction of inter-frame prediction for the current image frame. The direction of inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction includes forward or backward inter-frame prediction, while bidirectional inter-frame prediction includes both forward and backward prediction. When the direction of inter-frame prediction is unidirectional, the reference frame list for the current image frame can include a forward reference frame list (corresponding to forward inter-frame prediction) or a backward reference frame list (corresponding to backward inter-frame prediction). When the direction of inter-frame prediction is bidirectional, the reference frame list for the current image frame can include both a forward reference frame list and a backward reference frame list. The forward reference frame list can contain several image frames whose encoding order and playback order are before the current image frame, while the backward reference frame list can contain several image frames whose encoding order is before the current image frame but whose playback order is after the current image frame. When performing unidirectional inter-frame prediction (including forward inter-frame prediction or backward inter-frame prediction) on the CU in the current image frame, the encoder can select one image frame from the forward reference frame list or the backward reference frame list as the reference frame for the CU in the current image frame; when performing bidirectional inter-frame prediction on the CU in the current image frame, the encoder can select one image frame from each of the two reference frame lists, for a total of two image frames as the reference frames for the CU in the current image frame.

[0063] The basic idea of ​​inter-frame prediction is to utilize the temporal correlation of video content. In one previously encoded image frame (one-way inter-frame prediction) or two image frames (two-way inter-frame prediction), a region with the most similar pixel distribution is selected to predict the current CU. Then, the location information of this similar region and the pixel difference between the current CU and the similar region are encoded. For example... Figure 3As shown, taking bidirectional inter-frame prediction as an example, image frame 301 is selected from the forward reference frame list, and image frame 302 is selected from the backward reference frame list. Image frames 301 and 302 are used as reference frames for the current CU, and regions with similar pixel distribution to the current CU are identified in the reference frames (e.g., the dashed region 3011 in image frame 301 and the dashed region 3021 in image frame 302). Generally, the smaller the pixel difference, the fewer bytes need to be transmitted, and the higher the coding efficiency. If the encoder does not ultimately select the most reasonable region for prediction, it can still generate a standard-compliant bitstream, but the coding efficiency will be compromised. Finding this most reasonable region is a computationally very complex process, which the encoder often achieves by comparing pixels one by one; this process is also called motion search.

[0064] Based on the above introduction to the basic principles of video coding technology, it can be seen that block-based coding involves prediction and reference frame selection. For a certain image block in a video, the sub-blocks obtained by dividing the image block through different methods all require reference frame selection during inter-frame prediction. The decision-making process for reference frame selection is complex and inefficient, resulting in low video coding efficiency. Therefore, this application proposes a video processing method. In this method, if the current image block to be encoded in the video is translated as a whole, a reference frame constraint can be applied to the current image block. The reference frame constraint means that the reference frame used for inter-frame prediction during the translation of the current image block is set as a restricted reference frame. Thus, when the sub-blocks obtained from the current image block and the further subdivisions of the sub-blocks perform inter-frame prediction, no reference frame selection is required. This simplifies the decision-making process for reference frame selection in video coding, accelerates the efficiency of reference frame selection during video coding, and thereby improves video coding efficiency.

[0065] The video processing method provided in this application can determine whether an image block is translated as a whole from the following two perspectives: First, the prediction effect of the image block for translational inter-frame prediction. If the image block is translated as a whole, the effect of the image block for translational inter-frame prediction is better than the effect of other predictions (such as intra-frame prediction and affine inter-frame prediction); Second, the motion consistency of the sub-blocks obtained by dividing the image block. If the image block is translated as a whole, the possibility of further dividing the sub-blocks is relatively small, and the motion information of the sub-blocks for inter-frame prediction is consistent with the motion information of the image block for inter-frame prediction when it is not divided.

[0066] In specific implementations, the video processing method proposed in this application embodiment can be executed by an encoder that uses H.266 as the video coding standard. H.266, also known as VVC (Versatile Video Coding), is the latest generation video coding standard. The video coding standard specifies the process and syntax for H.266 encoding and decoding. Therefore, the video processing method proposed in this application embodiment is applicable to the H.266 video coding standard. The encoder can be software or hardware deployed in a computer device, which can be a terminal or a server. The terminals mentioned in this application embodiment can include, but are not limited to, smartphones, tablets, laptops, desktop computers, smartwatches, smart home appliances, smart vehicle terminals, and aircraft. The servers mentioned in this application embodiment can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0067] It should be noted that the video processing method proposed in this application can also be combined with cloud computing technology. Cloud computing technology can provide powerful computing support for the video encoding process in the video processing method, further improving video encoding efficiency. Wherein:

[0068] Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0069] Cloud computing refers to the delivery and usage model of IT (Internet Technology) infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.

[0070] The video processing method provided in the embodiments of this application will now be described in more detail with reference to the accompanying drawings.

[0071] This application provides a video processing method, which can be executed by a computer device, such as a terminal or a server. Figure 4 As shown, the video processing method may include, but is not limited to, the following steps S401-S404:

[0072] S401, Identify the image blocks to be encoded in the video.

[0073] A video can be composed of multiple consecutive image frames. The image block to be encoded can be determined in the image frame currently being encoded in the video. The image frame currently being encoded can include encoded image blocks and unencoded image blocks. Alternatively, the image blocks included in the image frame currently being encoded may not be encoded. Any unencoded image block in the image frame currently being encoded can be determined as the image block to be encoded.

[0074] Image blocks may be left undivided, or they may be divided in multiple ways, including at least: horizontal two-part division, vertical two-part division, four-part division, horizontal three-part division, and vertical three-part division.

[0075] S402, Encode the image block, and divide the image block according to the first division method among multiple division methods, and encode the multiple sub-blocks obtained from the division.

[0076] As described above, the video processing method provided in this application can determine whether an image block is translated as a whole from two perspectives: the prediction effect of inter-frame prediction for image blocks and the motion consistency of the sub-blocks obtained from dividing the image block. Therefore, the image block can be encoded directly without division, i.e., encoded without division. During the encoding process without division, it is determined whether the image block uses inter-frame coding for translation, and the prediction effect when using inter-frame coding for translation. Then, the image block can be divided according to a first division method among multiple division methods, and the resulting sub-blocks can be encoded, i.e., the image block is divided and encoded according to the first division method. During the division and encoding process, it is determined whether the sub-blocks obtained from dividing the image block have inter-frame motion consistency with the image block without division. The first division method can be any of multiple division methods. Wherein:

[0077] The process of encoding an image block (i.e., encoding an image block without partitioning) may include: performing intra-frame prediction, inter-frame translation prediction, and inter-frame affine prediction on the image block respectively; if the prediction effect of inter-frame translation prediction on the image block is better than the prediction effect of intra-frame prediction on the image block, and better than the prediction effect of inter-frame affine prediction on the image block, then it can be determined that inter-frame translation prediction is performed on the image block; if the prediction effect of inter-frame translation prediction on the image block is worse than the prediction effect of any one or both of intra-frame prediction and inter-frame affine prediction on the image block, then it can be determined that inter-frame translation prediction is not performed on the image block.

[0078] The prediction performance of inter-frame translation prediction for image patches can be measured by the coding rate-distortion loss information after encoding the image patches. This coding rate-distortion loss information can be calculated based on the coding rate and coding distortion information after encoding the image patches. The coding rate-distortion loss information can be the ratio between the coding rate and the coding distortion information; where the coding distortion information refers to an estimate of the degree of distortion during the encoding process after inter-frame translation prediction. The prediction performance of intra-frame prediction and affine inter-frame prediction for image patches is similar to that of inter-frame translation prediction; for details, please refer to the above description of the prediction performance of inter-frame translation prediction for image patches, which will not be repeated here.

[0079] The process of encoding sub-blocks obtained by dividing an image block according to the first division method (i.e., dividing and encoding the image block according to the first division method) may include: for any sub-block obtained by dividing an image block according to the first division method, one may try encoding the sub-block without dividing it, i.e., directly encoding the sub-block, and one may try encoding the sub-block according to multiple division methods; then, the encoding effect of encoding the sub-block without dividing it can be compared with the encoding effect of encoding the sub-block according to multiple division methods; if the encoding effect of encoding the sub-block without dividing it is better than the encoding effect of encoding the sub-block according to multiple division methods, it can be determined that the sub-block will not be divided, i.e., the sub-block will not be further divided; if the encoding effect of encoding the sub-block without dividing it is worse than the encoding effect of any one or more of the multiple division methods, it can be determined that the sub-block will be divided, i.e., the sub-block needs to be further divided.

[0080] If it is determined that the sub-block will not be further divided, i.e., if it is determined that the sub-block will be directly encoded, then similar to directly encoding the image block, the prediction mode of the sub-block can be determined. If it is determined that the sub-block will be inter-coded, then the motion information during inter-frame coding of the sub-block can be obtained. The motion information may include, but is not limited to: prediction mode, reference frame selection mode, and reference frame index. The prediction mode can refer to the prediction mode used for inter-frame prediction, which may include translational inter-frame prediction mode or affine inter-frame prediction mode. The reference frame selection mode can refer to the mode of selecting reference frames during inter-frame prediction. The reference frame selection mode is related to the direction of inter-frame prediction. When the inter-frame prediction is forward inter-frame prediction, the reference frame selection mode can refer to the mode of selecting reference frames from the forward reference frame list; when the inter-frame prediction is backward inter-frame prediction, the reference frame selection mode can refer to the mode of selecting reference frames from the backward reference frame list; when the inter-frame prediction is bidirectional inter-frame prediction, the reference frame selection mode can refer to the mode of selecting reference frames from both the forward and backward reference frame lists. The reference frame index refers to the index of the reference frame selected during inter-frame prediction within the reference frame list.

[0081] Similar to the prediction effect of image patches, the coding effect of sub-blocks can be measured by the coding rate-distortion loss information of sub-blocks with or without partitioning. That is, the coding effect of sub-blocks without partitioning can be measured by the coding rate-distortion information, and vice versa. The coding rate-distortion loss information for sub-blocks without partitioning can be calculated based on the coding rate and coding distortion information. This information can be the ratio between the coding rate and the coding distortion information; where the coding distortion information refers to an estimate of the degree of distortion during the coding process without partitioning. The coding effect of sub-blocks with partitioning is similar to that of sub-blocks without partitioning; for details, please refer to the section on coding effect of sub-blocks without partitioning, which will not be repeated here.

[0082] S403, during the encoding process of an image block and multiple sub-blocks, if the image block performs translational inter-frame prediction, and none of the multiple sub-blocks are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block performs translational inter-frame prediction is obtained.

[0083] S404, set the reference frame used for inter-frame prediction of image patch translation to the restricted reference frame of image patch.

[0084] In one implementation of steps S403-S404, if the following three conditions are met: the image block performs translational inter-frame prediction, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then it can be determined that the image block is performing translational motion as a whole, and the reference frame used when the image block performs translational inter-frame prediction can be obtained, and the reference frame used when the image block performs translational inter-frame prediction can be restricted; if any one or more of the above three conditions are not met, then it can be determined that the image block is not performing translational motion as a whole, and there is no need to restrict the reference frame for the image block.

[0085] The motion information of multiple sub-blocks during inter-frame prediction is consistent with that of the image block during inter-frame prediction. This can include: the prediction mode of multiple sub-blocks during inter-frame prediction is consistent with that of the image block during inter-frame prediction, and both are translation inter-frame predictions; the reference frame selection mode of multiple sub-blocks during inter-frame prediction is consistent with that of the image block during inter-frame prediction; and the reference frame index of multiple sub-blocks during inter-frame prediction is consistent with that of the image block during inter-frame prediction. The reference frame index refers to the index of the selected reference frame in the reference frame list during inter-frame prediction. The consistent reference frame index means that the reference frame selected by multiple sub-blocks during inter-frame prediction is the same as the reference frame selected by the image block during inter-frame prediction.

[0086] In another implementation of steps S403-S404, if the following four conditions are met: the image block performs translational inter-frame prediction, and the prediction residual signal of the image block when performing translational inter-frame prediction satisfies the inter-frame correlation condition, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction, then it can be determined that the image block performs translational motion as a whole, and the reference frame used when the image block performs translational inter-frame prediction can be obtained, and the reference frame used when the image block performs translational inter-frame prediction can be restricted; if any one or more of the above four conditions are not met, it can be determined that the image block does not perform translational motion as a whole, and then there is no need to restrict the reference frame of the image block.

[0087] The prediction residual signal for image patch inter-frame prediction can be obtained in the following way: a reference region associated with the image patch can be determined in the reference frame used for image patch inter-frame prediction. The reference region is obtained by motion search of the reference frame used for image patch inter-frame prediction. The reference region is the region in the reference frame used for image patch inter-frame prediction whose pixel distribution is most similar to that of the image patch. Then, the prediction residual signal for image patch inter-frame prediction can be determined based on the pixel difference between the image patch and the reference region.

[0088] Whether the prediction residual signal of an image patch satisfies the inter-frame association condition during inter-frame translation prediction can be determined in the following way: The prediction residual signal can be compressed to reduce its data size. Compression can include quantization, transform, or transform followed by quantization. Then, the compressed result of the prediction residual signal can be compared with a residual threshold. If the compressed result is less than the residual threshold, the prediction residual signal satisfies the inter-frame association condition. If the compressed result is greater than or equal to the residual threshold, the prediction residual signal does not satisfy the inter-frame association condition.

[0089] Furthermore, the prediction residual signal during inter-frame translation prediction of an image patch can include multiple signal components. The compression result of the prediction residual signal is compared with a residual threshold. Specifically, the compression result of the target signal component among the multiple signal components is compared with the residual threshold. If the compression result of the target signal component is less than the residual threshold, it can be determined that the prediction residual signal during inter-frame translation prediction of the image patch satisfies the inter-frame association condition. If the compression result of the target signal component is greater than or equal to the residual threshold, it can be determined that the prediction residual signal during inter-frame translation prediction of the image patch does not satisfy the inter-frame association condition.

[0090] Furthermore, when the residual threshold is sufficiently small, for example, when the residual threshold is 0, the comparison between the compression result of the target signal component and the residual threshold will change as follows: if the compression result of the target signal component is equal to the residual threshold, it can be determined that the prediction residual signal of the image block when performing translation inter-frame prediction satisfies the inter-frame association condition; if the compression result of the target signal component is greater than the residual threshold, it can be determined that the prediction residual signal of the image block when performing translation inter-frame prediction does not satisfy the inter-frame association condition.

[0091] The target signal component can be the signal component with the highest correlation to the image patch presentation among the multiple signal components of the prediction residual signal. In other words, the target signal component plays a decisive role in the presentation of the image patch. For example, in a video generated according to YUV (a color encoding method), the video can include Y components and UV components. The Y component refers to the Luminance or Luma component (luminance component), which is the grayscale value, and the UV component refers to the Chrominance or Chroma component (chrominance component), which is used to describe color and saturation. Therefore, the prediction residual signal can include Y signal components and UV signal components, and the target signal component can be the Y signal component. When the compression result of the Y signal component is 0, it can be determined that the prediction residual signal of the image patch does not meet the inter-frame correlation condition when performing translation inter-frame prediction.

[0092] It should be noted that restricting the reference frames used when performing inter-frame translation prediction on image patches can specifically mean setting the reference frames used for inter-frame translation prediction on image patches as restricted reference frames for the image patches. Reference frame restriction can be performed within the reference frame list of the image patch, which can include one or more reference frames. The reference frames used for inter-frame translation prediction on the image patch can be selected from this list. The methods for restricting reference frames in the reference frame list of the image patch can include any of the following: First, a state setting method, where the reference frames used for inter-frame translation prediction on the image patch can be set as available reference frames, and other reference frames in the reference frame list other than those used for inter-frame translation prediction on the image patch can be set as unavailable reference frames; where available reference frames refer to reference frames that can be used in the reference frame list of the image patch, and unavailable reference frames refer to reference frames that cannot be used in the reference frame list of the image patch. The second method, deletion, can retain the reference frames used when performing inter-frame prediction for the image patch in the reference frame list, and can delete other reference frames in the reference frame list besides those used when performing inter-frame prediction for the image patch.

[0093] After applying reference frame restrictions, the restricted reference frames can be used as reference frames for inter-frame prediction when an image block is divided into other sub-blocks according to the second partitioning method among multiple partitioning methods. The second partitioning method refers to any partitioning method other than the first partitioning method allowed for the image block. In other words, whether reference frame restrictions are applied through state settings or deletion, the purpose is to ensure that image frames used for translational inter-frame prediction within the image block's reference frame list can be used for inter-frame prediction in other sub-blocks divided according to the second partitioning method and their subsequent partitioning blocks. Conversely, image frames not used for translational inter-frame prediction within the image block's reference frame list cannot be used for inter-frame prediction in other sub-blocks divided according to the second partitioning method and their subsequent partitioning blocks. This is because when determining that an image block is being translated as a whole, it can be assumed that the image block has already found a reference frame that matches the actual motion trajectory, making further reference frame selection unnecessary.

[0094] Here, we also need to introduce the construction method of the reference frame list: In general, the reference frame list is constructed on a per-image-frame basis. The encoder can construct the reference frame list for each image frame according to the default reference frame list construction rules, so that all image blocks within a frame have the same reference frame list. However, in the embodiments of this application, both the above-mentioned state setting method and deletion method restrict the range of reference frames available in the reference frame list of the image block. The restricted reference frame list of the parent block is still valid for its child blocks. The image block that is divided into child blocks can be called the parent block of the child block. Therefore, when the parent block exists, the reference frame list of the parent block should be reused, that is, the reference frame list of the parent block of the image block can be determined as the reference frame list of the image frame. When the parent block does not exist, that is, when the current image block is the largest size image block, the reference frame list of the image frame to which the image block belongs in the video can be determined as the reference frame list of the image block. The reference frame list of the image frame can be constructed using the default reference frame list construction rules. Based on the parent-child inheritance relationship of the reference frame list in the embodiments of this application, after the reference frame list of an image block is restricted, the restricted reference frame list of the image block can be reused by other sub-blocks obtained by dividing the image block according to the second partitioning method, as well as the further partitioning blocks of other sub-blocks, thereby simplifying the decision-making process for reference frame selection.

[0095] If no reference frame is imposed on the image block, when the image block is divided into other sub-blocks according to the second division method, and when the other sub-blocks are further divided into further sub-blocks, reference frame selection is required when performing inter-frame prediction for the other sub-blocks and the further sub-blocks of the other sub-blocks.

[0096] After performing undivided encoding on image blocks and dividing and encoding image blocks according to various division methods, the method with the best encoding effect can be selected to process the image block. Specifically, if the undivided encoding effect is good, the image block can be used as a CU and directly encoded; if the division encoding effect under a certain division method is good, the image block can be divided according to that division method, and the resulting sub-blocks can be used as new image blocks. It is then determined whether reference frame constraints need to be applied to the new image blocks, and whether to perform undivided encoding or division encoding on the new image blocks. This is a recursive process.

[0097] In this embodiment, when an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks obtained by dividing the image block according to the first division method among multiple division methods are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, the reference frame used when the image block is subjected to translational inter-frame prediction can be set as the restricted reference frame of the image block. By setting the restricted reference frame, the restricted reference frame can be used as the reference frame when the image block is divided into other sub-blocks according to the second division method among multiple division methods, and other sub-blocks and their further division blocks are subjected to inter-frame prediction. That is, after setting the restricted reference frame, when the image block is divided into other sub-blocks according to the second division method among multiple division methods, the other sub-blocks and their further division blocks can directly use the restricted reference frame as the reference frame used during inter-frame prediction without the need for reference frame selection. This simplifies the decision-making process for reference frame selection in video encoding, accelerates the selection efficiency of reference frames in video encoding, and thus improves video encoding efficiency.

[0098] As described above, image blocks need to be able to be divided in multiple ways, including: horizontal binary division, vertical binary division, quadruple division, horizontal tri-division, and vertical tri-division. The following description uses an example where the first division method is quadruple division, and the second division methods are horizontal binary division, vertical binary division, horizontal tri-division, and vertical tri-division, to illustrate the video processing method provided in this application. Figure 5 As shown, the video processing method may include, but is not limited to, the following steps S501-S510:

[0099] S501, determine the current image block to be encoded in the video.

[0100] S502, perform a parent block check on the image block. If the image block has a parent block, proceed to step S503; if the image block does not have a parent block, proceed to step S504.

[0101] S503, determine the reference frame list of the parent block of the image block as the reference frame list of the image block.

[0102] S504: Construct a reference frame list for the image patch according to the default reference frame list construction rules.

[0103] As mentioned above, the reference frame list is constructed on a per-image-frame basis. The encoder can construct a reference frame list for each image frame according to the default reference frame list construction rules. Thus, all image blocks within a frame have the same reference frame list. In other words, the reference frame list of the image frame to which the image block belongs can be constructed according to the default reference frame list construction rules, and the reference frame list of the image frame to which the image block belongs is determined as the reference frame list of the image block.

[0104] S505 performs undivided encoding on image blocks.

[0105] S506 divides and encodes image blocks according to a four-part division method.

[0106] The image block is divided and encoded using a four-part partitioning method. Specifically, this includes dividing the image block into four equal-sized sub-blocks and then encoding each of the four sub-blocks. The reason for choosing the four-part partitioning method is that it produces four equal-sized sub-blocks, resulting in more reliable motion consistency.

[0107] S507 performs inter-frame correlation checks on image blocks when encoding them without segmentation.

[0108] Inter-frame correlation check refers to determining whether to perform inter-frame prediction for the image patch and whether the prediction residual signal of the image patch satisfies the inter-frame correlation condition. If inter-frame prediction for the image patch is performed and the prediction residual signal of the image patch satisfies the inter-frame correlation condition, then the inter-frame correlation check is determined to be passed, and step S508 is executed; if inter-frame prediction for the image patch is not performed, or if inter-frame prediction for the image patch is performed but the prediction residual signal of the image patch does not satisfy the inter-frame correlation condition, then the inter-frame correlation check is determined to be passed, and step S510 is executed.

[0109] S508, when dividing and encoding an image block, performs a motion consistency check on the four sub-blocks obtained from the division.

[0110] The motion consistency check refers to: determining whether the four sub-blocks need to be further divided. If none of the four sub-blocks are further divided, then the motion information predicted by the four sub-blocks for inter-frame prediction is consistent with the motion information predicted by the image block for inter-frame prediction. If none of the four sub-blocks are further divided, and the motion information predicted by the four sub-blocks for inter-frame prediction is consistent with the motion information predicted by the image block for inter-frame prediction, then the consistency check is considered passed, and step S509 is executed. If there are sub-blocks among the four sub-blocks that need to be further divided, or if none of the four sub-blocks are further divided, but the motion information predicted by the four sub-blocks for inter-frame prediction is inconsistent with the motion information predicted by the image block for inter-frame prediction, then the consistency check is considered failed, and step S510 is executed.

[0111] S509, reference frame constraint is applied to the image block.

[0112] Reference frame constraint refers to setting the reference frame used for inter-frame prediction of an image patch as a constraint reference frame in the reference frame list of the image patch.

[0113] S510 divides and encodes image blocks according to horizontal two-part division, vertical two-part division, horizontal three-part division, and vertical three-part division.

[0114] If a reference frame is imposed on the image block, then when inter-frame prediction is needed for other sub-blocks obtained from dividing the image block according to any of the horizontal two-partition, vertical two-partition, horizontal three-partition, and vertical three-partition methods, as well as for the further sub-blocks of those sub-blocks, the reference frame used for the translational inter-frame prediction of the image block can be used as the reference frame for inter-frame prediction of those sub-blocks and their further sub-blocks. If no reference frame is imposed on the image block, then when inter-frame prediction is needed for other sub-blocks obtained from dividing the image block according to any of the horizontal two-partition, vertical two-partition, horizontal three-partition, and vertical three-partition methods, a reference frame must be selected from the reference frame list.

[0115] After trying to encode the image block without partitioning it, and trying to partition and encode the image block according to various partitioning methods, a suitable method for processing the image block can be determined, and the encoding process of the current image block ends.

[0116] In this embodiment, when an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks obtained by dividing the image block according to the first division method among multiple division methods are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, the reference frame used when the image block is subjected to translational inter-frame prediction can be set as the restricted reference frame of the image block. By setting the restricted reference frame, the restricted reference frame can be used as the reference frame when the image block is divided into other sub-blocks according to the second division method among multiple division methods, and other sub-blocks and their further division blocks are subjected to inter-frame prediction. That is, after setting the restricted reference frame, when the image block is divided into other sub-blocks according to the second division method among multiple division methods, the other sub-blocks and their further division blocks can directly use the restricted reference frame as the reference frame used during inter-frame prediction without the need for reference frame selection. This simplifies the decision-making process for reference frame selection in video encoding, accelerates the selection efficiency of reference frames in video encoding, and thus improves video encoding efficiency.

[0117] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0118] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application. The video processing device can be installed in the computer equipment provided in the embodiment of this application. The computer equipment can be a terminal or a server. Figure 6 The video processing apparatus shown may be a computer program (including program code) running on a computer device, which can be used to execute... Figure 4 or Figure 5 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 6 The video processing apparatus may include the following units:

[0119] Processing unit 601 is used to determine the image blocks to be encoded in the video, and the image blocks can be divided according to multiple division methods;

[0120] The processing unit 601 is also used to encode the image block, and to divide the image block according to a first division method among multiple division methods, and to encode the multiple sub-blocks obtained from the division.

[0121] The acquisition unit 602 is used to acquire the reference frame used when the image block performs translational inter-frame prediction during the encoding process of the image block and multiple sub-blocks, if the image block performs translational inter-frame prediction and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction.

[0122] Processing unit 601 is further configured to set the reference frame used when performing translational inter-frame prediction of the image block as a limiting reference frame for the image block; the limiting reference frame is used as a reference frame when the image block is divided into other sub-blocks according to the second division method among the multiple division methods, and other sub-blocks are further divided into blocks for inter-frame prediction.

[0123] In one implementation, the acquisition unit 602 is used to acquire the reference frame used when the image block performs translational inter-frame prediction if the image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction. Specifically, it is used to perform the following steps:

[0124] If an image block is subjected to inter-frame translation prediction, and the prediction residual signal of the image block satisfies the inter-frame association condition, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block is subjected to inter-frame translation prediction is obtained.

[0125] In one implementation, the processing unit 601 is further configured to perform the following steps:

[0126] Acquire the prediction residual signal when performing inter-frame translation prediction on image patches;

[0127] The predicted residual signal is compressed.

[0128] If the compression result of the predicted residual signal is less than the residual threshold, then the predicted residual signal of the image block during inter-frame prediction satisfies the inter-frame correlation condition.

[0129] In one implementation, the prediction residual signal includes multiple signal components; the processing unit 601 is configured to determine that the prediction residual signal during the inter-frame prediction of the image block satisfies the inter-frame correlation condition if the compression result of the prediction residual signal is less than the residual threshold, specifically by performing the following steps:

[0130] If the compression result of the target signal component among multiple signal components is less than the residual threshold, then it is determined that the compression result of the predicted residual signal is less than the residual threshold.

[0131] In one implementation, when processing unit 601 acquires the prediction residual signal during inter-frame translation prediction of image blocks, it specifically performs the following steps:

[0132] In the reference frame used for inter-frame prediction of image patch translation, the reference region associated with the image patch is determined;

[0133] Based on the pixel difference between the image patch and the reference region, the prediction residual signal is determined when the image patch is used for inter-frame prediction with translation.

[0134] In one implementation, when the processing unit 601 sets the reference frame used for inter-frame prediction of image blocks as the limiting reference frame of the image blocks, it specifically performs the following steps:

[0135] Obtain the reference frame list for the image patch; the reference frame list for the image patch includes one or more reference frames, and the reference frames used when performing inter-frame prediction for translation of the image patch are selected from the reference frame list of the image patch;

[0136] In the reference frame list of the image patch, the reference frame used for inter-frame prediction of the image patch during translation is set as the limiting reference frame of the image patch.

[0137] In one implementation, when processing unit 601 sets the reference frame used for inter-frame prediction of the image block during translation into the reference frame list of the image block as the limiting reference frame of the image block, it specifically performs the following steps:

[0138] Set the reference frames used for inter-frame prediction of image blocks in the reference frame list to available reference frames;

[0139] Additionally, the reference frames in the reference frame list of the image patch, except for the reference frames used when performing inter-frame prediction for translation of the image patch, are set to unavailable reference frames.

[0140] In one implementation, when processing unit 601 sets the reference frame used for inter-frame prediction of the image block during translation into the reference frame list of the image block as the limiting reference frame of the image block, it specifically performs the following steps:

[0141] The reference frames in the list of retained image patches are the reference frames used when performing translational inter-frame prediction for the image patches;

[0142] In addition, the list of reference frames to be removed from the image patch includes all reference frames except those used for inter-frame prediction of the image patch during translation.

[0143] In one implementation, when processing unit 601 obtains the reference frame list of image blocks, it specifically performs the following steps:

[0144] If an image block has a parent block, then the reference frame list of the parent block of the image block is determined as the reference frame list of the image block.

[0145] If an image block does not have a parent block, the reference frame list of the image frame to which the image block belongs in the video is determined as the reference frame list of the image block; the reference frame list of the image frame to which the image block belongs in the video is constructed according to the reference frame list construction rules.

[0146] In one implementation, the motion information includes a prediction mode, a reference frame selection mode, and a reference frame index;

[0147] The motion information of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction includes: the prediction mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction, and both are translation inter-frame predictions; the reference frame selection mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction; and the reference frame index of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction.

[0148] In one implementation, the first partitioning method is any one of multiple partitioning methods, and the second partitioning method is any other partitioning method besides the first partitioning method among the multiple partitioning methods;

[0149] The various division methods include at least: horizontal two-part division, vertical two-part division, four-part division, horizontal three-part division, and vertical three-part division; the first division method is the four-part division method.

[0150] According to another embodiment of this application, Figure 6 The various units in the video processing apparatus shown can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the video processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0151] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 4 or Figure 5 The computer program (including program code) for each step involved in some or all of the methods shown, to construct such... Figure 6 The video processing apparatus shown herein, and the video processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0152] In this embodiment, when an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks obtained by dividing the image block according to the first division method among multiple division methods are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, the reference frame used when the image block is subjected to translational inter-frame prediction can be set as the restricted reference frame of the image block. By setting the restricted reference frame, the restricted reference frame can be used as the reference frame when the image block is divided into other sub-blocks according to the second division method among multiple division methods, and other sub-blocks and their further division blocks are subjected to inter-frame prediction. That is, after setting the restricted reference frame, when the image block is divided into other sub-blocks according to the second division method among multiple division methods, the other sub-blocks and their further division blocks can directly use the restricted reference frame as the reference frame used during inter-frame prediction without the need for reference frame selection. This simplifies the decision-making process for reference frame selection in video encoding, accelerates the selection efficiency of reference frames in video encoding, and thus improves video encoding efficiency.

[0153] Based on the above methods and apparatus embodiments, this application provides a computer device, which may be a terminal or a server. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 The computer device shown includes at least a processor 701, an input interface 702, an output interface 703, and a computer-readable storage medium 704. The processor 701, input interface 702, output interface 703, and computer-readable storage medium 704 can be connected via a bus or other means.

[0154] Computer-readable storage medium 704 can be stored in the memory of a computer device. Computer-readable storage medium 704 is used to store computer programs, including computer instructions. Processor 701 is used to execute the program instructions stored in computer-readable storage medium 704. Processor 701 (or CPU (Central Processing Unit)) is the computing and control core of the computer device. It is adapted to implement one or more computer instructions, specifically to load and execute one or more computer instructions to achieve corresponding method flows or corresponding functions.

[0155] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores one or more computer instructions suitable for loading and execution by a processor. These computer instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0156] In some embodiments, the processor 701 may load and execute one or more computer instructions stored in the computer-readable storage medium 704 to implement the aforementioned related... Figure 4 or Figure 5 The corresponding steps of the video processing method shown are as follows. In a specific implementation, the computer instructions in the computer-readable storage medium 704 are loaded by the processor 701 and executed as follows:

[0157] Identify the image blocks to be encoded in the video; these image blocks can be divided in multiple ways.

[0158] Encode the image block, and divide the image block according to the first division method among multiple division methods, and encode the multiple sub-blocks obtained from the division;

[0159] In the process of encoding an image block and multiple sub-blocks, if the image block is subjected to translational inter-frame prediction, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block is subjected to translational inter-frame prediction is obtained.

[0160] The reference frame used when performing translational inter-frame prediction on an image block is set as the limiting reference frame for the image block; the limiting reference frame is used as the reference frame when the image block is divided into other sub-blocks according to the second division method among the multiple division methods, and other sub-blocks are further divided into blocks for inter-frame prediction.

[0161] In one implementation, the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701. If the image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then when obtaining the reference frame used for the image block's translational inter-frame prediction, the following steps are specifically performed:

[0162] If an image block is subjected to inter-frame translation prediction, and the prediction residual signal of the image block satisfies the inter-frame association condition, and no sub-block among the multiple sub-blocks is further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used when the image block is subjected to inter-frame translation prediction is obtained.

[0163] In one implementation, the computer instructions in the computer-readable storage medium 704 are loaded by the processor 701 and are also used to perform the following steps:

[0164] Acquire the prediction residual signal when performing inter-frame prediction with translation of image patches;

[0165] The predicted residual signal is compressed.

[0166] If the compression result of the predicted residual signal is less than the residual threshold, then the predicted residual signal of the image block during inter-frame prediction satisfies the inter-frame correlation condition.

[0167] In one implementation, the predicted residual signal includes multiple signal components; the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701. If the compression result of the predicted residual signal is less than the residual threshold, and the predicted residual signal during the inter-frame prediction of the image block satisfies the inter-frame correlation condition, the following steps are specifically performed:

[0168] If the compression result of the target signal component among multiple signal components is less than the residual threshold, then it is determined that the compression result of the predicted residual signal is less than the residual threshold.

[0169] In one implementation, when the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701 to acquire the prediction residual signal during inter-frame translation prediction of image blocks, they are specifically used to perform the following steps:

[0170] In the reference frame used for inter-frame prediction of image patch translation, the reference region associated with the image patch is determined;

[0171] Based on the pixel difference between the image patch and the reference region, the prediction residual signal is determined when the image patch is used for inter-frame prediction with translation.

[0172] In one implementation, when the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701 to set the reference frame used for inter-frame prediction of image patch translation as the limiting reference frame of the image patch, the instructions specifically perform the following steps:

[0173] Obtain the reference frame list for the image patch; the reference frame list for the image patch includes one or more reference frames, and the reference frames used when performing inter-frame prediction for translation of the image patch are selected from the reference frame list of the image patch;

[0174] In the reference frame list of the image patch, the reference frame used for inter-frame prediction of the image patch during translation is set as the limiting reference frame of the image patch.

[0175] In one implementation, the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701. When setting the reference frame used for inter-frame prediction of the image block in the reference frame list of the image block as the limiting reference frame of the image block, the instructions specifically perform the following steps:

[0176] Set the reference frames used for inter-frame prediction of image blocks in the reference frame list to available reference frames;

[0177] Additionally, the reference frames in the reference frame list of the image patch, except for the reference frames used when performing inter-frame prediction for translation of the image patch, are set to unavailable reference frames.

[0178] In one implementation, the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701. When setting the reference frame used for inter-frame prediction of the image block in the reference frame list of the image block as the limiting reference frame of the image block, the instructions specifically perform the following steps:

[0179] The reference frames in the list of retained image patches are the reference frames used when performing translational inter-frame prediction for the image patches;

[0180] In addition, the list of reference frames to be removed from the image patch includes all reference frames except those used for inter-frame prediction of the image patch during translation.

[0181] In one implementation, when the computer instructions in the computer-readable storage medium 704 are loaded and executed by the processor 701 to obtain a reference frame list of image blocks, they are specifically used to perform the following steps:

[0182] If an image block has a parent block, then the reference frame list of the parent block of the image block is determined as the reference frame list of the image block.

[0183] If an image block does not have a parent block, the reference frame list of the image frame to which the image block belongs in the video is determined as the reference frame list of the image block; the reference frame list of the image frame to which the image block belongs in the video is constructed according to the reference frame list construction rules.

[0184] In one implementation, the motion information includes a prediction mode, a reference frame selection mode, and a reference frame index;

[0185] The motion information of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction includes: the prediction mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction, and both are translation inter-frame predictions; the reference frame selection mode of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction; and the reference frame index of multiple sub-blocks when performing inter-frame prediction is consistent with that of the image block when performing inter-frame prediction.

[0186] In one implementation, the first partitioning method is any one of multiple partitioning methods, and the second partitioning method is any other partitioning method besides the first partitioning method among the multiple partitioning methods;

[0187] The various division methods include at least: horizontal two-part division, vertical two-part division, four-part division, horizontal three-part division, and vertical three-part division; the first division method is the four-part division method.

[0188] In this embodiment, when an image block is subjected to translational inter-frame prediction, and none of the multiple sub-blocks obtained by dividing the image block according to the first division method among multiple division methods are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, the reference frame used when the image block is subjected to translational inter-frame prediction can be set as the restricted reference frame of the image block. By setting the restricted reference frame, the restricted reference frame can be used as the reference frame when the image block is divided into other sub-blocks according to the second division method among multiple division methods, and other sub-blocks and their further division blocks are subjected to inter-frame prediction. That is, after setting the restricted reference frame, when the image block is divided into other sub-blocks according to the second division method among multiple division methods, the other sub-blocks and their further division blocks can directly use the restricted reference frame as the reference frame used during inter-frame prediction without the need for reference frame selection. This simplifies the decision-making process for reference frame selection in video encoding, accelerates the selection efficiency of reference frames in video encoding, and thus improves video encoding efficiency.

[0189] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video processing methods provided in the various alternative embodiments described above.

[0190] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video processing method, characterized in that, include: Identify the image blocks to be encoded in the video, wherein the image blocks can be divided according to multiple partitioning methods; The image block is encoded, and the image block is divided according to the first division method among the multiple division methods, and the resulting sub-blocks are encoded. During the encoding of the image block and the plurality of sub-blocks, if the image block is subjected to translational inter-frame prediction, and none of the plurality of sub-blocks are further divided, and the motion information of the plurality of sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used by the image block during translational inter-frame prediction is obtained. The motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction includes: the prediction mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the prediction mode of the image block when performing inter-frame prediction, and both are translation inter-frame prediction; the reference frame selection mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the reference frame selection mode of the image block when performing inter-frame prediction; and the reference frame selected by the multiple sub-blocks when performing inter-frame prediction is the same reference frame as the reference frame selected by the image block when performing inter-frame prediction. The reference frame used when performing translational inter-frame prediction on the image block is set as the limiting reference frame of the image block; the limiting reference frame is used as the reference frame when the image block is divided into other sub-blocks according to the second division method among the multiple division methods, and when the other sub-blocks and the further division blocks of the other sub-blocks are performed for inter-frame prediction.

2. The method as described in claim 1, wherein if the image block undergoes translational inter-frame prediction, and none of the plurality of sub-blocks is further divided, and the motion information of the plurality of sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then obtaining the reference frame used for the translational inter-frame prediction of the image block includes: If the image block is subjected to translational inter-frame prediction, and the prediction residual signal of the image block during translational inter-frame prediction satisfies the inter-frame association condition, and none of the multiple sub-blocks are further divided, and the motion information of the multiple sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then the reference frame used by the image block during translational inter-frame prediction is obtained.

3. The method as described in claim 2, characterized in that, The method further includes: Acquire the prediction residual signal when performing inter-frame translation prediction on the image block; The predicted residual signal is compressed. If the compression result of the predicted residual signal is less than the residual threshold, then it is determined that the predicted residual signal of the image block when performing translation inter-frame prediction satisfies the inter-frame correlation condition.

4. The method as described in claim 3, characterized in that, The predicted residual signal includes multiple signal components; if the compression result of the predicted residual signal is less than the residual threshold, then determining that the predicted residual signal of the image patch satisfies the inter-frame correlation condition during translation inter-frame prediction includes: If the compression result of the target signal component among the plurality of signal components is less than the residual threshold, then it is determined that the compression result of the predicted residual signal is less than the residual threshold.

5. The method as described in claim 3, characterized in that, The step of acquiring the prediction residual signal when performing inter-frame prediction for the image patch includes: In the reference frame used when performing inter-frame prediction of the image patch, a reference region associated with the image patch is determined; Based on the pixel difference between the image block and the reference region, the prediction residual signal for the image block during inter-frame translation prediction is determined.

6. The method as described in claim 1, characterized in that, The step of setting the reference frame used for inter-frame prediction of the image patch by translation includes: Obtain a reference frame list for the image patch; the reference frame list for the image patch includes one or more reference frames, and the reference frames used when performing translation inter-frame prediction for the image patch are selected from the reference frame list for the image patch; In the reference frame list of the image patch, the reference frame used when performing translation inter-frame prediction of the image patch is set as the limiting reference frame of the image patch.

7. The method as described in claim 6, characterized in that, Setting the reference frame used for inter-frame prediction of the image patch in the reference frame list of the image patch as the limiting reference frame of the image patch includes: In the reference frame list of the image patch, the reference frame used for inter-frame prediction of the image patch during translation is set as an available reference frame; and, In the reference frame list of the image block, set all reference frames other than the reference frames used when performing inter-frame prediction for translation of the image block as unusable reference frames.

8. The method as described in claim 6, characterized in that, Setting the reference frame used for inter-frame prediction of the image patch in the reference frame list of the image patch as the limiting reference frame of the image patch includes: The reference frame list that retains the image patch includes the reference frames used when performing inter-frame prediction for the image patch; and, Remove all reference frames from the reference frame list of the image block except those used when performing inter-frame prediction for translation of the image block.

9. The method as described in claim 6, characterized in that, The reference frame list for obtaining the image patch includes: If the image block has a parent block, then the reference frame list of the parent block of the image block is determined as the reference frame list of the image block; If the image block does not have a parent block, then the reference frame list of the image frame to which the image block belongs in the video is determined as the reference frame list of the image block; the reference frame list of the image frame to which the image block belongs in the video is constructed according to the reference frame list construction rules.

10. The method as described in claim 1, characterized in that, The motion information includes prediction mode, reference frame selection mode, and reference frame index; The motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction includes: the prediction mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the prediction mode of the image block when performing inter-frame prediction, and both are translation inter-frame prediction; the reference frame selection mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the reference frame selection of the image block when performing inter-frame prediction; and the reference frame index of the multiple sub-blocks when performing inter-frame prediction is consistent with the reference frame index of the image block when performing inter-frame prediction.

11. The method according to any one of claims 1-10, characterized in that, The first division method is a four-part division method among the multiple division methods, and the second division method is a division method other than the first division method among the multiple division methods; The various division methods include at least: horizontal two-part division, vertical two-part division, four-part division, horizontal three-part division, and vertical three-part division.

12. A video processing apparatus, characterized in that, include: The processing unit is used to determine the image blocks to be encoded in the video, wherein the image blocks can be divided according to multiple division methods; The processing unit is further configured to encode the image block, and to divide the image block according to a first division method among the multiple division methods, and to encode the multiple sub-blocks obtained from the division. The acquisition unit is configured to, during the encoding process of the image block and the plurality of sub-blocks, if the image block is subjected to translational inter-frame prediction, and none of the plurality of sub-blocks are further divided, and the motion information of the plurality of sub-blocks during inter-frame prediction is consistent with the motion information of the image block during inter-frame prediction, then acquire the reference frame used when the image block is subjected to translational inter-frame prediction. The motion information of the multiple sub-blocks when performing inter-frame prediction is consistent with the motion information of the image block when performing inter-frame prediction includes: the prediction mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the prediction mode of the image block when performing inter-frame prediction, and both are translation inter-frame prediction; the reference frame selection mode of the multiple sub-blocks when performing inter-frame prediction is consistent with the reference frame selection mode of the image block when performing inter-frame prediction; and the reference frame selected by the multiple sub-blocks when performing inter-frame prediction is the same reference frame as the reference frame selected by the image block when performing inter-frame prediction. The processing unit is further configured to set the reference frame used when performing translation inter-frame prediction on the image block as a limiting reference frame for the image block; the limiting reference frame is used as a reference frame when the image block is divided into other sub-blocks according to the second division method among the multiple division methods, and when the other sub-blocks and the further division blocks of the other sub-blocks are performed for inter-frame prediction.

13. A computer device, characterized in that, The computer device includes: A processor is a tool for implementing computer programs. A computer-readable storage medium storing a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the video processing method as described in any one of claims 1-11.