Method and apparatus for predictive coding of moving images
By determining whether to skip motion estimation in video encoding based on pre-analysis information, the method reduces computational overhead and enhances encoding speed and efficiency, addressing the complexity of HEVC interframe prediction.
Patent Information
- Application Number
- JP2023535576
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-12-11
- Filing Date
- 2021-12-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-03
AI Technical Summary
The computational overhead in video encoding is high due to the need for motion estimation and compensation processes in block-based motion compensation technology, particularly in HEVC, which involves complex interframe prediction for various PU types, including subpixel motion estimation.
A method and apparatus for predictive video encoding that determines whether to skip motion estimation (ME) encoding of current blocks based on pre-analysis information, such as downsampling block costs, sub-block encoding results, and executed mode information, reducing unnecessary ME processes.
This approach significantly reduces encoding complexity and improves encoding speed by up to 18% with minimal loss in compression rate, saving server resources and reducing transcoding costs.
Smart Images

Figure 0007589929000001 
Figure 0007589929000002 
Figure 0007589929000003
Abstract
Description
[Technical field]
[0001] The present invention claims priority to a Chinese patent application with application number 202011459744.1 filed with the China Patent Office on December 11, 2020, the entire contents of which are incorporated herein by reference.
[0002] FIELD OF THE DISCLOSURE Embodiments of the present invention relate to video coding techniques, for example to methods and apparatus for video predictive coding. [Background technology]
[0003] In the coding framework, predictive coding is one of the core technologies of video coding, and predictive coding can be divided into intraframe prediction and interframe prediction. Intraframe prediction is to predict a current pixel using adjacent pixels coded in an image based on the spatial correlation of the video image. Interframe coding is to predict an image to be coded using a coded image based on the temporal correlation of the video image. Through intraframe prediction and interframe prediction, the encoder can remove the spatiotemporal correlation of the video, and perform transformation, quantization, and entropy coding on the predicted residual instead of the original pixel value, thereby greatly improving the coding efficiency.
[0004] At present, the standard interframe prediction part of the main video coding adopts block-based motion compensation (MC) technology. Its main principle is that the prediction unit (PU) performs motion estimation (ME) in the reference frame, searches for a matching block, and uses motion compensation technology to generate a predicted block, and then obtains a residual block, and then uses the residual block as the input of the subsequent coding module for the next coding process. In addition, since the movement of the object is not all full pixel movement, but may only move half a pixel or 1 / 4 pixel, it is necessary to perform fractional pixel ME, that is, fractional pixel data is generated on the full pixel by an interpolation method, and then matching blocks are searched and compensated in fractional pixels, and the highest accuracy of the luma component fractional pixel of High Efficiency Video Coding (HEVC) is 1 / 4 pixel, and the highest accuracy of the chroma component fractional pixel is 1 / 8 pixel.
[0005] In HEVC, inter-frame coding PUs include at least seven types: 2Nx2N / BIDIR, 2NxN / Nx2N / 2NxnD / 2NxnU / nRx2N / nLx2N (RECT / AMP), and each PU must go through the above inter-frame coding process, which results in a very large computational overhead. Summary of the Invention [Means for solving the problem]
[0006] The present invention provides a method and apparatus for video predictive coding to avoid the case where the computational overhead of coding is large because the PU of each mode needs to perform an inter-frame coding process including motion compensation in the inter-frame coding process in the related art.
[0007] In a first aspect, an embodiment of the invention comprises:
[0008] During inter-frame predictive coding, obtain judgment information for a current coding block, the judgment information including one of pre-analysis information determined by an encoder during pre-analysis of lightweight video coding, coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, and performed mode information determined according to a performed mode;
[0009] Based on the determination information, it is determined whether to skip the motion estimation ME coding of the current coding block. Wherein, the process of skipping ME includes skipping all ME and skipping part of ME And, Including, The present invention relates to a method for predictive coding of moving images.
[0010] In a second aspect, an embodiment of the invention comprises:
[0011] a determination information acquisition module configured to acquire, during inter-frame predictive coding, determination information for a current coding block, the determination information including one of pre-analysis information determined by an encoder when performing pre-analysis of lightweight video coding, coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, and executed mode information determined according to an executed mode;
[0012] and determining whether to skip motion estimation ME coding of the current coding block based on the determination information. , where the process of skipping MEs includes skipping all MEs and skipping a part of MEs. An ME skip determination module; Including, The present invention relates to a video predictive coding device.
[0013] In a third aspect, an embodiment of the present invention relates to an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device implementing the above method when the processor executes the program.
[0014] In a fourth aspect, an embodiment of the present invention relates to a computer readable storage medium having a computer program stored thereon, the computer program implementing the above method when the program is executed by a processor. [Brief description of the drawings]
[0015] [Figure 1] FIG. 2 is a schematic diagram of a PU division mode according to an embodiment of the present invention; [Diagram 2] FIG. 13 is a schematic diagram of a process for determining an optimal mode of an inter-frame predictive coding process for a single PU according to an embodiment of the present invention; [Diagram 3] FIG. 2 is a schematic diagram of an encoding process in single inter-frame mode according to an embodiment of the present invention; [Figure 4] 4 is a flow chart of a method embodiment of video predictive coding according to an embodiment of the present invention; [Diagram 5] 1 is a flowchart showing ME high-speed skip determination based on pre-analysis information according to an embodiment of the present invention; [Figure 6] FIG. 2 is a schematic diagram illustrating a relationship between recursive division of a coding block and a downsampling block according to an embodiment of the present invention. [Figure 7] 1 is a flowchart showing ME fast skip decision based on coding information of a sub-block according to an embodiment of the present invention; [Figure 8] FIG. 2 is a schematic diagram of recursive division of a coding block according to an embodiment of the present invention; [Figure 9] 11 is a flowchart showing ME fast skip judgment for a non-square PU based on executed mode information of an executed mode according to an embodiment of the present invention. [Figure 10] FIG. 2 is a structural block diagram of an embodiment of a video predictive coding device according to an embodiment of the present invention; [Figure 11] 1 is a structural schematic diagram of an electronic device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] The present invention will be described in more detail below with reference to the drawings and examples. It is understood that the specific examples described in this specification are not intended to limit the present invention, but are used only to explain the present invention. For ease of explanation, the drawings show only parts related to the present invention, not the entire configuration.
[0017] This embodiment can be applied to the predictive coding stage in video coding, where predictive coding means using one or more coded sample values to predict the current sample value based on a model or method, and coding is performed on the difference between the true value of the sample and the predicted value, and the coding efficiency can be greatly improved by performing transformation, quantization, and entropy coding on the residual after prediction instead of the original pixel value. Predictive coding can include intra-frame prediction Intra mode and inter-frame prediction Inter mode.
[0018] The new video compression standard HEVC defines new syntax units for image division, including a coding unit (CU), a prediction unit (PU), and a transform unit (TU). Here, a CU is a basic unit that performs processes such as prediction, transformation, quantization, and entropy coding, a PU is a basic unit that performs predictive coding including intraframe prediction or interframe prediction, and a TU is a basic unit that performs transformation and quantization. A PU specifies all prediction modes of a CU, and all information related to prediction can be defined in a PU. For example, the direction of intraframe prediction, the division method of interframe prediction, motion vector prediction, and interframe prediction reference image index number all belong to the category of a PU.
[0019] Referring to FIG. 1, a schematic diagram of a PU partition mode is shown. With respect to the PU partition mode included in one 2Nx2N CU, for one 2Nx2N CU mode, there are two selectable modes of intra-frame prediction PU, namely, 2Nx2N and NxN, and there are eight selectable modes of inter-frame prediction PU, namely, four symmetric modes (2Nx2N, 2NxN, Nx2N, NxN) and four asymmetric modes (2NxnD, 2NxnU, nRx2N, nLx2N). Here, according to the shape of the PU, 2Nx2N and NxN are called square PUs, and 2NxN, Nx2N, 2NxnD, 2NxnU, nRx2N, nLx2N are called non-square PUs. SKIP mode is a type of inter-frame prediction, and the motion information that needs to be coded only has a motion parameter set index, and the coding residual does not need to be coded. If this is the case, it is the 2Nx2N SKIP mode.
[0020] Referring to Figure 2, a schematic diagram of a process for determining an optimal mode of an inter-frame predictive coding process for a single PU is shown in Figure 2, and as shown in Figure 2, the determination process may include the following process (NxN mode is not used in Figure 2):
[0021] First, 2Nx2N level MERGE / SKIP mode is performed, and if the recursion condition is satisfied, CU recursion mode is performed, and after recursion is completed, it returns to the current block size CU, and continues to perform intra-frame prediction Intra mode, and after Intra is completed, 2Nx2N inter-frame mode is performed (forward ME and backward ME are performed respectively to obtain the best matching block), then 2NxN BIDIR is performed (only bidirectional search is performed to obtain the best matching block), and then 2NxN and Nx2N mode types are performed. Each time one type of mode is completed, one mode judgment is performed, and the step-optimal mode tempBestMode is obtained.
[0022] After running Nx2N, If tempBestMode == 2NxN, continue to run 2NxnU, 2NxnD and compare to get the final best mode, BestMode. If tempBestMode == Nx2N, continue to run nLx2N, nRx2N, and compare to get the final best mode, BestMode. If tempBestMode != 2NxN && tempBestMode != Nx2N, then continue to run 2NxnU, 2NxnD, nLx2N, nRx2N and compare to get the final best mode BestMode.
[0023] If the current block has performed CU recursion, the above decision process is executed for each sub-block, and the BestMode cost is compared with the recursion cost to decide whether or not to perform CU recursion division.
[0024] In addition, non-square modes such as 2NxN, Nx2N, 2NxnU, 2NxnD, nLx2N, and nRx2N include two PUs, and each PU needs to perform MERGE / SKIP mode, forward ME, backward ME, and both-way ME individually and compare them to obtain the optimal mode.
[0025] In addition, since the movement of an object may not be the movement of all pixels, but may be only a half pixel or a quarter pixel, it is necessary to perform fractional pixel ME, that is, fractional pixel data is generated on the full pixels by an interpolation method, and then matching blocks are searched and compensated for in the fractional pixels, and the highest accuracy of the fractional pixel of the HEVC luma component is 1 / 4 pixel, and the highest accuracy of the chroma component is 1 / 8 pixel. Referring to FIG. 3, the encoding process of the single inter-frame mode among the multiple modes in FIG. 2 above is shown, in which for one PU, it is first determined whether to perform fractional pixel ME, and fractional pixel motion estimation and motion compensation are performed according to the determination result of performing fractional pixel ME, and full pixel motion estimation and motion compensation are performed according to the determination result of not performing fractional pixel ME, and then a rate distortion optimization (RDO) encoding flow is performed on the obtained residual block.
[0026] As can be seen from the above, each inter-frame Inter PU must go through the inter-frame encoding process of FIG. 3, which causes a very large overhead. If unnecessary PU ME can be reduced, the resource overhead of the encoder can be greatly reduced, the encoding speed can be improved, and the encoding cost can be reduced.
[0027] It should be noted that the following embodiments of the present invention can be applied to any mixed coding architecture that complies with blocks, and any encoder that meets the HEVC coding standard can be applied, and other encoders that meet standards such as AVS1, H.264, VP8, VP9, AVS2, AVS3, AV1, VVC, etc. can also be used directly after adjustment. And, the encoder mentioned in this embodiment can be applied not only to transcoding services but also to real-time coding services on the mobile side.
[0028] FIG. 4 is a flowchart of a method embodiment of video predictive coding according to an embodiment of the present invention, which may include the following steps.
[0029] In step 410, during inter-frame predictive coding, determination information for a current coding block is obtained, the determination information including pre-analysis information determined by an encoder when performing pre-analysis for lightweight video coding, or coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, or performed mode information determined based on a performed mode.
[0030] In step 420, it is determined based on the judgment information whether to skip the motion estimation ME coding of the current coding block.
[0031] For example, for the decision process shown in Fig. 2, when deciding the optimal mode of the inter-frame prediction mode for the current PU, it can be determined whether to skip the ME processes of multiple modes in the decision process based on the acquired judgment information. Here, skipping the ME process can be divided into two cases, one is to skip all MEs in the decision process, and the other is to skip some of the MEs in the decision process. For example, only 2Nx2N ME is performed, and MEs of other modes after 2Nx2N are skipped.
[0032] For a scene in which all MEs are skipped in the above-mentioned decision process, in this embodiment, it is possible to determine whether or not to skip the ME process in different ways depending on the acquired determination information.
[0033] In one embodiment, a decision to quickly skip ME encoding can be made based on pre-analysis information, where the decision information may include pre-analysis information determined by the encoder when performing pre-analysis of lightweight video encoding, and as shown in FIG. 5, step 420 may include the following sub-steps:
[0034] In sub-step S11, the number of corresponding down-sampling blocks is determined based on the size of the current coding block, and the pre-analysis information of each down-sampling block is indexed.
[0035] When implementing this, the encoder may first use a pre-analysis module to perform a pre-analysis operation on the input video before performing an encoding process on the input video. For example, the pre-analysis operation may involve the pre-analysis module performing downsampling on the input video and then performing lightweight video encoding to obtain the frame structure, frame type, and encoding quantization parameter (QP) that will be used when actually encoding.
[0036] The coding unit for performing pre-analysis is an 8x8 downsampling block. For example, assuming that the original block size is w*h, the block size after downsampling is w / 2*h / 2, and the 8x8 downsampling block corresponds to a 16x16 coding block, that is, when the method of sub-step S11 is adopted, the size of the coding block in this embodiment is at least 16x16, including 16x16, 32x32, 64x64.
[0037] In one example, taking a 32x32 coding block as an example, as shown in Figure 6, the 32x32 coding block can be divided into four 16x16 coding sub-blocks, and each 16x16 coding sub-block further corresponds to one downsampling block (Lowres shown in Figure 6), so the number of downsampling blocks collected by the 32x32 coding block is 4. Similarly, in the case of a 64x64 coding block, it can be divided into 16 16x16 coding sub-blocks corresponding to the 16 downsampling blocks.
[0038] Since ME coding is also performed in the pre-analysis stage, if the pre-analysis information of the downsampling block corresponding to the current coding block in the pre-analysis stage can be found, ME fast skip judgment of the current block can be made based on the pre-analysis information. In one embodiment, the corresponding downsampling block can be indexed based on the coordinate information of the current coding block. For example, in FIG. 6, for a 32x32 coding block, it can be divided into four 16x16 coding sub-blocks, and the coordinate information of the coding block can be known, so the coordinate information of each 16x16 coding sub-block can be obtained, and then the downsampling block that falls within the coordinate range of each coding sub-block can be found as the downsampling block corresponding to the coding sub-block, and after finding the downsampling block corresponding to each coding block, the pre-analysis information of the downsampling block can be obtained.
[0039] As an example, the pre-analysis information of a downsampling block may include, but is not limited to, an interframe coding cost, an intraframe coding cost, and an optimal mode predicted by the downsampling block.
[0040] In sub-step S12, the pre-analysis information of the down-sampling block is aggregated, and it is determined whether to skip ME coding of the current coding block based on the aggregation result.
[0041] After obtaining all downsampling blocks corresponding to the current coding block, the pre-analysis information of all downsampling blocks corresponding to the current coding block is aggregated, and it is possible to determine whether to skip the ME process of the inter-frame prediction mode based on the aggregation result.
[0042] In one embodiment, sub-step S12 may include the following sub-steps:
[0043] In sub-step S121, an estimated interframe coding cost of the current coding block is obtained by calculating the sum of the interframe coding costs of all down-sampling blocks corresponding to the current coding block.
[0044] For example, for the 32x32 coding block in Figure 6, four downsampling blocks are indexed, and the inter-frame coding cost of each downsampling block is obtained. Then, the sum of the inter-frame coding costs of the four downsampling blocks can be calculated as the estimated inter-frame coding cost InterCost of the 32x32 coding block.
[0045] In sub-step S122, an estimated intra-frame coding cost of the current coding block is obtained by calculating the sum of the intra-frame coding costs of all down-sampled blocks corresponding to said current coding block.
[0046] For example, for the 32x32 coded block in FIG. 6, after the intra-frame coding costs of the four down-sampling blocks are indexed, the sum of the intra-frame coding costs of the four down-sampling blocks can be calculated as the estimated intra-frame coding cost IntraCost of the 32x32 coded block.
[0047] In sub-step S123, the number of downsampling blocks whose optimal mode is intra-frame prediction mode among all downsampling blocks corresponding to the current coding block is calculated, and a first ratio in which the optimal mode of the current coding block is intra-frame prediction mode is determined based on said number.
[0048] For example, for the 32x32 coding block of Figure 6, after the optimal modes of the four downsampling blocks are indexed, the number of downsampling blocks whose optimal modes are intra-frame prediction Intra modes and the first ratio IntraRatio can be calculated. For example, if the number of downsampling blocks whose optimal modes are Intra modes is two, the IntraRatio will be 2 / 4, and if the number of downsampling blocks whose optimal modes are Intra modes is three, the IntraRatio will be 3 / 4.
[0049] In sub-step S124, if the first ratio is greater than a first preset ratio threshold and the product of the estimated intraframe coding cost and a first adjustment factor is less than the product of the estimated interframe coding cost and a second adjustment factor, where the first adjustment factor is less than the second adjustment factor, skip ME coding of the current coding block.
[0050] In this step, after obtaining the InterCost, IntraCost, and IntraRatio of the current coding block, if the following conditional formula (1) is satisfied, ME coding can be skipped. if (IntraRatio > a && IntraCost * b < c * InterCost) (1) skipME = true
[0051] Here, a is the first preset ratio threshold, b is the first adjustment coefficient, c is the second adjustment coefficient, and skipME is the skip of ME. Note that a, b, and c can be adjusted according to requirements. On the premise of maintaining the same acceleration performance, the coding speed can be linearly increased or decreased to meet different service requirements. For example, b < c can be set according to service requirements, and for example, the default values of a, b, and c can be set to 0.5, 1, and 4.
[0052] According to the above conditional expression (1), when IntraCost * b < c * InterCost and IntraRatio is relatively large, it indicates that the spatial region correlation of the current block is stronger than the temporal region correlation, so the ME process can be directly skipped.
[0053] In another embodiment, when the current coded block performs recursive coding and can quickly determine to skip ME based on the coded information of sub-blocks, the determination information may include the coded information of a plurality of sub-blocks obtained after the recursive coding by the coded block. Exemplarily, the coded information of the sub-block may include at least the optimal mode of the sub-block and whether the sub-block is further recursively divided. As shown in FIG. 7, step 420 may include the following sub-steps.
[0054] In sub-step S21, based on the optimal mode of each sub-block of the current coded block, a second ratio that the optimal mode of the current coded block is the intra-frame prediction mode and a third ratio that the optimal mode of the current coded block is the SKIP mode are determined.
[0055] In one embodiment, the second ratio in which the optimal mode of the current coding block is intra-frame prediction mode may be a ratio of the number of sub-blocks in the current coding block whose optimal mode is intra-frame prediction Intra mode to the total number of all sub-blocks in the current coding block, and the third ratio in which the optimal mode of the current coding block is SKIP mode may be a ratio of the number of sub-blocks in the current coding block whose optimal mode is intra-frame prediction SKIP mode to the total number of all sub-blocks in the current coding block.
[0056] For example, as shown in FIG. 8, if the current coding block is divided into four 16x16 coding sub-blocks, and among the four coding sub-blocks, the number of sub-blocks whose optimal mode is Intra mode is 1 and the number of sub-blocks whose optimal mode is SKIP mode is 2, then the second ratio=1 / 4 and the third ratio=2 / 4.
[0057] In sub-step S22, a sub-block recursion ratio of the current coding block is determined based on whether each sub-block of the current coding block is further divided recursively.
[0058] In this step, the sub-block recursion ratio is used to reflect the ratio of the sub-blocks that can be further divided in the current coding block. In one embodiment, the sub-block recursion ratio of the current coding block may be the ratio of the number of sub-blocks that can be further recursively coded in the current coding block to the total number of all sub-blocks in the current coding block, for example, as shown in Figure 8, if the current coding block is divided into four 16x16 coding sub-blocks, and the number of sub-blocks that can be further recursively coded among the four coding sub-blocks is 1 (shown in the Split sub-block in Figure 8), the sub-block recursion ratio = 1 / 4.
[0059] In sub-step S23, it is determined whether to skip ME coding of the current coding block based on the second ratio, the third ratio and / or the sub-block recursion ratio.
[0060] In one embodiment, sub-step S23 may include the following sub-steps:
[0061] If the sub-block recursion ratio is greater than a preset recursion ratio threshold and the second ratio is greater than a second preset ratio threshold, skip the ME coding of the current coding block.
[0062] In this step, after obtaining the sub-block recursion ratio and the second ratio, if the following conditional expression (2) is satisfied, ME encoding can be skipped. if(SubSplitRatio > threshold_A && SubIntraRatio > threshold_B) (2) skipME = true
[0063] where SubSplitRatio is a sub-block recursion ratio, SubIntraRatio is a second ratio, threshold_A is a preset recursion ratio threshold, and threshold_B is a second preset ratio threshold. Note that threshold_A and threshold_B can be set according to actual business needs.
[0064] If SubSplitRatio>threshold_A and SubIntraRatio>threshold_B, it indicates that the current coding block has many sub-block Split blocks and a fixed Intra ratio. In this case, the current coding block is simply divided into four sub-blocks for coding, so that the ME process of the current coding block can be skipped.
[0065] In another embodiment, sub-step S23 may include the following sub-steps:
[0066] If the third ratio is greater than a third preset ratio threshold and the sub-block recursion ratio is greater than a fourth preset ratio threshold, skip the ME coding of the current coding block.
[0067] In this step, after obtaining the sub-block recursion ratio and the third ratio, if the following conditional expression (3) is satisfied, ME encoding can be skipped. if(SubSkipRatio > threshold_C && SubSplitRatio > threshold_D) (3) skipME = true
[0068] Here, SubSplitRatio is a sub-block recursion ratio, SubSkipRatio is a third ratio, threshold_C is a third preset ratio threshold, and threshold_D is a fourth preset ratio threshold, and threshold_C and threshold_D can be set according to actual business needs.
[0069] If SubSkipRatio > threshold_C and SubSplitRatio > threshold_D, it indicates that the current coding block has many sub-block splits and a high SKIP mode ratio, which indicates that the movement of the current coding block is weak, so the remaining ME processes can be skipped.
[0070] In another embodiment, when the current coding block CU does not perform recursive coding, it can determine whether to skip ME coding based on the mode information of the executed mode, and in this scene, the determination information may include executed mode information that is determined based on the executed mode. Illustratively, the executed mode information may include at least a tiered optimal mode SATD cost and an intra-frame prediction SATD cost. Step 420 may include the following substeps.
[0071] If the current coding block is not recursively coded and the product of the intra-frame prediction SATD cost and the third adjustment factor is less than the product of the hierarchical optimal mode SATD cost and the fourth adjustment factor, skip ME coding of the current coding block, where the third adjustment factor is greater than the fourth adjustment factor.
[0072] In this embodiment, the incrementally optimal mode SATD cost is the SATD cost of the incrementally optimal mode obtained by performing mode decision after performing this mode. The intra prediction SATD cost may be the SATD cost of the optimal intra mode determined by SATD coarse selection when performing intra prediction.
[0073] After obtaining the hierarchical optimal mode SATD cost and the intra-frame prediction SATD cost, if the following conditional expression (4) is satisfied, ME encoding can be skipped. if(IntraSATDCost * α< β* InterSATDCost) (4) skipME = true
[0074] Where IntraSATDCost is the intra-frame prediction SATD cost, InterSATDCost is the stepwise optimal mode SATD cost, α is the third adjustment factor, and β is the fourth adjustment factor. α and β can be adjusted according to actual business needs, for example, the default value of α may be 5, and the default value of β may be 3.
[0075] If IntraSATDCost*α is smaller than β*InterSATDCost, it indicates that the spatial domain correlation of the current block is stronger than the temporal domain correlation, so the ME coding process of the current coding block can be directly skipped.
[0076] In another embodiment, for a scene in which some of the MEs in the above decision process are skipped, for example, for non-square PUs such as 2NxN / Nx2N / 2NxnD / 2NxnU / nRx2N / nLx2N (RECT / AMP), each PU needs to perform MERGE mode, forward ME, backward ME, and bidirectional ME separately and obtain the optimal mode by comparison, and the whole ME process is complicated, and in this embodiment, a decision can be made to quickly skip the ME of the non-square PU according to the already performed ME mode. The judgment information may include performed mode information determined based on the performed mode, where the performed mode may include at least the 2Nx2N mode and the BIDIR mode in FIG. 2, and the performed mode information may include the MERGE mode SATD cost of the MERGE mode performed in the 2Nx2N mode and the step-optimal mode SATD cost.
[0077] As shown in FIG. 9, step 420 may include the following sub-steps.
[0078] In sub-step S31, if the current prediction unit is a non-square prediction unit, it is determined whether the MERGE mode SATD cost is less than the tiered optimal mode SATD cost.
[0079] In sub-step S32, if the MERGE mode SATD cost is less than the incremental optimal mode SATD cost, the MERGE mode is performed for the non-square prediction unit, and forward ME encoding, backward ME encoding and bidirectional ME encoding of the non-square prediction unit are skipped.
[0080] In this embodiment, if the MERGE mode SATD cost of the MERGE mode executed in 2Nx2N mode is smaller than the stage-optimal mode SATD cost, the MERGE mode is only executed for the current target non-square PU, and the forward ME encoding, backward ME encoding and bidirectional ME encoding processes are not performed.
[0081] In another embodiment, when the current coding block has performed recursive coding, in the decision process of determining whether to skip the forward ME, backward ME, and bidirectional ME of the non-square PU mode, the ratio of the optimal mode of the current coding block being SKIP mode, i.e., the ratio of sub-blocks in the coding sub-blocks of the current coding block being SKIP mode, the calculation method can refer to the third ratio above, and step 420 can include the following sub-steps:
[0082] If the MERGE mode SATD cost is smaller than the tiered optimal mode SATD cost and the ratio of the optimal mode of the current coding block being SKIP mode is greater than a preset ratio threshold, the MERGE mode is performed for the non-square prediction unit, and the forward ME coding, backward ME coding and bidirectional ME coding of the non-square prediction unit are skipped.
[0083] In implementation, if the current non-square mode satisfies the following condition (5), the encoding process of forward ME, backward ME, and bidirectional ME can be skipped, and only the MERGE mode of the mode is executed. if(SubSkipRatio > θ && MrgCost * a1 < b1 * InterSATDCost) (5) skipMELeft = true
[0084] Here, SubSkipRatio is the ratio in which the optimal mode of the current coding block is SKIP mode, MrgCost is the MERGE mode SATD cost of the MERGE mode executed in 2Nx2N mode, InterSATDCost is the step-optimal mode SATD cost, and θ, a1, and b1 are adjustment coefficients, which can be set according to actual business needs, for example, the default value of θ can be set to 0.8, and the default values of a1 and b1 are both 1. skipMELeft means that RECT / AMP only executes the MERGE mode and does not execute forward ME, backward ME, or bidirectional ME.
[0085] In this embodiment, during interframe predictive coding, based on the obtained judgment information on the current coding block, it is judged whether to skip the motion estimation ME coding of the current coding block, and the judgment information may include pre-analysis information determined by the encoder when performing pre-analysis of lightweight video coding, or coding information of multiple sub-blocks obtained after recursive coding by the current coding block, or executed mode information determined based on the executed mode, and based on different judgment information, different fast judgment methods can be adopted to make ME skip judgment, which can reduce unnecessary PU ME, greatly reduce the coding complexity, reduce the resource overhead of the encoder, and further improve the encoding compression speed of the encoder. After improving the compression speed of the encoder, the throughput of the transcoding service can be increased, and the server resources can be saved and the transcoding cost can be reduced. The judgment to skip ME coding in this embodiment can improve the coding speed by 18% under the condition that the encoder compression rate is lost by 0.2%. From the perspective of online transcoding applications, it can save the server computing resources by 18% with a slight image quality loss, and further reduce the transcoding cost.
[0086] FIG. 10 is a structural block diagram of an embodiment of an apparatus for video predictive coding according to an embodiment of the present invention, which may include the following modules:
[0087] The judgment information acquisition module 1001 is configured to acquire judgment information regarding a current coding block during inter-frame predictive coding, where the judgment information includes pre-analysis information determined by an encoder when performing pre-analysis of lightweight video coding, or coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, or performed mode information determined based on a performed mode.
[0088] The ME skip decision module 1002 is configured to determine whether to skip motion estimation ME coding of the current coding block based on the decision information.
[0089] In one embodiment, if the determination information is pre-analysis information determined by an encoder when performing pre-analysis of lightweight video coding, the ME skip determination module 1002 may include the following sub-modules:
[0090] The pre-analysis information index sub-module is configured to determine, based on the size of the current coding block, the number of corresponding downsampling blocks, and index the pre-analysis information of each downsampling block.
[0091] The pre-analysis information determining sub-module is configured to aggregate pre-analysis information of the downsampling block, and determine whether to skip ME coding of the current coding block according to a result of the aggregation.
[0092] In one embodiment, the pre-analysis information of the downsampling block includes an interframe coding cost, an intraframe coding cost, and an optimal mode predicted by the downsampling block.
[0093] The pre-analysis information determination submodule:
[0094] obtaining an estimated inter-frame coding cost of the current coding block by calculating a sum of inter-frame coding costs of all down-sampled blocks corresponding to the current coding block;
[0095] obtaining an estimated intra-frame coding cost of the current coding block by calculating a sum of the intra-frame coding costs of all down-sampled blocks corresponding to the current coding block;
[0096] Calculate the number of downsampling blocks whose optimal mode is an intra-frame prediction mode among all downsampling blocks corresponding to the current coding block, and determine a first ratio whose optimal mode of the current coding block is the intra-frame prediction mode based on the number;
[0097] The ME coding method is configured to skip ME coding of the current coding block when the first ratio is greater than a first preset ratio threshold and the product of the estimated intraframe coding cost and a first adjustment factor is less than the product of the estimated interframe coding cost and a second adjustment factor, where the first adjustment factor is less than the second adjustment factor.
[0098] In one embodiment, when the decision information is encoding information of a plurality of sub-blocks obtained after recursive encoding by the current encoding block, the encoding information of the sub-blocks includes an optimal mode of the sub-block and whether the sub-block is to be further recursively divided.
[0099] The ME skip determination module 1002 may include the following sub-modules:
[0100] The ratio calculation submodule is configured to determine, based on the optimal mode of each sub-block of the current coding block, a second ratio in which the optimal mode of the current coding block is an intra-frame prediction mode, and a third ratio in which the optimal mode of the current coding block is a SKIP mode, and to determine a sub-block recursive ratio of the current coding block based on whether each sub-block of the current coding block is further divided recursively.
[0101] The ratio determination sub-module is configured to determine whether to skip ME coding of the current coding block based on the second ratio, the third ratio and / or the sub-block recursion ratio.
[0102] In one embodiment, the ratio determination submodule:
[0103] skipping ME coding of the current coding block when the sub-block recursion ratio is greater than a preset recursion ratio threshold and the second ratio is greater than a second preset ratio threshold;
[0104] or
[0105] configured to skip ME coding of the current coding block when the third ratio is greater than a third preset ratio threshold and the sub-block recursion ratio is greater than a fourth preset ratio threshold;
[0106] In one embodiment, when the decision information is executed mode information determined based on an executed mode, the executed mode information includes a tiered optimal mode SATD cost and an intra-frame prediction SATD cost.
[0107] The ME skip determination module 1002 may include the following sub-modules:
[0108] The SATD cost determination submodule is configured to skip ME coding of the current coding block if the current coding block is not recursively coded and the product of the intra-frame prediction SATD cost and a third adjustment factor is less than the product of the stepwise optimal mode SATD cost and a fourth adjustment factor, where the third adjustment factor is greater than the fourth adjustment factor.
[0109] In one embodiment, when the determination information is executed mode information determined based on an executed mode, the executed modes include at least a 2Nx2N mode and a BIDIR mode, and the executed mode information includes a MERGE mode SATD cost and a tiered optimal mode SATD cost of a MERGE mode executed in the 2Nx2N mode.
[0110] The ME skip determination module 1002 may include the following sub-modules:
[0111] The first partial ME skip determination submodule is configured to determine whether the MERGE mode SATD cost is smaller than the tiered optimal mode SATD cost when the current prediction unit is a non-square prediction unit, and if the MERGE mode SATD cost is smaller than the tiered optimal mode SATD cost, perform a MERGE mode for the non-square prediction unit, and skip forward ME encoding, backward ME encoding and bidirectional ME encoding of the non-square prediction unit.
[0112] In one embodiment, if the current coding block has undergone recursive coding, the performed mode information further includes a proportion in which the optimal mode of the current coding block is a SKIP mode.
[0113] The ME skip determination module 1002 may include the following sub-modules:
[0114] The first partial ME skip judgment submodule is configured to perform a MERGE mode for the non-square prediction unit and skip forward ME encoding, backward ME encoding and bidirectional ME encoding of the non-square prediction unit when the MERGE mode SATD cost is smaller than the tiered optimal mode SATD cost and the ratio in which the optimal mode of the current encoding block is a SKIP mode is greater than a preset ratio threshold.
[0115] It should be noted that the above-mentioned video predictive coding apparatus according to the embodiment of the present invention can execute the video predictive coding method according to the embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution of the method.
[0116] Fig. 11 is a structural schematic diagram of an electronic device according to an embodiment of the present invention, and as shown in Fig. 11, the electronic device includes a processor 1110, a memory 1120, an input device 1130, and an output device 1140. In the electronic device, the number of processors 1110 may be one or more, and Fig. 11 takes one processor 1110 as an example. The processor 1110, the memory 1120, the input device 1130, and the output device 1140 in the electronic device may be connected by a bus or other methods, and Fig. 11 takes the example of connecting by a bus.
[0117] The memory 1120 can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the methods in the embodiments of the present invention, as a computer-readable storage medium. The processor 1110 executes the software programs, instructions and modules stored in the memory 1120 to perform various functional applications and data processing of the electronic device, i.e., to realize the above-mentioned methods.
[0118] The memory 1120 may mainly include a program storage area and a data storage area, where the program storage area includes:
[0119] The memory 1120 may store an operating system, an application required for at least one function, and a data storage area may store data created in response to the use of the terminal. The memory 1120 may also include a high-speed random access memory, such as at least one magnetic disk storage device, a flash memory device, or a non-volatile memory such as other non-volatile solid-state storage devices. In some examples, the memory 1120 may include memories located remotely with respect to the processor 1110, and these remote memories may be connected to the electronic device via a network. Examples of the network include, but are not limited to, the Internet, a corporate intranet, a local area network, a mobile communication network, and combinations thereof.
[0120] The input device 1130 may be configured to receive entered numeric or textual information and generate key signal inputs related to user settings and function control of the electronic device. The output device 1140 may include a display device such as a display.
[0121] In an embodiment of the present invention, there is further provided a storage medium containing computer-executable instructions, the computer-executable instructions being configured to perform a method in any of the embodiments of the present invention when executed by a processor of the server.
[0122] From the above description of the embodiments, those skilled in the art will clearly understand that the present invention can be realized by software and necessary general-purpose hardware, and of course, can also be realized by hardware. Based on this understanding, the technical solution of the present invention can be essentially or partly contributed to the related art embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk, or an optical disk of a computer, and includes some instructions for making a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in the multiple embodiments of the present invention. The computer-readable storage medium may be a non-transitory computer-readable storage medium.
[0123] In the above embodiment of the device, the included multiple units and modules are simply divided according to functional logic, but are not limited to the above divisions as long as they can realize the corresponding functions. In addition, the specific names of the multiple functional units are given only to facilitate mutual distinction, and are not intended to limit the scope of protection of the present invention. [Explanation of symbols]
[0124] 1001 Judgment information acquisition module 1002 ME skip judgment module 1110 Processor 1120 Memory 1130 Input Device 1140 Output Device
Claims
1. During inter-frame predictive coding, obtain judgment information for a current coding block, the judgment information including one of pre-analysis information determined by an encoder during pre-analysis of lightweight video coding, coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, and performed mode information determined according to a performed mode; determining whether to skip motion estimation ME coding of the current coding block based on the determination information, where the process of skipping ME includes skipping all ME and skipping a part of ME; Including, In response to determining that the determination information is coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, the coding information of the sub-blocks includes an optimal mode of each sub-block and whether each sub-block is recursively divided; Determining whether to skip motion estimation ME coding of the current coding block based on the determination information includes: determining a second ratio in which the optimal mode of the current coding block is an intra-frame prediction mode and a third ratio in which the optimal mode of the current coding block is a SKIP mode according to an optimal mode of each of the sub-blocks of the current coding block; determining a sub-block recursion ratio of the current coding block according to whether each of the sub-blocks of the current coding block is recursively divided; determining whether to skip ME coding of the current coding block based on at least one of the second ratio, the third ratio, and the sub-block recursion ratio; Including, A method for predictive coding of video.
2. determining whether to skip motion estimation ME coding of the current coding block based on the determination information in response to determining that the determination information is pre-analysis information determined by an encoder when performing pre-analysis of lightweight video coding, Determine the number of corresponding downsampling blocks according to the size of the current coding block, and index pre-analysis information of each downsampling block, where the number of downsampling blocks is plural; aggregating pre-analysis information of the downsampling block, and determining whether to skip ME encoding of the current coding block according to a result of the aggregation; Including, The method of claim 1.
3. The pre-analysis information of the downsampling block includes an interframe coding cost, an intraframe coding cost, and an optimal mode predicted by the downsampling block; aggregating pre-analysis information of the downsampling block and determining whether to skip ME coding of the current coding block based on a result of the aggregation, obtaining an estimated inter-frame coding cost of the current coding block by calculating a sum of inter-frame coding costs of all down-sampled blocks corresponding to the current coding block; obtaining an estimated intra-frame coding cost of the current coding block by calculating a sum of the intra-frame coding costs of all down-sampled blocks corresponding to the current coding block; Calculating the number of downsampling blocks whose optimal mode is an intra-frame prediction mode among all downsampling blocks corresponding to the current coding block, and determining a first ratio whose optimal mode of the current coding block is the intra-frame prediction mode according to the number; skipping ME encoding of the current coding block in response to determining that the first ratio is greater than a first preset ratio threshold and that a product of the estimated intraframe coding cost and a first adjustment factor is less than a product of the estimated interframe coding cost and a second adjustment factor, where the first adjustment factor is less than the second adjustment factor; Including, The method of claim 2.
4. Determining whether to skip ME coding of the current coding block based on at least one of the second ratio, the third ratio, and the sub-block recursion ratio includes: skipping ME encoding of the current coding block in response to determining that the sub-block recursion ratio is greater than a preset recursion ratio threshold and that the second ratio is greater than a second preset ratio threshold; or skipping ME encoding of the current coding block in response to determining that the third ratio is greater than a third preset ratio threshold and that the sub-block recursion ratio is greater than a fourth preset ratio threshold; Including, The method of claim 1.
5. In response to determining that the determination information is executed mode information determined based on an executed mode, the executed mode information includes a tiered optimal mode SATD cost and an intra-frame prediction SATD cost; Determining whether to skip motion estimation ME coding of the current coding block based on the determination information includes: skipping ME coding of the current coding block in response to determining that the current coding block is not recursively coded and that a product of the intra-frame prediction SATD cost and a third adjustment factor is less than a product of the tiered optimal mode SATD cost and a fourth adjustment factor, where the third adjustment factor is greater than the fourth adjustment factor; The method of claim 1.
6. In response to determining that the determination information is executed mode information determined based on an executed mode, the executed mode includes at least a 2Nx2N mode and a BIDIR mode, and the executed mode information includes a MERGE mode SATD cost of a MERGE mode executed in the 2Nx2N mode and a tiered optimal mode SATD cost, where N is a positive integer; Determining whether to skip motion estimation ME coding of the current coding block based on the determination information includes: determining whether the MERGE mode SATD cost is less than the tiered optimal mode SATD cost in response to determining that the current prediction unit is a non-square prediction unit; in response to determining that the MERGE mode SATD cost is less than the tiered optimal mode SATD cost, performing a MERGE mode on the non-square prediction unit and skipping forward ME encoding, backward ME encoding, and bidirectional ME encoding of the non-square prediction unit; Including, The method of claim 1.
7. In response to determining that the current coding block has been recursively coded, the performed mode information further includes a ratio in which an optimal mode of the current coding block is a SKIP mode; Determining whether to skip motion estimation ME coding of the current coding block based on the determination information includes: and performing a MERGE mode on the non-square prediction unit and skipping forward ME encoding, backward ME encoding and bidirectional ME encoding of the non-square prediction unit in response to determining that the MERGE mode SATD cost is less than the tiered optimal mode SATD cost and a ratio in which the optimal mode of the current coding block is a SKIP mode is greater than a preset ratio threshold. The method according to claim 6.
8. a determination information acquisition module configured to acquire, during inter-frame predictive coding, determination information for a current coding block, the determination information including one of pre-analysis information determined by an encoder when performing pre-analysis of lightweight video coding, coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, and executed mode information determined according to an executed mode; an ME skip determination module configured to determine whether to skip motion estimation ME coding of the current coding block based on the determination information, where a process of skipping ME includes skipping all ME and skipping a part of ME; Including, In response to determining that the determination information is coding information of a plurality of sub-blocks obtained after recursive coding by the current coding block, the coding information of the sub-blocks includes an optimal mode of each sub-block and whether each sub-block is recursively divided; The ME skip determination module includes a ratio calculation submodule and a ratio determination submodule; The ratio calculation submodule is configured to determine, according to an optimal mode of each of the sub-blocks of the current coding block, a second ratio in which the optimal mode of the current coding block is an intra-frame prediction mode, and a third ratio in which the optimal mode of the current coding block is a SKIP mode; and determine a sub-block recursion ratio of the current coding block according to whether each of the sub-blocks of the current coding block is recursively divided; the ratio determination submodule is configured to determine whether to skip ME coding of the current coding block based on at least one of the second ratio, the third ratio, and the sub-block recursion ratio. A video predictive coding device.
9. 8. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the electronic device implementing the method of any one of claims 1 to 7 when the processor executes the program.
10. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1 to 7 when executed by a processor.
Citation Information
Patent Citations
Moving image encoder
JP2012065176A
Moving image encoding device, moving image encoding method, and recording medium for recording moving image encoding program
WO2017122604A1
