Inter prediction method and device, electronic device, and computer program
By determining a historical prediction mode and generating a reference frame template using adjacent and sub-prediction block information, the method addresses inefficient computation in inter-prediction, improving video encoding efficiency and bitstream quality.
Patent Information
- Application Number
- JP2023518518
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-12-31
- Filing Date
- 2021-12-17
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-12-17
AI Technical Summary
The complexity of the priority selection process for reference frame modes in video encoding using inter-prediction leads to inefficient computation and reduced encoding efficiency, particularly in modes involving NEWMV, which affects the adaptability and quality of the bitstream.
An inter prediction method that determines a historical prediction mode for a current prediction block, acquiring adjacent block and sub-prediction block information, and generating a reference frame template based on this history to reduce computational complexity by leveraging existing information, thereby improving encoding efficiency.
This approach significantly reduces computational complexity and enhances video encoding efficiency by up to 15% while maintaining high adaptability and bitstream quality, without deleting reference frames during template generation.
Smart Images

Figure 0007592157000006 
Figure 0007592157000007 
Figure 0007592157000008
Abstract
Description
[Technical field]
[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed based on a Chinese patent application bearing application number 202011629460.2, filed with the China Patent Office on December 31, 2020, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.
[0002] The present application relates to video encoding technology, and to an inter-prediction method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]
[0003] Video coding is widely applied to video transmission, and in the future, video has the trend of developing to high definition, high frame rate, and high compression rate. When encoding a video frame using inter prediction (also called "interframe prediction"), the video frame needs to be divided into individual coding units (CUs), and when a CU uses inter prediction to calculate a predicted value, it is only after the CU selects an appropriate reference frame for the prediction mode that prediction can be started and the predicted value can be obtained.
[0004] When selecting a suitable reference frame, first, some reference frame modes that are considered to be unimportant are forcibly deleted, and then the prediction mode is combined with the remaining reference frame modes, and a suitable reference frame mode can be obtained after a high-complexity priority selection process for each mode combination. However, the calculation of the priority selection process is complex, which reduces the efficiency of video encoding. Summary of the Invention [Problem to be solved by the invention]
[0005] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present embodiments provide an inter-prediction method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product that improve the efficiency of video encoding. [Means for solving the problem]
[0006] The technical solution of the present embodiment is realized as follows.
[0007] An embodiment of the present application provides an inter prediction method, the method comprising: determining a history prediction mode corresponding to the current prediction block when the current prediction mode of the current prediction block is a preset prediction mode, the history prediction mode being a prediction mode that has completed prediction before the preset prediction mode; Obtaining neighboring block information of the neighboring block of the current prediction block, sub-prediction block information of the sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, the sub-prediction block being obtained by partitioning the current prediction block using a sub-block partition type before a current sub-block partition type; generating a reference frame template based on the historical optimal reference frame type, the neighboring block information, the sub-prediction block information, and a frame type corresponding to the current prediction block; The method includes a step of determining a reference frame for the predetermined prediction mode using the reference frame template, and performing inter prediction on the current prediction block using the reference frame to obtain a prediction value corresponding to the current prediction block.
[0008] An embodiment of the present application provides an inter prediction device, comprising: A mode decision module configured to determine a history prediction mode corresponding to a current prediction block when a current prediction mode of the current prediction block is a preset prediction mode, the history prediction mode being a prediction mode that has completed prediction before the preset prediction mode; An information obtaining module configured to obtain neighboring block information of a neighboring block of the current prediction block, a sub-prediction block information of a sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, wherein the sub-prediction block is obtained by partitioning the current prediction block using a sub-block partition type before a current sub-block partition type; a template generation module configured to generate a reference frame template based on the historical optimal reference frame type, the neighboring block information, the sub-prediction block information, and a frame type corresponding to the current prediction block; and an information prediction module configured to use the reference frame template to determine a reference frame for the pre-set prediction mode, and to perform inter prediction on the current prediction block using the reference frame to obtain a prediction value corresponding to the current prediction block.
[0009] An embodiment of the present application provides an electronic device for inter prediction, the electronic device comprising: A memory storing executable instructions; and a processor that, when executing executable instructions stored in the memory, implements an inter-prediction method according to an embodiment of the present disclosure. An embodiment of the present application provides a computer-readable storage medium having executable instructions stored thereon that, when executed by a processor, implements an inter-prediction method according to an embodiment of the present application.
[0010] An embodiment of the present application provides a computer program product including a computer program or instructions, which, when executed by a processor, implements an inter prediction method according to an embodiment of the present application. Effect of the Invention
[0011] The present embodiment has the following beneficial effects: when the current prediction mode is a preset prediction mode, determine a history prediction mode corresponding to the current preset prediction mode; then obtain the history optimal reference frame type of the current prediction block in the history prediction mode, the information of the neighboring blocks of the current prediction block in the history prediction mode, and the sub-prediction block information corresponding to the sub-prediction block obtained by partitioning by the current prediction block in the history prediction mode, so that the current prediction block directly inherits various information corresponding to the history prediction mode already performed before the preset prediction mode; and then adaptively generate a reference frame template for the current prediction block when in the preset prediction mode; in this way, by fully considering the characteristics of the preset prediction mode in video encoding, directly utilizing some existing information to generate a reference frame template, greatly reducing the computational complexity, and thereby improving the efficiency of video encoding. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a schematic diagram of the AV1 coding framework. [Diagram 2] FIG. 13 is a schematic diagram of a division rule for a CU. [Diagram 3] 1 is a schematic diagram of MVP selection for different single reference frame modes. [Figure 4] FIG. 1 is a schematic diagram of a process for preferentially selecting the most suitable mode combination. [Diagram 5] 1 is a schematic diagram of an alternative architecture of a video encoding system according to an embodiment of the present invention; [Figure 6]FIG. 2 is an exemplary configuration diagram of an electronic device for inter prediction according to an embodiment of the present application. [Figure 7] 1 is an optional flowchart 1 of an inter prediction method according to an embodiment of the present application. [Figure 8] 2 is a schematic diagram illustrating a positional relationship between a current prediction block and neighboring blocks according to an embodiment of the present invention; [Figure 9] FIG. 2 is a schematic diagram of a sequence of ten different sub-block partition types according to an embodiment of the present invention. [Figure 10] 2 is an optional flowchart 2 of an inter prediction method according to an embodiment of the present application. [Figure 11] 1 is a schematic diagram of a reference relationship between an I frame, a P frame, a B frame, a b frame, and a non-reference B frame according to an embodiment of the present invention. [Figure 12] FIG. 2 is a schematic diagram of the reference relationship of GOP16 according to an embodiment of the present invention. [Figure 13] 1 is a schematic diagram of a process for generating a reference frame template for a NEWMV mode according to an embodiment of the present invention. [Figure 14] FIG. 1 is a schematic diagram of the process of generating an initialization template. [Figure 15] FIG. 2 is a schematic diagram of the process of generating an initialization template. [Figure 16] 1 is a schematic diagram of a reference frame template application process in NEWMV mode according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is hereinafter described in more detail with reference to the drawings, and the described embodiments should not be regarded as limitations on the present application, and all other embodiments obtained by those skilled in the art without creative efforts shall be included in the protection scope of the present application.
[0014] In the following description, the term "some embodiments" refers to a subset of all possible embodiments, and it will be understood that the terms "some embodiments" may refer to the same or different subsets of all possible embodiments, which may be combined with each other without conflict.
[0015] It should be noted that in the following description, the use of the terms "first / second" is merely intended to distinguish between similar objects and does not represent a particular sorting of the objects; it should be understood that, where permitted, "first / second" may be interchanged to indicate a particular order or priority such that the embodiments of the present application described herein may be implemented in an order other than that shown in the drawings and described.
[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are for the purpose of describing the present application examples only and are not intended to be limiting of the present application.
[0017] Before describing the present embodiment in more detail, the nouns and terms related to the present embodiment are explained, and the nouns and terms related to the present embodiment are applicable to the following description.
[0018] 1) Intra-coding is a coding scheme that exploits the relationships that exist between adjacent pixels within a single video frame of a video image to reduce spatial redundancy between adjacent pixels.
[0019] 2) Inter-coding is a coding scheme that exploits the similarity between adjacent frames in a video image to eliminate temporal redundancy between adjacent frames, thereby improving the efficiency of video coding.
[0020] 3) Motion Vector (MV) is a vector that marks the positional relationship between a current block and a reference block when performing inter prediction. In inter coding, since there is a certain correlation between the image content in adjacent frames, a frame image is divided into several blocks, the position of each block in adjacent video frames is searched, and the relative offset amount of the spatial position between the two is calculated, and the obtained relative offset amount is the motion vector.
[0021] 4) Motion Estimation (ME) refers to the process of estimating motion vectors.
[0022] 5) Motion Vector Prediction (MVP) refers to the initial position of the MV derived according to the neighboring blocks of the current block.
[0023] 6) Motion Vector Difference (MVD) refers to the difference between MV and MVP, that is, MVD=MV-MVP. In use, the difference between the MV predicted value and the actual value (MVD) can be further coded to reduce the consumption of bits.
[0024] 7) Rate Distortion Cost (RDCost) is used to preferentially select multiple coding modes. The calculation formula for RDCost is shown in the following formula (1):
number
[0025] 8) The Sum of Absolute Difference (SAD) reflects the time domain difference of the residual signal, and cannot effectively reflect the size of the bitstream.
[0026] 9) Sum of Absolute Transformed Difference (SATD) is a method for calculating distortion, which is obtained by performing Hadamard transform on the residual signal and then calculating the sum of the absolute values of each element. Compared to SAD, SATD requires more calculations but has higher calculation accuracy.
[0027] 10) Sum of the Squared Errors (SSE) is another way to calculate distortion, which is the sum of the squares of the errors between the original pixel and the reconstructed pixel. To calculate the SSE, the residual signal needs to be transformed, quantized, inverse quantized, and inverse transformed, which is computationally complex, but the estimated codeword is the same as the actual coding, which saves the most codewords for the selected coding mode.
[0028] Video coding is widely applied in video transmission, and the future video has the trend of high definition, high frame rate, and high compression ratio, so the compression efficiency of video coding needs to be continuously improved. The first generation video coding standard (AV1) has attracted great attention since its emergence.
[0029] Compared with other video coding technologies such as High Efficiency Video Coding (HEVC) and Advance Video Coding (AVC), AV1 has a higher compression rate and can reduce the occupied bandwidth by 30% when the transmission quality is the same. Moreover, whether it is streaming media or pictures, AV1 can be used to perform coding transmission, which can be widely used in screen sharing and video game streaming.
[0030] FIG. 1 is a schematic diagram of the encoding framework of AV1. As shown in FIG. 1, the electronic device first divides a current video frame 1-1 transmitted in the electronic device into multiple 128×128 coding tree units (CTUs), and then divides each CTU into rectangular coding units (CUs) according to 10 different division rules, and each CU includes multiple prediction modes and transform units (TUs). The electronic device performs inter prediction 1-2 or intra prediction 1-3 on each CU to obtain a prediction value. Here, inter prediction 1-2 includes motion estimation (ME) 1-21 and motion compensation (MC) 1-22, and requires a reference frame 1-12; intra prediction 1-3 includes prediction mode selection 1-31 and prediction 1-32. The electronic device subtracts the input value of each CU from the prediction value to obtain a residual value, then performs transform 1-4 and quantization 1-5 on the residual value to obtain a residual coefficient, and performs entropy coding 1-6 on the residual coefficient to obtain an output bitstream. At the same time, the electronic device further performs inverse quantization 1-7 and inverse transform 1-8 on the residual coefficient to obtain a residual value of a reconstructed image, and the electronic device can add the residual value of the reconstructed image to the prediction value to obtain a reconstructed image, and performs intra prediction mode selection 1-31 according to the reconstructed image to perform intra prediction 1-32. The electronic device further performs filtering 1-9 on the reconstructed image, and the filtered reconstructed image is a reconstructed frame 1-11 corresponding to the current video frame 1-1, and the reconstructed frame 1-11 enters a reference frame queue as a reference frame of the next video frame, thereby encoding backward in sequence.
[0031] Furthermore, the division rule for a CU is not limited to one. FIG. 2 is a schematic diagram of the division rules for a CU. As shown in FIG. 2, a CU has ten types of division rules, which are: no division (NONE) 2-1, quartering (SPLIT) 2-2, horizontal halving (HORZ) 2-3, vertical halving (VERT) 2-4, horizontal quartering (HORZ_4) 2-5, first horizontal thirds (HORZ_A) 2-6, second horizontal thirds (HORZ_B) 2-7, first vertical thirds (VERT_A) 2-8, second vertical thirds (VERT_B) 2-9, and vertical quartering (VERT_4) 2-10. These 10 division rules correspond to 22 different block sizes: 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, 32x16, 32x32, 32x64, 64x32, 64x64, 64x128, 128x64, 128x128, 4x16, 16x4, 8x32, 32x8, 16x64, and 64x16.
[0032] During prediction, each CU includes multiple prediction modes, that is, each CU includes an intra prediction mode and an inter prediction mode. The electronics first compares between different prediction modes within the same prediction type to obtain an optimal prediction mode, and then compares the intra prediction mode with the inter prediction mode to find the optimal prediction mode for each CU. At the same time, since each CU includes multiple TUs, the electronics needs to select an optimal TU from the multiple TUs included in each CU, and then divide the current video frame into individual CUs.
[0033] The intra prediction modes include average prediction based on upper and left reference pixels (DC_PRED), horizontal and vertical differential value combined prediction (SMOOTH_PRED), vertical interpolation prediction (SMOOTH_V_PRED), horizontal interpolation prediction (SMOOTH_H_PRED), gradient minimum directional prediction (PEATH_PRED), and eight different main direction predictions, which are vertical direction prediction (V_PRED), horizontal direction prediction (H_PRED), 45 degree angular direction prediction (D45_PRED), 67 degree angular direction prediction (D67_PRED), 113 degree angular direction prediction (D113_PRED), 135 degree angular direction prediction (D135_PRED), 157 degree angular direction prediction (D157_PRED), and 203 degree angular direction prediction (D203_PRED). Each main direction further includes six angular offsets, respectively at plus or minus 3 degrees, plus or minus 6 degrees, and plus or minus 9 degrees. In some cases, the intra prediction modes may further include palette prediction mode and intra block copy prediction.
[0034] Inter prediction has four single reference frame modes and eight combined reference frame modes. Here, the four single reference frame modes refer to prediction using a single reference frame, and the four single reference frame modes can be NEARESTMV, NEARMV, GLOBALMV, and NEWMV; the eight combined reference frame modes refer to prediction using combined reference frames, which include NEAREST_NEARESTMV, NEAR_NEARMV, NEAREST_NEWMV, NEW_NEARESTMV, NEAR_NEWMV, NEW_NEARMV, GLOBAL_GLOBALMV, and NEW_NEWMV. Here, NEARESTMV and NEARMV refer to the MV of a prediction block being derived based on surrounding block information without the need to transmit MVD, NEWMV refers to being obtained based on the transmitted MVD, and GLOBALMV refers to the MV information of a prediction block being derived based on the global motion. As we can see, NEARESTMV, NEARMV and NEWMV all depend on the derivation of MVP, and for a given reference frame, AV1 calculates four MVPs according to the derivation rules of MVP. The derivation rules of MVP are as follows:
[0035] The electronic device jumps and scans the block information of the 1st, 3rd, and 5th columns to the left of the current block and the 1st, 3rd, and 5th rows above the current block in a specific manner, and first selects a block that uses the same reference frame as the current block and performs deduplication on the MV. If the number of MVs after deduplication is less than 8, selects a reference frame in the same direction and continues to add MVs; if the number of MVs is still less than 8, fills using the motion vectors of all stations until 8 MVs are selected. Then, the electronic device sorts the selected 8 MVs and selects the most important 4 MVs according to the sorting result. Then, the electronic device selects corresponding MVPs from these 4 MVs for three single reference frame modes, NEARESTMV, NEARMV, and NEWMV. Figure 3 is a schematic diagram of MVP selection for different single reference frame modes. Referring to Figure 3, there are multiple reference frames in the single reference frame list, which are ref1, ref2, ref3, . . . The electronic device selects a block using the ref1 reference frame, and after selecting 8 MVs, selects the most important 4 MV1s (MV2, MV3 are the MVs of ref2 and ref3, respectively), and then uses the 0th MV1 as the MVP corresponding to the NEARESTMV, one of the 1st to 3rd MV1s as the MVP of the NEARMV, and one of the 0th to 2nd MV1s as the MVP of the NEWMV. At the same time, the electronic device can further determine ZEROMV as {0,0}.
[0036] For each of the four single reference frame modes of the inter prediction mode, there are seven different reference frame types, and the seven reference frame types and their meanings are shown in Table 1: [Table 1]
[0037] For each of the eight combination reference frame modes in inter prediction mode, all, {LAST_FRAME,ALTREF_FRAME}, {LAST2_FRAME,ALTREF_FRAME},{LAST3_FRAME,ALTREF_FRAME}, {GOLDEN_FRAME,ALTREF_FRAME},{LAST_FRAME,BWDREF_FRAME}, {LAST2_FRAME,BWDREF_FRAME}, {LAST3_FRAME,BWDREF_FRAME}, {GOLDEN_FRAME,ALTREF_FRAME},{LAST_FRAME,BWDREF_FRAME}, {GOLDEN_FRAME,ALTREF_FRAME},{LAST_FRAME,BWDREF_FRAME}, OLDEN_FRAME,BWDREF_FRAME}, {LAST_FRAME,ALTREF2_FRAME}, {LAST2_FRAME,ALTREF2_FRAME}, {LAST3_FRAME,ALTREF2_FRAME}, {GOLDEN_FRAME, ALTREF2_FRAME}, {LAST_FRAME,LAST2_FRAME}, {LAST_FRAME,LAST3_FRAME}, {LAST_FRAME,GOLDEN_FRAME}, {BWDREF_FRAME,ALTREF_FRAME}.
[0038] As can be seen from this, for inter prediction mode, there are a total of 156 (i.e., 7×4+16×8) mode combinations (i.e., combinations composed of prediction mode and reference frame mode), and for each mode combination, up to three types of MVPs are corresponding to each mode combination. Then, four processes are performed for the current MVP: motion estimation (motion estimation is performed only if the prediction mode includes NEWMV mode), inter_inter priority selection, interpolation method priority selection, and motion mode priority selection, thereby selecting the most appropriate mode combination.
[0039] Illustratively, FIG. 4 is a schematic diagram of a process of preferentially selecting the most appropriate mode combination. Referring to FIG. 4, the process of selecting the most appropriate mode combination may include the following steps:
[0040] It starts in step S1.
[0041] In step S2, the total number of MVPs ref_set is obtained and n=0 is set.
[0042] Here, n is the number of the current MVP.
[0043] In step S3, it is determined whether n is smaller than ref_set, and if it is smaller, S4 is executed, and if not, S5 is executed.
[0044] In step S4, the MVP is obtained, and n is automatically incremented by 1.
[0045] In step S5, the process ends.
[0046] In step S6, it is determined whether the current prediction mode includes NEWMV or not. If it does not include NEWMV, step S7 is executed. If it does include NEWMV, step S8 is executed, that is, motion estimation is executed.
[0047] This is because motion estimation requires a large amount of calculation and is slow, and it does not mean that all prediction modes need to perform motion estimation, but only when the prediction mode includes NEWMV, motion estimation is required.
[0048] In step S7, it is determined whether it is a dual reference frame, if not, S9 is executed, if it is a layer, S10 is executed.
[0049] In step S8, motion estimation is performed.
[0050] Step S9 is quickly terminated.
[0051] After S9, execute S11 immediately.
[0052] In step S10, an inter_inter selection is performed.
[0053] In step S11, the interpolation direction is preferentially selected under the optimum motion vector.
[0054] In step S12, the motion mode is preferentially selected.
[0055] After motion mode priority selection, cycle back to S3 until the most appropriate reference frame mode is selected.
[0056] As can be seen from the above analysis, the calculation of each mode combination's priority selection process is very complicated, and especially, the mode combination including the NEWMV mode needs to perform motion estimation further, which slows down the encoding speed and reduces the video encoding efficiency. If the number of mode combinations is reduced by forcibly deleting some reference frame modes, not only the consideration of the characteristics of the prediction mode itself but also the consideration of the characteristics of the video scenario is reduced, and the adaptation degree to the prediction scenario is reduced, which reduces the adaptation ability of the reference frame template, and the coding loss increases with the increase of the coding time, which may cause the obtained bitstream quality to be poor.
[0057] The present embodiment provides an inter-prediction method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product that can improve the efficiency of video encoding. The following describes an exemplary application of an electronic device for inter-prediction according to the present embodiment, and the electronic device according to the present embodiment may be implemented as a terminal or a server. Here, the electronic device may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms; the terminal may be, but is not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart home appliance, an in-vehicle terminal, etc. The terminal and the server may be directly or indirectly connected by a wired or wireless communication method, and the present application is not limited thereto. The following describes an exemplary application of the electronic device.
[0058] Referring to FIG. 5, FIG. 5 is a schematic diagram of an alternative architecture of a video encoding system according to an embodiment of the present application. In order to realize support for one inter prediction application, in the video encoding system 100, an electronic device 400 reads a video frame 200 to be encoded, and then divides the video frame 200 into a number of image blocks, and selects one of them to be used as a current prediction block 300. The electronic device 400 first determines a current prediction mode of the current prediction block 300, and if the current prediction mode of the current prediction block is a preset prediction mode, determines a historical prediction mode corresponding to the current prediction block, that is, determines a prediction mode in which the current prediction block has completed prediction before the preset prediction mode. The electronic device 400 obtains adjacent block information of the current prediction block 300, sub-prediction block information of the sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, where the sub-prediction block is obtained by partitioning the current prediction block using a sub-block partition type previous to the current sub-block partition type. The electronic device 400 generates a reference frame template 500 according to the history optimal reference frame type, the neighboring block information, the sub-prediction block information, and the frame type corresponding to the current prediction block. Then, the electronic device 400 uses the reference frame template 500 to determine a reference frame corresponding to a preset prediction mode, and uses the reference frame to perform inter prediction on the current prediction block to obtain a prediction value corresponding to the current prediction block. After obtaining the prediction value, the electronic device 400 calculates a residual corresponding to the prediction value, and further performs transformation, quantization, and entropy coding on the residual to obtain a final bitstream.
[0059] Referring to FIG. 6, FIG. 6 is an exemplary block diagram of an electronic device for inter prediction according to an embodiment of the present application, in which the electronic device 400 shown in FIG. 6 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. Each component in the electronic device 400 is coupled via a bus system 440. It can be understood that the bus system 440 is used to realize the connection communication between these components. In addition to a data bus, the bus system 440 includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, various buses are represented as the bus system 440 in FIG. 6.
[0060] The processor 410 may be an integrated circuit chip having signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.
[0061] The user interface 430 comprises one or more output devices 431 that enable media content to be presented, said output devices 431 comprising one or more speakers and / or one or more visual display screens, and further comprises one or more input devices 432 that comprise user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch-screen display screen, a camera, other input buttons and controls, etc.
[0062] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 450 optionally includes one or more storage devices that are remote in physical location from processor 410.
[0063] The memory 450 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in the present embodiments is intended to include any suitable type of memory.
[0064] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as illustratively described below.
[0065] The operating system 451 includes system programs for processing various basic system services and executing tasks related to hardware, such as a framework layer, a core library layer, a driver layer, etc., and is used to realize various basic services and process hardware-based tasks.
[0066] The network communications module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420; exemplary network interfaces 420 include Bluetooth® technology, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), among others.
[0067] The presentation module 453 is used to enable presentation of information via one or more output devices 431 (e.g., a display screen, speakers, etc.) associated with the user interface 430 (e.g., the user interface for operating peripherals and displaying content and information).
[0068] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and to interpret the detected inputs or interactions.
[0069] In some embodiments, the inter-prediction device according to the embodiments of the present application can be realized by adopting a software method, and Fig. 6 shows an inter-prediction device 455 stored in a memory 450, which may be software in the form of a program, plug-in, etc., and includes software modules such as a mode determination module 4551, an information acquisition module 4552, a template generation module 4553, and an information prediction module 4554, and these modules are logical, so that any combination or further division can be performed based on the implementation function. The functions of each module are described below.
[0070] In some other embodiments, the inter-prediction device according to the embodiments of the present application may be realized by adopting a hardware scheme. For example, the inter-prediction device according to the embodiments of the present application may be a processor adopting the form of a hardware decoding processor, which is programmed to execute the inter-prediction method according to the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.
[0071] Exemplarily, an embodiment of the present application provides an electronic device for inter prediction, the electronic device comprising: A memory storing executable instructions; and a processor that, when executing executable instructions stored in the memory, implements an inter-prediction method according to an embodiment of the present disclosure. Hereinafter, an inter prediction method according to an embodiment of the present application will be described with reference to exemplary applications and implementations of an electronic device according to an embodiment of the present application. The embodiment of the present application can be realized by cloud technology. Here, cloud technology refers to a hosting technology that integrates a series of resources such as hardware, software, and networks in a wide area network or a local area network to realize computing, storage, processing, and sharing of data.
[0072] Please refer to FIG. 7, which is an optional flowchart 1 of an inter prediction method according to an embodiment of the present application, which will be described in combination with the steps shown in FIG.
[0073] In step S101, if the current prediction mode of the current prediction block is a preset prediction mode, a history prediction mode corresponding to the current prediction block is determined.
[0074] The present embodiment is realized in a scenario of encoding a video. The electronic device first divides an input video frame into a plurality of image blocks, and the current prediction block is an image block among the plurality of image blocks that is predicted at a current time. The electronic device predicts the current prediction block using a different reference frame mode each time, and the current prediction mode is a reference frame mode for predicting the current prediction block at a current time. If the electronic device determines that the current prediction mode is a preset prediction mode, it collects prediction modes that have been completed (determined corresponding reference frame templates) before the preset prediction mode, and determines these prediction modes as history prediction modes. That is, the history prediction modes are prediction modes that have completed prediction before the preset prediction mode.
[0075] It should be noted that in the embodiment of the present application, the preset prediction mode may be a single reference frame mode called NEWMV, or a combined reference frame mode including NEWMV, and the embodiment of the present application is not limited thereto.
[0076] It can be understood that there is a certain priority for prediction of the four single reference frame modes of inter prediction, for example, NEAEESTMV, NEARMV, and GLOBALMV complete prediction before NEWMV mode prediction, and therefore the history prediction mode in the present embodiment may be any one of NEAEESTMV, NEARMV, and GLOBALMV modes, or may be a combination composed of multiple of NEAEESTMV, NEARMV, and GLOBALMV.
[0077] In step S102, neighboring block information of the neighboring block of the current prediction block, sub-prediction block information of the sub-prediction block, and the history optimal reference frame type of the current prediction block in the history prediction mode are obtained. Current prediction block has been predicted using history prediction mode, and after prediction using history prediction mode, to determine the history optimal reference frame type of current prediction block in history prediction mode, electronic device can directly obtain history optimal reference frame type. At the same time, for current prediction block, there are multiple different sub-block division methods, and the sub-prediction block is obtained by partitioning current prediction block using sub-block division type before current sub-block division type, so it is known from the sub-prediction block, and the information of the sub-prediction block is all known, and the neighboring block information corresponding to the neighboring block is also known. Therefore, in the embodiment of the present application, electronic device can directly obtain sub-prediction block information and neighboring block information.
[0078] In addition, the neighboring block information may include a motion vector of the neighboring block and a reference frame type corresponding to the neighboring block, i.e., an adjacent reference frame type, or may include the number of neighboring blocks, and the present application is not limited thereto.
[0079] The sub-prediction block information may include the reference frame type of the sub-prediction block, i.e., the history sub-reference frame type, and may include the number of the sub-prediction block, etc., and the present application is not limited thereto.
[0080] As can be understood, the adjacent blocks may refer to the image blocks to the left, upper left, upper, or upper right of the current prediction block, or may refer to the image blocks in the first, third, and fifth columns located to the left of the current prediction block, and the image blocks in the first, third, and fifth rows located above the current prediction block, and the present application is not limited thereto.
[0081] For illustrative purposes, the present embodiment provides a schematic diagram of the positional relationship between a current prediction block and adjacent blocks. Referring to FIG. 8, image block E is the current prediction block, and image block A, image block B, image block C and image block D are all adjacent blocks of the current prediction block.
[0082] In step S103, a reference frame template is generated based on the history optimal reference frame type, the neighboring block information, the sub-prediction block information, and the frame type corresponding to the current prediction block.
[0083] Since the optimal reference frame types selected in different prediction modes of the same image block may all be the same reference frame type, in order to reduce the computational complexity, the historical optimal reference frame type can be directly inherited and used as part of the reference frame template. Furthermore, if the adjacent block is close to the current prediction block, its contents may also be similar, so a partial reference frame template can be generated for the current prediction block based on information such as the reference frame type and motion vector of the adjacent block. At the same time, since the sub-prediction block is determined using the sub-block division type before the current sub-block division type and is closely related to the current prediction block, the sub-prediction block information can be directly used to generate a partial reference frame template.
[0084] For example, the current predicted block may have 10 different sub-block partition types, which are NONE partition prediction, HORZ partition prediction, VERT partition prediction, SPLIT partition prediction, HORZ_4 partition prediction, HORZ_A partition prediction, HORZ_B partition prediction, VERT_A partition prediction, VERT_B partition prediction, and VERT_4 partition prediction. The order of the 10 sub-block partition types is not fixed, and there are multiple order combinations. The present embodiment provides a schematic diagram of one sequence of ten different sub-block division types. Referring to FIG. 9, the ten sub-block division types can be executed in the following order: NONE division prediction 9-1, HORZ division prediction 9-2, VERT division prediction 9-3, SPLIT division prediction 9-4, HORZ_A division prediction 9-5, HORZ_B division prediction 9-6, VERT_A division prediction 9-7, VERT_B division prediction 9-8, HORZ_4 division prediction 9-9 and VERT_4 division prediction 9-10. When the current sub-block division type is HORZ_A division prediction 9-5, the electronic device can directly obtain sub-prediction block information of sub-prediction blocks corresponding to several sub-block division types, which are NONE division prediction 9-1, HORZ division prediction 9-2, VERT division prediction 9-3 and SPLIT division prediction 9-4.
[0085] In step S104, a reference frame of a preset prediction mode is determined using the reference frame template, and inter-prediction is performed on a current prediction block using the reference frame to obtain a prediction value corresponding to the current prediction block.
[0086] After determining the reference frame template, the electronic device performs a consistency comparison of the candidate reference frame type with the reference frame template, and if the candidate reference frame type is consistent with the reference frame template, the electronic device uses the candidate reference frame type as the reference frame of the current prediction block, and then performs inter prediction on the current prediction block according to the reference frame until completing the circulation of all the candidate reference frame types, thereby obtaining a prediction value corresponding to the current prediction block, thereby completing the application of the reference frame template.
[0087] In the present embodiment, when the current prediction mode is a preset prediction mode, determine a history prediction mode corresponding to the current prediction preset mode, and then obtain the history optimal reference frame type of the current prediction block in the history prediction mode, the information of the neighboring blocks of the current prediction block in the history prediction mode, and the sub-prediction block information corresponding to the sub-prediction block obtained by partitioning by the current prediction block in the history prediction mode, so that the current prediction block directly inherits various information corresponding to the history prediction mode already performed before the preset prediction mode, and then adaptively generate a reference frame template for the current prediction block when in the preset prediction mode. In this way, by fully considering the characteristics of the preset prediction mode in video encoding, directly utilizing some existing information to generate a reference frame template, greatly reducing the computational complexity, and thereby improving the efficiency of video encoding.
[0088] In some embodiments of the present application, the neighboring block information includes the motion vector of the neighboring block, the reference frame type of the neighboring block, and the number of neighboring blocks, and the sub-prediction block information includes the reference frame type of the sub-prediction block, and the number of sub-prediction blocks. In this case, referring to FIG. 10, FIG. 10 is an optional flowchart 2 of the inter-prediction method according to an embodiment of the present application, and the step of generating a reference frame template according to the historical optimal reference frame type, the neighboring block information, the sub-prediction block information, and the frame type corresponding to the current prediction block, i.e., the specific implementation process of S103 may include steps S1031 to S1034.
[0089] In step S1031, an initial template corresponding to a current prediction block is determined according to the history optimal reference frame type, the reference frame type of the sub-prediction block and the reference frame type of the neighboring block.
[0090] The range of candidate reference frame types is limited, and the electronic device first uses the reference frame type of the sub-prediction block and the reference frame type of the neighboring block to determine the number of times each candidate reference frame type is selected, and then selects the appropriate reference frame type of the current prediction block in the preset prediction mode from each candidate reference frame type according to the number of times it is selected. At the same time, because the most appropriate reference frame type of the current prediction block in different prediction modes may be the same, the electronic device can further directly inherit the history optimal reference frame type, i.e., the most matching reference frame type of the current prediction block in the history prediction mode. The electronic device determines the selected appropriate reference frame type and the inherited history optimal reference frame type as the initial template of the current prediction block.
[0091] In step S1032, a master template corresponding to the current prediction block is generated according to the motion vector of the neighboring block.
[0092] The electronic device constructs an error situation when making predictions using each candidate reference frame type according to the motion vector of an adjacent block, and then determines a candidate reference frame type with a small error generated when predicting a current prediction block from each candidate reference frame type according to the error situation, and uses the candidate reference frame type as a master template corresponding to the current prediction block.
[0093] As can be understood, in some embodiments of the present application, the electronic device can further perform category division for each candidate reference frame type, for example, into forward reference category, backward reference category and long-term reference type, and then select the most appropriate candidate reference frame type from each reference category, and use the selected candidate reference frame type to generate a master template corresponding to the current prediction block.
[0094] In step S1033, the extension template corresponding to the current prediction block is determined according to the frame type of the current prediction block, the number of adjacent blocks and the number of sub-prediction blocks.
[0095] Besides determining the initial template and the master template, the electronic device also needs to determine an extension template for the current prediction block, so that when the effect of both the initial template and the master template is not good, the extension template can be used to ensure the quality of the finally determined reference frame template.The electronic device first uses the frame type of the current prediction block to determine a threshold value, and then uses the number of adjacent blocks and the number of sub-prediction blocks to calculate the number of inter-prediction blocks, and compares the number of inter-prediction blocks with the determined threshold value, thereby determining whether some candidate reference frame types are used as extension templates.In this way, the electronic device completes the generation of the extension template.
[0096] In step S1034, a reference frame template corresponding to the current prediction block is generated using the initial template, the master template and the extended template.
[0097] In some embodiments, after the electronic device obtains the initial template, the master template and the extended template, the initial template, the master template and the extended template are combined into a set, which constitutes a reference frame template of a current prediction block in a pre-set prediction mode.
[0098] For example, if the initial template is denoted by mask_init, the master template is denoted by mask_main, and the extended template is denoted by mask_add, the reference frame template can be denoted by mask_newmv=mask_init|mask_main|mask_add (| denotes a union).
[0099] In some other embodiments, the electronic device may further fuse the initial template, the master template and the enhanced template to obtain a reference frame template corresponding to the current prediction block.
[0100] In this embodiment, the electronics uses the inherited information and parameters to generate a master template, an initial template and an extended template for the current predicted block, respectively, and then integrates the three templates to obtain a reference frame template corresponding to the current predicted block, so as to facilitate subsequent use of the reference frame template to determine the reference frame.
[0101] In some embodiments of the present application, the step of determining an initial template corresponding to a current prediction block based on a historical optimal reference frame type, a reference frame type of a sub-prediction block and a reference frame type of an adjacent block, i.e., a specific implementation process of S1031 may include steps S1031a to S1031c.
[0102] In step S1031a, a first initial template is determined according to the reference frame type of the sub-prediction block and the reference frame type of the neighboring block.
[0103] It should be explained that the electronic device can determine, from all candidate reference frame types, what candidate reference frame types are selected as the sub-prediction block and the adjacent block according to the reference frame type of the sub-prediction block and the reference frame type of the adjacent block, and then determine the selected candidate reference frame type according to the selection number of times of the selected candidate reference frame type, and determine whether the first initial template is a selected reference frame type or an idle template.
[0104] In step S1031b, the historically optimal reference frame type is used as a second initial template.
[0105] The electronic device directly inherits the history optimal reference frame type, and uses the optimal reference frame type as the second initial template. It can be understood that when the history prediction mode is NEARESTMV mode, NEARMV mode and GLOBALMV mode, the electronic device sequentially compares the optimal reference frame type values of these three modes (the optimal reference frame types all belong to the seven reference frame types shown in Table 1) with 0, and when the value of the optimal reference frame type of a mode is greater than 0, the optimal reference frame type of this mode can be added to the second initial template.
[0106] Exemplarily, if the value of the optimal reference frame type corresponding to the NEARESTMV mode is greater than 0, the optimal reference frame type in the NEAREST mode is added to the second initial template.
[0107] In step S1031c, an initial template corresponding to a current prediction block is determined using the first initial template and the second initial template.
[0108] In some embodiments, after obtaining the first initial template and the second initial template, the electronic device aggregates the first initial template and the second initial template, and the obtained set is the initial template of the current prediction block.
[0109] Exemplarily, if a first initial template is denoted by mask_init1 and a second initial template is denoted by mask_init2, the initial templates may be denoted as mask_init=mask_init1|mask_init2.
[0110] In some other embodiments, the electronic device may further weight the first initial template with the second initial template to obtain an initial template corresponding to the current prediction block.
[0111] In another embodiment of the present application, the electronic device first determines a part of the initial template using the reference frame type of the sub-prediction block and the reference frame type of the adjacent block, and then the historical optimal reference template can be used as another part of the initial template, and the process of determining the initial template is realized by integrating the two parts.
[0112] In this embodiment, a first initial template is determined according to the reference frame type of the sub-prediction block and the reference frame type of the adjacent block, and the historical optimal reference frame type is inherited as a second initial template. Then, the first initial template and the second initial template are integrated into an initial template, thereby realizing the process of determining an initial template in an electronic device.
[0113] In some embodiments of the present application, the step of determining a first initial template according to the reference frame type of the sub-prediction block and the reference frame type of the neighboring block, ie, a specific implementation process of S1031a, may include steps S201 to S203.
[0114] In step S201, at least one historically selected reference frame type is determined by using the reference frame type of the neighboring block and the reference frame type of the sub-prediction block.
[0115] Since the reference frame type of the adjacent block and the reference frame type of the sub-prediction block are actually determined from all candidate reference frame types, the electronic device can merge the reference frame type of the adjacent block with the reference frame type of the sub-prediction block to clarify which candidate reference frame type is selected in the history prediction mode among all candidate reference frame types, and thereby determine the candidate reference frame type selected in the history prediction mode as the history selected reference frame type.Since the selected candidate reference frame type may not be limited to one, the electronic device can obtain at least one history selected reference frame type.
[0116] In step S202, the number of times each historically selected reference frame type is selected in at least one historically selected reference frame type is counted.
[0117] After obtaining at least one historically selected reference frame type, the electronic device calculates the number of times each historically selected reference frame type is selected in the historical prediction mode, thereby obtaining the number of times each historically selected reference frame type is selected.
[0118] In some embodiments, when counting the number of times a history selected reference frame is selected, the electronic device respectively counts the number of times each history selected reference frame is used as a reference frame type of a sub-prediction block, and counts the number of times each history selected reference frame is used as a reference frame type of an adjacent block, and then adds these two numbers for each history selected reference frame type to obtain the number of times each history selected reference frame type is selected.
[0119] Exemplarily, LAST_FRAME is one historically selected reference frame type, where LAST_FRAME is selected three times by sub-prediction blocks and selected twice by adjacent blocks, so that the number of times LAST_FRAME is selected is five.
[0120] In some other embodiments, when counting the number of times a historically selected reference frame is selected, the electronic device counts the number of times each historically selected reference frame is used as a reference frame type for a sub-prediction block, and counts the number of times each historically selected reference frame is used as a reference frame type for an adjacent block and calculates the average value to obtain the number of times each historically selected reference frame type is selected.
[0121] In step S203, the selected times are used to select a first initial template from each historically selected reference frame type.
[0122] The electronic device sorts the selected times of each historically selected reference frame type by size, thereby determining the order of the selected times of each historically selected reference frame type, and then selects the historically selected reference frame type corresponding to the largest selected times to be used as the first initial template. The electronic device further compares all the selected times of each historically selected reference frame type with a set threshold, and can use the historically selected reference frame type whose selected times are greater than the set threshold as the first initial template.
[0123] In the present embodiment, the electronic device first determines at least one historically selected reference frame type, and then selects a first initial template from each historically selected reference frame type according to the selection number of times of each historical reference frame type, so that the historically selected reference frame type selected by most adjacent blocks, sub-prediction blocks is used as the first initial template to make the first initial template more accurate.
[0124] In some embodiments of the present application, the step of selecting a first initial template from each historically selected reference frame type using the selected number of times, ie, a specific implementation process of S203 may include steps S2031 to S2034.
[0125] In step S2031, the maximum number of times selected is selected from the number of times selected for each historically selected reference frame type.
[0126] In step S2032, the selected number of times for each historical selected reference frame type is expanded to obtain the expanded selected number of times.
[0127] It can be understood that the electronic device can amplify the selected number of times according to a preset multiple, or can amplify the selected number of times according to a random multiple.
[0128] Furthermore, by expanding the number of times selected according to a preset multiple, as many reference frames as possible can be selected from each historically selected reference frame type; that is, when the number of times selected of a certain historically selected reference frame type reaches a certain level, it may be used as a reference frame template, which can be selected as the first initial template by the method of expanding according to a preset multiple, and thereby added to the reference frame template.
[0129] It can be understood that the preset multiple can be set to 4, 6, or other values as required, and the present application is not limited thereto.
[0130] In step S2033, the expanded selected number is compared with the maximum selected number to obtain a comparison result corresponding to each historical selected reference frame type.
[0131] Here, the comparison result indicates whether the expanded number of times to be selected is equal to or greater than the maximum number of times to be selected.
[0132] In step S2034, the historical selected reference frame type, the comparison result of which indicates that the extended selected number is equal to or greater than the maximum selected number, is used as the first initial template.
[0133] The electronic device compares the expanded selected times of each historical selected reference frame type with the maximum selected times selected, to determine the size relationship between the expanded selected times of each historical selected reference frame type and the largest selected times among all historical selected reference frame types, and uses the historical selected reference frame type whose expanded selected times are greater than or equal to the maximum selected times as the first initial template.
[0134] Exemplarily, the selected times of each historical selected reference frame type can be indicated by ref_num[i], and the maximum selected times can be indicated by ref_num[0] (ranked first after sorting by size). When the preset multiple is 4, if ref_num[i] * 4 ≥ ref_num[0], then mask_init1 |= (1 << i) (that is, the frame number i of the historical selected reference frame that meets the condition is stored in the number string for recording the first initial template).
[0135] In the embodiment of the present application, after expanding the selected times of each historical selected reference frame type, the electronic device compares it with the largest selected times, so that the first initial template can be selected from each historical selected reference frame type to facilitate the generation of subsequent initial templates.
[0136] In some embodiments of the present application, there may be sub-prediction information and adjacent block information that cannot be collected (i.e., the sub-prediction information and adjacent block information are idle information), so that both the reference frame type of the sub-prediction block and the reference frame type of the adjacent block are idle, so that the first initial template becomes idle, and if the historical optimal reference frame type is not collected (e.g., NEARESTMV, NEARMV and GLOBALMV modes are not predicted before the NEWMV mode), the second initial template also becomes idle, in which case the initial template is idle, and in such a case, after using the first initial template and the second initial template to configure the initial template of the current prediction block, i.e., after step S1031c, the method may further include step S1031d.
[0137] In step S1031d, at least one pre-defined reference frame type is added to the initial template to obtain a calibrated initial template.
[0138] If the initial template is idle, the electronic device corrects and complements the initial template to ensure the validity of the final reference frame template. At this time, the electronic device adds at least one preset reference frame type to the idle initial template, so that the initial template to which the at least one preset reference frame type is added is a calibrated initial template. In this case, the step of generating a reference frame template corresponding to a current prediction block using the initial template, the master template and the extended template, i.e., the implementation process of S1034, is changed to the step of generating a reference frame template corresponding to a current prediction block using the calibrated initial template, the master template and the extended template.
[0139] It should be noted that the at least one pre-defined reference frame type may be a selected candidate reference frame type, such as LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, etc., or may be a video frame selected from the video frames, such as the first video frame, and the present application is not limited thereto.
[0140] As can be understood, the at least one pre-defined reference frame type may include only one pre-defined reference frame type, such as including only LAST_FRAME, or may include three pre-defined reference frame types, such as including LAST_FRAME, BWDREF_FRAME, and ALTREF_FRAME, and the present application is not limited thereto.
[0141] In this embodiment, the electronic device can further add at least one pre-defined reference frame type to the initial template when the initial template is idle, thereby realizing calibration and complementation of the initial template and ensuring the validity of the final reference frame template.
[0142] In some embodiments of the present application, the neighboring block information includes a reference frame type of the neighboring block, and the sub-prediction block information includes a reference frame type of the sub-prediction block; the step of obtaining neighboring block information of the neighboring block of the current prediction block and sub-prediction block information of the sub-prediction block, i.e., a specific implementation process of S102 may include steps S1021 to S1023.
[0143] In step S1021, determine the optimal prediction mode of the neighboring block to obtain a first determination result, and determine the optimal prediction mode of the sub-prediction block to obtain a second determination result.
[0144] It should be noted that the first judgment result indicates whether the optimal prediction mode of the adjacent block is a first pre-set mode, and the second judgment result indicates whether the optimal prediction mode of the sub-prediction block is a second pre-set mode.
[0145] In step S1022, if the first determination result indicates that the optimal prediction mode of the neighboring block is the first preset mode, obtain the reference frame type of the neighboring block.
[0146] In step S1023, if the second determining result indicates that the optimal prediction mode of the sub-prediction block is the second preset mode, obtain a reference frame type of the sub-prediction block.
[0147] When obtaining adjacent block information corresponding to an adjacent block, the electronic device first determines an optimal prediction mode of the adjacent block, and records a reference frame type of the adjacent frame only when it determines that the optimal prediction mode of the adjacent block is the first preset mode. Similarly, the electronic device records a reference frame type of the sub-prediction block only when it determines that the optimal prediction mode of the sub-prediction block is the second preset mode, thereby obtaining adjacent block information and sub-prediction block information in this manner. If the optimal prediction mode of the adjacent block is not the first preset mode, the reference frame type of the adjacent frame is idle, and if the optimal prediction mode of the sub-prediction block is not the second preset mode, the reference frame type of the sub-prediction block is idle.
[0148] In addition, the first preset mode may be the NEARMV mode or the NEARESTMV mode, and the second preset mode may be the NEARESTMV mode or the NEARMV mode, and the present application is not limited thereto.
[0149] For example, when the adjacent blocks are the four adjacent blocks shown in FIG. 8, if the optimal prediction mode of each adjacent block is inter mode, the electronic device records the reference frame type of the adjacent block to obtain ref_nb[4]={ref_A, ref_B, ref_C, ref_D}, thereby obtaining adjacent block information.
[0150] In this embodiment, the electronic device obtains the reference frame type of the adjacent block and the reference frame type of the sub-prediction block only when the optimal prediction modes of the adjacent block and the sub-prediction block satisfy a condition, thereby realizing the process of obtaining adjacent block information and sub-prediction block information.
[0151] In some embodiments of the present application, the step of generating a master template corresponding to a current prediction block according to the motion vector of the neighboring block, ie, a specific implementation process of S1032 may include steps S1032a to S1032d.
[0152] In step S1032a, the motion vectors of the neighboring blocks are used to calculate a selection parameter for each candidate reference frame type within the total number of candidate reference frame types.
[0153] It should be noted that the all candidate reference frame types represent all available reference frame types during inter prediction, and in some embodiments of the present application, the all candidate reference frame types may be the seven reference frame types shown in Table 1, or some reference frame types selected from the seven reference frame types. The selection parameter represents the difference between the input value and the predicted value of the neighboring block, and such difference can be calculated using SAD or SATD. Although the accuracy of SATD is higher, the calculation complexity is correspondingly higher.
[0154] In step S1032b, the total amount of candidate reference frame types are respectively divided into candidate forward reference frame types, candidate backward reference frame types and candidate long-term reference frame types.
[0155] The electronic device divides all candidate reference frame types into three groups, i.e., candidate forward reference frame types, candidate backward reference frame types and candidate long-term reference frame types, according to the reference direction of each candidate reference frame type, and subsequently facilitates performing a reference frame type selection process for the three groups, that is, selecting forward reference frame types, backward reference frame types and long-term reference frame types from the three groups, respectively.
[0156] For example, when the total number of candidate reference frame types is seven candidate reference frame types according to Table 1, the electronic device divides LAST_FRAME, LAST_FRAME2 and LAST3_FRAME into candidate forward reference frame types, BWDREF_FRAME, ALTREF2_FRAME and ALTREF_FRAME into candidate backward reference frame types, and GOLDEN_FRAME into candidate long-term reference frame types according to the reference direction of each candidate reference frame type.
[0157] In step S1032c, a forward reference frame type, a backward reference frame type and a long-term reference frame type are selected using a selection parameter corresponding to the candidate forward reference frame type, a selection parameter corresponding to the candidate backward reference frame type and a selection parameter corresponding to the candidate long-term reference frame type.
[0158] In step S1032d, synthesize a master template corresponding to the current prediction block using the forward reference frame type, the backward reference frame type and the long-term reference frame type.
[0159] The electronic device performs a size comparison for the selection parameters of the candidate reference frame types in each group, and according to the comparison result, selects a forward reference frame type from the candidate forward reference frame types, selects a backward reference frame type from the candidate backward reference frame types, and selects a long-term reference frame type from the candidate long-term reference frame types. Then, the electronic device collects the selected forward reference frame type, backward reference frame type and long-term reference frame type into one set, and the obtained set is the master template.
[0160] In some other embodiments, the electronics may further perform weighted fusion on the forward reference frame type, the backward reference frame type and the long-term reference frame type to generate a master template for the current prediction block.
[0161] Exemplarily, if the current reference frame type is denoted by ref_list0, the backward reference frame type is denoted by ref_list1, and the long-term reference frame type is GOLDEN_FRAME, the master template is mask_main=ref_list0|ref_list1|GOLDEN_FRAME.
[0162] In the present embodiment, the electronic device calculates selection parameters for each candidate reference frame type according to the motion vectors of adjacent blocks, and then selects forward reference frame types, backward reference frame types and long-term reference frame types from the candidate forward reference frame types, candidate backward reference frame types and candidate long-term reference frame types obtained by dividing the total number of candidate reference frame types according to the reference direction according to the selection parameters, thereby aggregating these selected reference frame types into a master template.
[0163] In some embodiments of the present application, the step of selecting a forward reference frame type, a backward reference frame type and a long-term reference frame type using a selection parameter corresponding to a candidate forward reference frame type, a selection parameter corresponding to a candidate backward reference frame type, and a selection parameter corresponding to a candidate long-term reference frame type, i.e., a specific implementation process of S1032c may include steps S301 to S303.
[0164] In step S301, the candidate reference frame type with the smallest selection parameter among the candidate forward reference frame types is used as the forward reference frame type.
[0165] The electronic device compares the selection parameters of each candidate reference frame type in the candidate forward reference frame types with each other, selects the smallest selection parameter from among them, and then uses the candidate reference frame type corresponding to the smallest selection parameter as the forward reference frame type.
[0166] In step S302, the candidate reference frame type with the smallest selection parameter among the candidate backward reference frame types is used as the backward reference frame type.
[0167] The electronic device compares the selection parameters of each candidate reference frame type in the candidate backward reference frame types with each other, selects the smallest selection parameter from among them, and uses the candidate reference frame type corresponding to the smallest selection parameter of the candidate backward reference frame types as the backward reference frame type.
[0168] In addition, in the present application, whether S301 or S302 is performed first does not affect the selection of the forward reference frame type and the backward reference frame type, and therefore, in some embodiments, the electronic device may perform S302 first and then perform S301, or may perform S301 and S302 simultaneously.
[0169] In step S303, a candidate reference frame type whose selection parameter is smaller than the sum of the selection parameter corresponding to the forward reference frame type and the selection parameter corresponding to the backward reference frame type is used as the long-term reference frame type.
[0170] The electronic device adds the selection parameter of the previously selected forward reference frame type to the selection parameter of the candidate reference frame type to obtain a sum result, and then performs a size comparison of the selection parameter of each candidate reference frame type in the candidate long-term reference frame types with the sum result, thereby selecting a candidate reference frame type whose selection parameter of the candidate long-term reference frame type is smaller than the sum result to be used as the long-term reference frame type.
[0171] Exemplarily, the selection parameters for the forward reference frame type are denoted by sad_list0, and the selection parameters for the backward reference frame type are denoted by sad_list1. If there is only one candidate reference frame type GLODEN_FRAME in the candidate long-term reference frame types, then if the selection parameters for GLODEN_FRAME are less than sad_list0+sad_list1, then GLODEN_FRAME is used as the long-term reference frame type.
[0172] In the present embodiment, the electronic device finds the smallest selection parameters of the candidate forward reference frame type and the candidate backward reference frame type, respectively, thereby determining the forward reference frame type and the backward reference frame type, and then can select a long-term reference frame type from the candidate long-term reference frame types according to the sum of the selection parameter of the forward reference frame type and the selection parameter corresponding to the backward reference frame type, and in this way, the electronic device can select a forward reference frame type, a backward reference frame type and a long-term reference frame type.
[0173] In some embodiments of the present application, the step of calculating a selection parameter for each candidate reference frame type in the total number of candidate reference frame types by using the motion vectors of neighboring blocks, i.e., a specific implementation process of S1032a may include steps S401 to S403.
[0174] In step S401, each candidate reference frame type in the total amount of candidate reference frame types is matched with the motion vector of an adjacent block to obtain a matching result.
[0175] Here, the matching result indicates whether or not there is a matching motion vector for each candidate reference frame type.
[0176] There is a correspondence between the motion vector of the neighboring block and the reference frame type of the neighboring block, and the electronic device determines a candidate reference frame type to match the reference frame type of the neighboring block, thereby determining whether the reference frame type of the neighboring block is the same as the candidate reference frame type. If the candidate reference frame type is the same, the electronic device considers that the candidate reference frame type matches the motion vector of the neighboring block, that is, there is a matching motion vector. According to this process, the electronic device divides the motion vector of the neighboring block under each candidate reference frame type, thereby obtaining the matching result of each candidate reference frame type.
[0177] It should be noted that after matching, some candidate reference frame types may have matching motion vectors and some candidate reference frame types may not have matching motion vectors, so one matching result is needed to record these situations.
[0178] In step S402, if the matching result indicates that there is no matching motion vector in each candidate reference frame type, a preset value is used as a selection parameter for each candidate reference frame type.
[0179] If the matching result indicates that there is no matching motion vector in a candidate reference frame type, the electronic device initializes selection parameters for the candidate reference frame type, i.e., uses pre-set numerical values as selection parameters for each candidate reference frame type.
[0180] It can be understood that the preset value can be INT32_MAX, i.e. the maximum number of 32 bits, or can be a binary representation of 50000, and the present application is not limited thereto.
[0181] In step S403, if the matching result indicates that there is a matching motion vector for each candidate reference frame type, a selection parameter for each candidate reference frame type is calculated using a predicted value and an input value when an adjacent block predicts based on the motion vector of the adjacent block.
[0182] If the matching result for a candidate reference frame type indicates that a matching motion vector exists in the candidate reference frame type, the electronic device calculates selection parameters for each candidate reference frame type using a predicted value obtained by an adjacent block predicting using the motion vector of the adjacent block and an input value of the adjacent block itself.
[0183] In some embodiments of the present application, there may be a plurality of matching motion vectors, in this case, these matching motion vectors can all be regarded as sub-vectors, and the matching motion vectors are collectively referred to as a single name, so that the matching motion vectors include a plurality of sub-vectors.Furthermore, each sub-vector corresponds to a predicted value, and the predicted value corresponding to each sub-vector is a sub-predicted value, so that the predicted value includes a plurality of sub-predicted values, and the plurality of sub-vectors correspond to the plurality of sub-predicted values.In this case, the step of calculating the selection parameters of each candidate reference frame type using the predicted value and the input value when the neighboring block predicts based on the motion vector of the neighboring block, i.e., the specific implementation process of S403, may include steps S4031 to S4033.
[0184] In step S4031, a difference between a pixel of a sub-predicted value corresponding to each of a plurality of sub-vectors and a pixel of an input value of an adjacent block is calculated to obtain a pixel difference value corresponding to each sub-vector.
[0185] In step S4032, the absolute values of the pixel difference values corresponding to each sub-vector are accumulated to obtain a temporary selection parameter corresponding to each sub-vector.
[0186] Illustratively, the present embodiment provides a method for calculating the temporary selection parameter, see equation (2).
number
[0187] In step S4033, the smallest temporary selection parameter among the temporary selection parameters of each sub-vector is used as the selection parameter corresponding to each candidate reference frame type.
[0188] The electronics selects, for each candidate reference frame type, one smallest temporary selection parameter from the temporary selection parameters in each sub-vector and uses the smallest temporary selection parameter as the final selection parameter, and in this manner, the electronics calculates the selection parameters for each candidate reference frame type.
[0189] In some embodiments of the present application, the step of matching each candidate reference frame type in the total number of candidate reference frame types with the motion vector of the neighbor block to obtain a matching result, i.e., the specific implementation process of S401 may include step S4011.
[0190] In step S4011, if the neighboring block is available, it is determined that the optimal prediction mode of the neighboring block is the second pre-set mode, and if the reference frame type of the neighboring block is the same as each candidate reference frame type, there is a matching motion vector for each reference frame type.
[0191] When matching each candidate reference frame type with the motion vector of the neighboring block, the electronic device first determines whether the neighboring block is available, and if the neighboring block is available, determines whether the optimal prediction mode of the neighboring block is a second preset mode, and determines whether the reference frame type of the neighboring block is the same as each candidate reference frame type. Because there is a correspondence between the motion vector of the neighboring block and the reference frame type of the neighboring block, if the electronic device determines that the optimal prediction mode of each neighboring block is indeed the second preset mode and the reference frame type of the neighboring block is the same as each candidate reference frame type, the electronic device considers that the motion vector of the neighboring block corresponding to the reference frame type of the neighboring block matches each candidate reference frame type, and therefore there is a matching motion vector for each candidate reference frame type.
[0192] In the present embodiment, the electronic device can first match the motion vector of the adjacent block to each candidate reference frame type, and calculate a selection parameter for each candidate reference frame type according to whether there is a matching motion vector in each candidate reference frame type, and in this way, the electronic device can obtain a selection parameter corresponding to each candidate reference frame type.
[0193] In some embodiments of the present application, a specific implementation process of determining an extension template corresponding to a current predicted block based on the frame type of the current predicted block, the number of adjacent blocks, and the number of sub-prediction blocks of the current predicted block, i.e., S1033, may include steps S1033a to S1033d.
[0194] In step S1033a, a frame type weight corresponding to a current prediction block is determined according to a corresponding relationship between the frame type of the current prediction block and a preset frame type weight.
[0195] Although the video frame to which the current prediction block belongs has been determined, the frame type of each video frame has already been determined before prediction. Different frame types have different reference relationships, and therefore different importance. The importance of a video frame referenced by more video frames is higher than the importance of a video frame referenced by fewer video frames. The electronic device can determine the importance according to the frame type of the video frame corresponding to the current prediction block, and then determine the corresponding frame type weight according to the importance.
[0196] For example, FIG. 11 is a schematic diagram of the reference relationships between I frames, P frames, B frames, b frames and non-reference B frames according to an embodiment of the present application, and as can be seen from FIG. 11, according to the reference relationships, the importance order determined for these frame types is: I frames > P frames > B frames > b frames > non-reference B frames.
[0197] In some embodiments, the importance is also related to the structure of a group of pictures (GOP). Figure 12 is a schematic diagram of the reference relationship of GOP16 according to an embodiment of the present application, and as can be seen from Figure 12, POC16 refers to POC0; POC8 refers to POC0 and POC16; POC4 refers to POC0 and POC8; POC2 refers to POC0 and POC4; and the remaining POCs are not referenced at all. This allows the weighting levels shown in Table 2 to be determined, see Table 2. [Table 2]
[0198] Therefore, the weight sorting of each video frame in GOP16 is: POC0>POC16>POC8>POC4>POC2>POC1.
[0199] Thus, the electronic device can select the frame type weight according to the frame type.
[0200] In step S1033b, an expansion threshold is generated according to the frame type weight.
[0201] The electronic device can select a certain video frame and use the frame type weight of the selected video frame as the expansion threshold, and further can first set different threshold parameters for different frame type weights, and then use the threshold parameter corresponding to the frame type weight of the selected video frame as the expansion threshold.
[0202] For illustrative purposes, the present embodiment provides a formula for generating the expansion threshold, see formula (3).
number
[0203] In step S1033c, the number of adjacent blocks and the number of sub-prediction blocks are added to obtain a sum result.
[0204] In step S1033d, if the sum result is less than or equal to the expansion threshold, at least one pre-defined reference frame type is used as the expansion template corresponding to the current prediction block.
[0205] The electronic device adds the number of the neighboring blocks to the number of the sub-prediction blocks to obtain a sum result, and performs a size comparison of the sum result with an expansion threshold. If the electronic device finds that the sum result is less than or equal to the expansion threshold, the electronic device obtains at least one pre-defined reference frame type, and uses the at least one pre-defined reference frame type as an expansion template. In this way, the electronic device completes the process of generating the expansion template.
[0206] It should be noted that the at least one preset reference frame type may be LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, or LAST_FRAME, BWDREF_FRAME, GOLDEN_FRAME, and the present application is not limited thereto.
[0207] In the present embodiment, the electronic device first determines a frame type weight according to the frame type of the current prediction block, and then generates an expansion threshold according to the frame type weight, thereby selecting whether to add at least one pre-defined reference frame type to the expansion template according to the sum of the number of adjacent blocks and the number of sub-prediction blocks and the size relationship between the generated expansion thresholds, thereby ensuring the validity of the finally generated reference frame template.
[0208] An exemplary application of the present embodiment in a practical application scenario will now be described.
[0209] The present embodiment is realized in a scenario where an encoder (electronic device) generates a reference frame template for a NEWMV mode (pre-set prediction mode). The spirit of this process is to: before NEWMV mode prediction, NEARSETMV, NEARMV and GLOBALMV have been determined, and inherit the optimal reference frame of these modes, and inherit the information of adjacent blocks and predicted different CU divisions; according to each reference frame adjacent MV, calculate the SAD corresponding to each MV, and then use the minimum SAD as the SAD of this reference frame; divide the reference frames into three groups, forward reference, backward reference and long-term reference, and from each group, select the reference frame corresponding to the minimum SAD in the group as the representative of the regrouped reference frames to form the master template of the NEWMV mode; and further combine with the collected information to synthesize a final reference frame template of the NEWMV mode. Figure 13 is a schematic diagram of a process of generating a reference frame template for a NEWMV mode according to an embodiment of the present application, and referring to Figure 13, the process may include the following steps:
[0210] In step S501, sad (a selection parameter) is determined for each reference frame (candidate reference frame type).
[0211] In step S502, reference frame information in adjacent blocks and different CU divisions (obtaining sub-prediction blocks) is obtained.
[0212] In step S503, a reference frame template for the NEWMV mode is generated.
[0213] In step S504, a reference frame template is applied.
[0214] Each step is explained below.
[0215] In S501, the encoder initializes the SAD corresponding to each reference frame to INT32_MAX (a preset value), where INT32_MAX is the maximum number of 32 bits. Step S501 is divided into three parts: In step S5011, a neighboring block motion vector candidate list is obtained.
[0216] The positional relationship between the current block (current prediction block) and the neighboring blocks may be as shown in Figure 8. The electronic device sequentially determines the position of each neighboring block, and records the current MV if the current neighboring block position is possible, and the optimal mode is inter prediction, and the reference frame is the same as the current reference frame (each candidate reference frame type). There may be multiple MVs under the reference frame of the current neighboring block, or there may be no MV at all (determine a matching motion vector for each candidate reference frame type, and a motion vector matching one candidate reference frame type may have multiple sub-vectors).
[0217] In step S5012, the SAD (temporary selection parameter) corresponding to each MV (sub-vector) is calculated in sequence.
[0218] The encoder can calculate the SAD according to equation (2). Here, the encoder can also calculate the SATD in each MV. Although the SATD is more accurate, the calculation complexity is relatively large.
[0219] In step S5013, the minimum SAD (smallest temporary selection parameter) corresponding to each reference frame is selected.
[0220] Since there may be multiple MVs under the current reference frame, multiple SADs are obtained and the smallest SAD is taken as the SAD of the current reference frame; there may be no MVs under the current reference frame, in which case the SAD of the current reference frame is INT32_MAX (if there is no matching motion vector for each candidate reference frame type, a pre-set numerical value is used as the selection parameter for each candidate reference frame type).
[0221] Step S502 is divided into three parts, which are as follows:
[0222] In step S5021, adjacent block reference frame information is obtained.
[0223] The positional relationship between the current block and the neighboring blocks is as shown in Figure 8. If a neighboring block exists and the optimal mode is the inter mode (first preset mode), the reference frame information of the neighboring block is recorded (the reference frame type of the neighboring block is obtained, i.e., the neighboring block information is obtained).
[0224] In step S5022, reference frame information for different CU partition types is obtained.
[0225] There are 10 CU partition types for the current block, and the partitioning is as shown in Figure 9. Since the encoding prediction process is performed sequentially, when the current partition type (current sub-block partition type) is performed, other CU partition types (previous sub-block partition types) may have already been performed. For example, when the current partition type is HORZ_A, the NONE partition type has been performed previously, and therefore the partition information of the NONE partition type can be used.
[0226] Obtain reference frame information (sub-prediction block information) of each sub-block (sub-prediction block) in each determined CU partition type. The judgment method of this process is similar to that of adjacent blocks, and records the reference frame information of each sub-block if a sub-block position exists and the optimal mode is inter mode (second predefined mode).
[0227] In step S5023, the data is organized.
[0228] The number of inter-predicted blocks divided according to CUs different from those of adjacent blocks (the sum of the number of adjacent blocks and the number of sub-predicted blocks) is counted and set as inter_total_num. The collected reference frames (history selected reference frame types) and the number of times each reference frame is selected (number of times selected) are counted, and the reference frames are arranged from largest to smallest according to the number of times selected.
[0229] The reference frame template generated in S503 is defined as mask_newmv, and is composed of three parts, that is, an initialization template, a master template, and an extension template, that is, mask_newmv=mask_init|mask_main|mask_add.
[0230] Step S503 includes the following steps: In step S5031, an initialization template (initial template) is generated.
[0231] The mask_init is generated according to the collected reference frame information (ie, the reference frame type of the neighboring block and the reference frame type of the sub-prediction block). The generation process mainly includes the following steps:
[0232] In step S50311, according to the reference frame information in the CU partition type different from that of the neighboring block, determine whether to add the current reference frame to the initialization template. Figure 14 provides a schematic diagram 1 of the process of generating the initialization template. Referring to Figure 14, the process includes the following steps:
[0233] The process starts in step S601.
[0234] In step S602, the selected reference frame number is obtained.
[0235] In step S603, it is determined whether the number is equal to or smaller than the number of reference frame types, which refers to the number of types of reference frame in the collected reference frame information.
[0236] If it is equal to or less than this, execute step S604; otherwise, execute step S607.
[0237] In step S604, it is determined whether or not four times the number of times to be selected (the expanded number of times to be selected) is equal to or greater than the maximum number of times to be selected (the maximum number of times to be selected).
[0238] If it is equal to or less than, execute step S605; if not, execute step S606.
[0239] In step S605, the number of the selected reference frame is recorded.
[0240] In step S606, the number is automatically incremented by 1, and the process advances to S602 again.
[0241] In step S607, the process ends.
[0242] It should be noted that the number of the selected reference frame here is determined by arranging the reference frames from largest to smallest according to the selected number, so the smaller the selected number, the smaller the reference value. If they are all added to the initialization template, it will increase the number of unnecessary reference frames and slow down the encoding speed.
[0243] Up to this point, the process of step S50311 is completed, and the process proceeds to step S50312.
[0244] In step S50312, an initialization template is generated according to the reference frame information of the predicted mode.
[0245] FIG. 15 is a schematic diagram 2 of the process of generating an initialization template. Referring to FIG. 15, the process includes the following steps:
[0246] The process starts in step S701.
[0247] In step S702, it is determined whether or not the best reference frame in the NEARESTMV mode (history best reference frame type) has been collected.
[0248] If so, execute step S703; if not, execute step S704.
[0249] In step S703, the optimal reference frame for the NEARESTMV mode is added to the initialization template.
[0250] In step S704, it is determined whether the optimal reference frame for the NEAR MV mode (historical optimal reference frame type) has been collected.
[0251] If so, execute step S705; if not, execute step S706.
[0252] In step S705, the optimal reference frame for the NEAR MV mode is added to the initialization template.
[0253] In step S706, it is determined whether an optimal reference frame for the GLOBALMV mode (history optimal reference frame type) has been collected.
[0254] If so, execute step S707; if not, execute step S708.
[0255] In step S707, the optimal reference frame for the GLOBALMV mode is added to the initialization template.
[0256] In step S708, the process ends.
[0257] Up to this point, the process of step S50312 is completed, and the process proceeds to step S50313.
[0258] In step S50313, correction and complementation are performed.
[0259] After steps S50311 and S50312, mask_init may be 0 (the initial template is idle), in which case values corresponding to LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME (at least one pre-configured reference frame type) are written to mask_init (for example, the value of LAST_FRAME can be recorded in mask_init by shifting mask_init LAST_FRAME bits to the left), thereby allowing these frames to be added to the initialization template.
[0260] Note that mask_init being 0 may be because no reference frame information has been collected and the optimal mode has not been selected from other single reference frame modes. In this case, it must be forcibly added to LAST_FRAME, BWDREF_FRAME, and ALTREF_FRAME.
[0261] Up to this point, the process of generating the initialization template is complete and it is necessary to proceed to step S5032, ie, the process of generating the master template.
[0262] In step S5032, a master template is generated.
[0263] Let mask_main be a,mask,, which is initialized to 0 and generated according to the,SAD.
[0264] The seven reference frames (all candidate reference frame types) in Table 1 are divided into three types: forward reference frames, backward reference frames, and long-term reference frames, where the forward reference frames include LAST_FRAME, LAST2_FRAME, and LAST3_FRAME (these are candidate forward reference frame types), the backward reference frames include BWDREF_FRAME, ALTREF2_FRAME, and ALTREF_FRAME (these are candidate backward reference frame types), and the long-term reference frames include GOLDEN_FRAME (i.e., a candidate long-term reference frame type). Then, find the reference frame with the smallest SAD of each type.
[0265] The specific process is as follows: Step 1: Find the forward reference frame (forward reference frame type).
[0266] Compare the SADs corresponding to LAST_FRAME, LAST2_FRAME and LAST3_FRAME, find a reference frame with the minimum SAD and not INT32_MAX, and set it as ref_list0. Record the minimum forward SAD and set it as sad_list0.
[0267] Step 2: Find the backward reference frame (backward reference frame type).
[0268] Compare the SADs corresponding to BWDREF_FRAME, ALTREF2_FRAME and ALTREF_FRAME, find the reference frame with the minimum SAD and not INT32_MAX, and put it into ref_list1; record the back minimum SAD and put it into sad_list1.
[0269] Step 3: Find the long-term reference frame (long-term reference frame type).
[0270] If the SAD corresponding to the reference frame GOLDEN_FRAME is not INT32_MAX and is smaller than sad_list0+sad_list1, then the long-term reference frame is GOLDEN_FRAME.
[0271] In the above process, if there is no satisfactory MV in the current reference frame, the SAD corresponding to such current reference frame is INT32_MAX, which indicates that the reference frame is not important and can be directly skipped.
[0272] Up to this point, the process of generating the master template is complete, and it is necessary to proceed to step S5033, that is, the process of generating the extension template.
[0273] In step S5033, an expansion template is generated.
[0274] Let mask_add be initialized to 0 and relate to the frame type weight and the number of inter blocks collected.
[0275] First, generate a threshold value thr according to the current frame type weight, where the current frame type has already been determined before prediction, so the current frame type weight can also be determined, as slice_level, and generate a threshold value according to equation (3).
[0276] If inter_total_num<=thr, write the values of LAST_FRAME, BWDREF_FRAME, and ALTREF_FRAME into mask_add (completed by shifting mask_add left).
[0277] Up to this point, the generation of the reference frame template for the NEWMV mode has been completed, and it is necessary to proceed to the application process of the reference frame template.
[0278] In step S504, a reference frame template is applied.
[0279] All reference frames are cycled through, and the current reference frame is compared to whether it matches the reference frame template, for example, if the first and fourth bits of the reference frame template mask_newmv are all 1 and the current reference frame is 1 or 4, the current reference frame can be used for prediction, otherwise, the next reference frame is determined. Figure 16 is a schematic diagram of a process applied to a reference frame template in NEWMV mode according to an embodiment of the present application, and referring to Figure 16, the process may include the following steps.
[0280] The process starts in step S801.
[0281] In step S802, the number of the current reference frame is obtained.
[0282] In step S803, it is determined whether the current reference frame number is 7 or less.
[0283] If it is equal to or less than this, execute step S804; otherwise, execute step S807.
[0284] In step S804, it is determined whether the current reference frame matches the reference frame template.
[0285] If they match, step S805 is executed; if they do not match, step S806 is executed.
[0286] In step S805, the current reference frame is predicted.
[0287] In step S806, the number of the current reference frame is automatically incremented by 1, and the process returns to step S802.
[0288] In step S807, the process ends.
[0289] With the above method, the encoder does not introduce any new calculations, takes into full consideration the characteristics of the NEWMV mode, and directly uses the collected information to generate the reference frame template, which achieves a very high speed-up rate of 15% when encoding 65 video frames compared with the encoding speed in the related art. Moreover, the obtained reference frame template has high adaptability, and there is no need to delete reference frames during the generation process, so the quality of the bitstream is also guaranteed.
[0290] The following describes an exemplary structure of the inter prediction device 455 implemented as a software module according to an embodiment of the present application. In some embodiments, as shown in FIG. 6, the software module in the inter prediction device 455 stored in the memory 450 includes: A mode determination module 4551 configured to determine a history prediction mode corresponding to the current prediction block when a current prediction mode of the current prediction block is a preset prediction mode, the history prediction mode being a prediction mode that has completed prediction before the preset prediction mode; An information acquisition module 4552 configured to acquire neighboring block information of the neighboring block of the current prediction block, sub-prediction block information of the sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, where the sub-prediction block is obtained by partitioning the current prediction block using a sub-block partition type before the current sub-block partition type; a template generation module 4553 configured to generate a reference frame template based on the historical optimal reference frame type, the neighboring block information, the sub-prediction block information, and a frame type corresponding to the current prediction block; and an information prediction module 4554 configured to use the reference frame template to determine a reference frame for the preset prediction mode, and to use the reference frame to perform inter prediction on the current prediction block to obtain a prediction value corresponding to the current prediction block.
[0291] In some embodiments of the present application, the neighboring block information includes a motion vector of the neighboring block, a reference frame type of the neighboring block, and the number of the neighboring blocks, and the sub-prediction block information includes a reference frame type of the sub-prediction block, and the number of the sub-prediction blocks.
[0292] The template generation module 4553 is further configured to determine an initial template corresponding to the current prediction block based on the historical optimal reference frame type, the reference frame type of the sub-prediction block, and the reference frame type of the neighboring block; generate a master template corresponding to the current prediction block according to the motion vector of the neighboring block; determine an extended template corresponding to the current prediction block using the frame type of the current prediction block, the number of the neighboring blocks, and the number of the sub-prediction blocks; and generate the reference frame template corresponding to the current prediction block using the initial template, the master template, and the extended template.
[0293] In some embodiments of the present application, the template generation module 4553 is further configured to determine a first initial template according to a reference frame type of the sub-prediction block and a reference frame type of the adjacent block; use the historical optimal reference frame type as a second initial template; and use the first initial template and the second initial template to determine the initial template corresponding to the current prediction block.
[0294] In some embodiments of the present application, the template generation module 4553 is further configured to determine at least one historically selected reference frame type using the reference frame type of the adjacent block and the reference frame type of the sub-prediction block; to calculate the number of times each historically selected reference frame type has been selected in the at least one historically selected reference frame type; and to select a first initial template from each of the historically selected reference frame types using the number of times of selection.
[0295] In some embodiments of the present application, the template generation module 4553 is further configured to: select a maximum selected number from the selected number of times of each of the history selected reference frame types; expand the selected number of times of each of the history selected reference frame types to obtain an expanded selected number; compare the expanded selected number with the maximum selected number to obtain a comparison result corresponding to each of the history selected reference frame types; the comparison result indicates whether the expanded selected number is greater than or equal to the maximum selected number; and determine the history selected reference frame type for which the comparison result indicates that the expanded selected number is greater than or equal to the maximum selected number as the first initial template.
[0296] In some embodiments of the present application, the initial template is idle, and the template generation module 4553 is further configured to add at least one pre-configured reference frame type to the initial template to obtain a calibrated initial template; The template generation module 4553 is further configured to generate the reference frame template corresponding to the current prediction block using the calibrated initial template, the master template and the enhanced template.
[0297] In some embodiments of the present application, the neighboring block information includes a reference frame type of the neighboring block, and the sub-prediction block information includes a reference frame type of the sub-prediction block, and the template generation module 4553 is further configured to: determine an optimal prediction mode of the neighboring block to obtain a first judgment result; determine an optimal prediction mode of the sub-prediction block to obtain a second judgment result; the first judgment result indicates whether the optimal prediction mode of the neighboring block is a first predetermined mode, and the second judgment result indicates whether the optimal prediction mode of the sub-prediction block is a second predetermined mode; if the first judgment result indicates that the optimal prediction mode of the neighboring block is the first predetermined mode, obtain the reference frame type of the neighboring block; and if the second judgment result indicates that the optimal prediction mode of the sub-prediction block is the second predetermined mode, obtain the reference frame type of the sub-prediction block.
[0298] In some embodiments of the present application, the template generation module 4553 is further configured to calculate a selection parameter for each candidate reference frame type in the total amount of candidate reference frame types using the motion vector of the neighboring block; the total amount of candidate reference frame types represent all available candidate reference frame types during inter prediction, and the selection parameter represents the difference between the input value and the predicted value of the neighboring block; the total amount of candidate reference frame types are respectively divided into candidate forward reference frame types, candidate backward reference frame types and candidate long-term reference frame types; select a forward reference frame type, a backward reference frame type and a long-term reference frame type using the selection parameter corresponding to the candidate forward reference frame type, the selection parameter corresponding to the candidate backward reference frame type, and the selection parameter corresponding to the candidate long-term reference frame type; and integrate the master template corresponding to the current prediction block using the forward reference frame type, the backward reference frame type and the long-term reference frame type.
[0299] In some embodiments of the present application, the template generation module 4553 is further configured to use the candidate reference frame type having the smallest selection parameter among the candidate forward reference frame types as the forward reference frame type; use the candidate reference frame type having the smallest selection parameter among the candidate backward reference frame types as the backward reference frame type; and use the candidate reference frame type whose selection parameter is smaller than the sum of the selection parameter corresponding to the forward reference frame type and the selection parameter corresponding to the backward reference frame type as the long-term reference frame type.
[0300] In some embodiments of the present application, the template generation module 4553 is further configured to: match each candidate reference frame type in the total number of candidate reference frame types to the motion vector of the adjacent block to obtain a matching result; the matching result indicates whether a matching motion vector exists in each candidate reference frame type; if the matching result indicates that a matching motion vector does not exist in each candidate reference frame type, use a predetermined numerical value as a selection parameter for each candidate reference frame type; if the matching result indicates that a matching motion vector exists in each candidate reference frame type, calculate a selection parameter for each candidate reference frame type using a prediction value and an input value when the adjacent block predicts based on the motion vector of the adjacent block.
[0301] In some embodiments of the present application, the template generation module 4553 is further configured to calculate a difference between a pixel of a sub-prediction value corresponding to each sub-vector in the plurality of sub-vectors and a pixel of an input value of the adjacent block to obtain a pixel difference value corresponding to each sub-vector; accumulate an absolute value of the pixel difference value corresponding to each sub-vector to obtain a temporary selection parameter corresponding to each sub-vector; and use the smallest temporary selection parameter among the temporary selection parameters of each sub-vector as a selection parameter corresponding to each candidate reference frame type.
[0302] In some embodiments of the present application, the template generation module 4553 is further configured to: determine a frame type weight corresponding to the current prediction block according to a correspondence relationship between the frame type of the current prediction block and a predetermined frame type weight; generate an expansion threshold according to the frame type weight; add the number of the neighboring blocks and the number of the sub-prediction blocks to obtain a sum result; and use at least one predetermined reference frame type as an expansion template corresponding to the current prediction block if the sum result is less than or equal to the expansion threshold.
[0303] Thus, the description of the inter prediction device according to the embodiment of the present application is similar to the description of the inter prediction method according to the embodiment of the present application, and has similar beneficial effects.
[0304] The present embodiment provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium, the computer instructions being read by a processor of a computer device (electronic device for inter-prediction) from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the inter-prediction method described in the present embodiment.
[0305] An embodiment of the present application provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform an inter-prediction method according to an embodiment of the present application, for example, the method shown in FIG.
[0306] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, or may be any of a variety of devices including one or any combination of the above memories.
[0307] In some embodiments, the executable instructions take the form of a program, software, software module, script or code, may be written in any type of programming language (including compiled or parsed languages, or declarative or procedural languages), and may be deployed in any form, such as as a separate program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0308] As an example, the executable instructions may, but do not necessarily, correspond to a file in a file system and may be stored in a portion of a file that stores other programs or data, for example in one or more scripts within a HyperText Markup Language (HTML) document, in a single file for the program under discussion, or in multiple shared files (e.g., files that store one or more modules, subprograms, or code portions).
[0309] As an example, the executable instructions may be deployed to be executed on one computing device, or may be deployed to be executed on multiple computing devices that are co-located, or may be deployed to be executed on multiple computing devices that are distributed across multiple locations and interconnected by a communications network.
[0310] The above is merely an embodiment of the present application, and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are all included in the scope of protection of the present application.
Claims
1. 1. An inter prediction method implemented by an electronic device, comprising: determining a history prediction mode corresponding to the current prediction block when the current prediction mode of the current prediction block is a preset prediction mode, the preset prediction mode being a single reference frame mode of NEWMV or a combined reference frame mode including NEWMV, and the history prediction mode being a prediction mode that has completed prediction before the preset prediction mode; obtaining neighboring block information of neighboring blocks of the current prediction block, sub-prediction block information of a sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, the neighboring block information including a motion vector of the neighboring block, a reference frame type of the neighboring block, and the number of the neighboring blocks, the sub-prediction block information including a reference frame type of the sub-prediction block, and the number of the sub-prediction blocks, the sub-prediction block being obtained by partitioning the current prediction block using a sub-block partition type previous to a current sub-block partition type; determining a union of a reference frame type determined according to a selection frequency of the reference frame type of the neighboring block and the reference frame type of the sub-prediction block and a history optimal reference frame type as an initial template of the current prediction block; Calculating an error when predicting using each candidate reference frame type according to the motion vector of the neighboring block, classifying each candidate reference frame type into three types, i.e., forward reference frame, backward reference frame, and long-term reference frame, and determining a candidate reference frame type that gives the smallest error in each of the three types as a master template corresponding to the current prediction block; determining an extension threshold value according to a frame type of the current prediction block, and when the sum of the number of the neighboring blocks and the number of the sub-prediction blocks is equal to or less than the extension threshold value, determining a preset reference frame type as an extension template; generating a reference frame template corresponding to the current prediction block as a union of the initial template, the master template and the enhanced template; performing inter prediction on the current prediction block by using a reference frame corresponding to a reference frame type included in the reference frame template to obtain a prediction value corresponding to the current prediction block; The inter prediction method includes:
2. The step of determining a union of a reference frame type determined according to a selection frequency of the reference frame type of the adjacent block and the reference frame type of the sub-prediction block and a historical optimal reference frame type as an initial template corresponding to the current prediction block, determining a first initial template determined according to a selected number of reference frame types of the neighboring blocks and the reference frame type of the sub-prediction block; using the historically optimal reference frame type as a second initial template; determining the initial template corresponding to the current prediction block as a union of the first initial template and the second initial template; Including, The inter prediction method according to claim 1 .
3. The step of determining a first initial template determined according to a selected number of times of a reference frame type of the adjacent block and a reference frame type of the sub-prediction block, determining at least one historically selected reference frame type as a union of the reference frame types of the neighboring blocks and the reference frame types of the sub-prediction blocks; Counting the number of times each historically selected reference frame type has been selected within the at least one historically selected reference frame type; selecting each of the historically selected reference frame types having the greatest number of selected times as a first initial template; Including, The inter prediction method according to claim 2 .
4. The step of determining a first initial template determined according to a selected number of times of a reference frame type of the adjacent block and a reference frame type of the sub-prediction block, selecting a maximum number of times selected from the number of times selected for each historical selected reference frame type; Increasing the number of selections of each of the historical selected reference frame types to obtain an increased number of selections; comparing the expanded number of selections with the maximum number of selections to obtain a comparison result corresponding to each of the historical selected reference frame types, the comparison result indicating whether the expanded number of selections is equal to or greater than the maximum number of selections; determining a historically selected reference frame type, the comparison result of which indicates that the expanded selected number is equal to or greater than the maximum selected number, as the first initial template; Including, The inter prediction method according to claim 2 .
5. The initial template is empty, and after determining the initial template corresponding to the current prediction block as a union of the first initial template and the second initial template, the inter prediction method includes: adding at least one pre-defined reference frame type to the initial template to obtain a calibrated initial template; The step of generating the reference frame template corresponding to the current prediction block as a union of the initial template, the master template and the extended template includes: generating the reference frame template corresponding to the current prediction block as a union of the calibrated initial template, the master template and the enhanced template; The method of inter prediction according to claim 2 .
6. The step of obtaining neighboring block information of a neighboring block of the current prediction block and sub-prediction block information of a sub-prediction block includes: determining an optimal prediction mode of the adjacent block to obtain a first determination result, and determining an optimal prediction mode of the sub-prediction block to obtain a second determination result, the first determination result indicating whether the optimal prediction mode of the adjacent block is a first preset mode, the first preset mode being a NEARESTMV mode or a NEARMV mode, and the second determination result indicating whether the optimal prediction mode of the sub-prediction block is a second preset mode, the second preset mode being a NEARESTMV mode or a NEARMV mode; When the first determination result indicates that the optimal prediction mode of the neighboring block is the first preset mode, obtaining a reference frame type of the neighboring block; When the second determination result indicates that the optimal prediction mode of the sub-prediction block is the second preset mode, obtaining a reference frame type of the sub-prediction block; Including, The inter prediction method according to claim 1 .
7. calculating an error when predicting using each candidate reference frame type according to the motion vector of the neighboring block, classifying each candidate reference frame type into three types, i.e., forward reference frame, backward reference frame, and long-term reference frame, and generating a candidate reference frame type that gives the smallest error in each of the three types as a master template corresponding to the current prediction block; Calculating a selection parameter for each candidate reference frame type in a total number of candidate reference frame types using the motion vectors of the neighboring blocks, the total number of candidate reference frame types representing all available candidate reference frame types during inter prediction, and the selection parameter representing a difference between an input value and a predicted value of the neighboring blocks; the total number of candidate reference frame types are divided into candidate forward reference frame types, candidate backward reference frame types and candidate long-term reference frame types; using a selection parameter corresponding to the candidate forward reference frame type, a selection parameter corresponding to the candidate backward reference frame type, and a selection parameter corresponding to the candidate long-term reference frame type, the candidate reference frame type having the smallest selection parameter among the candidate forward reference frame types is a forward reference frame type, the candidate reference frame type having the smallest selection parameter among the candidate backward reference frame types is a backward reference frame type, and the candidate reference frame type having a selection parameter among the candidate long-term reference frame types that is smaller than the sum of the selection parameter corresponding to the forward reference frame type and the selection parameter corresponding to the backward reference frame type is selected as a long-term reference frame type; generating the master template corresponding to the current prediction block as a union of the forward reference frame type, the backward reference frame type and the long-term reference frame type; Including, The inter prediction method according to claim 1 .
8. Calculating a selection parameter for each candidate reference frame type in the total number of candidate reference frame types using the motion vectors of the neighboring blocks includes: Matching each of the candidate reference frame types in the total number of candidate reference frame types to the motion vector of the neighboring block to obtain a matching result, determining whether the reference frame type of the neighboring block is the same as each of the candidate reference frame types, and the matching result indicates whether there is a matching motion vector in each of the candidate reference frame types; if the matching result indicates that there is no matching motion vector in each of the candidate reference frame types, using a preset value as a selection parameter for each of the candidate reference frame types; If the matching result indicates that there is a matching motion vector for each of the candidate reference frame types, calculating a selection parameter for each of the candidate reference frame types using a prediction value and an input value when the neighboring block predicts based on the motion vector of the neighboring block; Including, The inter prediction method according to claim 7 .
9. The plurality of matching motion vectors constitute a plurality of sub-vectors, the prediction value includes a plurality of sub-prediction values, the plurality of sub-vectors and the plurality of sub-prediction values correspond to each other, and the step of calculating a selection parameter for each of the candidate reference frame types using a prediction value and an input value when the adjacent block predicts based on the motion vector of the adjacent block, comprises: calculating a difference between a pixel of a sub-predicted value corresponding to each sub-vector of the plurality of sub-vectors and a pixel of an input value of the neighboring block to obtain a pixel difference value corresponding to each sub-vector; accumulating absolute values of pixel difference values corresponding to each of the sub-vectors to obtain a temporary selection parameter corresponding to each of the sub-vectors; using the smallest one of the temporary selection parameters for each of the sub-vectors as a selection parameter corresponding to each of the candidate reference frame types; Including, The inter prediction method according to claim 8 .
10. An inter prediction device A mode decision module configured to determine a history prediction mode corresponding to a current prediction block when a current prediction mode of the current prediction block is a preset prediction mode, the preset prediction mode being a single reference frame mode of NEWMV or a combined reference frame mode including NEWMV, and the history prediction mode being a prediction mode that has completed prediction before the preset prediction mode; an information acquiring module configured to acquire neighboring block information of a neighboring block of the current prediction block, a sub-prediction block information of a sub-prediction block, and a historical optimal reference frame type of the current prediction block in the historical prediction mode, the neighboring block information including a motion vector of the neighboring block, a reference frame type of the neighboring block, and a number of the neighboring blocks; the sub-prediction block information including a reference frame type of the sub-prediction block, and a number of the sub-prediction blocks; and the sub-prediction block is obtained by partitioning the current prediction block using a sub-block partition type before a current sub-block partition type; a template generation module, determining a union of a reference frame type determined according to a selection frequency of the reference frame type of the neighboring block and the reference frame type of the sub-prediction block and a history optimal reference frame type as an initial template of the current prediction block; Calculating an error when predicting using each candidate reference frame type according to the motion vector of the neighboring block, classifying each candidate reference frame type into three types, i.e., forward reference frame, backward reference frame, and long-term reference frame, and determining a candidate reference frame type that gives the smallest error in each of the three types as a master template corresponding to the current prediction block; determining an extension threshold value according to a frame type of the current prediction block, and when the sum of the number of the neighboring blocks and the number of the sub-prediction blocks is equal to or less than the extension threshold value, determining a preset reference frame type as an extension template; generating a reference frame template corresponding to the current prediction block as a union of the initial template, the master template and the enhanced template; An inter prediction device comprising: an information prediction module configured to perform inter prediction on the current prediction block using a reference frame corresponding to a reference frame type included in the reference frame template to obtain a prediction value corresponding to the current prediction block.
11. An electronic device for inter prediction, comprising: A memory storing executable instructions; A processor which, when executing executable instructions stored in said memory, implements the inter prediction method of claim 1 ; An electronic device comprising:
12. A computer program comprising a computer program or instructions, which, when executed by a processor, implements the inter prediction method according to claim 1 .
Citation Information
Patent Citations
Motion picture coding method and motion picture coding apparatus
JP2003284091A
Apparatus and method for reducing the search for reference frames in video encoding
JP2010509850A
Encoding device, imaging device, encoding method, and program
JP2017098601A
