Inter-frame prediction methods and apparatus, storage media and electronic devices

Abstract

This disclosure provides an inter-frame prediction method and apparatus, a storage medium, and an electronic device. The inter-frame prediction method includes: acquiring a valid reference frame corresponding to the current decoded frame and assist frames for storing motion information; dividing each valid reference frame into several first-level blocks and calculating the address information of each first-level block; acquiring motion information based on each assist frame; and projecting each first-level block to the block position corresponding to the current decoded frame based on the address information and the motion information. This technical solution can reduce the total bandwidth usage and power consumption.

Description

Technical Field

[0001] This disclosure relates to the field of video encoding and decoding technology, and in particular to inter-frame prediction methods and apparatus, storage media and electronic devices. Background Technology

[0002] Video codec protocols stipulate that, in the process of inter-frame prediction, in addition to reading the reference frame data, it is also necessary to access the motion vector information of the assisting frame to parse the prediction block motion vector of the current decoding frame.

[0003] The existing hardware architecture does not perform compression processing when predicting and writing inter-frame prediction assistance frame data. The assistance frame data is directly stored in the DDR memory, which will consume additional DDR memory access bandwidth, resulting in a large amount of hardware computation and power consumption. Summary of the Invention

[0004] The purpose of this disclosure is to provide an inter-frame prediction method and apparatus, storage medium and electronic device for reducing total bandwidth usage and power consumption.

[0005] Firstly, the purpose of this disclosure is to provide an inter-frame prediction method. The method includes: acquiring a valid reference frame corresponding to the current decoded frame and an assist frame for storing motion information; dividing each of the valid reference frames into several first-level blocks and calculating the address information of each first-level block; acquiring motion information based on the assist frame; and projecting each first-level block to a block position corresponding to the current decoded frame based on the address information of each first-level block and the motion information.

[0006] In one implementation of the first aspect, obtaining a valid reference frame corresponding to the current decoded frame includes: determining whether the sequence number of the reference frame is before the sequence number of the current decoded frame; if so, determining whether the image size of the reference frame is consistent with that of the current decoded frame; if consistent, using the reference frame as a valid reference frame for the current decoded frame; if inconsistent, discarding the reference frame; otherwise, discarding the reference frame.

[0007] In one implementation of the first aspect, dividing each of the valid reference frames into several primary blocks includes: when there are at least two valid reference frames, sorting each of the valid reference frames in reverse chronological order; and further dividing each of the valid reference frames into several primary blocks of a set size.

[0008] In one implementation of the first aspect, projecting each first-level block to a block position corresponding to the current decoded frame based on the address information and motion information of each first-level block includes: calculating the motion vector coordinates of first-level blocks located at the same block position within each valid reference frame based on the motion information to obtain the projection position of each first-level block; writing the projection position of each first-level block and the projection flag of the projection position into a corresponding temporary storage unit; the temporary storage unit includes a first temporary storage unit, a second temporary storage unit, and a third temporary storage unit; when the loading and projection of each first-level block are completed, releasing the information in the first temporary storage unit, shifting the information in the second temporary storage unit to the first temporary storage unit, shifting the information in the third temporary storage unit to the second temporary storage unit, and clearing the third temporary storage unit to zero.

[0009] In one implementation of the first aspect, projecting each of the first-level blocks to the block position corresponding to the current decoded frame includes: dividing the first-level blocks into several second-level blocks; performing motion vector coordinate projection calculations on the second-level blocks in reverse Z-shaped order to obtain the projection position of each second-level block; sequentially judging the range validity and image boundary validity of the projection position of each second-level block; if the projection position of the second-level block satisfies both range validity and image boundary validity, then projecting the second-level block to the corresponding second-level block position of the current decoded frame, and storing the projection position and projection flag of the second-level block in the temporary storage unit.

[0010] In one implementation of the first aspect, projecting each of the first-level blocks to the block position corresponding to the current decoded frame further includes: if the projection position of the second-level block cannot simultaneously satisfy the range validity and image boundary validity, then the projection of the second-level block is considered invalid and the temporary storage unit is not updated; and the projection position and flag information of the temporary storage unit are cleared to invalid values ​​under initial conditions or before being reactivated.

[0011] In one implementation of the first aspect, determining the range validity of the projection position of the secondary block includes: determining whether the projection position of the secondary block is within the range of the primary block where the target coordinate is located, and the primary blocks to the left and right of the primary block where the target coordinate is located; if the projection position of the secondary block is within the range of the primary block where the target coordinate is located, and the primary blocks to the left and right of the primary block where the target coordinate is located, then the projection position of the secondary block is valid; otherwise, the projection position of the secondary block is invalid.

[0012] In one implementation of the first aspect, determining the image boundary validity of the projection position of the secondary block includes: determining whether the projection position of the secondary block is within the range of the image where the target coordinates are located; if the projection position of the secondary block is within the range of the image where the target coordinates are located, then the projection position of the secondary block is valid; otherwise, the projection position of the secondary block is invalid.

[0013] In one implementation of the first aspect, projecting each of the first-level blocks to the block position corresponding to the current decoded frame further includes: if the projection position of the second-level block currently being projected is the same as the projection position of the second-level block located in the same first-level block that has already been projected, then canceling the projection and information storage of the current second-level block.

[0014] Secondly, this disclosure provides an inter-frame prediction apparatus. The apparatus includes: an acquisition module configured to acquire a valid reference frame corresponding to a current decoded frame and an assist frame for storing motion information; a pre-read control module configured to divide each of the valid reference frames into several primary blocks and calculate address information for each primary block; a decoding module configured to acquire motion information based on the assist frames; and a projection module configured to project each primary block to a block position corresponding to the current decoded frame based on the address information of each primary block and the motion information.

[0015] Thirdly, this disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by an electronic device, it implements the inter-frame prediction method according to any one of the first aspects of this disclosure.

[0016] Fourthly, this disclosure provides an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to cause the electronic device to perform the inter-frame prediction method according to any one of the first aspects of this disclosure.

[0017] According to the inter-frame prediction method, apparatus, storage medium, and electronic device described in this disclosure, projection calculation and processing are performed based on the motion information of assisting frames from inter-frame prediction. Valid reference frames are determined at the frame level and divided into blocks. Simultaneously, projection processing is performed within each block in reverse order, reducing the number of post-projection processing calculations and memory read / write operations, thereby reducing overall bandwidth usage and power consumption. Furthermore, projection processing is performed based on the motion vector data of assisting frames from inter-frame prediction and the image display order. For CTU64 or CTU128 video sequences, a uniform 64×64 block projection rotation method is used, improving the reuse rate of hardware design resources. Attached Figure Description

[0018] Figure 1 The diagram shown is an application scenario illustration of an embodiment of this disclosure.

[0019] Figure 2 The flowchart shown is a process for an inter-frame prediction method provided in an embodiment of this disclosure.

[0020] Figure 3 This is a flowchart illustrating the process of obtaining a valid reference frame corresponding to the current decoded frame in an embodiment of this disclosure.

[0021] Figure 4 The diagram shown is a set of reference frames that can be used as the current decoding frame in an embodiment of this disclosure.

[0022] Figure 5 This is a flowchart illustrating the process of obtaining a valid reference frame corresponding to the current decoded frame, as shown in another embodiment of this disclosure.

[0023] Figure 6 The flowchart shown is an embodiment of this disclosure of how a first-level block is projected to the block position corresponding to the current decoded frame.

[0024] Figures 7a to 7d This diagram illustrates the processing order of projection operations on first-level blocks within a valid reference frame in the video sequences of CTU64 and CTU128 in this embodiment of the present disclosure.

[0025] Figure 8 This diagram illustrates the information translation during the projection of first-level blocks of each valid reference frame in an embodiment of this disclosure.

[0026] Figure 9 The flowchart shown is a projection flowchart of a first-level block in an embodiment of this disclosure.

[0027] Figure 10 This diagram illustrates the projection operation performed on each secondary block in reverse Z-shaped order according to an embodiment of this disclosure.

[0028] Figure 11 This diagram illustrates the range validity of the projection position of the secondary block and the validity of the image boundary in an embodiment of this disclosure.

[0029] Figure 12 The diagram shown is a schematic representation of projection within a first-level block in an embodiment of this disclosure.

[0030] Figure 13 The diagram shown is a structural schematic of the inter-frame prediction device in an embodiment of this disclosure. Detailed Implementation

[0031] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0032] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0033] Furthermore, the use of terms such as "first" and "second" in this disclosure is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this disclosure.

[0034] The existing hardware architecture does not perform compression processing when reading and writing inter-frame prediction assistance frame data. The assistance frame data is directly stored in DDR, which also consumes additional DDR access bandwidth. Furthermore, AV1 format assistance frame data requires the calculation of motion vector information of up to 3 reference frames from 5 reference frames as reference frames for the current decoding frame. Before use, a 64×64 projection operation is required, resulting in a large amount of hardware computation and power consumption.

[0035] Current market products support increasingly higher image resolutions and larger chip data throughput, leading to continuous increases in DDR access bandwidth. To meet market and consumer experience demands, more functionalities need to be supported while maintaining good battery life in electronic products. Given the ever-expanding scale of chip-level SoCs, SoC design that reduces DDR access efficiency, bus bandwidth, and power consumption is crucial. Video encoding and decoding based on inter-frame prediction assists in lossless compression of frame information, and the decompressed data is optimized through frame motion vector projection, which can reduce bus bandwidth and overall bus power consumption.

[0036] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram illustrating an application scenario of the inter-frame prediction method provided according to an embodiment of this disclosure. For example... Figure 1 As shown, the motion vector information lossless compression device 100 includes: a motion vector data decoding module 110, a motion vector data encoding module 120, an encoding / decoding control module 130, and an inter-frame prediction module 140.

[0038] The encoding / decoding control module 120 is connected to the inter-frame prediction module 140 and is used to receive motion vector data from the inter-frame prediction module 140 as auxiliary frame data for the subsequent frames of the current image encoding / decoding. After the inter-frame prediction module 140 finishes processing the current encoding / decoding frame, it will store the motion vector data in DDR.

[0039] The motion vector data encoding module 120 is connected to the encoding and decoding control module 130. It is used to receive motion vector data from the encoding and decoding control module 130, compress the motion vector data through a lossless compression algorithm, and store the compressed bit stream data in DDR.

[0040] When the inter-frame prediction module 140 issues a request for compressed bitstream data, the motion vector data decoding module 110 obtains the compressed bitstream data from the DDR and decompresses it to obtain the corresponding motion data, which is used by the inter-frame prediction module 140 when performing assisted frame projection.

[0041] Figure 2 The flowchart shown is an embodiment of an inter-frame prediction method provided in this disclosure, which is applicable to... Figure 1 The inter-frame prediction module in [the system]. For example... Figure 2 The inter-frame prediction method provided in this disclosure includes the following steps S21 to S24.

[0042] Step S21: Obtain the valid reference frame corresponding to the current decoding frame and the assist frame used to store motion information.

[0043] In some embodiments, the assist frame is defined relative to the reference frame. It is used to store motion information and may be referenced and used by other frames. The process of referencing and using it is called view prediction (MV). The specific content represented by the assist frame can be referred to as view motion information.

[0044] Step S22: Divide each valid reference frame into several first-level blocks and calculate the address information of each first-level block.

[0045] In some embodiments, the size of the first-level block can be 64×64.

[0046] Step S23: Obtain motion information based on the assist frame.

[0047] Step S24: Project each first-level block to the block position corresponding to the current decoding frame according to the address information and motion information of each first-level block.

[0048] It should be noted that the labels S21 to S24 are only used to identify different steps, and not to restrict the execution order of these steps.

[0049] In some implementations, before acquiring the valid reference frame corresponding to the current decoded frame and the assist frame used to store motion information, the read or write enable signal for the assist frame and the enable signal for the current decoded frame to use the assist frame are acquired. If the read or write of the assist frame is invalid and / or the current decoded frame does not allow the use of the assist frame, then no inter-frame prediction projection processing is performed on the current decoded frame. `enable_ref_frame_mvs` indicates whether the read or write of the assist frame is valid, and `use_ref_frame_mvs` indicates whether the current decoded frame allows the use of the assist frame.

[0050] Next, we will describe in detail how to obtain the valid reference frame corresponding to the current decoded frame.

[0051] During projection, not all frames are suitable as valid reference frames for the current decoding frame. Therefore, it is necessary to filter out the valid reference frames that are allowed as the current decoding frame. Table 1 below shows the reference frames for the current decoding frame.

[0052]

[0053] In Table 1, INTRA_FRAME indicates that the first position defaults to storing an INTRA frame, which is an I-frame and is not temporally projected. LAST_FRAME represents the nearest reference frame in the previous frame. LAST2_FRAME represents the second nearest reference frame in the previous frame. LAST3_FRAME represents the third nearest reference frame in the previous frame and is not temporally projected. GOLDEN_FRAME represents a relatively distant frame in the previous frame; Golden is generally an I-frame and is not temporally projected. BWDREF_FRAME represents the nearest frame in the future frame and is not temporally projected. ALTREF2_FRAME represents the second nearest reference frame in the future frame. ALTREF_FRAME represents the farthest reference frame in the future frame.

[0054] It should be noted that I-frames are not included in the filtering process when obtaining a valid reference frame corresponding to the current decoded frame.

[0055] For example, Figure 3 This is a flowchart illustrating the process of obtaining a valid reference frame corresponding to the current decoded frame in an embodiment of this disclosure. For example... Figure 3 As shown, obtaining a valid reference frame corresponding to the current decoded frame includes the following steps S31 to S32.

[0056] Step S31: Determine whether the sequence number of the reference frame is before the sequence number of the current decoded frame.

[0057] If the sequence number of the reference frame is before the sequence number of the current decoded frame, then continue to step S32; otherwise, discard the reference frame and proceed to the next reference frame.

[0058] Step S32: Determine whether the image size of the reference frame is consistent with that of the current decoded frame.

[0059] If the reference frame has the same image size as the current decoded frame, then the reference frame is used as the valid reference frame for the current decoded frame; otherwise, the reference frame is discarded, and step S31 is performed for the next reference frame.

[0060] In some implementations, Figure 4 Displayed as a set of reference frames that can be used as the current decoding frame. For example... Figure 4 As shown, the current decoding frame has 8 reference frames numbered 0 to 7. Among them, INTRA_FRAME, LAST3_FRAME, and GLODEN_FRAME are I-frames and do not participate in inter-frame prediction. The remaining five reference frames are... Figure 4 The arrows shown are arranged in order from ① to ⑤.

[0061] When selecting valid reference frames corresponding to the current decoded frame, according to Figure 4 The middle arrow indicates the order of execution from ① to ⑤. Figure 3 Steps S31 to S32 are performed to obtain a valid reference frame corresponding to the current decoded frame.

[0062] In some implementations, if no valid reference frame is found in the current decoded frame, the current decoded frame is not projected.

[0063] For example, Figure 5 This is a flowchart illustrating the acquisition of a valid reference frame corresponding to the current decoded frame, as shown in another embodiment of this disclosure. Figure 4 and Figure 5 As shown, obtaining a valid reference frame corresponding to the current decoded frame includes the following steps S51 to S57.

[0064] Step S51: Has the determination of the five reference frames been completed?

[0065] If yes, proceed to step S56; otherwise, proceed to step S52 for the next reference frame according to the sequence number.

[0066] Step S52: Is the current reference frame located before the current decoded frame sequence number?

[0067] If yes, proceed to step S53; otherwise, the current reference frame is invalid, the judgment of the current reference frame ends, and step S51 is performed.

[0068] Step S53: Is the current reference frame a full I-frame?

[0069] If yes, proceed to step S54; otherwise, the current reference frame is invalid, the judgment of the current reference frame ends, and step S51 is performed.

[0070] Step S54: Are the sizes of the current reference frame and the current decoded frame consistent?

[0071] If yes, proceed to step S55; otherwise, the current reference frame is invalid, the judgment of the current reference frame ends, and step S51 is performed.

[0072] Step S55: Are there already three valid reference frames?

[0073] If yes, output at most three valid reference frame flags and at most three reference frame indices; otherwise, jump to step S51.

[0074] Step S56: Does at least one frame satisfy the conditions in steps S52 to S54?

[0075] If yes, proceed to step S57; otherwise, if there is no valid reference frame matching the current decoded frame, no projection operation is performed on the current decoded frame.

[0076] Step S57: Output the valid flags of up to three reference frames and the indexes of up to three reference frames.

[0077] It should be noted that the labels S52 to S54 are only used to identify different steps, and not to restrict the execution order of these steps.

[0078] Next, we will provide a detailed explanation of how each valid reference frame is divided into several first-level blocks.

[0079] The protocol stipulates that each frame of the assistance frame shall perform projection operations in a 64×64 raster sequence.

[0080] In some implementations, the effective reference frame is divided into several first-level blocks of size 64×64 according to the 64×64 raster order of the assistance frames.

[0081] In some implementations, the number of valid reference frames corresponding to the current decoded frame is at most three. When there are at least two valid reference frames, each valid reference frame is processed according to... Figure 4The middle arrow indicates that the frames 1 to 5 are sorted in reverse order, and then each valid reference frame is divided into several first-level blocks of a set size.

[0082] Next, we will provide a detailed explanation of how to project each first-level block to the block position corresponding to the current decoded frame.

[0083] For example, Figure 6 This is a flowchart illustrating how, in this embodiment of the disclosure, each first-level block is projected to the block position corresponding to the current decoded frame based on its address information and motion information. For example... Figure 6 As shown, the process of projecting each first-level block to the block position corresponding to the current decoding frame according to the address information and motion information of each first-level block includes the following steps S61 to S63.

[0084] Step S61: Calculate the motion vector coordinates of the first-level blocks located at the same position within each valid reference frame based on the motion information, and obtain the projection position of each first-level block.

[0085] Step S62: Write the projection position and projection flag of each first-level block into the corresponding temporary storage unit. The temporary storage unit includes a first temporary storage unit, a second temporary storage unit, and a third temporary storage unit.

[0086] Step S63: When the loading and projection of each first-level block is completed, release the information of the first temporary storage unit, shift the information of the second temporary storage unit to the first temporary storage unit, shift the information of the third temporary storage unit to the second temporary storage unit, and clear the third temporary storage unit.

[0087] In some implementations, motion information includes: motion vector information, first-level block size, and first-level block target coordinates.

[0088] In some implementations, for a valid reference frame divided into several first-level blocks of a predetermined size, each first-level block within the valid reference frame is projected horizontally in a 64×64 raster order to the block position corresponding to the current decoded frame. When any first-level block has been projected to the block position corresponding to the current decoded frame, it means that at most three valid reference frames containing first-level blocks at the same position have been projected.

[0089] In some implementations, for video sequences with tile divisions, when projecting onto a valid reference frame corresponding to the current decoded frame, the 64×64 blocks to the left and right of the tile currently being projected need to be added to the projection operation. For CTU64 or CTU128 video sequences, when projecting onto a valid reference frame corresponding to the current decoded frame, CTU128 is decomposed into the same 64×64 blocks as CTU64 for projection. The difference between CTU128 and CTU64 lies in the different projection order of the 64×64 blocks and the need to adapt to the tile division method.

[0090] For example, Figures 7a to 7d This diagram illustrates the processing order of projection operations on the first-level blocks within the valid reference frame corresponding to the current decoded frame in video sequences of CTU64 and CTU128. Figures 7a to 7d As shown, one square represents a first-level block, that is, one square represents a 64×64 block.

[0091] like Figures 7a to 7b As shown, for a CTU64 video sequence, when projecting onto a valid reference frame corresponding to the current decoded frame, if a 64×64 block scanning sequence is encountered at an image boundary, the 64×64 blocks are projected sequentially in the horizontal direction. If a 64×64 block scanning sequence is encountered at a tile boundary, the 64×64 block to the left of the first 64×64 block in each row and the 64×64 block to the right of the last 64×64 block in each row need to be added to the projection, and these are also projected sequentially in the horizontal direction.

[0092] like Figures 7c to 7d As shown, for a CTU128 video sequence, it can be divided into groups of four 64×64 blocks for operation. When projecting onto a valid reference frame corresponding to the current decoded frame, when encountering a 64×64 block scanning sequence at an image boundary, the next group of 64×64 blocks is scanned only after completing the scan of one group. During the scan of each group of 64×64 blocks, the adjacent 64×64 blocks of the rightmost 64×64 block need to be added to the scan. When encountering a 64×64 block scanning sequence at a tile boundary, it is similarly divided into groups of four 64×64 blocks for operation. However, for the first group of 64×64 blocks within a tile, the 64×64 blocks to its left need to be added to the scan, and for the last group of 64×64 blocks within a tile, the 64×64 blocks to its right need to be added to the scan.

[0093] In some implementations, the projection position and projection flag of each first-level block are written into the corresponding temporary storage unit. The temporary storage unit includes a first temporary storage unit, a second temporary storage unit, and a third temporary storage unit. When the loading and projection of each first-level block are completed, the information in the first temporary storage unit is released, the information in the second temporary storage unit is shifted to the first temporary storage unit, the information in the third temporary storage unit is shifted to the second temporary storage unit, and the third temporary storage unit is cleared.

[0094] For example, Figure 8 This is a schematic diagram illustrating the information translation during the projection of first-level blocks of each valid reference frame. For example... Figure 8 As shown, the temporary storage units include blk0_proj_info to blk2_proj_info, where the first temporary storage unit is blk0_prj_info, the second temporary storage unit is blk1_prj_info, and the third temporary storage unit is blk2_prj_info.

[0095] The projection positions and projection flags of each acquired first-level block are written into temporary storage units blk0~2_prj_info. The completion of projection and information loading for each first-level block indicates that the first-level blocks with the most three valid reference frames numbered 0 to 2 located at the same position have been loaded and projected. The information in the first temporary storage unit blk0_prj_info is released into the memory, and the information in the second temporary storage unit blk1_prj_info is moved and replaced in the first temporary storage unit blk0_prj_info. The information in the third temporary storage unit blk2_prj_info is moved into the second temporary storage unit blk1_prj_info, and the third temporary storage unit blk2_prj_info is initialized and cleared to zero in order to facilitate the projection and information loading of the next first-level block.

[0096] Next, the projection process within each first-level block will be described in detail.

[0097] For example, Figure 9 The flowchart is shown as a projection within a first-level block. For example... Figure 9 As shown, projecting each first-level block to the block position corresponding to the current decoded frame includes steps S91 to S95.

[0098] Step S91: Divide the first-level block into several second-level blocks.

[0099] Step S92: Perform motion vector coordinate projection calculations on each secondary block in reverse Z-shaped order to obtain the projection position of each secondary block.

[0100] Step S93: Sequentially determine the range validity and image boundary validity of the projection position of each secondary block.

[0101] Step S94: If the projection position of the secondary block satisfies both range validity and image boundary validity, then the secondary block is projected to the corresponding secondary block position of the current decoded frame, and the projection position and projection flag of the secondary block are stored in the temporary storage unit.

[0102] Step S95: If the projection position of the secondary block cannot simultaneously satisfy the range validity and image boundary validity, the projection of the secondary block is considered invalid, the temporary storage unit is not updated, and the projection operation of the secondary block is exited; and the projection position and flag information of the temporary storage unit are cleared to invalid values ​​under the initial conditions or before being reactivated.

[0103] In some implementations, a 64×64 primary block is divided into 64 8×8 secondary blocks.

[0104] Figure 10 This is a schematic diagram showing the projection operation performed on each secondary block in reverse Z-shaped order. (See diagram below.) Figure 10 As shown, a first-level block of size 64×64 contains 64 second-level blocks of size 8×8 numbered 0-63.

[0105] When performing projection operations on secondary blocks within the same first-level block, follow the following... Figure 10 The arrows shown are scanned sequentially.

[0106] In some implementations, determining the range validity of the projection position of the second-level block includes: determining whether the projection position of the second-level block is within the range of the first-level block containing the target coordinates, as well as the first-level blocks to its left and right. If the projection position of the second-level block is within the range of the first-level block containing the target coordinates, as well as the first-level blocks to its left and right, then the projection position of the second-level block is valid; if it is outside the range, then the projection position of the second-level block is invalid.

[0107] In some implementations, determining the image boundary validity of the projection position of the secondary block includes: determining whether the projection position of the secondary block is within the range of the image containing the target coordinates. If the projection position of the secondary block is within the range of the image containing the target coordinates, then the projection position of the secondary block is valid; if it is outside the range, then the projection position of the secondary block is invalid.

[0108] For example, Figure 11 This diagram illustrates the range validity of the projection position of the secondary block and the validity of the image boundary.

[0109] like Figure 11As shown, the effective range of each secondary block projection position requires that the target coordinates in both the horizontal and vertical directions fall within the range of the current left, center, and right primary blocks. Positions outside this range are invalid. Each secondary block projection position is located on the left or right boundary of the image. If the projection position exceeds the image range, even if it falls within the range of the three primary blocks, it is considered invalid.

[0110] In some implementations, projecting each first-level block to the block position corresponding to the current decoded frame also includes: if the projection position of the second-level block being projected is the same as the projection position of a second-level block located in the same first-level block that has been projected, then cancel the projection and information storage of the second-level block being projected.

[0111] For example, Figure 12 This is a diagram illustrating the projection operation within a first-level block.

[0112] Using a maximum of three valid reference frames 0-2 arranged in reverse order, when projecting a first-level block located at the same position, the second-level blocks within that block are projected in reverse order from bottom right to top left. The projection position and projection flag of each second-level block are recorded in temporary storage units blk0_proj_info to blk2_proj_info. When all second-level blocks within a first-level block located at the same position using a maximum of three valid reference frames have been loaded and projected, the information in the first temporary storage unit blk0_prj_info is released to memory, and the information in the second temporary storage unit blk1_prj_info is moved and replaced in the first temporary storage unit blk0_prj_info. The information in the third temporary storage unit blk2_prj_info is moved to the second temporary storage unit blk1_prj_info, and the third temporary storage unit blk2_prj_info is initialized and cleared.

[0113] Each temporary storage unit (blk0_proj_info to blk2_proj_info) corresponds to a tag information for each second-level block. Each tag information consists of 5 bits: the high 2 bits represent the frame number of the valid reference frames 0-2 containing the first-level block, and the low 3 bits represent the row number of the second-level block within that first-level block (0-7). For reference frames numbered 0-2 that have been reversed, the high 2 bits represent the projection flag of the second-level block, where 0 indicates that the projection source first-level block is invalid, 1 indicates that the valid reference frame sequence number of the projection source first-level block is 0, 2 indicates that the valid reference frame sequence number of the projection source first-level block is 1, and 3 indicates that the valid reference frame sequence number of the projection source first-level block is 2.

[0114] In some implementations, after all second-level blocks within a first-level block have been projected, the projection positions and projection flags of all second-level blocks within that first-level block, which are stored in a temporary storage unit, are output and stored in SRAM.

[0115] It should be noted that the inter-frame prediction method disclosed herein is applicable to inter-frame prediction of AV1 format video sequences.

[0116] It should be noted that the scope of protection of the inter-frame prediction method disclosed herein is not limited to the execution order of the steps listed in this embodiment. Any scheme implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the scope of protection of this disclosure.

[0117] This disclosure also provides an inter-frame prediction apparatus. Figure 13 The diagram shows the structure of an inter-frame prediction device. (Example:) Figure 13 As shown, the inter-frame prediction device 1300 includes an acquisition module 1310, a pre-read control module 1320, a decoding module 1330, and a projection module 1340. The acquisition module 1310 is configured to acquire a valid reference frame corresponding to the current decoded frame and an assist frame for storing motion information. The pre-read control module 1320 is configured to divide each valid reference frame into several first-level blocks and calculate the address information of each first-level block. The decoding module 1330 is configured to acquire motion information based on the assist frames. The projection module 1340 is configured to project each first-level block to the block position corresponding to the current decoded frame based on the address information and motion information of each first-level block.

[0118] In some implementations, the inter-frame prediction device 1300 may further include a first buffer unit 1350, a second buffer unit 1360, and an external storage unit 1370. The first buffer unit 1350 is configured to store motion information for use by the projection module 1340. The second buffer unit 1360 is configured to save the projection position and validity of a valid reference frame that has been projected, to provide compressed bitstream information as the next decoded frame. The external storage unit 1370 is configured to receive requests from the acquisition module 1310 and provide assistance frames.

[0119] Each module in the inter-frame prediction apparatus provided in this embodiment can be further configured to perform steps corresponding to the steps of the inter-frame prediction method described above, which will not be repeated here.

[0120] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0121] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0122] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0123] This disclosure also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the compression and / or decompression methods provided in this disclosure. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The above storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0124] This disclosure also provides an electronic device including a memory and a processor. The memory is used to store a computer program. In some implementations, the memory may include a computer system readable medium in the form of volatile memory, such as RAM and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0125] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the inter-frame prediction method provided in the embodiments of this disclosure.

[0126] In some implementations, the processor can be a general-purpose processor, including a central processing unit (CPU) or a network processor (NP). In other implementations, the processor can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0127] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0128] In summary, this disclosure performs projection processing based on inter-frame predicted assist frame motion information and image display order, identifies up to three valid reference frames for projection at the frame level, and employs an 8×8 raster reversal method within a 64×64 block for projection. This reduces the number of post-projection processing calculations and memory read / write operations, thereby reducing overall bandwidth usage and power consumption. This disclosure uses inter-frame predicted assist frame motion vector data and image display order for projection processing, with CTU64 or CTU128 uniformly processing in a 64×64 block projection rotation manner, improving the reuse rate of hardware design resources. This disclosure uses inter-frame predicted assist frame motion vector data and image display order for projection processing, with a 64×64 block motion vector projection device adapted to the decoding CTU block order, comprehensively handling image or tile size and boundaries, and supporting projection scenarios for all AV1 assist frames. Therefore, this disclosure effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0129] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. An inter-frame prediction method, characterized in that, include: Obtain the valid reference frame corresponding to the current decoded frame and the assist frame used to store motion information; Each of the valid reference frames is divided into several first-level blocks, and the address information of each first-level block is calculated; Motion information is obtained based on the assist frame; and Based on the address information and motion information of each first-level block, each first-level block is projected to the block position corresponding to the current decoded frame. The process of projecting each first-level block to the block position corresponding to the current decoded frame based on the address information and motion information of each first-level block includes: calculating the motion vector coordinates of first-level blocks located at the same block position within each valid reference frame based on the motion information to obtain the projection position of each first-level block; writing the projection position of each first-level block and the projection flag of the projection position into a corresponding temporary storage unit, wherein the temporary storage unit includes a first temporary storage unit, a second temporary storage unit, and a third temporary storage unit; when the loading and projection of each first-level block are completed, releasing the information in the first temporary storage unit, shifting the information in the second temporary storage unit to the first temporary storage unit, shifting the information in the third temporary storage unit to the second temporary storage unit, and clearing the third temporary storage unit to zero. Projecting each of the first-level blocks to the block position corresponding to the current decoded frame includes: dividing the first-level block into several second-level blocks; and projecting the second-level blocks to the corresponding second-level block position of the current decoded frame, and storing the projection position and projection flag of the second-level blocks in the temporary storage unit.

2. The inter-frame prediction method according to claim 1, characterized in that, Obtaining the valid reference frame corresponding to the current decoded frame includes: Determine whether the sequence number of the reference frame is before the sequence number of the current decoded frame; If so, determine whether the image size of the reference frame is consistent with that of the current decoded frame. If they are consistent, use the reference frame as the valid reference frame of the current decoded frame. If they are inconsistent, discard the reference frame. Otherwise, discard the reference frame.

3. The inter-frame prediction method according to claim 1, characterized in that, Each of the aforementioned valid reference frames is divided into several first-level blocks, including: When there are at least two valid reference frames, the valid reference frames are sorted in reverse temporal order; and Each of the valid reference frames is then divided into several first-level blocks of a set size.

4. The inter-frame prediction method according to claim 1, characterized in that, Projecting the secondary block to the corresponding secondary block position of the current decoded frame, and storing the projection position and projection flag of the secondary block in the temporary storage unit includes: The motion vector coordinate projection calculations are performed on the secondary blocks in reverse Z-shaped order to obtain the projection position of each secondary block; The projection positions of each of the secondary blocks are sequentially evaluated for range validity and image boundary validity. If the projection position of the secondary block satisfies both range validity and image boundary validity, then the secondary block is projected to the corresponding secondary block position of the current decoded frame, and the projection position and projection flag of the secondary block are stored in the temporary storage unit.

5. The inter-frame prediction method according to claim 4, characterized in that, Projecting each of the first-level blocks to the block position corresponding to the current decoded frame further includes: If the projection position of the secondary block cannot simultaneously satisfy both range validity and image boundary validity, then the projection of the secondary block is considered invalid, and the temporary storage unit is not updated; and The projection position and flag information of the temporary storage unit are cleared to invalid values ​​under initial conditions or before being reactivated.

6. The inter-frame prediction method according to claim 4, characterized in that, Determining the range validity of the projection position of the secondary block includes: Determine whether the projection position of the secondary block is within the range of the primary block where the target coordinates are located, as well as the primary blocks to the left and right of the primary block where the target coordinates are located; If the projection position of the secondary block is within the range of the primary block where the target coordinate is located, as well as the primary blocks to the left and right of the primary block where the target coordinate is located, then the projection position of the secondary block is valid; otherwise, the projection position of the secondary block is invalid.

7. The inter-frame prediction method according to claim 4, characterized in that, Determining the validity of image boundaries for the projection positions of the secondary blocks includes: Determine whether the projection position of the secondary block is within the range of the image where the target coordinates are located; If the projection position of the secondary block is within the range of the image where the target coordinates are located, then the projection position of the secondary block is valid; otherwise, the projection position of the secondary block is invalid.

8. The inter-frame prediction method according to claim 4, characterized in that, Projecting each of the first-level blocks to the block position corresponding to the current decoded frame further includes: If the projection position of the secondary block currently being projected is the same as the projection position of a secondary block located in the same primary block that has already been projected, then the projection and information storage of the secondary block currently being projected are cancelled.

9. An inter-frame prediction device, characterized in that, include: The acquisition module is configured to acquire a valid reference frame corresponding to the current decoded frame and an assist frame for storing motion information. The pre-read control module is configured to divide each of the valid reference frames into several first-level blocks and calculate the address information of each first-level block; The decoding module is configured to acquire motion information based on the assist frame; and The projection module is configured to project each of the first-level blocks to the block position corresponding to the current decoded frame based on the address information of each first-level block and the motion information. The projection module is configured to: calculate the motion vector coordinates of first-level blocks located at the same position within each valid reference frame based on the motion information, and obtain the projection position of each first-level block; write the projection position of each first-level block and the projection flag of the projection position into a corresponding temporary storage unit, wherein the temporary storage unit includes a first temporary storage unit, a second temporary storage unit, and a third temporary storage unit; when the loading and projection of each first-level block are completed, release the information in the first temporary storage unit, shift the information in the second temporary storage unit to the first temporary storage unit, shift the information in the third temporary storage unit to the second temporary storage unit, and clear the third temporary storage unit. The projection module is configured to: divide the primary block into several secondary blocks; project the secondary blocks to the corresponding secondary block position of the current decoded frame, and store the projection position and projection flag of the secondary blocks in the temporary storage unit.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the inter-frame prediction method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the inter-frame prediction method according to any one of claims 1 to 8.

Images