Intra-frame prediction method, device, computer equipment and readable medium

Abstract

The present disclosure provides an intra-frame prediction method, applied to a Transformer network, comprising: dividing an image to be predicted into a preset number of image blocks and generating an image block sequence including the image blocks; performing dimensional processing on the image block sequence to obtain an image block embedding output sequence; encoding the image to be predicted based on the image block embedding output sequence and first position information of the image blocks to obtain an image block encoding output sequence, the image block encoding output sequence including first intra-frame global information; decoding the image block encoding output sequence based on second position information of the image blocks and a predicted image block prediction sequence to obtain a current image block prediction sequence, the current image block prediction sequence including second intra-frame global information; generating a predicted image based on the current image block prediction sequence; the present disclosure embodiment can more accurately obtain an intra-frame encoded predicted image. The present disclosure also provides an intra-frame prediction apparatus, a computer device, and a readable medium.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding technology, and in particular to an intra-frame prediction method, apparatus, computer equipment, and readable medium. Background Art

[0002] As user demand for high-definition video increases, the amount of video data used in multimedia is also increasing. Because videos contain a lot of redundant information, uncompressed videos are difficult to store and transmit. Therefore, coding technologies are needed to compress videos and reduce storage and transmission burdens. Intra-frame prediction is a crucial coding technique. Among all video frame types, I-frames (intra-coded frames) all use intra-frame prediction. The compression ratio of I-frames is typically lower than that of P-frames (predictive-coded frames) and B-frames (bidirectionally predictive-coded frames). Therefore, the efficiency of intra-frame prediction coding has a significant impact on the overall average bitrate of the video. Furthermore, I-frames are often used as reference frames in the decoding process of P-frames and B-frames. If an I-frame encoding error occurs, not only will the I-frame itself be erroneous, but the P-frames and B-frames that reference it will also be incorrectly decoded.

[0003] Taking H.265 as an example, the traditional intra-frame prediction method defines 35 prediction modes based on PU (Prediction Unit). PU can divide TU (Transform Unit) in the form of a quadtree, and all TUs in a PU share the same prediction mode. The H.265 intra-frame prediction process is as follows: determine whether the adjacent reference pixels of the current TU are available and perform corresponding processing, filter the reference pixels, and calculate the predicted pixel value of the current TU based on the filtered reference pixels. Traditional intra-frame prediction has multiple prediction modes, resulting in high computational overhead.

[0004] In addition to the traditional intra-frame prediction method, the current related technologies also use convolutional neural networks in deep learning for intra-frame prediction. However, convolution extracts features through local receptive fields, which to some extent ignores the correlation between coding blocks over longer distances. Summary of the Invention

[0005] The present disclosure provides an intra-frame prediction method, apparatus, computer device, and readable medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an intra-frame prediction method, which is applied to a Transformer network and includes:

[0007] Dividing the image to be predicted into a preset number of image blocks, and generating an image block sequence including the image blocks;

[0008] Performing dimension processing on the image block sequence to obtain an image block embedding output sequence;

[0009] encoding the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes first intra-frame global information;

[0010] decoding the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, wherein the current image block prediction sequence includes the second intra-frame global information;

[0011] A predicted image is generated according to the current image block prediction sequence.

[0012] In another aspect, an embodiment of the present disclosure further provides an intra-frame prediction apparatus, the apparatus being a Transformer network device, comprising a partitioning module, a dimensionality processing module, an encoding module, a decoding module, and a generation module, wherein the partitioning module is configured to partition an image to be predicted into a preset number of image blocks and generate an image block sequence including the image blocks;

[0013] The dimension processing module is used to perform dimension processing on the image block sequence to obtain an image block embedding output sequence;

[0014] The encoding module is configured to encode the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes the first intra-frame global information;

[0015] The decoding module is configured to decode the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, wherein the current image block prediction sequence includes the second intra-frame global information;

[0016] The generating module is used to generate a predicted image according to the current image block prediction sequence.

[0017] On the other hand, an embodiment of the present disclosure further provides a computer device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the intra-frame prediction method as described above.

[0018] On the other hand, an embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed, the intra-frame prediction method as described above is implemented.

[0019] The intra-frame prediction method provided by the embodiment of the present disclosure is applied to a Transformer network, including: dividing the image to be predicted into a preset number of image blocks, and generating an image block sequence including the image blocks; performing dimensionality processing on the image block sequence to obtain an image block embedding output sequence; encoding the image to be predicted based on the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, the image block encoding output sequence including first intra-frame global information; decoding the image block encoding output sequence based on the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, the current image block prediction sequence including second intra-frame global information; generating a predicted image based on the current image block prediction sequence; the embodiment of the present disclosure implements intra-frame prediction coding through a Transformer network, which not only utilizes local information within the image block, but also utilizes the self-attention mechanism layer in the Transformer to obtain intra-frame global information, effectively overcoming the limitations brought by convolutional induction bias, making information interaction more sufficient, and thus more accurately obtaining the intra-frame encoded predicted image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram comparing intra-frame prediction using a Transformer network in an embodiment of the present disclosure with traditional intra-frame prediction;

[0021] Figure 2 A schematic diagram of an intra-frame prediction process provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the encoding process provided in an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a decoding process provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a process for generating an image block sequence according to an embodiment of the present disclosure;

[0025] Figure 6 Schematic diagram of the process of generating a predicted image provided by the embodiment of the present disclosure Figure 1 ;

[0026] Figure 7 Schematic diagram of the process of generating a predicted image provided by the embodiment of the present disclosure Figure 2 ;

[0027] Figure 8 A schematic diagram of a process for determining encoding and decoding times provided in an embodiment of the present disclosure;

[0028] Figure 9A schematic diagram of the structure of the intra-frame prediction device provided in the embodiment of the present disclosure Figure 1 ;

[0029] Figure 10 A schematic diagram of the structure of the intra-frame prediction device provided in the embodiment of the present disclosure Figure 2 . DETAILED DESCRIPTION

[0030] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.

[0031] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0032] The terms used herein are used only to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof is not excluded.

[0033] The embodiments described herein may be described with reference to plan views and / or cross-sectional views, with the aid of idealized schematic diagrams of the present disclosure. Thus, the example illustrations may be modified based on manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings are schematic in nature, and the shapes of the regions shown in the drawings illustrate specific shapes of the regions of the elements, but are not intended to be limiting.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0035] When using convolutional neural networks in deep learning for intra-frame prediction, convolution extracts features through local receptive fields, which to some extent ignores the correlation between coding blocks over longer distances. The self-attention mechanism of the Transformer network, which has rapidly emerged in natural language processing tasks, can effectively overcome the limitations brought about by convolutional inductive bias and take more global information into account. Therefore, in order to obtain more accurate intra-frame prediction values, the embodiments of the present disclosure propose an intra-frame prediction method for video coding based on the Transformer network architecture.

[0036] like Figure 1 As shown, the embodiment of the present disclosure uses the Transformer network to perform intra-frame prediction coding, replacing the traditional intra-frame estimation and intra-frame prediction coding, and the output prediction image is used for subsequent quantization operations. Figure 1 and Figure 2 , the intra-frame prediction process of the embodiment of the present disclosure is described in detail.

[0037] The present disclosure provides an intra-frame prediction method, which is applied to a Transformer network and combined with Figure 1 and Figure 2 As shown, the method includes the following steps:

[0038] Step S11 : dividing the image to be predicted into a preset number of image blocks, and generating an image block sequence including the image blocks.

[0039] The width of the image to be predicted is W and the height is H. In this step, the Extracted Patches module divides the image to be predicted of size W*H into a preset number S of image patches of the same size P i , (i=1, 2, ..., S), S image blocks form an image block sequence P, P=[P1, P2, ..., P s ].

[0040] Step S12: performing dimension processing on the image block sequence to obtain an image block embedding output sequence.

[0041] In this step, the Embedding module embeds the image block sequence P = [P1, P2, ..., P s Each image block P in i The dimension d is obtained through the fully connected layer t The first one-dimensional characteristic sequence P x_i =[x1, x2, ..., x t ], the dimension of each image block is d t , the Embedding module outputs the image block embedded in the output sequence P x , Px =[P x_1 , P x_2 ,...,P x_s ].

[0042] Step S13 , encoding the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes the first intra-frame global information.

[0043] The Encoder module is composed of N identical encoder submodules stacked together, and encodes the predicted image N times. The number of encoding times N is pre-configured and N is an integer greater than 1. In this step, the Encoder module embeds the output sequence P according to the image block. x and the first position information of the image block (Positional Encoding 1), encode the predicted image to obtain the image block encoding output sequence P e , P e =[P e_1 , P e_2 ,...,P e_s ]. Each image block gets a dimension of d t The one-dimensional second characteristic sequence P e_i (i=1, 2, ..., S), the second feature sequence P of each image block e_i Composed image block encoding output sequence P e .

[0044] It should be noted that the image block encoding output sequence P e The first intra-frame global information is included, and the first intra-frame global information is generated during the encoding process of the image to be predicted.

[0045] Step S14 , decoding the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, where the current image block prediction sequence includes the second intra-frame global information.

[0046] The decoder module is composed of M identical encoder submodules stacked together to encode the image block and output the sequence P e Decoding is performed, and the number of decoding times M is pre-configured, where M is an integer greater than 1. In this step, the Decoder module predicts the sequence P according to the second position information (Positional Encoding 2) of the image block and the predicted image block. d ', P d '=[P d_1 , P d_2 ,...,P t-1_s], encode the image block and output the sequence P e Decode and get the current image block prediction sequence P d , P d =[P d_1 , P d_2 ,...,P d_s ]. Each image block gets a dimension of d t The one-dimensional third characteristic sequence P d_i (i=1, 2, ..., S), the third feature sequence P of each image block d_i The current image block prediction sequence P d .

[0047] It should be noted that the current image block prediction sequence P d Including the global information in the second frame, the global information in the second frame is encoded in the image block output sequence P e Generated during the decoding process.

[0048] Step S15: Generate a predicted image based on the current image block prediction sequence.

[0049] The Fusion module predicts the sequence P for the current image block. d The unified dimensions are transformed and then spliced ​​to obtain a predicted image with a width of W and a height of H.

[0050] The intra-frame prediction method provided by the embodiment of the present disclosure is applied to a Transformer network, including: dividing the image to be predicted into a preset number of image blocks, and generating an image block sequence including the image blocks; performing dimensionality processing on the image block sequence to obtain an image block embedding output sequence; encoding the image to be predicted based on the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, the image block encoding output sequence including first intra-frame global information; decoding the image block encoding output sequence based on the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, the current image block prediction sequence including second intra-frame global information; generating a predicted image based on the current image block prediction sequence; the embodiment of the present disclosure implements intra-frame prediction coding through a Transformer network, which not only utilizes local information within the image block, but also utilizes the self-attention mechanism layer in the Transformer to obtain intra-frame global information, effectively overcoming the limitations brought by convolutional induction bias, making information interaction more sufficient, and thus more accurately obtaining the intra-frame encoded predicted image.

[0051] The self-attention mechanism is the core of the Transformer network and is also an important means for the embodiment of the disclosure to obtain global information within the frame. Figure 3 and Figure 4 , the process of obtaining the global information within the first frame and the global information within the second frame is described.

[0052] In some embodiments, the global information in the first frame is embedded into the output sequence P according to the image block to which the first position information has been added. x Calculated. Figure 3 As shown, the encoder submodule first embeds the output sequence P according to the image block to which the first position information has been added. x Perform self-attention processing to obtain the global information in the first frame, and embed the output sequence P with the image block x After addition, normalization and positive feedback processing, the predicted image is coded and the image block coding output sequence P is obtained. e .

[0053] In some embodiments, the second intra-frame global information is encoded according to the image block output sequence P e and the predicted image block prediction sequence P to which the second position information has been added d 'Calculated. Figure 4 As shown, the decoder submodule first predicts the sequence P based on the predicted image blocks to which the second position information has been added. d 'Perform the first self-attention process, and then predict the sequence P of the image blocks obtained by the prediction d 'After addition, normalization and other processing, the output sequence P is encoded according to the image block e Perform the second self-attention process, and output the sequence P after encoding with the image block e After addition, normalization, positive feedback and other processing, the decoding process is completed and the current image block prediction sequence P is obtained. d .

[0054] The following takes the global information in the first frame as an example to illustrate the calculation process. The global information in the first frame can be calculated using the following formula (1):

[0055]

[0056] Among them, Attention is the global information in the first frame, d t is the first feature sequence P x_i , softmax() is the activation function, and the Q, K, and V matrices represent the first feature sequence P of each dimension with the first position information added. x_i The weight values ​​between them (i.e., dependency), Q, K, V matrices are embedded in the output sequence P according to the image block xAfter matrix transformation (multiplication) with the three preset matrices, the parameters of the three preset matrices can be obtained in a learnable way. It can be seen that the Q, K, V matrices are only embedded in the output sequence P with the image block. x It is related to the image block embedding output sequence P x of self-attention.

[0057] In some embodiments, as Figure 5 As shown, the step of dividing the image to be predicted into a preset number of image blocks and generating an image block sequence including the image blocks (i.e., step S11) includes the following steps:

[0058] Step S111 : dividing the image to be predicted into a preset number of image blocks of equal size.

[0059] Each, S = num*num, so The size of the image to be predicted is W*H. The image to be predicted with a size of W*H is divided into S image blocks. The size of each image block is That is, the width of the image block is W / num and the height is H / num.

[0060] Step S112 , sorting the image blocks in order from left to right and from top to bottom to generate an image block sequence.

[0061] Through steps S111-S112, the traditional coding method of calculating CTU (Coding Tree Units) block division is abandoned, and the direct equal block division method is adopted to improve the efficiency of image block division and solve the problem of high computational overhead of traditional intra-frame prediction.

[0062] In some embodiments, as Figure 6 As shown, the step of generating a predicted image according to the current image block prediction sequence (i.e., step S15) includes the following steps:

[0063] Step S151: linearize the current image block prediction sequence to obtain a first sequence, where the first sequence includes a one-dimensional array of each image block.

[0064] Step S152: convert the one-dimensional array of each image block into a two-dimensional matrix, and generate a second sequence according to the two-dimensional matrix.

[0065] Step S153 : According to the second sequence, the two-dimensional matrices are spliced ​​in order from left to right and from top to bottom to obtain a predicted image.

[0066] Combine Figure 6 and Figure 7As shown in Figure 1, the Fusion module includes three processing units: Linear, Reshape and Concat. Linear is used to predict the sequence P of the input current image block. d =[P d_1 , P d_2 ,...,P d_s ] linearization process, and obtain the first sequence P L , P L =[P L_1 , P L_2 ,...,P L_s ], P L_i The dimension of (i=1, 2, ..., S) is H*W / s. The first sequence P L Input Reshape, Reshape the one-dimensional array P L_i Convert to a two-dimensional matrix P R_i , the two-dimensional matrix P R_i The width is W / num, the height is H / num, and each two-dimensional matrix P R_i The second sequence P R =[P R_1 , P R_2 ,...,P R_s ]. The second sequence P R Input Concat, Concat will convert the second sequence P R The predicted image is spliced ​​in the order from left to right and from top to bottom. The width of the predicted image is W and the height is H. The final predicted image of the Transformer network is output and provided for subsequent quantization processing.

[0067] In some embodiments, when different images to be predicted are encoded a different number of times and / or decoded a different number of times, the step of determining N and M includes:

[0068] Step S21: Calculate the texture complexity of the image to be predicted.

[0069] In some embodiments, the texture complexity is the variance μ2 of the gray level histogram of the image, such as Figure 1 As shown, the texture estimation module calculates the texture complexity of the image to be predicted. The texture complexity can be calculated by the following formula (2):

[0070]

[0071] Among them, z represents the grayscale of the image to be predicted, p(z i ) is the corresponding histogram, L is the number of gray levels. m is the mean of z, which can be calculated by the following formula (3):

[0072]

[0073] It should be noted that texture complexity is not limited to the above calculation method, and can also be calculated using other methods, such as gradient-based calculation, deep learning-based method, etc.

[0074] Step S22 : determining N and M according to the texture complexity and a preset reference threshold, where N and M are one of the pre-configured thresholds.

[0075] In some embodiments, the reference threshold includes a first reference threshold and a second reference threshold, and determining the N and M according to the texture complexity and the preset reference threshold (i.e., step S22) includes the following steps: when the texture complexity is less than the first reference threshold, determining N as the preconfigured first encoding threshold N1, and determining M as the preconfigured first decoding threshold M1; when the texture complexity is greater than or equal to the first reference threshold and less than or equal to the second reference threshold, determining N as the preconfigured second encoding threshold N2, and determining M as the preconfigured second decoding threshold M2; when the texture complexity is greater than the second reference threshold, determining N as the preconfigured third encoding threshold N3, and determining M as the preconfigured third decoding threshold M3; wherein, N3>N2>N1, M3>M2>M1, N1, N2, N3 and M1, M2, M3 are set according to actual application.

[0076] like Figure 1 As shown, the threshold judgment module performs texture complexity judgment. If the texture complexity is less than the first reference threshold, it indicates that the image to be predicted has weak texture, and the number of encodings N and the number of decodings M are set to smaller values ​​(N1 and M1); if the texture complexity is greater than or equal to the first reference threshold and less than or equal to the second reference threshold, it indicates that the image to be predicted has medium texture, and the number of encodings N and the number of decodings M are set to relatively intermediate values ​​(N2 and M2); if the texture complexity is greater than the second reference threshold, it indicates that the image to be predicted has strong texture, and the number of encodings N and the number of decodings M are set to larger values ​​(N3 and M3).

[0077] Through the above steps S21-S22, dynamic adjustment of the number of encoding times N (i.e., the number of stacks of the encoder submodule) and the number of decoding times M (i.e., the number of stacks of the decoder submodule) can be achieved. In this case, the Transformer network includes a texture complexity estimation module and a threshold judgment module, and the Transformer network is a Dynamic Transformer network. It should be noted that the number of encoding times N (i.e., the number of stacks of the encoder submodule) and the number of decoding times M (i.e., the number of stacks of the decoder submodule) can also be configured as constants, without the need for dynamic adjustment according to the image to be predicted. In this case, the Transformer network does not include a texture complexity estimation module and a threshold judgment module.

[0078] In some embodiments, the first position information and the second position information may be calculated using the following formulas (4) and (5):

[0079]

[0080]

[0081] Among them, pos represents the number of the image block, PE(pos,2i) is the position of the even-numbered image block, PE(pos,2i+1) is the position of the odd-numbered image block, and i represents d t The identifier of the dimension.

[0082] It should be noted that the first position information and the second position information can also be obtained by deep learning.

[0083] In the embodiment of the present disclosure, the cross entropy can be used as the loss function for training the Transformer network, and other loss functions such as L1 loss function, L2 loss function, etc. can also be used.

[0084] The embodiments of the present disclosure are applied to the Transformer network and can be replaced with other Transformer network variants, such as swin-Transformer, Sparse Transformer, Image Transformer and other networks.

[0085] The disclosed embodiment provides an intra-frame prediction coding method based on a Dynamic Transformer network. It abandons the traditional coding method of calculating CTU block division and adopts the method of directly dividing the patch into equal blocks to improve the efficiency of block division. The intra-frame prediction coding is realized by the Transformer network, which utilizes the local information within the image block and obtains the global information within the frame through the self-attention mechanism layer in the Transformer network, so that the information interaction in the network is more sufficient, thereby obtaining the predicted image of the intra-frame coding more accurately and ensuring the coding quality. The Dynamic Transformer combines the evaluation of the texture complexity within the frame and uses the reference threshold for judgment. It can adaptively adjust the number of stacking of the encoding and decoding submodules, dynamically change the network depth, and reduce the network computing resources. Compared with CNN (Convolution neural network), the Transformer network can obtain global information within the frame with a shallower model depth, and the overall computing resources are less.

[0086] Based on the same technical concept, the embodiment of the present disclosure further provides an intra-frame prediction device, which is a Transformer network device, such as Figure 9 As shown, the intra-frame prediction device includes a division module 101, a dimension processing module 102, an encoding module 103, a decoding module 104 and a generation module 105. The division module 101 is used to divide the image to be predicted into a preset number of image blocks and generate an image block sequence including the image blocks.

[0087] The dimension processing module 102 is configured to perform dimension processing on the image block sequence to obtain an image block embedding output sequence.

[0088] The encoding module 103 is configured to encode the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes first intra-frame global information.

[0089] The decoding module 104 is configured to decode the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, where the current image block prediction sequence includes the second intra-frame global information.

[0090] The generating module 105 is configured to generate a predicted image according to the current image block prediction sequence.

[0091] In some embodiments, the first intra-frame global information is calculated based on an image block embedded in an output sequence to which the first position information has been added.

[0092] In some embodiments, the second intra-frame global information is calculated based on the image block encoding output sequence and a predicted image block prediction sequence to which the second position information has been added.

[0093] In some embodiments, the division module 101 is configured to divide the image to be predicted into a preset number of image blocks of equal size; and sort the image blocks in order from left to right and from top to bottom to generate the image block sequence.

[0094] In some embodiments, the generation module 105 is used to linearize the current image block prediction sequence to obtain a first sequence, where the first sequence includes a one-dimensional array of each image block; convert the one-dimensional array of each image block into a two-dimensional matrix, and generate a second sequence based on the two-dimensional matrix; and according to the second sequence, splice each two-dimensional matrix in a left-to-right and top-to-bottom order to obtain a predicted image.

[0095] In some embodiments, the encoding module 103 is configured to encode the image to be predicted N times; and the decoding module 104 is configured to decode the image block encoding output sequence M times. N and M are preconfigured, and are integers greater than 1.

[0096] In some embodiments, as Figure 10 As shown, the intra-frame prediction device also includes a coding and decoding times determination module 106, which is used to calculate the texture complexity of the image to be predicted; determine N and M based on the texture complexity and a preset reference threshold, and N and M are one of the pre-configured thresholds.

[0097] In some embodiments, the reference threshold includes a first reference threshold and a second reference threshold.

[0098] The encoding and decoding times determination module 106 is used to, when the texture complexity is less than the first reference threshold, determine N as the preconfigured first encoding threshold N1, and determine M as the preconfigured first decoding threshold M1; when the texture complexity is greater than or equal to the first reference threshold and less than or equal to the second reference threshold, determine N as the preconfigured second encoding threshold N2, and determine M as the preconfigured second decoding threshold M2; when the texture complexity is greater than the second reference threshold, determine N as the preconfigured third encoding threshold N3, and determine M as the preconfigured third decoding threshold M3; wherein, N3>N2>N1, M3>M2>M1.

[0099] An embodiment of the present disclosure also provides a computer device, which includes: one or more processors and a storage device; wherein one or more programs are stored on the storage device, and when the one or more programs are executed by the one or more processors, the one or more processors implement the intra-frame prediction method provided in the aforementioned embodiments.

[0100] An embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed, the intra-frame prediction method provided in the aforementioned embodiments is implemented.

[0101] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, and the functional modules / units in the device can be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0102] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. An intra-frame prediction method, the method being applied to a Transformer network, comprising: Dividing the image to be predicted into a preset number of image blocks, and generating an image block sequence including the image blocks; Performing dimension processing on the image block sequence to obtain an image block embedding output sequence; encoding the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes first intra-frame global information; decoding the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, wherein the current image block prediction sequence includes the second intra-frame global information; A predicted image is generated according to the current image block prediction sequence.

2. The method according to claim 1, wherein The first intra-frame global information is calculated based on an image block embedded in an output sequence to which the first position information has been added.

3. The method according to claim 1, wherein The second intra-frame global information is calculated based on the image block encoding output sequence and a predicted image block prediction sequence to which the second position information has been added.

4. The method according to claim 1, wherein The step of dividing the image to be predicted into a preset number of image blocks and generating an image block sequence including the image blocks comprises: Dividing the image to be predicted into a preset number of image blocks of equal size; The image blocks are sorted in order from left to right and from top to bottom to generate the image block sequence.

5. The method according to claim 1, wherein Generating a predicted image according to the current image block prediction sequence includes: performing linearization processing on the current image block prediction sequence to obtain a first sequence, where the first sequence includes a one-dimensional array of each of the image blocks; Converting the one-dimensional array of each image block into a two-dimensional matrix, and generating a second sequence according to the two-dimensional matrix; According to the second sequence, the two-dimensional matrices are spliced ​​in order from left to right and from top to bottom to obtain a predicted image.

6. The method according to any one of claims 1 to 5, wherein: The encoding of the image to be predicted includes: encoding the image to be predicted N times; The decoding of the image block coding output sequence includes: decoding the image block coding output sequence M times; The N and M are pre-configured, and N and M are integers greater than 1.

7. The method according to claim 6, wherein In the case where different to-be-predicted images are encoded and / or decoded at different times, N and M are determined in the following manner: Calculating the texture complexity of the image to be predicted; The N and M are determined according to the texture complexity and a preset reference threshold, where the N and M are one of the pre-configured thresholds.

8. The method according to claim 7, wherein The reference thresholds include a first reference threshold and a second reference threshold, and determining N and M according to the texture complexity and the preset reference thresholds includes: In a case where the texture complexity is less than the first reference threshold, determining N as a preconfigured first encoding threshold N1, and determining M as a preconfigured first decoding threshold M1; When the texture complexity is greater than or equal to the first reference threshold and less than or equal to the second reference threshold, determining N as a preconfigured second encoding threshold N2, and determining M as a preconfigured second decoding threshold M2; In a case where the texture complexity is greater than the second reference threshold, determining N as a preconfigured third encoding threshold N3, and determining M as a preconfigured third decoding threshold M3; Among them, N3>N2>N1, M3>M2>M1.

9. An intra-frame prediction apparatus, the apparatus being a Transformer network device, comprising a partitioning module, a dimensionality processing module, an encoding module, a decoding module, and a generation module, wherein the partitioning module is configured to partition an image to be predicted into a preset number of image blocks and generate an image block sequence including the image blocks; The dimension processing module is used to perform dimension processing on the image block sequence to obtain an image block embedding output sequence; The encoding module is configured to encode the image to be predicted according to the image block embedding output sequence and the first position information of the image block to obtain an image block encoding output sequence, wherein the image block encoding output sequence includes the first intra-frame global information; The decoding module is configured to decode the image block coding output sequence according to the second position information of the image block and the predicted image block prediction sequence to obtain a current image block prediction sequence, wherein the current image block prediction sequence includes the second intra-frame global information; The generating module is used to generate a predicted image according to the current image block prediction sequence.

10. A computer device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the intra-frame prediction method according to any one of claims 1 to 8.

11. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed, the intra-frame prediction method according to any one of claims 1 to 8 is implemented.

Images