Image coding method and apparatus

Abstract

The present disclosure provides an image coding method and device, relates to the technical field of computer image processing, and can solve the problem of excessive redundant data caused by the video coding technology in the prior art when encoding after layering. The specific technical scheme is as follows: first, an original frame to be encoded is obtained, and the original frame is divided into M macroblocks; then, the M macroblocks are encoded to obtain code stream data of each macroblock; the M macroblocks are divided into N layers, and code stream data of macroblocks in each layer is obtained, wherein 1≤N≤M; finally, at least one layer of macroblocks in the N layers is sent to a decoding end according to current bandwidth data. The present disclosure is used for image encoding and decoding.

Description

Technical Field

[0001] This disclosure relates to the field of computer image processing technology, and in particular to an image encoding and decoding method and apparatus. Background Technology

[0002] Since computer-generated images need to cover various scenarios such as office work and gaming, the bandwidth and frame rate of the actual environment also vary when encoding and decoding images. Especially when network bandwidth is limited, images are generally displayed in a progressive manner to improve the user experience.

[0003] Existing video encoding and decoding technologies are mostly based on the JPEG (Joint Photographic Experts Group) format. Its characteristic is that it performs DCT (Discrete Cosine Transform) transformation and quantization on a block-by-block basis, transmitting the results in layers. However, this technology limits the number of layers because a block has a fixed 64 coefficients. Furthermore, since the JPEG layering process occurs before the run-length encoding stage, run-length encoding is still required after layering. The more layers there are, the more markers are added to the end of each layer's data, potentially increasing the amount of redundant data. Summary of the Invention

[0004] This disclosure provides an image encoding / decoding method and apparatus, which can solve the problem of excessive redundant data caused by layered encoding in existing video encoding / decoding technologies. The technical solution is as follows:

[0005] According to a first aspect of the present disclosure, an image encoding method is provided, the method comprising:

[0006] Obtain the original frame to be encoded and divide the original frame into M macroblocks;

[0007] The M macroblocks are encoded to obtain the bitstream data of each macroblock;

[0008] The M macroblocks are divided into N layers, and the bitstream data of each macroblock is obtained, where 1≤N≤M;

[0009] Based on the current bandwidth data, send at least one macroblock from the N-layer macroblocks to the decoding end.

[0010] The image encoding method provided in this disclosure first acquires the original frame to be encoded and divides it into M macroblocks; then, it encodes the M macroblocks to obtain the bitstream data of each macroblock; it divides the M macroblocks into N layers and obtains the bitstream data of each layer of macroblocks, where 1≤N≤M; finally, based on the current bandwidth data, at least one layer of macroblocks from the N layers is sent to the decoding end. The image encoding method provided in this disclosure can set any number of layers, adapting to various extreme bandwidth conditions, and the layering after encoding does not introduce any redundant data.

[0011] In one embodiment, before encoding the M macroblocks to obtain the bitstream data of each macroblock, the method further includes:

[0012] The M macroblocks are divided into text and image categories. The text category is used to indicate macroblocks whose pixel value changes exceed a first preset threshold, and the image category is used to indicate macroblocks whose pixel value changes are less than a second preset threshold.

[0013] The encoding of the M macroblocks includes:

[0014] The macroblocks of the text class are encoded using a text block encoder;

[0015] The macroblocks of the image class are encoded using an image encoder.

[0016] The embodiments of this disclosure, through the above method, classify macroblocks into text and image types based on different macroblock pixel value changes. This facilitates the use of appropriate encoders for encoding different macroblock types, resulting in better encoding performance.

[0017] In one embodiment, the above method further includes:

[0018] The M macroblocks are further divided into other categories, and the macroblocks of the other categories are encoded using an extended encoder.

[0019] The embodiments of this disclosure, through the methods described above, can support the encoding of macroblocks by other encoders to be extended in the future.

[0020] In one embodiment, dividing the M macroblocks into N layers includes:

[0021] The first template block is divided into N*N / 2 macroblocks from the original frame;

[0022] If N is even, the i-th row on the left side of the first template block is divided into the odd-increasing layers from 1 to N-1, and the i-th row on the right side of the first template block is divided into the even-decreasing layers from N to 2, where i≤N / 2.

[0023] If N is odd, the left i-th row of the first template block is divided into odd-increasing layers from 1 to N, and the right i-th row of the first template block is divided into even-decreasing layers from N-1 to 2.

[0024] The left part of the (i+1)th row of the first template block is shifted to the right by L macroblocks, and the rightmost macroblock is looped to the leftmost position; the right part of the (i+1)th row of the first template block is shifted to the left by L macroblocks, and the leftmost macroblock is looped to the rightmost position.

[0025] Loop through the layers of each macroblock in the first template block.

[0026] The embodiments of this disclosure use the above method to discretely layer the macroblocks of the original frame. When transmitting progressively, the more discrete the positions between macroblocks are, the better the overall frame effect after filling, which is more conducive to improving the user experience.

[0027] In one embodiment, dividing the M macroblocks into N layers includes:

[0028] The original frame is divided into a second template block consisting of (2*N)*N macroblocks;

[0029] Divide the j-th row on the left side of the second template block into layers 1 to N, and divide the j-th row on the right side of the second template block into layers N to 1, where j≤N;

[0030] The left part of the (j+1)th row of the second template block is shifted to the right by K macroblocks, and the rightmost macroblock is looped to the leftmost position; the right part of the (j+1)th row of the second template block is shifted to the left by K macroblocks, and the leftmost macroblock is looped to the rightmost position.

[0031] Loop through the layers of each macroblock in the second template block.

[0032] The embodiments of this disclosure use the above method to discretely layer the macroblocks of the original frame. When transmitting progressively, the more discrete the positions between macroblocks are, the better the overall frame effect after filling, which is more conducive to improving the user experience.

[0033] In one embodiment, sending at least one macroblock from N macroblock layers to the decoding end based on current bandwidth data includes:

[0034] Get the current bandwidth data and frame rate;

[0035] Based on the current bandwidth data and frame rate, obtain the maximum bitrate that the current frame is allowed to send;

[0036] Based on the bitstream data of each macroblock in the N-layer macroblock, the macroblocks between layers A and B are sent to the decoding end, where 0 < A ≤ B ≤ N.

[0037] The embodiments of this disclosure use the above method to calculate the available size of the bitstream for each frame based on the current bandwidth data and frame rate, thereby determining the maximum size of the bitstream encoded for the current frame that can be sent.

[0038] According to a second aspect of the present disclosure, an image decoding method is provided, the method comprising:

[0039] Receive encoded data sent by the encoding end;

[0040] The encoded data is layered and decoded to obtain layered data;

[0041] Fill in the unreceived blank macroblocks with the layered data to obtain filling data;

[0042] The layered data and the padding data are concatenated to obtain frame data.

[0043] The image decoding method provided in this disclosure first receives encoded data sent by an encoding end; then, it performs layering and decoding on the encoded data to obtain layered data; it fills in unreceived blank macroblocks according to the layered data to obtain filling data; finally, it concatenates the layered data and the filling data to obtain frame data. The image encoding method provided in this disclosure can fill in macroblocks not received by the decoding end, resulting in good display effects and a good user experience.

[0044] In one embodiment, filling unreceived blank macroblocks based on the hierarchical data includes:

[0045] Obtain the horizontal pixel value of Xi in the blank macroblock based on the nearest decoded pixels to the left and right of the blank macroblock;

[0046] The vertical pixel value of Xi in the blank macroblock is obtained based on the nearest decoded pixels on the top and bottom sides of the blank macroblock;

[0047] The final pixel value of Xi is obtained based on the horizontal pixel value and the vertical pixel value of Xi;

[0048] The blank macroblock is filled according to the final pixel value of Xi.

[0049] In one embodiment, the horizontal pixel value of the blank macroblock Xi is obtained according to a first formula, the first formula including:

[0050] The horizontal pixel value of Xi = the pixel value of Li + Dist LX * (the pixel value of Ri - the pixel value of Li) / Dist LR;

[0051] Wherein, the Li pixel value is the value of the nearest decoded pixel to the left of the blank macroblock; the Ri pixel value is the value of the nearest decoded pixel to the right of the blank macroblock; Dist LX is the distance between Li and Xi in pixels; and Dist LR is the distance between Li and Ri in pixels.

[0052] The vertical pixel value of the blank macroblock Xi is obtained according to the second formula, which includes:

[0053] The vertical pixel value of Xi = the pixel value of Ui + Dist UX * (the pixel value of Di - the pixel value of Ui) / Dist UD;

[0054] Wherein, the Ui pixel value is the value of the nearest decoded pixel on the upper side of the blank macroblock; the Di pixel value is the value of the nearest decoded pixel on the lower side of the blank macroblock; the Dist UX is the distance between Ui and Xi in pixels; and the Dist UD is the distance between Ui and Di in pixels.

[0055] The final pixel value of the blank macroblock Xi is obtained according to a third formula, which includes:

[0056] The final pixel value of Xi = (the horizontal pixel value of Xi + the horizontal pixel value of Xi) / 2.

[0057] According to a third aspect of the present disclosure, an image encoding and decoding apparatus is provided, comprising: an encoding end and a decoding end;

[0058] The encoding end is used to execute the encoding method mentioned above;

[0059] The decoding end is used to execute the decoding method mentioned above.

[0060] According to a fourth aspect of the present disclosure, an image encoding / decoding apparatus is provided, the image encoding / decoding apparatus including a processor and a memory, the memory storing at least one computer instruction, the instruction being loaded and executed by the processor to perform the steps performed in any of the above-described image encoding / decoding methods.

[0061] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing at least one computer instruction, the instruction being loaded and executed by a processor to perform the steps performed in the image encoding / decoding method described in any of the preceding claims.

[0062] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0064] Figure 1 This is a flowchart of an image encoding method provided in an embodiment of this disclosure;

[0065] Figure 2 This is a template block for macroblock layering provided in this embodiment of the disclosure;

[0066] Figure 3 This is a schematic diagram of the original frame and template block division provided in the embodiments of this disclosure;

[0067] Figure 4 This is another macroblock layered template block provided in the embodiments of this disclosure;

[0068] Figure 5 This is a flowchart of an image decoding method provided in an embodiment of this disclosure;

[0069] Figure 6 This is a schematic diagram of the variables involved in the macroblock filling algorithm at the decoding end provided in this embodiment of the disclosure;

[0070] Figure 7 This is a schematic diagram of the structure of an image encoding and decoding device provided in an embodiment of this disclosure;

[0071] Figure 8 This is a schematic diagram of the structure of an image encoding and decoding device provided in an embodiment of this disclosure. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0073] This disclosure provides an image encoding method, such as... Figure 1 As shown, the image encoding method includes the following steps:

[0074] Step 101: Obtain the original frame to be encoded and divide the original frame into M macroblocks;

[0075] Specifically, the original frame image to be encoded can be acquired in YUV format. The entire frame pixels can be divided into macroblocks of 16*16 pixels or macroblocks of other pixels. This disclosure does not limit the scope of the data.

[0076] Step 102: Encode the M macroblocks to obtain the bitstream data of each macroblock;

[0077] In one embodiment, before encoding M macroblocks to obtain the bitstream data of each macroblock, the method further includes:

[0078] The M macroblocks are divided into text and image categories. The text category is used to indicate macroblocks whose pixel value changes exceed a first preset threshold, and the image category is used to indicate macroblocks whose pixel value changes are less than a second preset threshold.

[0079] Encoding M macroblocks includes:

[0080] Text-based macroblocks are encoded using a text block encoder;

[0081] Image macroblocks are encoded using an image encoder.

[0082] This embodiment of the disclosure, through the above method, categorizes macroblocks into text and image types based on different macroblock pixel value variations. This facilitates the use of appropriate encoders for different macroblock types, resulting in better encoding performance. Specifically, macroblocks with gradual pixel value changes are classified as JPEG type, while macroblocks with more dramatic gradient changes are classified as text type. This classification method allows for the use of JPEG encoding to obtain a smaller bitstream in smooth areas, while macroblocks with more detail are encoded into a bitstream with less distortion using a text block encoder. The text block encoder primarily uses a Huffman encoder for encoding, and a Huffman decoder for decoding.

[0083] In one embodiment, the above method further includes:

[0084] The M macroblocks are further divided into other classes, and the macroblocks in other classes are encoded using an extended encoder.

[0085] The embodiments of this disclosure, through the above-described method, are not limited to a single type of macroblock but are applicable to all types of macroblocks, and are also applicable to newly added macroblock types and encoding methods; they possess strong compatibility and scalability. For example, if the scenario requires lossless compression encoding, an encoder such as PNG can be used.

[0086] Step 103: Divide the M macroblocks into N layers and obtain the bitstream data of each macroblock layer, where 1≤N≤M;

[0087] In one embodiment, dividing M macroblocks into N layers includes:

[0088] The first template block is used to divide the original frame into N*N / 2 macroblocks;

[0089] If N is even, the left i-th row of the first template block is divided into odd increasing layers from 1 to N-1, and the right i-th row of the first template block is divided into even decreasing layers from N to 2, where i≤N / 2.

[0090] If N is odd, divide the i-th row on the left of the first template block into the odd-increasing layers from 1 to N, and divide the i-th row on the right of the first template block into the even-decreasing layers from N-1 to 2.

[0091] The left part of the (i+1)th row of the first template block is shifted to the right by L macroblocks, and the rightmost macroblock is looped to the leftmost position; the right part of the (i+1)th row of the first template block is shifted to the left by L macroblocks, and the leftmost macroblock is looped to the rightmost position.

[0092] Loop through the layers of each macroblock in the first template block.

[0093] like Figure 2 The image shows a layered template block, using 24 layers as an example. The method is similar for other layer counts. The image contains 24*12 small squares, each representing a macroblock. The labels indicate the layer number to which the macroblock is divided. The division method is as follows:

[0094] 1. If the total number of layers N is even, the template is designed to be N*N / 2 in size. The leftmost N / 2 macroblocks of the first row are assigned to layers 1, 3, 5, ..., N-1, i.e., layer numbers increase in odd order starting from 1; the rightmost N / 2 macroblocks are assigned to layers 24, 22, 20, ..., 2, i.e., starting from layer N, layer numbers decrease in even order. If N is odd, the left side increases in odd order from 1 to N, and the right side decreases in even order from N-1 to 2.

[0095] 2. The second line is the same as the first line, except that the left part starts from 1 and shifts to the right by one macroblock at a time until it ends at 21, and the original 21 to the right of 23 cycles to the leftmost position; the right part shifts to the left by one macroblock at a time, and the original 24 macroblock cycles to the rightmost position.

[0096] 3. Process the following rows in the same way until all rows return to the first row.

[0097] The 24*12 macroblock matrix formed by the above method is called a template block. Template blocks are concatenated together, and the final size matches the size of the entire frame image, as shown below. Figure 3As shown, this layered method can divide the entire frame's macroblocks into 24 relatively discrete layers. During progressive transmission, the more discrete the positions of the macroblocks, the better the user experience. Because the macroblock group received by the decoding end is incomplete, the missing macroblocks must first be filled in before displaying the blurred image. Without filling, areas with macroblocks will display real image blocks, while areas without macroblocks will display the default color, resulting in a blind-like effect that negatively impacts the user experience. Based on the filling approach, the more discrete the distribution of the received macroblocks, the better the overall frame effect after filling.

[0098] The embodiments of this disclosure use the above method to discretely layer the macroblocks of the original frame. When transmitting progressively, the more discrete the positions between macroblocks are, the better the overall frame effect after filling, which is more conducive to improving the user experience.

[0099] In one embodiment, dividing M macroblocks into N layers includes:

[0100] The second template block is used to divide the original frame into (2*N)*N macroblocks;

[0101] Divide the j-th row on the left of the second template block into layers 1 to N, and divide the j-th row on the right of the second template block into layers N to 1, where j≤N;

[0102] The left part of the (j+1)th row of the second template block is shifted to the right by K macroblocks, and the rightmost macroblock is looped to the leftmost position; the right part of the (j+1)th row of the second template block is shifted to the left by K macroblocks, and the leftmost macroblock is looped to the rightmost position.

[0103] Loop through the layers of each macroblock in the second template block.

[0104] like Figure 4 As shown, another layered template block is displayed. The image contains 48*24 small squares, each representing a macroblock. The template block is divided into two equal parts, left and right, with the layer numbers symmetrically distributed on both sides of the central axis. For example, the first row on the left is numbered 1, 2...24 from left to right, and the first row on the right is numbered 1, 2...24 from right to left; then, the second row on the left is numbered 24, 1, 2, 3...23 from left to right; the second row on the right is numbered 24, 1, 2, 3...23 from left to right; and so on, until each macroblock is numbered.

[0105] The embodiments of this disclosure use the above method to discretely layer the macroblocks of the original frame. When transmitting progressively, the more discrete the positions between macroblocks are, the better the overall frame effect after filling, which is more conducive to improving the user experience.

[0106] In practical use, taking a 1920*1080 resolution image as an example, the entire frame contains 120*68 = 8160 macroblocks, each with a size and position number (0-8159) that correspond one-to-one. Based on this data, the data size of each macroblock in N layers can be calculated. Theoretically, N can be any number between 1 and 8160. For ease of description, this disclosure can be divided into 8, 16, or 24 layers. The default is 24 layers. One requirement that must be guaranteed is that the bitstream size allowed to be sent in the current frame is greater than the size of the first layer after dividing into N layers. If this condition is not met, even the first layer cannot be transmitted, and the decoding end cannot decode. Based on this, depending on the computing power of the device, if the computing power is strong, such as a personal computing PC, a 24-layer mechanism can be used. If it is an embedded device with weak computing power, an 8-layer mechanism can be selected. That is, the more layers there are, the higher the computing power requirement.

[0107] Another condition is that when network fluctuations are detected, the maximum number of layers is set to 24. This is because network fluctuations result in significant differences in the amount of data that can be sent per frame. In order to ensure that several layers of data can still be sent even when network fluctuations are very small, if the number of layers is too small, the amount of data in each layer will be relatively large, and this guarantee cannot be achieved.

[0108] Another scenario is that a 24-layer mechanism can be set up. When the user is scrolling the mouse wheel or dragging the window quickly, the user is not very concerned about the specific content displayed on the desktop, but is more concerned about the smoothness of the operation. With more layers, there is more flexibility in choosing the number of layers for each frame, and the changes in each frame seen by the user on the decoding end are smoother.

[0109] Step 104: Based on the current bandwidth data, send at least one macroblock from the N-layer macroblocks to the decoding end.

[0110] In one embodiment, sending at least one macroblock from N macroblock layers to the decoding end based on current bandwidth data includes:

[0111] Get the current bandwidth data and frame rate;

[0112] Based on the current bandwidth data and frame rate, obtain the maximum bitrate that the current frame is allowed to send;

[0113] Based on the bitstream data of each macroblock in the N-layer macroblock, the macroblocks between layers A and B are sent to the decoding end, where 0 < A ≤ B ≤ N.

[0114] Specifically, the more layers there are, the greater the computational load during encoding, because the encoder needs to count the amount of data in the macroblocks from layer A to layer B and compare it with the bandwidth allocated to each frame. Under the premise of less than the available bandwidth, as many layers as possible are sent.

[0115] Based on the real-time monitored network bandwidth and the actual frame rate, the available bitstream size for each frame is calculated. The calculation method is to divide the network bandwidth by the frame rate (how many frames are encoded per second). This determines the maximum size of the bitstream encoded for the current frame that can be sent. The bitstream that needs to be transmitted for this frame is then sent to the decoding end through the network.

[0116] Progressive encoding and decoding aims to improve user experience. When the source data changes, if bandwidth is sufficient, all encoded data for each frame is transmitted, with A equal to 1 and B equal to N. The decoder decodes everything, and the user sees a complete and clear image on the screen after decoding – this is the ideal scenario. However, without a progressive scheme, when bandwidth is insufficient, frame drops occur when the network cannot transmit the encoded bitstream of certain frames, resulting in stuttering for the user. The significance of progressive encoding lies in the fact that when the encoder encodes data frame by frame, it can send out a portion of the encoded bitstream of the current frame based on the available network bandwidth. The decoder receives this and decodes it, displaying a potentially blurry image. If the source image is static, the encoder can gradually transmit the remaining data from the original frame at subsequent points in time, resulting in a gradually clearer image as seen on the decoder. If the source screen keeps changing, such as when a user is quickly dragging a window or a document, due to bandwidth limitations, only a very small portion of each frame may be transmitted. The user will see a blurry animated image until the screen stops and the image gradually becomes clear.

[0117] The image encoding method provided in this disclosure first acquires the original frame to be encoded and divides it into M macroblocks; then, it encodes the M macroblocks to obtain the bitstream data of each macroblock; it divides the M macroblocks into N layers and obtains the bitstream data of each layer of macroblocks, where 1≤N≤M; finally, based on the current bandwidth data, at least one layer of macroblocks from the N layers is sent to the decoding end. The image encoding method provided in this disclosure can set any number of layers, adapting to various extreme bandwidth conditions, and the layering after encoding does not introduce any redundant data.

[0118] Based on the above Figure 1 The corresponding embodiment provides an image encoding method, and another embodiment of this disclosure provides an image decoding method, such as... Figure 5 As shown.

[0119] Step 201: Receive encoded data sent by the encoding end;

[0120] Step 202: Layer and decode the encoded data to obtain layered data;

[0121] Specifically, text block decoders, image block decoders, and other block decoders are used to decode text, image, and other macroblocks to obtain decoded data, and the layering of each macroblock is managed.

[0122] Step 203: Fill the unreceived blank macroblocks with fill data based on the layered data;

[0123] In one embodiment, filling unreceived blank macroblocks with hierarchical data includes:

[0124] Obtain the horizontal pixel value of Xi in the blank macroblock based on the nearest decoded pixels to the left and right of the blank macroblock;

[0125] The vertical pixel value of Xi in the blank macroblock is obtained based on the nearest decoded pixels on the top and bottom sides of the blank macroblock;

[0126] The final pixel value of Xi is obtained based on the horizontal pixel value and the vertical pixel value of Xi.

[0127] The blank macroblock is filled based on the final pixel value of Xi.

[0128] In one embodiment, the lateral pixel value of the blank macroblock Xi is obtained according to a first formula, the first formula including:

[0129] The horizontal pixel value of Xi = the pixel value of Li + Dist LX * (the pixel value of Ri - the pixel value of Li) / Dist LR;

[0130] Wherein, Li pixel value is the value of the nearest decoded pixel to the left of the blank macroblock; Ri pixel value is the value of the nearest decoded pixel to the right of the blank macroblock; Dist LX is the distance between Li and Xi in pixels; Dist LR is the distance between Li and Ri in pixels.

[0131] The vertical pixel value of the blank macroblock Xi is obtained according to the second formula, which includes:

[0132] The vertical pixel value of Xi = the pixel value of Ui + Dist UX * (the pixel value of Di - the pixel value of Ui) / Dist UD;

[0133] Wherein, Ui pixel value is the value of the nearest decoded pixel above the blank macroblock; Di pixel value is the value of the nearest decoded pixel below the blank macroblock; Dist UX is the distance between Ui and Xi in pixels; Dist UD is the distance between Ui and Di in pixels.

[0134] The final pixel value of the blank macroblock Xi is obtained according to the third formula, which includes:

[0135] The final pixel value of Xi = (the horizontal pixel value of Xi + the horizontal pixel value of Xi) / 2.

[0136] like Figure 6 As shown, L1 to L16 represent a column of pixels, which is the rightmost column of pixels in a macroblock, and this macroblock has already been received by the decoder. R1 to R16 represent a column of pixels, which is the leftmost column of pixels in a macroblock, and this macroblock has also been received by the decoder. Pixels in the macroblock between columns L and R are not received by the decoder. Therefore, after decoding the macroblocks containing columns L and R, the decoder obtains the pixel values ​​of columns L and R. X1 to X16 represent a column of pixel values ​​that cannot be decoded; it is located in a column between columns L and R.

[0137] 1. Calculate the distance between column L and column X in pixels, denoted as Dist LX;

[0138] 2. Calculate the distance between column X and column R in pixels, denoted as Dist XR;

[0139] 3. Calculate the distance between column L and column R, denoted as Dist LR.

[0140] In column X, the calculation method for each pixel is the same, so Xi will be used as an example below, which corresponds to Li and Ri.

[0141] 4. Horizontal variation step value = (pixel value of Ri - pixel value of Li) / Dist LR

[0142] 5. The horizontal pixel value of Xi = the pixel value of Li + Dist LX * horizontal variation step value

[0143] The horizontal pixel value of Xi has been obtained above. The vertical pixel value of Xi also needs to be calculated, similar to the above method. This involves obtaining the pixel value of the nearest decoded pixel above the current pixel (Ui) and the pixel value of the nearest decoded pixel below the current pixel (Di), calculating the vertical gradient value, and then calculating the vertical pixel value of Xi using a similar method.

[0144] 6. Calculate the distance between column U and column X in pixels, denoted as Dist UX;

[0145] 7. Calculate the distance between column X and column D in pixels, denoted as Dist XD;

[0146] 8. Calculate the distance between column U and column D, denoted as Dist UD.

[0147] In column X, the calculation method for each pixel is the same, so Xi will be used as an example below, which corresponds to Ui and Di.

[0148] 9. Horizontal variation step value = (Pixel value of Di - Pixel value of Ui) / Dist UD

[0149] 10. The horizontal pixel value of Xi = the pixel value of Ui + Dist UX * horizontal variation step value.

[0150] The final pixel value of Xi = (the horizontal pixel value of Xi + the vertical pixel value of Xi) / 2.

[0151] In the decoding process, for each pixel in the macroblock of the layer that has not been received, the above filling algorithm is used to fill it. The filled frame is the frame in the progressive display process after decoding.

[0152] Step 204: Concatenate the layered data and the padding data to obtain frame data.

[0153] The image decoding method provided in this disclosure first receives encoded data sent by the encoding end; then, it performs layering and decoding on the encoded data to obtain layered data; it fills in unreceived blank macroblocks according to the layered data to obtain filling data; finally, it concatenates the layered data and the filling data to obtain frame data. The image encoding method provided in this disclosure can fill in macroblocks not received by the decoding end, resulting in good display effect and user experience.

[0154] Based on the above Figure 1 and Figure 5 The corresponding embodiment provides an image encoding and decoding method, and another embodiment of this disclosure provides an image encoding and decoding apparatus, such as... Figure 7 As shown, it includes: an encoding end 301 and a decoding end 302;

[0155] Encoding port 301 is used to execute the encoding method mentioned above;

[0156] Decoding end 302 is used to execute the decoding method mentioned above.

[0157] Specifically, such as Figure 8 As shown, the encoding end 301 includes an image acquisition module 101, a macroblock classification module 102, a text block encoder 103, an image encoder 104, an other block encoder 105, a hierarchical data manager 106, a network bandwidth monitoring module 107, a bitstream size prediction module 108, and a transmission and sending module 109.

[0158] Image acquisition module 101 acquires the original frame image to be encoded, usually in YUV format;

[0159] The macroblock classification module 102 divides the entire frame's pixels into 16x16 pixel macroblocks and classifies them to distinguish macroblocks suitable for different encoders. This scheme classifies macroblocks with gradual pixel value changes as JPEG type and macroblocks with significant gradient changes as text type. This classification method allows for smaller bitstreams with smooth pixel values ​​using JPEG encoding, while more detailed macroblocks are encoded into a less distorted bitstream using the text block encoder. The text block encoder primarily uses a Huffman encoder for encoding and a Huffman decoder for decoding.

[0160] After classifying macroblocks, this module assigns text macroblocks to text block encoder 103 for encoding and image blocks to image encoder 104 for encoding. Other block encoders 105 are reserved encoders in this disclosure, intended to demonstrate that this progressive scheme can support other encoders for future expansion. For example, if the scenario requires lossless compression encoding, encoders such as PNG can be used; these are represented by other block encoders 105.

[0161] The layered data manager 106 records the bitstream size of each macroblock after encoding. Taking a 1920*1080 resolution image as an example, the entire frame contains 120*68 = 8160 macroblocks, each with a size and position number (0-8159) corresponding one-to-one. Based on this data, module 106 can also calculate the data size of each macroblock when divided into N layers. Theoretically, N can be any number between 1 and 8160. For ease of description, this scheme supports three layering methods: 8, 16, and 24 layers. The default is 24 layers. The more layers, the greater the computational load during encoding, because the encoder needs to count the data size of macroblocks from layer A to layer B and compare it with the bandwidth allocated to each frame. Under the premise of less than the available bandwidth, as many layers as possible are sent.

[0162] The relationship between A / B / N is: 0 <A<B≤N。

[0163] The network bandwidth monitoring module 107 monitors the current network bandwidth in real time, in bits per second (bps).

[0164] The bitstream size prediction module 108 and the network bandwidth monitoring module 107 collect statistics on the current network bandwidth. These statistics, combined with the actual frame rate, are used to calculate the available bitstream size for each frame, thus determining the maximum size of the bitstream that can be transmitted for the current frame. Since both the frame rate and network bandwidth change in real time, this value cannot be perfectly accurate; therefore, it is called a predicted value. The bitstream size for each frame is calculated by dividing the network bandwidth by the frame rate (how many frames are encoded per second).

[0165] The transmission module 109 sends the bit stream that needs to be transmitted in this frame to the decoding end 302 via the network.

[0166] The layered data manager 106 divides all macroblocks of the entire frame into N layers, and the bitstream size of each layer can be obtained.

[0167] The layered data manager 106 focuses on two main aspects: how to layer the data and how many layers can be transmitted after the current frame is encoded. The following describes how the encoding end 301 determines how many layers the current frame is divided into.

[0168] The current frame's allowed bitstream size, denoted as FrameLenLimit, needs to be provided by the bitstream size prediction module 108. This scheme supports three specifications: 8, 16, and 24 layers. A crucial requirement is that FrameLenLimit > the size of the first layer after dividing into N layers. If this condition is not met, even the first layer cannot be transmitted, and the decoding end 302 will be unable to decode. Based on this, depending on the device's computing power, a 24-layer mechanism can be used if the computing power is strong, such as a PC. For embedded devices with weaker computing power, an 8-layer mechanism can be selected. In other words, the more layers, the higher the computing power requirement.

[0169] Another condition is that when the network bandwidth monitoring module 107 detects fluctuations in network conditions, it will feed back to the layered data manager 106 through the bitstream size prediction module 108. The layered data manager 106 then sets the maximum number of layers to 24. This is because network fluctuations result in a large difference in the amount of data that can be sent per frame. In order to ensure that several layers of data can still be sent even when network fluctuations are very small, if the number of layers is too small, the amount of data in each layer will be relatively large, and this guarantee cannot be achieved.

[0170] On the other hand, such as Figure 8 As shown, the decoding end 302 includes a transmission and receiving module 110, a layered data manager 111, a text block decoder 112, an image block decoder 113, other block decoders 114, a post-processing module 115, and a reconstructed frame 116.

[0171] The transmission receiving module 110 is used to receive encoded data sent by the encoding end 301;

[0172] The hierarchical data manager 111 is used for hierarchical management of encoded data;

[0173] The text block decoder 112 is used to decode text blocks in encoded data;

[0174] Image block decoder 113 is used to decode images in encoded data;

[0175] Other block decoder 114 is used to decode other blocks in the encoded data;

[0176] Post-processing module 115 is used to fill unreceived blank macroblocks with fill data based on the layered data.

[0177] Reconstruct frame 116 is used to concatenate the layered data and padding data to obtain frame data.

[0178] The image encoding and decoding method provided in this disclosure can perform progressive encoding, decoding, and transmission based on bandwidth. Using the method of this disclosure, layered encoding and decoding of encoded frames can be performed without adding redundant information even under bandwidth constraints. Furthermore, it has stronger compatibility, applicable to both existing macroblock types and any macroblock types that may be added in the future. It also improves the user experience at the decoding end.

[0179] Based on the above Figure 1 and Figure 5 In another embodiment of the present disclosure, an image encoding and decoding device is provided, comprising a processor and a memory. The memory stores at least one computer instruction, which is loaded and executed by the processor to implement the above-described image encoding and decoding method. Figure 1 and Figure 5 The image encoding / decoding method described in the corresponding embodiment.

[0180] Based on the above Figure 1 and Figure 5 In addition to the image encoding and decoding method described in the corresponding embodiments, this disclosure also provides a computer-readable storage medium. For example, a non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, or an optical data storage device. This storage medium stores at least one computer instruction for executing the above-described... Figure 1 and Figure 5 The image encoding and decoding methods described in the corresponding embodiments will not be repeated here.

[0181] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0182] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image encoding method, characterized in that, The method includes: Obtain the original frame to be encoded and divide the original frame into M macroblocks; The M macroblocks are encoded to obtain the bitstream data of each macroblock; The M macroblocks are divided into N layers, and the bitstream data of each macroblock is obtained, where 1≤N≤M; Based on the current bandwidth data, at least one macroblock from the N-layer macroblocks will be sent to the decoding end; The step of dividing the M macroblocks into N layers includes: The original frame is divided into a first template block of N*N / 2 macroblocks; where N is an even number. The first N / 2 macroblocks in the first row to the left of the first template block are divided into odd-increasing layers from 1 to N-1, and the first N / 2 macroblocks in the first row to the right of the first template block are divided into even-decreasing layers from N to 2, where i≤N / 2; The first template block's first row N / 2 macroblocks are shifted one macroblock to the right to obtain the N / 2th macroblock in the left part of the current row of the first template block, until the last row N / 2th macroblock is equivalent to the leftmost macroblock in the first row; the first template block's first row N / 2 macroblocks are shifted one macroblock to the left to obtain the N / 2th macroblock in the right part of the current row of the first template block, until the last row N / 2+1th macroblock is equivalent to the rightmost macroblock in the first row.

2. The image encoding method according to claim 1, characterized in that, Before encoding the M macroblocks to obtain the bitstream data of each macroblock, the method further includes: The M macroblocks are divided into text and image categories. The text category is used to indicate macroblocks whose pixel value changes exceed a first preset threshold, and the image category is used to indicate macroblocks whose pixel value changes are less than a second preset threshold. The encoding of the M macroblocks includes: The macroblocks of the text class are encoded using a text block encoder; The macroblocks of the image class are encoded using an image encoder.

3. The image encoding method according to claim 2, characterized in that, The method further includes: The M macroblocks are further divided into other categories, and the macroblocks of the other categories are encoded using an extended encoder.

4. The image encoding method according to claim 1, characterized in that, The step of sending at least one macroblock from the N-layer macroblocks to the decoding end based on the current bandwidth data includes: Get the current bandwidth data and frame rate; Based on the current bandwidth data and frame rate, obtain the maximum bitrate that the current frame is allowed to send; Based on the bitstream data of each macroblock in the N-layer macroblock, the macroblocks between layers A and B are sent to the decoding end, where 0 < A ≤ B ≤ N.

5. An image decoding method, characterized in that, The method includes: Receive encoded data sent by the encoding end; The encoded data is layered and decoded to obtain layered data; Fill in the unreceived blank macroblocks with the layered data to obtain filling data; The layered data and the padding data are concatenated to obtain frame data; The encoded data is encoded using the encoding method according to any one of claims 1-4.

6. The image decoding method according to claim 5, characterized in that, The step of filling unreceived blank macroblocks based on the hierarchical data includes: Obtain the horizontal pixel value of Xi in the blank macroblock based on the nearest decoded pixels to the left and right of the blank macroblock; The vertical pixel value of Xi in the blank macroblock is obtained based on the nearest decoded pixels on the upper and lower sides of the blank macroblock; The final pixel value of Xi is obtained based on the horizontal pixel value and the vertical pixel value of Xi; The blank macroblock is filled according to the final pixel value of Xi.

7. The image decoding method according to claim 6, characterized in that, The horizontal pixel value of the blank macroblock Xi is obtained according to a first formula, wherein the first formula includes: The horizontal pixel value of Xi = the pixel value of Li + Dist LX * (the pixel value of Ri - the pixel value of Li) / Dist LR; Wherein, the Li pixel value is the value of the nearest decoded pixel to the left of the blank macroblock; the Ri pixel value is the value of the nearest decoded pixel to the right of the blank macroblock; Dist LX is the distance between Li and Xi in pixels; and Dist LR is the distance between Li and Ri in pixels. The vertical pixel value of the blank macroblock Xi is obtained according to the second formula, which includes: The vertical pixel value of Xi = the pixel value of Ui + Dist UX * (the pixel value of Di - the pixel value of Ui) / Dist UD; Wherein, the Ui pixel value is the value of the nearest decoded pixel above the blank macroblock; the Di pixel value is the value of the nearest decoded pixel below the blank macroblock; Dist UX is the distance between Ui and Xi in pixels; and Dist UD is the distance between Ui and Di in pixels. The final pixel value of the blank macroblock Xi is obtained according to a third formula, which includes: The final pixel value of Xi = (the horizontal pixel value of Xi + the horizontal pixel value of Xi) / 2.

8. An image encoding and decoding apparatus, characterized in that, include: Encoding end and decoding end; The encoding end is used to execute the method in claims 1-4; The decoding end is used to execute the methods in claims 5-7.

Images