Word-containing image encoding method and apparatus, and word-containing image decoding method and apparatus

Abstract

The present application provides a method and apparatus for encoding a word-containing image, and a method and apparatus for decoding a word-containing image. The method for encoding a word-containing image in the present application includes: obtaining a word region in a first image, where the word region includes at least one character; filling the word region to obtain a word filling region, where the height or width of the word filling region is n times a preset size and n≥1; obtaining a second image based on the word filling region; and encoding the second image and word alignment side information to obtain a first bitstream, where the word alignment side information includes the height or width of the word filling region. The present application can avoid classifying pixels of the same character into different CUs as much as possible in order to improve the accuracy and efficiency of word content compression.

Description

Technical Field

[0001] This application relates to video encoding and decoding technologies, and in particular, to a word-containing image encoding method and apparatus, and a word-containing image decoding method and apparatus.

Background Art

[0002] In a High Efficiency Video Coding (HEVC) encoder, an input image is first divided into a plurality of Coding Tree Units (CTUs) that have the same size and do not overlap with each other. The size of the CTU may be determined by the encoder, and the maximum size of the CTU may be 64×64. One CTU may be directly used as one Coding Unit (CU), or may be further divided into a plurality of CUs with smaller sizes in a quadtree recursive partitioning manner. The CU partitioning depth is determined based on the rate-distortion cost obtained by calculation. The encoder compares different partitioning modes to balance and selects the partitioning mode with the lowest rate-distortion cost for encoding. Finally, the CU located at the leaf node of the quadtree is the basic unit for the encoder to perform subsequent prediction, transformation, and encoding.

[0003] However, when the above encoding method is applied to an image containing word content, pixels of the same word may be classified into different CTUs or CUs, and as a result, the accuracy and efficiency of word content compression during prediction, transformation, and quantization are reduced.

Summary of the Invention

Means for Solving the Problems

[0004] This application provides a word-containing image encoding method and apparatus, and a word-containing image decoding method and apparatus to avoid classifying pixels of the same character into different CUs as much as possible in order to improve the accuracy and efficiency of word content compression.

[0005] According to a first aspect, the present application provides a word-containing image encoding method, the method comprising: a step of obtaining a word region in a first image, wherein the word region contains at least one character; a step of filling the word region to obtain a word-filling region, wherein the height or width of the word-filling region is n times a preset size, n ≥ 1; a step of obtaining a second image based on the word-filling region; and a step of encoding the second image and word alignment side information to obtain a first bitstream, wherein the word alignment side information includes the height or width of the word-filling region.

[0006] In this embodiment of the present application, word regions acquired by recognition in the original image are filled such that the size of the acquired word filling region matches the size of the CU for encoding, and alignment between the word filling region and the CU is implemented to avoid classifying pixels of the same character into different CUs as much as possible, thereby improving the accuracy and efficiency of word content compression.

[0007] The first image may be an independent image or any video frame within a video. This is not specifically limited. In this embodiment of the present application, the first image may include word regions. There may be only one or more word regions. The number of word regions may depend on the number of characters included in the first image, the character distribution rules, the character size, etc. This is not specifically limited. In addition, a word region may contain one character, i.e., a word region may contain only one character, or a word region may contain one line of characters, and the line of characters may contain multiple characters, or a word region may contain one string of characters, and the string of characters may contain multiple characters. The aforementioned characters may be Chinese characters, Sino-Japanese characters, English letters, numbers, etc. This is not specifically limited.

[0008] In this embodiment of the present application, the word region in the first image may be obtained using the following four methods.

[0009] In the first method, character recognition is performed on the first image in order to obtain the word region.

[0010] Binarization and corrosion can be performed on the first image. Typically, in a binarized image, samples corresponding to word regions are white, and samples corresponding to non-word regions are black. Corrosion can be used to address pixel adhesion between characters. Then, horizontal projection is performed on the image obtained through binarization and corrosion to calculate the number of white samples in the horizontal direction (corresponding to word regions). The amount of white samples on the horizontal line containing characters is not zero. Therefore, all lines of characters in the image can be obtained based on this.

[0011] For each character line, the following operations are performed: extract the connected components (a connected component refers to a continuous white sample area, one character is a connected component, and correspondingly, a character line can have multiple connected components), and select a character box of an appropriate size. If a character box contains only one character, the size of the character box is greater than or equal to the size of the largest connected component in the character line, and all characters in the same character line use character boxes of the same size. If a character box contains a character line or string, the size of the character box is greater than or equal to the sum of the sizes of all connected components in the character line or string. For example, the connected component of character 1 in a character line occupies a pixel space of 6 (width) × 10 (height), and the connected component of character 2 occupies a pixel space of 11 × 8. In this case, at least 11 × 10 character boxes must be selected to contain the two characters, or at least 17 × 10 character boxes must be selected to contain the two characters in that character line. In this embodiment of the present application, the area covered by the text box in the first image may be referred to as the word area, i.e., the word area and the text box refer to the same area within the first image.

[0012] Therefore, the first image can be recognized using the method described above in order to obtain the word region.

[0013] In the second method, character recognition is performed on the first image to obtain an intermediate word region, which contains multiple lines of characters, and then line splitting is performed on each line of characters within the intermediate word region to obtain a word region.

[0014] The difference from the first method is that in the second method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple lines of characters, and then line splitting is performed on the intermediate word region to obtain multiple word regions, each containing only one line of characters. This method may be used when characters are arranged horizontally. For example, when multiple characters are arranged from left to right and from top to bottom.

[0015] In the third method, character recognition is performed on the first image to obtain an intermediate word region. The intermediate word region contains multiple strings, and column splitting is performed on the intermediate word region for each string to obtain a word region.

[0016] The difference from the first method is that in the third method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple lines of characters, and then column splitting is performed on the intermediate word region to obtain multiple word regions, each containing only one string. This method may be used when characters are arranged vertically, for example, when multiple characters are arranged from top to bottom and left to right.

[0017] In the fourth method, character recognition is performed on the first image to obtain an intermediate word region, which contains multiple lines of characters or multiple strings of characters, and then line splitting and column splitting are performed on each character of the intermediate word region to obtain a word region.

[0018] The difference from the first method is that in the fourth method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple character lines or multiple strings, and then row splitting and column splitting are performed on the intermediate word region to obtain multiple word regions, each containing only one character. This method may be used when characters are arranged horizontally or vertically. For example, multiple characters are arranged from top to bottom and left to right. In another example, multiple characters are arranged from top to bottom and left to right.

[0019] In addition to the four methods described above, it should be noted that in this embodiment of the present application, another method may be used to obtain the word region within the first image. This is not specifically limited.

[0020] The preset size may be any of 8, 16, 32, and 64. In this embodiment of the present application, an associated encoding and decoding method can be used to encode an image. In the associated encoding and decoding method, the image typically needs to be divided to obtain a CU, and subsequent processing is performed based on the CU. The size of the CU may be one of 8, 16, 32, and 64. In this embodiment of the present application, the preset size may be determined based on the size of the CU. For example, if the size of the CU is 8, the preset size may be 8 or 16, or if the size of the CU is 32, the preset size may be 32.

[0021] In this embodiment of the present application, a word-filling region can be obtained by filling a word region. Thus, the height of the word-filling region is greater than or equal to the height of the word region, and / or the width of the word-filling region is greater than or equal to the width of the word region. In the aforementioned case, where the height of the word-filling region is equal to the height of the word region, and / or the width of the word-filling region is equal to the width of the word region, it may mean that the height and / or width of the word region is exactly n times the preset size. Thus, the word region does not need to be filled, and the subsequent steps are performed using the word region directly as the word-filling region. In addition, when a word region is obtained, as described in step 801, the height and / or width of the word regions of characters in the same row or the same column are kept constant. Correspondingly, when a word-filling region is obtained, the height and / or width of the word regions in the same row or the same column are also kept constant. In this way, it can be ensured that the height of the word-filling region matches the height of the CU obtained by the partitioning in the relevant encoding and decoding method, and / or that the width of the word-filling region matches the width of the CU obtained by the partitioning in the relevant encoding and decoding method. In this embodiment of the present application, the aforementioned effect may also be referred to as the alignment of the word-filling region with the CU obtained by the partitioning in the relevant encoding and decoding method.

[0022] In possible embodiments, the height of the word filling area is n times the preset size. For example, if the height of the word area is 10 and the preset size is 8, the height of the word filling area may be 16 (twice 8). In another example, if the height of the word area is 10 and the preset size is 16, the height of the word filling area may be 16 (one times 16).

[0023] In a possible embodiment, the width of the word filling area is n times the preset size. For example, if the width of the word area is 17 and the preset size is 8, the width of the word filling area may be 24 (3 times 8). As another example, if the width of the word area is 17 and the preset size is 16, the width of the word filling area may be 32 (2 times 16).

[0024] In a possible embodiment, the height of the word filling area is n1 times the preset size, and the width of the word filling area is n2 times the preset size, where n1 and n2 may or may not be equal. For example, if the size of the word area is 6 (width) × 10 (height) and the preset size is 8, the size of the word filling area may be 8 (1 times 8) × 16 (2 times 8). As another example, if the size of the word area is 17 × 10 and the preset size is 16, the size of the word filling area may be 32 (2 times 16) × 16 (1 times 16).

[0025] In this embodiment of the present application, in order to obtain the word filling area, the following three methods can be used to fill the word area.

[0026] In the first method, the height of the word filling area is obtained based on the height of the word area. The height of the word filling area is n times the preset size. The word area is filled vertically in the word area to obtain the word filling area, and the filling height is the difference between the height of the word filling area and the height of the word area.

[0027] As described above, the height of the word filling area is greater than the height of the word area, and the height of the word filling area is n times the preset size. Based on this, the height of the word filling area can be obtained based on the height of the word area. Refer to the previous example. Optionally, when the height of the word area is exactly n times the preset size, the word area need not be filled. In this case, the height of the word filling area is equal to the height of the word area. After the height of the word filling area is obtained, the word area may be filled in the vertical direction of the word area (for example, the upper side, the center, or the lower side of the word area, which is not specifically limited), and the filling height is the difference between the height of the word filling area and the height of the word area. For example, when the height of the word filling area is 16 and the height of the word area is 10, the filling height is 6. The pixel value of the filling may be the pixel value of the background pixel other than the character pixel in the word area, or alternatively, the preset pixel value, for example, 0 or 255. This is not specifically limited.

[0028] In the second method, the width of the word filling area is obtained based on the width of the word area, the width of the word filling area is n times the preset size, the word area is filled in the horizontal direction of the word area to obtain the word filling area, and the filling width is the difference between the width of the word filling area and the width of the word area.

[0029] As mentioned above, the width of the word filling area is greater than the width of the word area, and the width of the word filling area is n times the preset size. Based on this, the width of the word filling area can be obtained based on the width of the word area. See the example above. Optionally, if the width of the word area is exactly n times the preset size, the word area does not need to be filled. In this case, the width of the word filling area is equal to the width of the word area. After the width of the word filling area is obtained, the word area may be filled horizontally (e.g., to the left, center, or right of the word area, and this is not specifically limited), and the width of the filling is the difference between the width of the word filling area and the width of the word area. For example, if the width of the word filling area is 24 and the width of the word area is 17, the width of the filling is 7. The pixel values ​​of the filling may be the pixel values ​​of background pixels other than character pixels within the word area, and the pixel values ​​of the filling may alternatively be preset pixel values, e.g., 0 or 255. This is not specifically limited.

[0030] The third method may be a combination of the first and second methods, that is, after the height and width of the word-filling area are obtained, the first two methods are used to fill the word area vertically and horizontally in order to obtain the word-filling area.

[0031] It can be observed that the size of the word-filling region obtained through filling has several relationships with the size of the CU. Therefore, after such a word-filling region is divided, the word-filling region can cover the entire CU, i.e., there are no cases where a word-filling region covers only a part of a CU, and the same CU does not cover two word-filling regions, thereby implementing alignment between the word region and the CU during division.

[0032] In this embodiment of the present application, a new image, i.e., a second image, can be obtained based on a word-filling region. The second image includes the characters in the original first image. Obtaining the second image includes the following four cases:

[0033] 1. If there is only one word-filling area, the word-filling area will be used as the second image.

[0034] If there is only one word-filling area, that word-filling area may be used directly as the second image.

[0035] 2. If there are multiple word-filling regions, and each of these regions contains one line of text, the multiple word-filling regions are joined from top to bottom to obtain a second image.

[0036] One word-filling region contains one line of text. In this case, the word-filling regions to which multiple lines of text belong may be joined from top to bottom based on the distribution of lines of text in the first image. Specifically, the word-filling region to which the topmost line of text in the first image belongs is joined to the top, the word-filling region to which the second line of text in the first image belongs is joined to the second line, and the rest may be inferred by analogy. Note that in this embodiment of the present application, joining may be performed alternatively from bottom to top, and the order of joining is not specifically limited.

[0037] 3. If there are multiple word-filling regions, and each of them contains one string, the multiple word-filling regions are joined from left to right to obtain a second image.

[0038] Each word-filling region contains one string. In this case, the word-filling regions to which multiple strings belong may be joined from left to right based on the distribution of strings in the first image. Specifically, the word-filling region to which the leftmost string in the first image belongs is joined to the leftmost position, the word-filling region to which the strings in the second column of the first image belong is joined to the second column, and the rest may be inferred by analogy. Note that in this embodiment of the present application, joining may be performed alternatively from right to left, and the order of joining is not specifically limited.

[0039] 4. If there are multiple word-filling regions, and each of them contains one character, the multiple word-filling regions are joined from left to right and top to bottom to obtain a second image, and multiple word-filling regions within the same row have the same height, or multiple word-filling regions within the same column have the same width.

[0040] Each word-filling region contains one character. In this case, based on the distribution of characters in the first image, multiple word-filling regions to which different characters belong may be joined from left to right and from top to bottom. The order in which the word-filling regions are arranged corresponds to the order in which the characters contained within the word-filling regions in the first image, and multiple word-filling regions in the same row have the same height, or multiple word-filling regions in the same column have the same width.

[0041] It should be noted that in this embodiment of the present application, the joining may be performed in a different order, and the order of joining is not specifically limited.

[0042] In this embodiment of the present application, standard video coding or IBC coding may be performed on the second image. This is not specifically limited. When IBC coding is used, if a character in the currently coded CU has already appeared in the coded CU, the coded CU may be determined as a predicted block of the currently coded CU. In this way, the probability that the acquired residual block is 0 is very high, thereby improving the coding efficiency of the current CU. Word alignment side information may be coded using an exponential Golomb coding scheme or another coding scheme. This is also not specifically limited. In this embodiment of the present application, the height or width of the word filling area may be coded directly, or n times a preset size of the height or width of the word filling area may be coded. This is also not specifically limited.

[0043] Optionally, word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region in the first image. This information can help the decoder recover the word region in the reconstructed image, thereby improving decoding efficiency.

[0044] In possible embodiments, in order to obtain a third image, the pixel values ​​in a word region within the first image may be filled with preset pixel values, and the third image is encoded in order to obtain a second bitstream.

[0045] In the steps described above, word regions are recognized from the first image, then the word regions are filled to obtain word-filling regions, and the second image is obtained by joining the word-filling regions. It can be seen that the second image contains only the text content from the first image and is encoded separately. However, the first image contains other content in addition to text. Therefore, in order to maintain the integrity of the image, other image content besides text needs to be further processed and encoded.

[0046] In this embodiment of the present application, the pixel values ​​of the pixels in the recognized word region in the original first image may be filled with a preset pixel value (e.g., 0, 1, or 255). This is equivalent to removing the word region from the first image and replacing its pixel values ​​with non-character pixel values. The difference between the third image thus obtained and the first image is that the pixel values ​​of the word region are different. Standard video coding or IBC coding may also be performed on the third image, but this is not specifically limited.

[0047] According to a second aspect, the present application provides a word-containing image decoding method, the method comprising: acquiring a bitstream; decoding the bitstream to acquire a first image and word alignment side information, wherein the word alignment side information includes the height or width of a word-filling region contained in the first image, the height or width of the word-filling region being n times a preset size, and n ≥ 1; acquiring a word-filling region based on the first image and the height or width of the word-filling region; and acquiring a word region in an image to be reconstructed based on the word-filling region.

[0048] In this embodiment of the present application, word region alignment information can be obtained by parsing the bitstream to obtain a character filling region containing characters. The size of the character filling region matches the size of the CU for decoding in order to implement alignment between the word filling region and the CU in order to improve the accuracy and efficiency of word content compression.

[0049] The decoder can obtain the first image and word alignment side information through decoding. The decoding method can correspond to the encoding method used on the encoder side. The encoder can encode the image using standard video coding or IBC coding, and the decoder can decode the bitstream using standard video decoding or IBC decoding to obtain the reconstructed image. The encoder can encode the word alignment side information using exponential Golomb coding, and the decoder can decode the bitstream using exponential Golomb decoding to obtain the word alignment side information.

[0050] The height or width of the word filling area is n times the preset size, where n ≥ 1, and the preset size is one of 8, 16, 32, or 64. For details, see the explanation below in step 802. Details are not explained again here.

[0051] After determining the height or width of the word-filling region, the decoder can extract the corresponding pixel values ​​from the first image based on the height or width in order to obtain the word-filling region.

[0052] In this embodiment of the present application, the word alignment side information further includes the height and width of the word region. Therefore, based on the height and width of the word region, the corresponding pixel values ​​can be extracted from the word filling region to obtain the word region in the image to be reconstructed. The word region corresponds to the word region in step 801 and may contain one character, or one line of characters, the line of characters may contain multiple characters, or one string of characters, the string of characters may contain multiple characters.

[0053] In possible embodiments, the decoder can decode the bitstream to obtain a second image and obtain a reconstruction target image based on the word region and the second image.

[0054] The word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region. Based on this, the replacement region in the second image may be determined based on the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region, and the pixel values ​​in the replacement region are filled with the pixel values ​​in the word region to obtain the image to be reconstructed. This process may be the reverse of the process of obtaining the third image in the embodiment of the first aspect, i.e., the process of filling the second image (corresponding to the third image in the embodiment of the first aspect) to obtain the reconstructed image (corresponding to the first image in the embodiment shown in Figure 8). After the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region in the second image, as well as the height and width of the word region, the replacement region in the second image may be determined, and then the pixel values ​​in the filling region are filled with the pixel values ​​of the word region. This is equivalent to removing the replacement region from the second image and replacing the pixel values ​​in the filling region with the pixel values ​​of the word region.

[0055] According to a third aspect, the present application provides an encoding device comprising: an acquisition module configured to acquire a word region in a first image, wherein the word region includes at least one character; a filling module configured to fill a word region to acquire a word filling region, wherein the height or width of the word filling region is n times a preset size, n ≥ 1, and the filling module configured to acquire a second image based on the word filling region; and an encoding module configured to encode the second image and word alignment side information to acquire a first bitstream, wherein the word alignment side information includes the height or width of the word filling region.

[0056] In possible embodiments, the filling module is specifically configured such that it obtains the height of a word filling area based on the height of the word area, the height of the word filling area is n times a preset size, and it fills the word area vertically to obtain the word filling area, and the height of the filling is the difference between the height of the word filling area and the height of the word area.

[0057] In possible embodiments, the filling module is specifically configured such that it obtains the width of a word filling area based on the width of the word area, the width of the word filling area is n times a preset size, and it fills the word area horizontally to obtain the word filling area, and the width of the filling is the difference between the width of the word filling area and the width of the word area.

[0058] In possible embodiments, a word region may contain one character, or a word region may contain one line of characters, and the line of characters may contain multiple characters, or a word region may contain one string of characters, and the string of characters may contain multiple characters.

[0059] In possible embodiments, the acquisition module is specifically configured to perform character recognition on a first image in order to acquire a word region.

[0060] In possible embodiments, the acquisition module is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region containing multiple character lines, and to perform line splitting on each character line of the intermediate word region to acquire a word region.

[0061] In possible embodiments, the acquisition module is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region containing multiple strings, and to perform string-by-string column splitting on the intermediate word region to acquire a word region.

[0062] In possible embodiments, the acquisition module is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region comprising multiple character lines or multiple strings, and to perform character-by-character row splitting and column splitting on the intermediate word region to acquire a word region.

[0063] In possible embodiments, the filling module is specifically configured to use the word filling region as a second image if there is only one word filling region.

[0064] In possible embodiments, the filling module is specifically configured to join multiple word filling regions from top to bottom in order to obtain a second image, provided that multiple word filling regions exist and each word filling region contains one line of characters.

[0065] In possible embodiments, the filling module is specifically configured to join multiple word filling regions from left to right to obtain a second image, given that multiple word filling regions exist, each containing one string.

[0066] In possible embodiments, the filling module is configured such that, if there are multiple word filling regions and each word filling region contains one character, it joins the multiple word filling regions from left to right and from top to bottom to obtain a second image, and the multiple word filling regions within the same row have the same height or the multiple word filling regions within the same column have the same width.

[0067] In possible embodiments, the word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region in the first image.

[0068] In possible embodiments, the preset size is one of 8, 16, 32, and 64.

[0069] In possible embodiments, a filling module is further configured to fill pixel values ​​in word regions within a first image with preset pixel values ​​in order to obtain a third image, and an encoding module is further configured to encode the third image in order to obtain a second bitstream.

[0070] According to a fourth aspect, the present application provides a decoding device comprising: an acquisition module configured to acquire a bitstream; a decoding module configured to decode the bitstream to acquire a first image and word alignment side information, wherein the word alignment side information includes the height or width of a word filling region contained in the first image, the height or width of the word filling region being n times a preset size, and n ≥ 1; and a reconstruction module configured to acquire a word filling region based on the first image and the height or width of the word filling region, and to acquire a word region in an image to be reconstructed based on the word filling region.

[0071] In possible embodiments, the word alignment side information further includes the height and width of the word region, and the reconstruction module is specifically configured to extract corresponding pixel values ​​from the word filling region based on the height and width of the word region in order to obtain the word region.

[0072] In possible embodiments, the decoding module is further configured to decode the bitstream to obtain a second image, and the reconstruction module is further configured to obtain a reconstruction target image based on the word region and the second image.

[0073] In possible embodiments, the word alignment side information further includes the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and the reconstruction module is specifically configured to determine a replacement region in a second image based on the height and width of the word region and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and to fill in the pixel values ​​in the replacement region with the pixel values ​​in the word region to obtain the image to be reconstructed.

[0074] In possible embodiments, a word region contains one character, or a word region contains one line of characters and the line of characters contains multiple characters, or a word region contains one string of characters and the string of characters contains multiple characters.

[0075] In possible embodiments, the preset size is one of 8, 16, 32, and 64.

[0076] According to a fifth aspect, the application provides an encoder including one or more processors and a memory configured to store one or more programs. When one or more programs are executed by one or more processors, one or more processors can implement a method according to any one of the possible embodiments of the first aspect.

[0077] According to a sixth aspect, the application provides a decoder including one or more processors and a memory configured to store one or more programs. When one or more programs are executed by one or more processors, one or more processors can implement a method according to any one of the possible embodiments of the second aspect.

[0078] According to the seventh aspect, the application provides a computer-readable storage medium containing a computer program. When the computer program is executed on a computer, the computer can implement a method according to any one of the possible embodiments of the first and second aspects.

[0079] According to the eighth aspect, the present application provides a computer program product. The computer program product includes instructions. When the instructions are executed on a computer or processor, the computer or processor becomes capable of carrying out a method according to any one of the possible embodiments of the first and second aspects. [Brief explanation of the drawing]

[0080] [Figure 1A] This is an exemplary block diagram of the coding system 10 according to this application. [Figure 1B] This is an illustrative block diagram of the video coding system 40 according to this application. [Figure 2] This is an illustrative block diagram of the video encoder 20 according to this application. [Figure 3] This is an illustrative diagram of the structure of the video decoder 30 according to this application. [Figure 4] This is a diagram of the video encoding device 400 according to this application. [Figure 5] This is a schematic block diagram of the apparatus 500 described in this application. [Figure 6] This is a diagram showing the PU division according to this application. [Figure 7] This is an exemplary block diagram of the coding system 70 according to this application. [Figure 8] This is a flowchart of process 800 of a word-containing image encoding method according to one embodiment of this application. [Figure 9] This is a flowchart of process 900 of a word-containing image decoding method according to one embodiment of this application. [Figure 10] This is an exemplary block diagram of the coding system 100 according to this application. [Figure 11a] This is a diagram illustrating word area detection and segmentation. [Figure 11b] This is a diagram illustrating adaptive word alignment. [Figure 11c] This is a diagram of Word recovery. [Figure 12a] This is a diagram illustrating word area detection and segmentation. [Figure 12b] This is a diagram illustrating adaptive word alignment. [Figure 13] This is an illustrative diagram of the structure of an encoding device 1300 according to one embodiment of this application. [Figure 14] This is an illustrative diagram of the structure of a decoding device 1400 according to one embodiment of the present application. [Modes for carrying out the invention]

[0081] To further clarify the purpose, technical solution, and merits of this application, the following will provide a clear and complete description of the technical solution of this application with reference to the accompanying drawings. It will be apparent that the embodiments described are only a selection, and not all, of the embodiments of this application. Any other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort will fall within the scope of protection of this application.

[0082] In the specification, embodiments, claims, and accompanying drawings of this application, terms such as “first” and “second” are intended merely for distinction and descriptive purposes and should not be understood as indicating or implying relative importance or order. In addition, the terms “includes,” “has,” and any variations thereof are intended to cover non-exclusive inclusion, such as including a set of steps or units. A method, system, product, or device is not necessarily limited to the steps or units expressly enumerated and may include other steps or units not expressly enumerated or specific to such process, method, product, or device.

[0083] In this application, it should be understood that "at least one" means one or more, and "multiple" means two or more. The terms "and / or" are used to describe the relationship between related objects and indicate that three relationships may exist. For example, "A and / or B" may mean the following three cases: only A exists, only B exists, and both A and B exist, where A and B may be singular or plural. The symbol " / " generally indicates an "or" relationship between related objects. "At least one of the following items" or similar expressions refer to any combination of these items, including one item or any combination of multiple items. For example, at least one of a, b or c may refer to a, b, c, "a and b", "a and c", "b and c", or "a, b and c", where a, b and c may be singular or plural.

[0084] Video coding generally refers to the processing of a sequence of images, which forms a video or video sequence. In the field of video coding, the terms “picture,” “frame,” and “image” may be used synonymously. Video coding (or coding in general) consists of two parts: video coding and video decoding. Video coding is performed on the source side and typically involves processing the original video image (e.g., by compression) to reduce the amount of data required to represent that video image (for more efficient storage and / or transmission). Video decoding is performed on the destination side and generally involves the reverse processing compared to the encoding process in order to reconstruct the video image. In embodiments, “coding” of a video image (or image in general) should be understood as “coding” or “decoding” of a video image or video sequence. The combination of the coding and decoding parts is also referred to as coding and decoding (coding and decoding, CODEC).

[0085] In lossless video coding, the original video image can be reconstructed. In other words, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss occurs during storage or transmission). In lossy video coding, further compression is performed, such as quantization, to reduce the amount of data required to represent the video image, and the video image cannot be completely reconstructed on the decoder side. In other words, the quality of the reconstructed video image is lower or worse than the original video image.

[0086] Several video encoding standards are used for "lossy hybrid video encoding and decoding" (i.e., spatial and temporal prediction in the pixel domain is combined with 2D transform encoding to apply quantization in the transform domain). Each image in a video sequence is typically divided into a set of non-overlapping blocks, and encoding is typically performed at the block level. Specifically, in the encoder, video is typically processed, i.e., encoded, at the block (video block) level. For example, predicted blocks are generated by spatial (intra) and temporal (inter) predictions, the predicted blocks are subtracted from the current block (the block being processed or to be processed) to obtain the residual block, and the residual block is transformed and quantized (compressed) in the transform domain to reduce the amount of data being transmitted. On the decoder side, the inverse processing part of the encoder is applied to the encoded or compressed block to reconstruct the current block for display. Furthermore, the encoder must replicate the decoder's processing steps so that the encoder and decoder process subsequent blocks, i.e., generate the same predictions (e.g., intra-predictions and inter-predictions) and / or reconstructed pixels for encoding.

[0087] In the following embodiments of the encoding system 10, the encoder 20 and decoder 30 are described with reference to Figures 1A to 3.

[0088] Figure 1A is an illustrative block diagram of the coding system 10 according to the present application, for example, a video coding system 10 (or coding system 10) that can use the technology described herein. The video encoder 20 (or encoder 20) and video decoder 30 (or decoder 30) of the video coding system 10 represent devices that may be configured to perform the technology according to the various examples described herein.

[0089] As shown in Figure 1A, the encoding system 10 includes a source device 12. The source device 12 is configured to provide encoded image data 21, for example, an encoded image for a destination device 14 to decode the encoded image data 21.

[0090] The source device 12 includes an encoder 20 and may additionally, or optionally, include an image source 16, a preprocessor (or preprocessing unit) 18, for example, an image preprocessor, and a communication interface (or communication unit) 22.

[0091] The image source 16 includes, or may include, any type of image acquisition device, such as a camera for capturing images of the real world, and / or any type of image generation device, such as a computer graphics processing unit for generating computer-animated images, or any other type of device for acquiring and / or providing images of the real world, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may also include any type of memory or storage device for storing any of the aforementioned images.

[0092] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the original image (or original image data) 17.

[0093] The preprocessor 18 is configured to receive (original) image data 17, perform preprocessing on the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., conversion from RGB to YCbCr), color correction, or denoising. It will be understood that the preprocessing unit 18 may be an optional component.

[0094] The video encoder (or encoder) 20 is configured to receive preprocessed image data 19 and provide encoded image data 21 (further explanation is provided below with reference to Figure 2, etc.).

[0095] The communication interface 22 of the source device 12 may be configured to receive the encoded image data 21 and transmit the encoded image data 21 (or any further processed version thereof) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0096] The destination device 14 may optionally include a decoder 30, a communication interface (or communication unit) 28, a post-processor (post-processing unit) 32, and a display device 34.

[0097] The communication interface 28 of the destination device 14 is configured to receive encoded image data 21 (or any further processed version thereof) from the source device 12 or directly from another source device, such as a storage device, for example, an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0098] Communication interfaces 22 and 28 may be configured to send and receive encoded image data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network or any combination thereof, any type of private and public network, or any type of combination thereof.

[0099] The communication interface 22 may be configured, for example, to package the encoded image data 21 into an appropriate format, such as a packet, and / or to process the encoded image data using any kind of transmit encoding or processing for transmission over a communication link or communication network.

[0100] The communication interface 28 forms the other end of the communication interface 22 and may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging to obtain encoded image data 21.

[0101] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows on the communication channel 13 in Figure 1A pointing from the source device 12 to the destination device 14, for example, and may be configured to send and receive messages, set up connections, and confirm and exchange any other information relating to the communication link and / or data transmission, such as the transmission of encoded image data.

[0102] The video decoder (or decoder) 30 is configured to receive encoded image data 21 and provide decoded image data 31 (further explanation is given below with reference to Figure 3, etc.).

[0103] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), for example, the decoded image, to obtain post-processed image data 33, for example, the post-processed image. Post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., conversion from YCbCr to RGB), color correction, cropping, resampling, or any other processing, for example, processing to prepare the decoded image data 31 for display by a display device 34.

[0104] The display device 34 is configured to receive post-processed image data 33 for displaying an image to, for example, a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0105] The coding system 10 further includes a training engine 25. The training engine 25 is configured to train one or more modules of the encoder 20 or decoder 30, or the encoder 20 or decoder 30.

[0106] In this embodiment of the present application, training data may be stored in a database (not shown), and the training engine 25 performs training based on the training data to obtain a target model. Note that the source of the training data is not limited to this embodiment of the present application. For example, the training data may be obtained from the cloud or other locations to perform model training.

[0107] The target model acquired through training by the training engine 25 may be applied to the coding system 10, for example, to the source device 12 (e.g., the encoder 20) or the destination device 14 (e.g., the decoder 30) shown in Figure 1A. The training engine 25 may acquire the target model through training on the cloud, after which the coding system 10 downloads the target model from the cloud and uses it. Alternatively, the training engine 25 may acquire the target model through training on the cloud and use it, and the coding system 10 may obtain the processing results directly from the cloud. This is not specifically limited.

[0108] Figure 1A shows that the source device 12 and the destination device 14 are independent devices. However, the device embodiment may include both the source device 12 and the destination device 14, or may include the functions of both the source device 12 and the destination device 14, that is, it may include both the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In these embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0109] As will be apparent to those skilled in the art based on the description, the presence of different units in the source device 12 and / or destination device 14 shown in Figure 1A, and the division of functions into different units, may vary depending on the actual device and application.

[0110] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented by processing circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video encoding, or any combination thereof, as shown in Figure 1B. The encoder 20 can be implemented via processing circuits 46 to embody various modules as described with respect to the encoder 20 in Figure 2 and / or any other encoder systems or subsystems described herein. The decoder 30 can be implemented via processing circuits 46 to embody various modules as described with respect to the decoder 30 in Figure 3 and / or any other decoder systems or subsystems described herein. The processing circuits 46 may be configured to perform various operations as described below. As shown in Figure 5, if the technology is partially implemented in software, the device may store the software instructions in a suitable non-temporary computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technology of this application. Either the video encoder 20 or the video decoder 30 may be integrated as part of a combination encoder / decoder (CODEC) within a single device, for example, as shown in Figure 1B.

[0111] The source device 12 and destination device 14 may include any type of handheld or stationary device, including a wide range of devices such as notebook or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiving devices, and broadcast transmitting devices, and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may be equipped with components for wireless communication. Thus, the source device 12 and destination device 14 may be wireless communication devices.

[0112] In some cases, the video encoding system 10 shown in Figure 1A is merely an example. The technology of this application may be applied to video encoding configurations (e.g., video encoding or video decoding) that do not necessarily involve data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory or transmitted over a network. A video encoding device may encode data and store the encoded data in memory, and / or a video decoding device may retrieve the data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve the data from memory and decode it.

[0113] Figure 1B is an illustrative block diagram of a video encoding system 40 according to the present application. The video encoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0114] As shown in Figure 1B, the imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 can communicate with each other. In different examples, the video encoding system 40 may include only the video encoder 20 or only the video decoder 30.

[0115] In some examples, the antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Furthermore, in some examples, the display device 45 may be configured to present video data. The processing circuit 46 may include application-specific integrated circuit (ASIC) logic, graphics processing units, general-purpose processors, etc. The video encoding system 40 may also include an optional processor 43. The optional processor 43 may also include application-specific integrated circuit (ASIC) logic, graphics processing units, general-purpose processors, etc. In addition, the memory 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)) or non-volatile memory (e.g., flash memory). In an unrestricted example, the memory 44 may be implemented by a cache memory. In another example, the processing circuit 46 may include memory (e.g., a cache) to implement an image buffer.

[0116] In some examples, the video encoder 20 implemented using logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may be included in the video encoder 20 implemented by processing circuit 46 to embody various modules described with reference to Figure 2 and / or any other encoder systems or subsystems described herein. The logic circuits may be configured to perform various operations described herein.

[0117] In some examples, the video decoder 30 may also be implemented by the processing circuit 46 to implement various modules and / or any other decoder systems or subsystems described herein, with reference to the video decoder 30 in Figure 3. In some examples, the video decoder 30 implemented using logic circuits may include an image buffer (implemented by the processing circuit 46 or memory 44) and a graphics processing unit (for example, implemented by the processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may be included in the video decoder 30 implemented by the processing circuit 46 to embody various modules and / or any other decoder systems or subsystems described herein, with reference to Figure 3.

[0118] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described herein, the encoded bitstream may include data related to encoding partitions (e.g., conversion coefficients or quantization conversion coefficients, optional indicators (as described), and / or data defining encoding partitions), such as data related to video frame encoding as described herein, indicators, index values, and mode selection data. The video encoding system 40 may further include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. A display device 45 is configured to present video frames.

[0119] In this embodiment of the present application, it should be understood that, with respect to the example described with reference to the video encoder 20, the video decoder 30 may be configured to perform the reverse process. With respect to the signaling of syntax elements, the video decoder 30 may be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, the video encoder 20 may perform entropy coding on the syntax elements to obtain an encoded video bitstream. In such examples, the decoder 30 may parse the syntax elements and decode the associated video data accordingly.

[0120] For the sake of clarity, embodiments of the present invention will be described with reference to Versatile video coding (VVC) reference software or High-Efficiency Video Coding (HEVC) reference software developed by the Joint Collaboration Team on Video Coding (JCT-VC), which is comprised of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0121] Encoder and encoding method Figure 2 is an exemplary block diagram of a video encoder 20 according to the present application. In the example in Figure 2, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transformation unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface 272). The mode selection unit 260 may include an inter-prediction unit 244, an intra-prediction unit 254, and a splitting unit 262. The inter-prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in Figure 2 may also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0122] The residual calculation unit 204, the transformation processing unit 206, the quantization unit 208, and the mode selection unit 260 are referred to as forming the forward signal path of the encoder 20, and the inverse quantization unit 210, the inverse transformation processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-prediction unit 244, and the intra-prediction unit 254 are referred to as forming the reverse signal path of the encoder, and the reverse signal path of the encoder 20 corresponds to the signal path of the decoder (see decoder 30 in Figure 3). The inverse quantization unit 210, the inverse transformation processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter-prediction unit 244, and the intra-prediction unit 254 are further referred to as forming the “built-in decoder” of the video encoder 20.

[0123] Images and image segmentation (images and blocks) The encoder 20 may be configured to receive, for example, an image (or image data) 17, for example, an image in a sequence of images forming a video or video sequence, via an input terminal 201. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the image 17 will be used in the following description. The image 17 may also be referred to as the current image or the image to be encoded (particularly in video encoding to distinguish the current image from other images, for example, previously encoded and / or decoded images in the same video sequence, i.e., the video sequence which also includes the current image).

[0124] A (digital) image can be considered, or may be considered, a two-dimensional array or matrix containing samples with intensity values. Samples within the array may also be referred to as pixels or pels (short forms of image elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. Three color components are typically used to represent color; specifically, an image may be represented as an array of three samples or contain an array of three samples. In the RGB format or color space, an image contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance / chromaticity format or color space, e.g., YCbCr, which contains a luminance component represented by Y (and sometimes by L) and two chromaticity components represented by Cb and Cr. The luminance (luma) component Y represents luminance or gray level intensity (for example, these two are the same in a grayscale image), and both chrominance (chroma) components Cb and Cr represent chromaticity or color information components. Therefore, an image in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chromaticity sample arrays of chromaticity values ​​(Cb and Cr). An image in RGB format may be converted to or transformed into an image in YCbCr format, and vice versa. This process is also called color conversion or color transformation. If the image is monochrome, the image may contain only a luminance sample array. Therefore, an image may be, for example, a luminance sample array in monochrome format, or a luminance sample array and two corresponding chromaticity sample arrays in 4:2:0, 4:2:2, and 4:4:4 color format.

[0125] In one embodiment, the video encoder 20 may include a splitting unit 262 configured to divide an image 17 into a plurality of (usually non-overlapping) image blocks 203 (which may also be abbreviated as block 203 or CTU 203). These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The splitting unit 262 may use the same block size for all images in the video sequence and a corresponding grid that defines the block size, or it may be configured to change the block size between images or subsets or groups of images to divide each image into a corresponding block.

[0126] In other embodiments, the video encoder may be configured to directly receive blocks 203 of the image 17, for example, one, some, or all of the blocks that make up the image 17. The image block 203 may also be referred to as the current image block or the image block to be encoded.

[0127] Similar to Image 17, Image Block 203 is a two-dimensional array or matrix containing samples with intensity values ​​(sample values), although it has fewer dimensions than Image 17, or may be considered as such. In other words, Block 203 may contain one sample array (e.g., a luminance array for monochrome image 17, or a luminance or chromaticity array for color images), three sample arrays (e.g., one luminance array and two chromaticity arrays for color image 17), or any other number and / or type of arrays, depending on the color format applied. The number of samples in Block 203 in the horizontal and vertical directions (or axes) determines the size of Block 203. Thus, a block can be an array of M × N (M columns × N rows) samples, an array of M × N conversion coefficients, and so on.

[0128] In one embodiment, the video encoder 20, as shown in Figure 2, may be configured to encode the image 17 block by block, for example, encoding and predicting each block 203.

[0129] In one embodiment, the video encoder 20, as shown in Figure 2, may be further configured to divide and / or encode an image using slices (also referred to as video slices), and the image may be divided or encoded using one or more slices (usually not overlapping). Each slice may contain one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC and VVC standards, and bricks in the VVC standard).

[0130] In one embodiment, the video encoder 20 shown in Figure 2 may be further configured to divide and / or encode an image by using slice / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles). The image may be divided or encoded using one or more slice / tile groups (usually non-overlapping), each slice / tile group may contain, for example, one or more blocks (e.g., CTUs) or one or more tiles. Each tile may be rectangular or of another shape and may contain one or more complete or fragmented blocks (e.g., CTUs).

[0131] Residual calculation The residual calculation unit 204 is configured to calculate the residual block 205 based on the image block 203 and the predicted block 265 (further details regarding the predicted block 265 will be provided later) by subtracting the sample value of the predicted block 265 from the sample value of the image block 203 for each sample (for each pixel) in order to obtain the residual block 205 in the pixel region.

[0132] conversion The transformation processing unit 206 is configured to apply transformations such as the discrete cosine transform (DCT) or discrete sine transform (DST) to the sample values ​​of the residual block 205 in order to obtain the transformation coefficients 207 of the transformation domain. The transformation coefficients 207 may also be called transformation residual coefficients and represent the residual block 205 in the transformation domain.

[0133] The conversion processing unit 206 may be configured to apply an integer approximation of the DCT / DST, such as the conversion defined in H.265 / HEVC. Compared to the orthogonal DCT conversion, such an integer approximation is typically scaled based on the coefficients. An additional scaling factor is applied as part of the conversion process to maintain the norm of the residual blocks processed using the forward and inverse conversions. The scaling factor is typically selected based on several constraints, such as the power of two of the shift operation, the bit depth of the conversion coefficients, and the trade-off between precision and embodiment cost. For example, a particular scaling factor may be specified for the inverse conversion by the inverse conversion processing unit 212 on the encoder 20 side (and for the corresponding inverse conversion by the inverse conversion processing unit 312 on the decoder 30 side, etc.), and correspondingly, the conversion processing unit 206 on the encoder 20 side may specify a scaling factor corresponding to the forward conversion.

[0134] In one embodiment, the video encoder 20 (correspondingly the conversion processing unit 206) may be configured to output conversion parameters such as one or more conversion types, for example, after the conversion parameters have been encoded or compressed by the entropy coding unit 270, or to output conversion parameters directly.

[0135] Quantization The quantization unit 208 is configured to quantize the transformation coefficients 207 by applying, for example, scalar quantization or vector quantization, to obtain the quantization transformation coefficients 209. The quantization transformation coefficients 209 are sometimes referred to as the quantization residual coefficients 209.

[0136] The quantization process can reduce the bit depth associated with some or all of the conversion coefficients 207. For example, if n is greater than m, n-bit conversion coefficients may be rounded to m-bit conversion coefficients during quantization. The degree of quantization may be changed by adjusting the quantization parameter (QP). For example, in the case of scalar quantization, different scales may be used to implement finer or coarser quantization. Smaller quantization steps correspond to finer quantization, and larger quantization steps correspond to coarser quantization. The appropriate quantization step may be indicated by the quantization parameter (QP). For example, the quantization parameter may be an index to a given set of appropriate quantization steps. For example, a smaller quantization parameter may correspond to finer quantization (smaller quantization steps), and a larger quantization parameter may correspond to coarser quantization (larger quantization steps), or vice versa. Quantization may include division by the quantization step, and the corresponding inverse quantization and / or inverse-inverse quantization by, for example, the inverse quantization unit 210 may include multiplication by the quantization step. In embodiments conforming to some standards such as HEVC, a quantization parameter may be used to determine the quantization step size. Generally, the quantization step can be calculated by a fixed-point approximation of the equations including division, based on the quantization parameter. Further scale factors may be introduced with respect to quantization and inverse quantization to restore the norm of the residual block, and the norm of the residual block may be modified due to the scale used in the fixed-point approximation of the equations for the quantization step and the quantization parameter. In exemplary embodiments, the scale of the inverse transform may be combined with the scale of the inverse quantization. Alternatively, a customized quantization table may be used to signal from the encoder to the decoder, for example, in a bitstream. Quantization is a lossy operation, and larger quantization steps result in greater losses.

[0137] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, after the quantization parameter has been encoded or compressed by the entropy coding unit 270, the quantization parameter may be output directly, or the quantization parameter may be output, and as a result, for example, the video decoder 30 can receive the quantization parameter and use it for decoding.

[0138] inverse quantization The inverse quantization unit 210 is configured to obtain inverse quantization coefficients 211 by applying the inverse scheme of the quantization scheme applied by the quantization unit 208 with respect to the quantization coefficients, for example, by applying the inverse scheme of the quantization scheme applied by the quantization unit 208 based on or using the same quantization steps as the quantization unit 208. The inverse quantization coefficients 211 are sometimes referred to as inverse quantization residual coefficients 211 and correspond to the transformation coefficients 207. However, the inverse quantization coefficients 211 are typically not identical to the transformation coefficients due to losses due to quantization.

[0139] Inverse Transform The inverse transform processing unit 212 is configured to apply the inverse transform of the transform applied by the transform processing unit 206, for example, the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), in order to obtain the reconstructed residual block 213 (or the corresponding inverse quantization coefficient 211) within the sample region. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0140] Reconstruction The reconstruction unit 214 (e.g., adder 214) is configured to add the transformed block 213 (i.e., the reconstructed residual block 213) to the predicted block 265 by adding the sample values ​​of the reconstructed residual block 213 and the sample values ​​of the predicted block 265, thereby obtaining the reconstructed block 215 in the pixel region.

[0141] filtering The loop filter unit 220 (also referred to simply as the "loop filter" 220) is configured to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed sample to obtain the filtered sample value. For example, the loop filter unit may be configured to perform a smooth pixel transformation or to improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be the deblocking filter, the SAO filter, and then the ALF filter. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may be applied to internal subblock edges such as affine subblock edges, ATMVP subblock edges, subblock transform (SBT) edges, and intra subpartition (ISP) edges. The loop filter unit 220 is shown as a loop filter in Figure 2, but in other forms, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as the filtered reconfigured block 221.

[0142] In one embodiment, the video encoder 20 (and correspondingly, the loop filter unit 220) may be configured to output loop filter parameters (such as SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, by directly outputting the loop filter parameters, or by outputting the loop filter parameters after entropy coding has been performed on the loop filter parameters by the entropy coding unit 270, so that, for example, the decoder 30 can receive and use the same or different loop filter parameters for decoding.

[0143] Decoded image buffer The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference image data used by the video encoder 20 during video data encoding. The DPB 230 may be formed by any one of various memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may be further configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, such as the same current image or a different image, or a previously reconstructed image, and may also provide a fully previously reconstructed, i.e., decoded image (and corresponding reference blocks and samples) and / or a partially reconstructed current image (and corresponding reference blocks and samples), for example, interpretation. The decoded image buffer 230 can be further configured to store one or more unfiltered reconstructed blocks 215, or generally, it can be further configured to store unfiltered reconstructed samples, for example, reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed samples on which no other processing has been performed.

[0144] Mode selection (split and predict) The mode selection unit 260 includes a splitting unit 262, an inter-prediction unit 244, and an intra-prediction unit 254, and is configured to receive or acquire original image data, e.g., the original block 203 (the current block 203 of the current image 17), and reconstructed image data, e.g., filtered and / or unfiltered reconstructed samples or blocks from the same (current) image and / or one or more previously decoded images, e.g., from a decoded image buffer 230 or other buffer (e.g., a column buffer not shown in the figure). The reconstructed image data is used as reference image data for predictions, e.g., inter-prediction or intra-prediction, to acquire predicted blocks 265 or predicted values ​​265.

[0145] The mode selection unit 260 may be configured to determine or select a division for the current block prediction mode (including no division) and prediction mode (e.g., intra or inter prediction mode), and to generate the corresponding predicted block 265 used for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0146] In one embodiment, the mode selection unit 260 may be configured to select a split and predict mode (e.g., from those supported or available by the mode selection unit 260) that provides best match or minimum residual (minimum residual refers to better compression for transmission or storage), or minimum signaling overhead (minimum signaling overhead refers to better compression for transmission or storage), or in which minimum residual and minimum signaling overhead are considered or balanced in the predict mode. The mode selection unit 260 may also be configured to determine the split and predict modes based on rate distortion optimization (RDO), i.e., to select a predict mode that provides minimum rate distortion optimization. The terms “best,” “lowest,” and “optimal” as used herein do not necessarily mean “best,” “lowest,” and “optimal” in general, but may refer to a situation in which the criteria for termination or selection are met. For example, values ​​above or below a threshold or other limit may be a “second-best choice” but reduce complexity and processing time.

[0147] In other words, the partitioning unit 262 may be configured to partition the images of a video sequence into a sequence of image blocks (also referred to as coding tree units, CTUs) by repeatedly using, for example, quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and the CTUs 203 may be further partitioned into smaller block partitions or subblocks (which again form blocks), and prediction may be performed for each of the block partitions or subblocks, for example, mode selection includes selecting the tree structure of the partitioned block 203, and prediction modes are applied to each of the block partitions or subblocks.

[0148] Next, we will describe in detail the partitioning (e.g., by the partitioning unit 262) and prediction (e.g., by the inter-prediction unit 244 and the intra-prediction unit 254) performed in the video encoder 20.

[0149] division The partitioning unit 262 can partition (or split) the coding tree unit 203 into smaller partitions, such as smaller square or rectangular blocks. For an image with three sample sequences, the CTU includes N × N blocks of luminance samples and two corresponding blocks of chromaticity samples. The maximum allowable size of a luminance block in a CTU is specified to be 128 × 128 in the Versatile Video Coding (VVC) under development, but may be specified to be 256 × 256 in the future, rather than 128 × 128. The CTUs of an image may be clustered / grouped as slice / tile groups, tiles, or bricks. A tile covers a rectangular area of ​​the image, and a tile may be divided into one or more bricks. A brick contains multiple rows of CTUs within it. A tile that is not divided into multiple bricks may be referred to as a brick. However, a brick is a true subset of a tile and is therefore not referred to as a tile. VVC supports tile groups in two modes: raster scan slice / tile group mode and rectangular slice mode. In raster scan tile group mode, a slice / tile group contains a sequence of tiles in a tile raster scan of an image. In rectangular slice mode, a slice contains several image bricks that collectively form a rectangular region of the image. The bricks within a rectangular slice are arranged in the order of a brick cluster scan of the slice. These smaller blocks (also called subblocks) can be further divided into smaller partitions. This is also called tree partitioning or hierarchical tree partitioning, and for example, a root block at root tree level 0 (hierarchy level 0, depth 0) can be recursively partitioned into two or more blocks at the next lower tree level, e.g., nodes at tree level 1 (hierarchy level 1, depth 1). These blocks may be further partitioned into two or more blocks at the next lower level, e.g., tree level 2 (hierarchy level 2, depth 2), until the partitioning is terminated (because a termination criterion is met, e.g., the maximum tree depth or minimum block size is reached).Blocks that are not further divided are also called leaf blocks or leaf nodes of a tree. A tree divided into two partitions is called a binary tree (BT), a tree divided into three partitions is called a ternary tree (TT), and a tree divided into four partitions is called a quad tree (QT).

[0150] For example, a coding tree unit (CTU) may be, or include, a CTB of luminance samples, two corresponding CTBs of chromaticity samples in an image with three sample arrays, a CTB of samples in a grayscale image, or a CTB of samples in an image encoded using three separate color planes and syntax structures (used to encode the samples). Correspondingly, a coding tree block (CTB) may be a block of N × N samples. By setting N to a specific value, the CTB can be obtained by partitioning its components. This is partitioning. A coding unit (CU) may be, or include, a coding block of luminance samples, two corresponding coding blocks of chromaticity samples in an image with three sample arrays, a coding block of samples in a grayscale image, or a coding block of samples in an image encoded by using three separate color planes and syntax structures (used to encode the samples). Correspondingly, a coding block (CB) may be a block of M × N samples. M and N can be set to specific values ​​to partition the CTB into coding blocks. This is partitioning.

[0151] In embodiments, for example, according to HEVC, a coding tree unit (CTU) may be divided into multiple CUs by using a quadtree structure represented as a coding tree. The decision of whether to encode an image region using inter-(time) prediction or intra-(spatial) prediction is made at the leaf CU level. Each leaf CU can be further divided into one, two, or four PUs based on the PU division type. Within a single PU, the same prediction process is applied, and relevant information is sent to the decoder on a PU-by-PU basis. After residual blocks are obtained by applying the prediction process based on the PU division type, the leaf CU can be divided into transform units (TUs) based on another quadtree structure similar to the coding tree of the CU.

[0152] In the embodiment, for example, a modern video coding standard currently under development (referred to as Multipurpose Video Coding (VVC)) uses a combination of quadtree nested multitype trees (e.g., binary and ternary trees) to segment the coding tree units. In the coding tree structure of the coding tree units, the CUs can be square or rectangular. For example, a coding tree unit (CTU) is first segmented by a quadtree structure. Then, the quadtree leaf nodes are further segmented by a multitype tree structure. There are four types of multitype tree structures: vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR). The multitype tree leaf nodes are referred to as coding units (CUs), and this segmentation is used for prediction and transformation processing without further segmentation, as long as the CUs are not excessively large relative to the maximum transformation length. This means that, in most cases, the CU, PU, ​​and TU have the same block size in the coded block structure where the quadtree is nested in a multitype tree. An exception is raised if the maximum supported translation length is shorter than the width or height of the CU's color components. The unique signaling mechanism for splitting or partitioning information in the coded structure where the quadtree is nested in a multitype tree is formulated in VVC. In the signaling mechanism, the coded tree unit (CTU) is treated as the root of the quadtree and is first split by the quadtree structure. Each quadtree leaf node (if it is large enough) is then further split by the multitype tree structure. In the multitype tree structure, a first flag (mtt_split_cu_flag) indicates whether the node is further split, and if the node is further split, a second flag (mtt_split_cu_vertical_flag) indicates the direction of the split, and then a third flag (mtt_split_cu_binary_flag) indicates whether the split is a binary or ternary split.Based on the values ​​of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the decoder can derive the multi-type tree splitting mode (MttSplitMode) of the CU based on a predefined rule or table. Note that in certain designs, such as 64x64 luminance block and 32x32 chromaticity pipeline designs in VVC hardware decoders, TT splitting is not permitted if the width or height of the luminance coding block is greater than 64, as shown in Figure 6. TT splitting is also not permitted if the width or height of the chromaticity coding block is greater than 32. In pipeline designs, the image is divided into multiple virtual pipeline data units (VPDUs), where VPDUs are defined as non-overlapping units within the image. In hardware decoders, consecutive VPDUs are processed simultaneously in multiple pipeline stages. The VPDU size is approximately proportional to the buffer size in most pipeline stages. Therefore, small VPDU sizes should be maintained. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVCs, ternary (TT) and binary (BT) partitioning can result in an increase in the VPDU size.

[0153] Additionally, note that if any part of a tree node block crosses the lower or right image boundary, the tree node block will be forced to split until all samples of all encoded CUs are located inside the image boundary.

[0154] For example, an intra sub-partition (ISP) tool can divide a luminance intra-predicted block vertically or horizontally into two or four sub-partitions based on the block size.

[0155] In one example, the mode selection unit 260 of the video encoder 20 may be configured to perform any combination of the division techniques described above.

[0156] As described above, the video encoder 20 is configured to determine or select the best or most optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, an intra-prediction mode and / or an inter-prediction mode.

[0157] Intra Prediction The set of intra-prediction modes may include 35 different intra-prediction modes, e.g., omnidirectional modes such as DC (or average) mode and planar mode, or directional modes such as modes defined in HEVC, or it may include 67 different intra-prediction modes, e.g., omnidirectional modes such as DC (or average) mode and planar mode, or directional modes such as modes defined in VVC. For example, some conventional angular intra-prediction modes are adaptively replaced by wide-angle intra-prediction modes for non-square blocks, such as those defined in VVC. In another example, only the longest side is used to calculate the average of non-square blocks in order to avoid division operations for DC prediction. In addition, the results of intra-prediction in planar mode may be further modified by using a position-dependent intra-prediction combination (PDPC) method.

[0158] The intra-prediction unit 254 is configured to use reconstructed samples of neighboring blocks of the same current image to generate an intra-predicted block 265 based on the intra-prediction mode of a set of intra-prediction modes.

[0159] The intra-prediction unit 254 (or typically the mode selection unit 260) is further configured to output intra-prediction parameters (or typically information indicating the intra-prediction mode selected for a block) to be sent to the entropy coding unit 270 in the form of syntax elements 266 included in the coded image data 21, so that the video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0160] Interpretation In possible embodiments, the set of interprediction modes depends on the available reference image (i.e., an image previously at least partially decoded and stored in, for example, DPB 230) and other interprediction parameters, such as whether the entire reference image or only a portion of it is used to search for the best-matching reference block, for example, by using a search window area around the region of the current block of the reference image, and / or whether pixel interpolation, such as half / semi-per, quarter-per, and / or 1 / 16-per interpolation, is applied.

[0161] In addition to the prediction mode described above, a skip mode and / or direct mode may also be applied.

[0162] For example, the merge candidate list for the extended merge prediction mode includes, in order, five types of candidates: spatial MVP from spatially neighboring CUs, temporal MVP from collated CUs, history-based MVP from FIFO tables, pairwise-averaged MVP, and zero MV. Bidirectional matching-based decoder-side motion vector refinement (DMVR) may be used to improve the accuracy of the MV in the merge mode. Merge mode with MVD (MMVD) is derived from merge mode with motion vector difference. The MMVD flag is sent immediately after the skip and merge flags and specifies whether the MMVD mode is used for the CU. Adaptive motion vector resolution (AMVR) schemes at the CU level may be used. AMVR supports encoding the MVD of CUs with different accuracies. The MVD of the current CU can be adaptively selected based on the prediction mode of the current CU. When a CU is encoded in merge mode, a combined inter / intra prediction (CIIP) mode may be applied to the current CU. To achieve CIIP prediction, a weighted average is performed on the inter-prediction signal and the intra-prediction signal. For affine motion compensation prediction, the affine motion field of a block is described using motion information from motion vectors of two control points (four parameters) or three control points (six parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vector of a subCU in the current CU. Bi-directional optical flow (BDOF), formerly known as BIO, is a simpler version requiring far fewer calculations, particularly in terms of the number of multiplications and multiplier values.In triangulation mode, the CU is equally divided into two triangular parts by diagonal and off-diagonal divisions. Furthermore, the dual prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals.

[0163] The interpretation unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which are shown in Figure 2). The motion estimation unit may be configured to receive or acquire, for motion estimation, an image block 203 (the current image block 203 of the current image 17) and a decoded image 231, or at least one or more previously reconstructed blocks, e.g., one or more reconstructed blocks of other / different previously decoded images 231. For example, a video sequence may include the current image and the previously decoded image 231, in other words, the current image and the previously decoded image 231 are part of, or can form part of, a sequence of images that make up the video sequence.

[0164] For example, the encoder 20 may be configured to select one reference block from multiple reference blocks of the same or different images of multiple other images, and to provide the motion estimation unit with an offset (spatial offset) between the reference image (or reference image index) and / or the position (x and y coordinates) of the reference block and the position of the current block as an interprediction parameter. This offset is also referred to as a motion vector (MV).

[0165] The motion compensation unit is configured, for example, to acquire interprediction parameters and to perform interprediction based on or using the interprediction parameters to acquire interpredicted blocks 246. Motion compensation performed by the motion compensation unit may include extracting or generating predicted blocks based on motion / block vectors determined by motion estimation, and may further include performing interpolation for sub-pixel precision. Interpolation filtering may generate further pixel samples from known pixel samples and thus potentially increase the amount of candidate predicted blocks that may be used to encode image blocks. Upon receiving the motion vector corresponding to the PU of the current image block, the motion compensation unit may identify the predicted block pointed to by the motion vector in one of the reference image lists.

[0166] The motion compensation unit may further generate syntax elements for blocks and video slices for use by the video decoder 30 when decoding image blocks of video slices. In addition to, or instead of, slices and their corresponding syntax elements, tile groups and / or tiles and their corresponding syntax elements may be generated or used.

[0167] Entropy coding The entropy coding unit 270 is configured to apply, for example, an entropy coding algorithm or scheme (e.g., variable length coding, VLC, context adaptive VLC (CAVLC), arithmetic coding, binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique) to the quantization residual coefficients 209, inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements in order to obtain encoded image data 21 that can be output via the output terminal 272 in the form of an encoded bitstream 21, for example, so that the video decoder 30 can receive the parameters and use them for decoding. The encoded bitstream 21 may be sent to the video decoder 30, or it may be stored in memory for later transmission or retrieval by the video decoder 30.

[0168] Other structural variations of the video encoder 20 may be used to encode video streams. For example, a non-conversion-based encoder 20 can directly quantize residual signals with respect to some blocks or frames without involving a conversion processing unit 206. In other embodiments, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0169] decoder and decoding method Figure 3 is an illustrative diagram of the structure of a video decoder 30 according to the present application. The video decoder 30 is configured to receive encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20 and to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks and associated syntax elements of encoded video slices (and / or tile groups or tiles).

[0170] In the example in Figure 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transformation unit 312, a reconstruction unit 314 (e.g., an adder 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an interpretation unit 344, and an intraprediction unit 354. The interpretation unit 344 may be a motion compensation unit or may include a motion compensation unit. In some examples, the video decoder 30 may perform a decoding process that is generally the reverse of the encoding process described with respect to the video encoder 20 in Figure 2.

[0171] As described for the encoder 20, the inverse quantization unit 210, the inverse transformation unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer DPB 230, the inter-prediction unit 244, and the intra-prediction unit 254 are further referred to as forming the “built-in decoder” of the video encoder 20. Thus, the function of the inverse quantization unit 310 may be the same as that of the inverse quantization unit 210, the function of the inverse transformation unit 312 may be the same as that of the inverse transformation unit 212, the function of the reconstruction unit 314 may be the same as that of the reconstruction unit 214, the function of the loop filter 320 may be the same as that of the loop filter 220, and the function of the decoded image buffer 330 may be the same as that of the decoded image buffer 230. Therefore, the descriptions provided for the corresponding units and functions of the video encoder 20 are applicable accordingly to the corresponding units and functions of the video decoder 30.

[0172] Entropy decoding The entropy decoding unit 304 is configured to analyze the bitstream 21 (or, generally, the encoded image data 21) and, for example, perform entropy decoding on the encoded image data 21 to obtain, for example, quantization coefficients 309 and / or decoded encoding parameters (not shown in Figure 3), such as inter-prediction parameters (e.g., reference image index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360, and the other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice level and / or video block level. In addition to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received or used instead.

[0173] inverse quantization The inverse quantization unit 310 may be configured to receive quantization parameters (QP) (or general information related to inverse quantization) and quantization coefficients from encoded image data 21 (e.g., analyzed and / or decoded by the entropy decoding unit 304), and to inverse quantize the quantization coefficients 309 decoded based on the quantization parameters to obtain inverse quantization coefficients 311. The inverse quantization coefficients 311 may also be referred to as transformation coefficients 311. The inverse quantization process may include the use of quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of inverse quantization that should be applied as well as the degree of quantization.

[0174] Inverse Transform The inverse transformation processing unit 312 may be configured to receive inverse quantization coefficients 311, also referred to as transformation coefficients 311, and to apply a transformation to the inverse quantization coefficients 311 in order to obtain a reconstructed residual block 313 in the pixel region. The reconstructed residual block 313 may also be referred to as a transformation block 313. The transformation may be an inverse transformation, such as an inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transformation processing unit 312 may be further configured to receive transformation parameters or corresponding information from the encoded image data 21 (for example, by parsing and / or decoding, e.g., by the entropy decoding unit 304) in order to determine the transformation to be applied to the inverse quantization coefficients 311.

[0175] Reconstruction The reconstruction unit 314 (for example, an adder 314) is configured to obtain the reconstructed block 315 in the pixel region by adding the sample value of the reconstructed residual block 313 to the predicted block 365, for example, by adding the sample value of the reconstructed residual block 313 to the sample value of the predicted block 365.

[0176] filtering The loop filter unit 320 (either during or after the coding loop) is configured to filter the reconstructed block 315 to obtain the filtered block 321, smoothing pixel transitions or improving image quality. The loop filter unit 320 may include one or more loop filters, e.g., a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, e.g., an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be the deblocking filter, the SAO filter, and then the ALF filter. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may be applied to internal subblock edges such as affine subblock edges, ATMVP subblock edges, subblock transform (SBT) edges, and intra subpartition (ISP) edges. Although the loop filter unit 320 is shown as a loop filter in Figure 3, in other forms the loop filter unit 320 may be implemented as a post-loop filter.

[0177] Decoded image buffer Next, the image buffer block 321 is stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. The reference image is used for subsequent motion compensation of other images and / or to output their respective displays.

[0178] The decoder 30 is configured to output the decoded image 331, for example, via the output terminal 332, for presentation or viewing to the user.

[0179] prediction The inter-prediction unit 344 may be functionally identical to the inter-prediction unit 244 (particularly the motion compensation unit), and the intra-prediction unit 354 may be functionally identical to the intra-prediction unit 254, and performs partitioning or partitioning decision and prediction based on partitioning parameters and / or prediction parameters, or the respective information received from the encoded image data 21 (for example, by analysis and / or decoding by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra or inter-prediction) for each block based on the reconstructed image, or block, or corresponding sample (filtered or unfiltered) in order to obtain predicted blocks 365.

[0180] When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 of the mode-applying unit 360 is configured to generate a predicted block 365 for the image block of the current video slice based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When a video image is encoded as an inter-coded (e.g., B or P) slice, the inter-prediction unit 344 (e.g., motion compensation unit) of the mode-applying unit 360 is configured to generate a predicted block 365 for the video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. In the case of inter-prediction, the predicted block may be generated from one reference image in one reference image list. The video decoder 30 may configure reference frame lists, i.e., List 0 and List 1, by using a default configuration technique based on reference images stored in the DPB 330. The same or similar processes may be applied to, or by, embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to, or instead of, slices (e.g., video slices), for example, video may be encoded using I, P, or B tile groups and / or tiles.

[0181] The mode application unit 360 is configured to determine predictive information about the video blocks of the current video slice by analyzing motion vectors and other syntax elements, and to use the predictive information to create predicted blocks about the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the predictive mode used to encode the video blocks of the video slice (e.g., intra-predictive or inter-predictive), the inter-predictive slice type (e.g., B-slice, P-slice, or GPB-slice), construction information about one or more of the reference image lists for the slice, the motion vector for each inter-encoded video block of the slice, the inter-predictive state for each inter-encoded video block of the slice, and other information for decoding the video blocks in the current video slice. The same or similar process may be applied to, or by, embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to, or instead of, slices (e.g., video slices), for example, video may be encoded using I, P, or B tile groups and / or tiles.

[0182] In one embodiment, the video decoder 30, as shown in Figure 3, may be further configured to divide and / or decode an image using slices (also referred to as video slices), and the image may be divided or decoded using one or more slices (usually not overlapping). Each slice may contain one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0183] In one embodiment, the video decoder 30 shown in Figure 3 may be further configured to divide and / or decode an image by using slice / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles). The image may be divided or decoded using one or more slice / tile groups (usually non-overlapping), each slice / tile group may contain, for example, one or more blocks (e.g., CTUs) or one or more tiles. Each tile may be rectangular or of another shape and may contain one or more complete or fragmented blocks (e.g., CTUs).

[0184] Other variations of the video decoder 30 may be used to decode encoded image data 21. For example, the decoder 30 may generate an output video stream without a loop filter unit 320. For example, a non-transformation-based decoder 30 may directly dequantize the residual signal with respect to some blocks or frames without an inverse transformation processing unit 312. In other embodiments, the video decoder 30 may have an inverse quantization unit 310 and an inverse transformation processing unit 312 that combine to form a single unit.

[0185] It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations, such as clipping or shifting, may be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering.

[0186] While video encoding and decoding are primarily described in the embodiments described above, it should be noted that the embodiments of the encoding system 10, encoder 20, and decoder 30 described herein, as well as other embodiments, may also be used for still image processing or encoding and decoding, i.e., for processing or encoding and decoding a single image unrelated to preceding or consecutive images in video encoding and decoding. Generally, when image processing is limited to a single image, the interpretation unit 244 (encoder) and interpretation unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30, such as the residual calculation unit 204, the transformation processing unit 206, the quantization unit 208, the inverse quantization units 210 / 310, the (inverse) transformation processing units 212 / 312, the partitioning unit 262, the intra-prediction units 254 / 354 and / or the loop filter units 220 / 320, the entropy coding unit 270, and the entropy decoding unit 304 may also be used for still image processing.

[0187] Figure 4 is a diagram of the video encoding device 400 according to this application. The video encoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video encoding device 400 may be a decoder, such as the video decoder 30 in Figure 1A, or an encoder, such as the video encoder 20 in Figure 1A.

[0188] The video coding device 400 includes an inlet port 410 (or input port 410) and a receiver unit (Rx) 420 configured to receive data, a processor, logic unit, or central processing unit (CPU) 430 for processing data, a transmitter unit (Tx) 440 and an exit port 450 (or output port 450) configured to transmit data, and memory 460 for storing data. The video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the inlet port 410, receiver unit 420, transmitter unit 440, and exit port 450 for the entry and exit of optical or electrical signals.

[0189] The processing unit 430 is implemented using hardware or software. The processing unit 430 may be implemented as one or more processor chips, cores (e.g., multicore processors), FPGAs, ASICs, and DSPs. The processing unit 430 communicates with the input port 410, the receiver unit 420, the transmitter unit 440, the output port 450, and the memory 460. The processing unit 430 includes an encoding module 470. The encoding module 470 implements the embodiments disclosed above. For example, the encoding module 470 implements, processes, prepares, or provides various encoding operations. Thus, the encoding module 470 provides a significant improvement to the functionality of the video encoding device 400, resulting in a switch between different states of the video encoding device 400. Alternatively, the encoding module 470 is implemented using instructions stored in the memory 460 and executed by the processing unit 430.

[0190] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0191] Figure 5 is a schematic block diagram of the device 500 according to this application. The device 500 may be used as both or either the source device 12 and the destination device 14 in Figure 1A.

[0192] The processor 502 within the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or multiple devices currently existing or to be developed in the future that are capable of manipulating or processing information. The disclosed embodiments may be implemented using a single processor such as the processor 502 shown in the figure, but advantages in speed and efficiency can be achieved by using multiple processors.

[0193] In one embodiment, the memory 504 in the device 500 may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by the processor 502 through the bus 512. Memory 504 may further include an operating system 508 and an application 510, the application 510 including at least one program that enables the processor 502 to perform the method described herein. For example, application 510 may include applications 1 to N, and further include a video encoding application that performs the method described herein.

[0194] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display, where the display is combined with a touch-sensitive element that can be configured to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0195] In this specification, the bus 512 of the device 500 is described as a single bus, but the bus 512 may include multiple buses. Furthermore, the secondary storage device may be directly coupled to other components of the device 500 or accessed via a network, and may include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, the device 500 can have a variety of configurations.

[0196] For ease of understanding, the nouns or terms used in the embodiments of this application are first described and explained below. These nouns or terms are also used as part of the subject matter of the invention.

[0197] 1. Neural Networks A neural network (NN) is a machine learning model. A neural network can include neurons. Neurons are x s It may also be an arithmetic unit that uses an intercept of 1 as input, and the output of the arithmetic unit may be as follows:

number

[0198] Here, s = 1, 2, ..., or n, where n is a natural number greater than 1, and W s is x s The weights are , b is the neuron's bias, and f is the neuron's activation function, which is used to introduce nonlinear features into the neural network and convert the neuron's input signal into an output signal. The output signal of the activation function can be used as the input to the next convolutional layer. The activation function may be a nonlinear function such as ReLU. A neural network is a network formed by connecting a large number of single neurons to each other. Specifically, the output of one neuron may be the input to another neuron. The input of each neuron may be connected to the local receptive field of the previous layer to extract features of the local receptive field. The local receptive field may be a region containing multiple neurons.

[0199] 2. Multilayer perceptron (MLP) MLP is a simple deep neural network (DNN) (where different layers are fully connected), also known as a multilayer neural network. An MLP can be understood as a neural network with many hidden layers. There is no specific metric for "many" in this specification. DNNs are divided based on the location of different layers, and a DNN neural network can be divided into three types: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the intermediate layers are hidden layers. The layers are fully connected. Specifically, any neuron in the i-th layer is always connected to any neuron in the (i+1)-th layer. While DNNs may seem complex, they are not complex in terms of the work done in each layer. Simply put, the work in each phase is governed by the following linear relationship:

number

number

number

number

number

number

number

number

number

[0200] 3. Convolutional Neural Networks A convolutional neural network (CNN) is a deep neural network with a convolutional structure and is also a deep learning architecture. In deep learning architectures, multi-layer learning is performed at different levels of abstraction according to machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network, where each neuron in the feed-forward artificial neural network can respond to image inputs to the feed-forward artificial neural network. A convolutional neural network includes a feature extractor that contains convolutional layers and pooling layers. The feature extractor can be thought of as a filter. The convolutional process can also be thought of as using a trainable filter to perform convolution on an input image or a convolutional feature map.

[0201] A convolutional layer is a neuron layer in a convolutional neural network that performs a convolution operation on an input signal. A convolutional layer can contain multiple convolution operators, also known as kernels. In image processing, convolution operators function as filters that extract specific information from an input image matrix. A convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix typically processes pixels at a granularity of one pixel (or two pixels, depending on the stride value) horizontally in the input image to extract specific features from the image. The size of the weight matrix must be related to the size of the image. Note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing a convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolutional output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows x columns), i.e., multiple matrices of the same type, are applied. The outputs of the weight matrices are stacked to form the depth dimension of the convolutional image. The dimension herein may be understood as being determined based on the aforementioned "multiple". Different weight matrices may be used to extract different features from the image. For example, one weight matrix may be used to extract edge information from the image, another to extract a specific color from the image, and yet another to blur unwanted noise in the image. The multiple weight matrices are the same size (rows x columns). The feature maps extracted by multiple weight matrices of the same size are also the same size, and then multiple extracted feature maps of the same size are combined to form the output of the convolutional operation. The weight values ​​of these weight matrices need to be acquired through extensive training during actual application. Each weight matrix containing the weight values ​​acquired through training may be used to extract information from the input image, so that the convolutional neural network makes correct predictions.When a convolutional neural network has multiple convolutional layers, a large number of general features are typically extracted in the initial convolutional layers. These general features may also be called low-level features. As the depth of the convolutional neural network increases, the features extracted in subsequent convolutional layers become more complex, for example, high-level semantic features. Features with high-level semantics are more applicable to the problem to be solved.

[0202] The number of training parameters often needs to be reduced. Therefore, pooling layers often need to be introduced periodically after convolutional layers. One pooling layer may follow one convolutional layer, or one or more pooling layers may follow multiple convolutional layers. During image processing, pooling layers are used solely to reduce the spatial size of the image. Pooling layers may include mean pooling operators and / or max pooling operators to perform sampling on the input image to obtain a smaller image. The mean pooling operator can be used to perform calculations on pixel values ​​within a specific range of the image to generate an average value, which is used as the mean pooling result. The max pooling operator may be used to select pixels with the maximum value within a specific range as the max pooling result. In addition, just as the size of the weight matrix in a convolutional layer needs to be related to the size of the image, the operators in a pooling layer also need to be related to the size of the image. The size of the processed image output from a pooling layer may be smaller than the size of the image input to the pooling layer. Each sample in the image output from a pooling layer represents the mean or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0203] After processing in the convolutional / pooling layers, the convolutional neural network is not yet ready to output the required output information because, as mentioned above, the convolutional / pooling layers only extract features, reducing the parameters derived from the input image. However, to generate the final output information (required type information or other relevant information), the convolutional neural network needs to use neural network layers to produce an output of one required type, or a group of required types. Therefore, the convolutional neural network layers may include multiple hidden layers. The parameters included in the multiple hidden layers can be obtained through pre-training based on relevant training data for a particular task type. For example, task types may include image recognition, image classification, and super-resolution image reconstruction.

[0204] Optionally, in a neural network layer, a multi-hidden layer is followed by an output layer for the entire convolutional neural network. The output layer has a loss function similar to that of categorical cross-entropy, and this loss function is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is complete, backpropagation is initiated to update the weights and deviations of each layer described above, reducing the loss of the convolutional neural network, as well as the error between the result output by the convolutional neural network and the ideal result using the output layer.

[0205] 4. Recurrent Neural Networks Recurrent neural networks (RNNs) are used to process sequence data. In conventional neural network models, the layers from the input layer to the hidden and output layers are fully connected, and the nodes in each layer are not connected. While such general neural networks can solve many problems, they still cannot solve many others. For example, to predict the next word in a sentence, the previous word is usually used because the previous and next words in the sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is also related to the previous output of the sequence. A specific form of representation is that the network remembers previous information and applies that information to the calculation of the current output. Specifically, the nodes in the hidden layer are no longer disconnected but connected, and the input to the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. Theoretically, RNNs can process sequence data of any length. Training an RNN is similar to training a conventional CNN or DNN. The backpropagation algorithm is also used, but there is a difference in that parameters such as W of the RNN are shared when the RNN is extended. This differs from the conventional neural networks described in the previous example. In addition, when using the gradient descent algorithm, the output at each step depends not only on the network state at the current step, but also on the network state at some previous steps. The learning algorithm is called the Backpropagation Through Time (BPTT) algorithm.

[0206] Why are recurrent neural networks still needed when convolutional neural networks are available? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, like "dog" and "cat." However, in the real world, many elements are interconnected. For example, stock prices change over time. Another example is when someone says, "I like to travel, and my favorite place is Yunnan. I'll go there if I get the chance." In this specification, people should know that this person will go to "Yunnan" because people perform inference from context. But how do machines do this? This is where RNNs come in. RNNs are intended to enable machines to remember like humans. Therefore, the output of an RNN must depend on the current input information and the historical stored information.

[0207] 5. Loss function In the process of training a deep neural network, the output of the deep neural network is expected to be as close as possible to the predicted value that is actually expected. Therefore, the current network's predicted value may be compared to the actually expected target value, and then the weight vectors of each layer of the neural network are updated based on the difference between the predicted value and the target value (indeed, there is usually an initialization process before the first update, specifically, the parameters are pre-configured for all layers of the deep neural network). For example, if the network's predicted value is large, the weight vectors are adjusted to decrease the predicted value, and this adjustment is continued until the deep neural network can predict the actually expected target value, or a value very close to the actually expected target value. Therefore, "how the difference between the predicted value and the target value is obtained by comparison" needs to be defined in advance. This is the loss function or objective function. The loss function and objective function are important equations that measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training a deep neural network is a process of minimizing losses as much as possible.

[0208] 6. Backpropagation Algorithm Convolutional neural networks may correct the parameter values ​​of the initial super-resolution model during the training process according to a backpropagation (BP) algorithm, resulting in smaller error losses when reconstructing the super-resolution model. Specifically, the input signal is forward-transmitted until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on backpropagated error loss information to converge the error loss. The backpropagation algorithm is an error loss-centered backpropagation motion intended to obtain parameters such as the weight matrix of the optimal super-resolution model.

[0209] 7. Generative Adversarial Networks A generative adversarial network (GAN) is a deep learning model. The model contains at least two modules: a generative model and a discriminative model. The two modules learn from each other through a game to produce better output. Both the generative and discriminative models may be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: Using a GAN to generate an image is used as an example, and we assume there are two networks, G (Generator) and D (Discriminator). G is the network for generating the image. G receives random noise z, uses the noise to generate an image, and the image is shown as G(z). D is the discriminative network used to determine whether the image is a "real image". The input parameter of D is x, where x represents an image, and the output D(x) represents the probability that x is a real image. A value of D(x) of 1 indicates that the image is 100% real. A value of 0 for D(x) indicates that the image cannot be a real image. In the process of training a generative adversarial network, the goal of the generative network G is to generate an image as close to a real image as possible in order to deceive the discriminative network D, and the goal of the discriminative network D is to distinguish as much as possible between the image generated by G and a real image. Thus, there is a dynamic "game" process between G and D, or an "adversary" in a "generative adversarial network". The final game result, in an ideal state, is that G can generate an image G(z) that is difficult to distinguish from a real image, and for D, it is difficult to determine whether the image generated by G is a real image, specifically D(G(z)) = 0.5. In this way, a good generative model G is obtained and can be used to generate images.

[0210] 8. Golomb coding In computer data storage devices, a bit is the smallest unit of memory, and one bit stores a binary code of logical 0 or 1. Generally, numbers are encoded in binary mode. However, binary mode generates a lot of redundant information because different numbers of the same length are recorded. For example, the binary codes for 255 and 1 of the same length are 11111111 and 00000001, with 8 and 1 being the effective bits of the binary code. If arbitrary data is transmitted using a binary encoding scheme, a large amount of redundancy is generated, increasing the network load.

[0211] Exponential Golomb coding is a lossless data compression method. It is a variable-length prefix coding scheme that is suitable for coding situations where smaller numbers have a higher probability of occurring than larger numbers. Shorter code lengths are used to code smaller numbers, and longer code lengths are used to code larger numbers. Exponential Golomb coding is a form of Golomb coding and is a variable-length coding scheme commonly used in audio and video coding standards. All numbers are divided into different groups of equal size. Groups with smaller symbol values ​​are assigned shorter code lengths, and symbol lengths within the same group are essentially equal, with the group size increasing exponentially.

[0212] The format of the number to be coded, num, obtained by performing k-th exponential Golomb coding is [MZeros][1][Info], where MZeros is assumed to represent M zeros. The coding process is as follows:

[0213] i.num is represented in binary mode (add 0 to the most significant bit if the number of bits is less than k), the least significant k bits are removed (if the number of bits is exactly k, num is 0), and the number n is obtained.

[0214] ii. M is the number of bits in the binary representation obtained by subtracting 1 from the number (n+1).

[0215] iii. Place the k-bit binary sequence removed in the i-th step into the least significant bit of (n+1) to obtain [1][Info].

[0216] Table 1 shows a diagram of codewords obtained by performing Golomb coding on numbers from 0 to 9.

[0217] [Table 1]

[0218] 9. Block division The HEVC standard introduces a set of fundamental units based on quadtree recursive partitioning, including coding units (CUs), prediction units (PUs), and transform units (TUs). In an HEVC encoder, the input image is first partitioned into multiple coding tree units (CTUs) of the same size that do not overlap each other. The size of the CTUs is determined by the encoder, and the maximum size of a CTU may be 64x64. In the subsequent coding process, one CTU may be used directly as one CU, or it may be further partitioned into multiple smaller CUs using the quadtree recursive partitioning scheme. The CU partitioning depth is determined based on the computationally obtained rate-distortion cost. The encoder compares and balances different partitioning modes and selects the partitioning mode with the lowest rate-distortion cost for coding. Finally, the CUs located at the leaf nodes of the quadtree are the fundamental units for which the encoder performs subsequent prediction, transformation, and coding. Typically, during block partitioning, a shallow partitioning depth and large CUs are used for flat and smooth areas, while a deep partitioning depth and small CUs are used for areas with complex colors. In this way, processing by modules such as the prediction module, transformation module, and quantization module becomes more convenient and accurate, and the selection of the encoding mode better matches the image features of the video content, thereby effectively improving encoding efficiency.

[0219] The PU is the basic unit for the encoder to perform predictions, containing all prediction-related information, and is obtained by further subdividing the CU. For a 2N×2N CU, the PU has a total of eight optional subdivision modes, including four symmetric modes of 2N×2N, N×N, 2N×N, and N×2N, as well as four asymmetric modes of 1:3 or 3:1 subdivision, as shown in Figure 6. Introducing multiple subdivision modes can make predictions of complex images more accurate.

[0220] A TU (Tuner Unit) is the fundamental unit from which an encoder performs transformations and quantization. The size of a TU depends on the CU (Cumulative Unit) to which it belongs, and the CU can be further divided using a quadtree partitioning scheme to obtain TUs. The minimum size of a TU is 4x4, and the maximum size can reach 32x32. Depending on the content characteristics, the encoder can flexibly select the best partitioning mode for the TU.

[0221] 10. Intra-block copy (IBC) IBC is a key tool for screen content coding in the HEVC Screen Content Coding (SCC) standard. It is a block-based prediction technique, and its mechanism is similar to interpretation or motion compensation. Motion compensation means that for the current prediction unit, the encoder finds the best matching block in a previously coded reference image as the prediction value according to a motion search algorithm, and the motion vector (MV) indicates the matching relationship. The difference obtained by subtracting the prediction value from the pixel value of the current prediction unit is used as the prediction residual. The prediction residual is processed by modules such as the transformation module, quantization module, and entropy coding module before being output to the bitstream.

[0222] The main difference between IBC and motion compensation is that in IBC, the reference sample is taken from within the current image (reconstructed portion), and there is a block vector (BV) similar to the motion vector to indicate block matching relationships. IBC can handle screen content with multiple similar graphics or words within a video frame better; that is, the current block can be a block with similar graphics in the current image, and a predicted residual close to 0 pixel value can be obtained. This residual occupies a very small bitrate in the bitstream. The syntax structure and encoding process of the associated information contained in the rest of IBC is almost the same as in motion compensation.

[0223] This application provides a word-containing image encoding method and a word-containing image decoding method, solving the problem in the word-containing image encoding and decoding process where pixels of the same word are classified into different CTUs or CUs, which can lead to low accuracy and efficiency of word content compression.

[0224] Figure 2 illustrates the structure and principle of a standard video encoding module system, and Figure 3 illustrates the structure and principle of a standard video decoding system. Based on this, an encoding scheme for words within an image is added in this application.

[0225] Figure 7 is an exemplary block diagram of the coding system 70 according to this application. As shown in Figure 7, compared to a standard video coding and decoding system (e.g., coding system 10), the encoder side (e.g., source device 12) adds a word processing module and a word alignment side information coding module in addition to the standard video coding module (e.g., encoder 20), and the decoder side (e.g., destination device 14) adds a word alignment side information decoding module and a word reconstruction module in addition to the standard video decoding module (e.g., decoder 30). The added modules are independent of the standard video coding / decoding process and, together with the standard video coding / decoding process, form an improved coding system 70.

[0226] The functions of the aforementioned module are as follows:

[0227] Word processing module 1. Detection: Extract word regions and non-word regions from the original image (i.e., the image to be encoded).

[0228] 2. Split: Splitting retrieves the complete characters or lines of characters within the word area.

[0229] 3. Adaptive alignment: Aligns divided characters or lines of characters with the block division grid. In this case, the position of the aligned characters or lines of characters is offset from their original position.

[0230] Word Alignment Side Information Encoding Module: Encodes the word alignment side information (e.g., original character coordinates, character box size, and filling box size) generated by the word processing module for word reconstruction on the decoder side.

[0231] Standard video encoding module: Performs standard mixed video encoding separately for non-word regions and aligned word regions.

[0232] Standard video decoding module: Performs standard mixed video decoding on the bitstream to obtain non-word regions and aligned word regions.

[0233] Word Alignment Side Information Decoding Module: Decodes the bitstream to obtain word alignment side information (e.g., original character coordinates, character box size, and filling box size).

[0234] Word Reconstruction Module 1. Word Restoration: Based on the word alignment side information, the correct word region is restored from the aligned word region.

[0235] 2. Region merging: Word regions and non-word regions in the correct positions are merged to form a complete reconstructed image.

[0236] Based on the encoding system 70 shown in Figure 7, the solution provided in this application will be described below.

[0237] Figure 8 is a flowchart of process 800 of a word-containing image encoding method according to one embodiment of the present application. Process 800 may be performed by a video encoder 20. Process 800 is described as a series of steps or operations. It should be understood that process 800 may be performed in various orders and / or simultaneously, but is not limited to the execution order shown in Figure 8. Assume that the current image block of the current video frame of a video data stream containing multiple video frames is encoded by the video encoder 20 using process 800, which includes the following steps.

[0238] Step 801: Obtain the word region in the first image, where the word region contains at least one character.

[0239] The first image may be an independent image or any video frame within a video. This is not specifically limited. In this embodiment of the present application, the first image may include word regions. There may be only one or more word regions. The number of word regions may depend on the number of characters included in the first image, the character distribution rules, the character size, etc. This is not specifically limited. In addition, a word region may contain one character, i.e., a word region may contain only one character, or a word region may contain one line of characters, and the line of characters may contain multiple characters, or a word region may contain one string of characters, and the string of characters may contain multiple characters. The aforementioned characters may be Chinese characters, Sino-Japanese characters, English letters, numbers, etc. This is not specifically limited.

[0240] In this embodiment of the present application, the word region in the first image may be obtained using the following four methods.

[0241] In the first method, character recognition is performed on the first image in order to obtain the word region.

[0242] Binarization and corrosion can be performed on the first image. Typically, in a binarized image, samples corresponding to word regions are white, and samples corresponding to non-word regions are black. Corrosion can be used to address pixel adhesion between characters. Then, horizontal projection is performed on the image obtained through binarization and corrosion to calculate the number of white samples in the horizontal direction (corresponding to word regions). The amount of white samples on the horizontal line containing characters is not zero. Therefore, all lines of characters in the image can be obtained based on this.

[0243] For each character line, the following operations are performed: extract the connected components (a connected component refers to a continuous white sample area, one character is a connected component, and correspondingly, a character line can have multiple connected components), and select a character box of an appropriate size. If a character box contains only one character, the size of the character box is greater than or equal to the size of the largest connected component in the character line, and all characters in the same character line use character boxes of the same size. If a character box contains a character line or string, the size of the character box is greater than or equal to the sum of the sizes of all connected components in the character line or string. For example, the connected component of character 1 in a character line occupies a pixel space of 6 (width) × 10 (height), and the connected component of character 2 occupies a pixel space of 11 × 8. In this case, at least 11 × 10 character boxes must be selected to contain the two characters, or at least 17 × 10 character boxes must be selected to contain the two characters in that character line. In this embodiment of the present application, the area covered by the text box in the first image may be referred to as the word area, i.e., the word area and the text box refer to the same area within the first image.

[0244] Therefore, the first image can be recognized using the method described above in order to obtain the word region.

[0245] In the second method, character recognition is performed on the first image to obtain an intermediate word region, which contains multiple lines of characters, and then line splitting is performed on each line of characters within the intermediate word region to obtain a word region.

[0246] The difference from the first method is that in the second method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple lines of characters, and then line splitting is performed on the intermediate word region to obtain multiple word regions, each containing only one line of characters. This method may be used when characters are arranged horizontally. For example, when multiple characters are arranged from left to right and from top to bottom.

[0247] In the third method, character recognition is performed on the first image to obtain an intermediate word region. The intermediate word region contains multiple strings, and column splitting is performed on the intermediate word region for each string to obtain a word region.

[0248] The difference from the first method is that in the third method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple lines of characters, and then column splitting is performed on the intermediate word region to obtain multiple word regions, each containing only one string. This method may be used when characters are arranged vertically, for example, when multiple characters are arranged from top to bottom and left to right.

[0249] In the fourth method, character recognition is performed on the first image to obtain an intermediate word region, which contains multiple lines of characters or multiple strings of characters, and then line splitting and column splitting are performed on each character of the intermediate word region to obtain a word region.

[0250] The difference from the first method is that in the fourth method, the intermediate word region can be obtained by character recognition, and the intermediate word region can contain multiple character lines or multiple strings, and then row splitting and column splitting are performed on the intermediate word region to obtain multiple word regions, each containing only one character. This method may be used when characters are arranged horizontally or vertically. For example, multiple characters are arranged from top to bottom and left to right. In another example, multiple characters are arranged from top to bottom and left to right.

[0251] In addition to the four methods described above, it should be noted that in this embodiment of the present application, another method may be used to obtain the word region within the first image. This is not specifically limited.

[0252] Step 802: Fill the word area to obtain the word filling area, where the height or width of the word filling area is n times the preset size, and n ≥ 1.

[0253] The preset size may be any one of 8, 16, 32, and 64. In this embodiment of the present application, related encoding and decoding methods can be used to encode an image. In the related encoding and decoding methods, an image usually needs to be divided to obtain CUs, and subsequent processing is performed based on the CUs. The size of the CU may be one of 8, 16, 32, and 64. In this embodiment of the present application, the preset size may be determined based on the size of the CU. For example, when the size of the CU is 8, the preset size may be 8 or 16, or when the size of the CU is 32, the preset size may be 32.

[0254] In this embodiment of the present application, the word filling area can be obtained by filling the word area. Therefore, the height of the word filling area is greater than or equal to the height of the word area, and / or the width of the word filling area is greater than or equal to the width of the word area. In the aforementioned case where the height of the word filling area is equal to the height of the word area and / or the width of the word filling area is equal to the width of the word area, it may mean that the height and / or width of the word area is exactly n times the preset size. Therefore, the word area does not need to be filled, and the subsequent steps are executed by directly using the word area as the word filling area. In addition, as described in step 801, when the word area is obtained, the height and / or width of the word area of the characters in the same row or the same column is kept constant. Correspondingly, when the word filling area is obtained, the height and / or width of the word area in the same row or the same column is also kept constant. In this way, it can be guaranteed that the height of the word filling area matches the height of the CU obtained by the division in the related encoding and decoding method, and / or the width of the word filling area matches the width of the CU obtained by the division in the related encoding and decoding method. In this embodiment of the present application, the aforementioned effect may also be referred to as the word filling area being aligned with the CU obtained by the division in the related encoding and decoding method.

[0255] In a possible embodiment, the height of the word filling area is n times the preset size. For example, when the height of the word area is 10 and the preset size is 8, the height of the word filling area may be 16 (2 times 8). As another example, when the height of the word area is 10 and the preset size is 16, the height of the word filling area may be 16 (1 times 16).

[0256] In possible embodiments, the width of the word-filling area is n times the preset size. For example, if the width of the word area is 17 and the preset size is 8, the width of the word-filling area may be 24 (3 times 8). In another example, if the width of the word area is 17 and the preset size is 16, the width of the word-filling area may be 32 (2 times 16).

[0257] In possible embodiments, the height of the word filling area is n1 times the preset size, and the width of the word filling area is n2 times the preset size, where n1 and n2 may or may not be equal. For example, if the size of the word area is 6 (width) × 10 (height) and the preset size is 8, the size of the word filling area may be 8 (1 times 8) × 16 (2 times 8). As another example, if the size of the word area is 17 × 10 and the preset size is 16, the size of the word filling area may be 32 (2 times 16) × 16 (1 times 16).

[0258] In this embodiment of the present application, the following three methods can be used to fill a word area in order to obtain a word-filling area.

[0259] In the first method, the height of the word filling area is obtained based on the height of the word area, and the height of the word filling area is n times a preset size, and the word area is filled vertically to obtain the word filling area, and the height of the filling is the difference between the height of the word filling area and the height of the word area.

[0260] As mentioned above, the height of the word filling area is greater than the height of the word area, and the height of the word filling area is n times the preset size. Based on this, the height of the word filling area can be obtained based on the height of the word area. See the example above. Optionally, if the height of the word area is exactly n times the preset size, the word area does not need to be filled. In this case, the height of the word filling area is equal to the height of the word area. After the height of the word filling area is obtained, the word area may be filled vertically (e.g., the top, middle, or bottom of the word area, and this is not specifically limited), and the height of the filling is the difference between the height of the word filling area and the height of the word area. For example, if the height of the word filling area is 16 and the height of the word area is 10, the height of the filling is 6. The pixel values ​​of the filling may be the pixel values ​​of background pixels other than character pixels within the word area, and the pixel values ​​of the filling may alternatively be preset pixel values, e.g., 0 or 255. This is not specifically limited.

[0261] In the second method, the width of the word filling area is obtained based on the width of the word area, which is n times a preset size, and the word area is filled horizontally to obtain the word filling area, and the width of the filling is the difference between the width of the word filling area and the width of the word area.

[0262] As mentioned above, the width of the word filling area is greater than the width of the word area, and the width of the word filling area is n times the preset size. Based on this, the width of the word filling area can be obtained based on the width of the word area. See the example above. Optionally, if the width of the word area is exactly n times the preset size, the word area does not need to be filled. In this case, the width of the word filling area is equal to the width of the word area. After the width of the word filling area is obtained, the word area may be filled horizontally (e.g., to the left, center, or right of the word area, and this is not specifically limited), and the width of the filling is the difference between the width of the word filling area and the width of the word area. For example, if the width of the word filling area is 24 and the width of the word area is 17, the width of the filling is 7. The pixel values ​​of the filling may be the pixel values ​​of background pixels other than character pixels within the word area, and the pixel values ​​of the filling may alternatively be preset pixel values, e.g., 0 or 255. This is not specifically limited.

[0263] The third method may be a combination of the first and second methods, that is, after the height and width of the word-filling area are obtained, the first two methods are used to fill the word area vertically and horizontally in order to obtain the word-filling area.

[0264] It can be observed that the size of the word-filling region obtained through filling has several relationships with the size of the CU. Therefore, after such a word-filling region is divided, the word-filling region can cover the entire CU, i.e., there are no cases where a word-filling region covers only a part of a CU, and the same CU does not cover two word-filling regions, thereby implementing alignment between the word region and the CU during division.

[0265] Step 803: Obtain the second image based on the word filling area.

[0266] In this embodiment of the present application, a new image, i.e., a second image, can be obtained based on a word-filling region. The second image includes the characters in the original first image. Obtaining the second image includes the following four cases:

[0267] 1. If there is only one word-filling area, the word-filling area will be used as the second image.

[0268] If there is only one word-filling area, that word-filling area may be used directly as the second image.

[0269] 2. If there are multiple word-filling regions, and each word-filling region contains one line of text, the multiple word-filling regions are joined from top to bottom to obtain a second image.

[0270] One word-filling region contains one line of text. In this case, the word-filling regions to which multiple lines of text belong may be joined from top to bottom based on the distribution of lines of text in the first image. Specifically, the word-filling region to which the topmost line of text in the first image belongs is joined to the top, the word-filling region to which the second line of text in the first image belongs is joined to the second line, and the rest may be inferred by analogy. Note that in this embodiment of the present application, joining may be performed alternatively from bottom to top, and the order of joining is not specifically limited.

[0271] 3. If there are multiple word-filling regions, and each of them contains one string, the multiple word-filling regions are joined from left to right to obtain a second image.

[0272] One word filling area contains one string. In this case, the word filling areas to which multiple strings belong may be joined from left to right based on the distribution of the strings in the first image. Specifically, the word filling area to which the string arranged on the leftmost side in the first image belongs is also joined on the leftmost side, the word filling area to which the string arranged in the second column in the first image belongs is also joined in the second column, and the rest may be estimated by analogy. In this embodiment of the present application, it should be noted that the joining may alternatively be performed from right to left, and the joining order is not specifically limited.

[0273] 4. When there are multiple word filling areas and each of the word filling areas contains one character, the multiple word filling areas are joined from left to right and from top to bottom to obtain the second image, and the multiple word filling areas in the same row have the same height or the multiple word filling areas in the same column have the same width.

[0274] One word filling area contains one character. In this case, based on the distribution of the characters in the first image, the word filling areas to which multiple characters belong may be joined from left to right and from top to bottom. The arrangement order of the word filling areas is consistent with the arrangement order of the characters included in the word filling areas in the first image, and the multiple word filling areas in the same row have the same height or the multiple word filling areas in the same column have the same width.

[0275] In this embodiment of the present application, it should be noted that the joining may alternatively be performed in another order, and the joining order is not specifically limited.

[0276] Step 804: Encoding the second image and the word alignment side information to obtain the first bit stream, where the word alignment side information includes the height or width of the word filling area.

[0277] In this embodiment of the present application, standard video coding or IBC coding may be performed on the second image. This is not specifically limited. When IBC coding is used, if a character in the currently coded CU has already appeared in the coded CU, the coded CU may be determined as a predicted block of the currently coded CU. In this way, the probability that the acquired residual block is 0 is very high, thereby improving the coding efficiency of the current CU. Word alignment side information may be coded using an exponential Golomb coding scheme or another coding scheme. This is also not specifically limited. In this embodiment of the present application, the height or width of the word filling area may be coded directly, or n times a preset size of the height or width of the word filling area may be coded. This is also not specifically limited.

[0278] Optionally, word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region in the first image. This information can help the decoder recover the word region in the reconstructed image, thereby improving decoding efficiency.

[0279] In possible embodiments, in order to obtain a third image, the pixel values ​​in a word region within the first image may be filled with preset pixel values, and the third image is encoded in order to obtain a second bitstream.

[0280] In the steps described above, word regions are recognized from the first image, then the word regions are filled to obtain word-filling regions, and the second image is obtained by joining the word-filling regions. It can be seen that the second image contains only the text content from the first image and is encoded separately. However, the first image contains other content in addition to text. Therefore, in order to maintain the integrity of the image, other image content besides text needs to be further processed and encoded.

[0281] In this embodiment of the present application, the pixel values ​​of the pixels in the recognized word region in the original first image may be filled with a preset pixel value (e.g., 0, 1, or 255). This is equivalent to removing the word region from the first image and replacing its pixel values ​​with non-character pixel values. The difference between the third image thus obtained and the first image is that the pixel values ​​of the word region are different. Standard video coding or IBC coding may also be performed on the third image, but this is not specifically limited.

[0282] In this embodiment of the present application, word regions acquired by recognition in the original image are filled such that the size of the acquired word filling region matches the size of the CU for encoding, and alignment between the word filling region and the CU is implemented to avoid classifying pixels of the same character into different CUs as much as possible, thereby improving the accuracy and efficiency of word content compression.

[0283] Figure 9 is a flowchart of process 900 of a word-containing image decoding method according to one embodiment of the present application. Process 900 may be performed by a video decoder 30. Process 900 is described as a series of steps or operations. It should be understood that process 900 is not limited to the execution order shown in Figure 9 and may be performed in various orders and / or simultaneously. It is assumed that the bitstream has been decoded by the video decoder 30 using process 900, which includes the following steps, to reconstruct the image.

[0284] Step 901: Obtain the bitstream.

[0285] The decoder can receive the bitstream from the encoder.

[0286] Step 902: Decode the bitstream to obtain the first image and word alignment side information, the word alignment side information including the height or width of the word filling area contained in the first image, the height or width of the word filling area being n times a preset size, and n ≥ 1.

[0287] The decoder can obtain the first image and word alignment side information through decoding. The decoding method can correspond to the encoding method used on the encoder side. The encoder can encode the image using standard video coding or IBC coding, and the decoder can decode the bitstream using standard video decoding or IBC decoding to obtain the reconstructed image. The encoder can encode the word alignment side information using exponential Golomb coding, and the decoder can decode the bitstream using exponential Golomb decoding to obtain the word alignment side information.

[0288] The height or width of the word filling area is n times the preset size, where n ≥ 1, and the preset size is one of 8, 16, 32, or 64. For details, please refer to the explanation above in step 802. Details will not be explained again here.

[0289] Step 903: Obtain the word filling area based on the height or width of the first image and word filling area.

[0290] After determining the height or width of the word-filling region, the decoder can extract the corresponding pixel values ​​from the first image based on the height or width in order to obtain the word-filling region.

[0291] Step 904: Obtain the word regions within the image to be reconstructed based on the word-filling regions.

[0292] In this embodiment of the present application, the word alignment side information further includes the height and width of the word region. Therefore, based on the height and width of the word region, the corresponding pixel values ​​can be extracted from the word filling region to obtain the word region in the image to be reconstructed. The word region corresponds to the word region in step 801 and may contain one character, or one line of characters, the line of characters may contain multiple characters, or one string of characters, the string of characters may contain multiple characters.

[0293] In possible embodiments, the decoder can decode the bitstream to obtain a second image and obtain a reconstruction target image based on the word region and the second image.

[0294] The word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper left corner of the word region. Based on this, the replacement region in the second image may be determined based on the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and the pixel values ​​in the replacement region are filled with the pixel values ​​in the word region to obtain the image to be reconstructed. This process may be the reverse of the process for obtaining the third image in the embodiment shown in Figure 8, i.e., the process of filling the second image (corresponding to the third image in the embodiment shown in Figure 8) with the word region to obtain the reconstructed image (corresponding to the first image in the embodiment shown in Figure 8). After the horizontal and vertical coordinates of the pixel at the upper left corner of the word region in the second image, as well as the height and width of the word region, are obtained, the replacement region in the second image may be determined, and then the pixel values ​​in the filling region are filled with the pixel values ​​of the word region. This is equivalent to removing the replacement region from the second image and replacing the pixel values ​​in the filling region with the pixel values ​​of the word region.

[0295] In this embodiment of the present application, word region alignment information can be obtained by parsing the bitstream to obtain a character filling region containing characters. The size of the character filling region matches the size of the CU for decoding in order to implement alignment between the word filling region and the CU in order to improve the accuracy and efficiency of word content compression.

[0296] Figure 10 is an exemplary block diagram of the encoding system 100 according to this application. As shown in Figure 10, the encoding system 100 includes an encoder side and a decoder side.

[0297] encoder side 1. Word region detection and segmentation The original image is processed using an existing word detection and segmentation method to output word regions. Any word region may contain one or more characters. If a word region contains multiple characters, the multiple characters are characters from the same character row in the original image. In this embodiment of the present application, a word region may be a rectangular region. Thus, a word region may be referred to as a character box. The position of one or more characters contained in a character box may be represented by the coordinates of the upper-left corner of the character box in the original image and its size (including height, or height and width).

[0298] In this embodiment of the present application, word region detection and segmentation may be implemented using existing methods based on horizontal projection histograms or connected components. Details are as follows:

[0299] (1) Preprocessing: The original image is subjected to binarization and corrosion. In the binarized image, samples corresponding to word regions are white, and samples corresponding to non-word regions are black. Corrosion is used to address pixel adhesion between characters.

[0300] (2) Perform horizontal projection on the preprocessed image and calculate the amount of white sample in the horizontal direction.

[0301] The amount of white samples on the horizontal line containing characters is not zero. Therefore, all lines of characters in the image can be obtained.

[0302] Next, for each character line, the following operations are performed: extract the connected components (a connected component refers to a continuous white sample area, one character is a connected component, and correspondingly, a character line can have multiple connected components), and select a character box of an appropriate size. If a character box contains only one character, the size of the character box is greater than or equal to the size of the largest connected component in the character line, and all characters in the same character line use character boxes of the same size. If a character box contains a character line, the size of the character box is greater than or equal to the sum of the sizes of all connected components in the character line. For example, the connected component of character 1 in a character line occupies a pixel space of 6 (width) × 10 (height), and the connected component of character 2 occupies a pixel space of 11 × 8. In this case, at least 11 × 10 character boxes must be selected to contain the two characters, or at least 17 × 10 character boxes must be selected to contain the two characters in that character line.

[0303] In this embodiment of the present application, the word region detection and segmentation method described above may be implemented by the word processing module shown in Figure 7. It should be understood that the word processing module may also implement word region detection and segmentation by using a deep neural network-based method, but is not specifically limited.

[0304] After the word region is obtained, the pixel values ​​within the word region in the original image are filled with preset pixel values, so the word region can be removed from the original image, thereby obtaining a non-word image from which all characters have been removed.

[0305] 2. Adaptive word alignment The character box is filled within a character filling box having a width Kw and a height Kh, where Kw and Kh may or may not be equal. The HEVC standard includes sizes 8, 16, 32, and 64. In this embodiment of the present application, Kw and Kh may each be multiples of 8, 16, 32, or 64. The character filling box may also be filled with preset pixel values ​​in areas other than those occupied by the character box, such as the background area within the character box.

[0306] It can be found that multiple character filling boxes can be obtained based on multiple character boxes, and that multiple character filling boxes can be joined. Character filling boxes to which characters of the same line in the original image belong are joined to the word region of the line, aligned with the block division grid. If there are multiple lines of characters, when the height of the character filling box for the current line of characters is the same as that of the previous line, the character filling box for the current line of characters may be joined to the right of the word region of the previous line, aligned with the block division grid, or the character filling box for the current line of characters may be joined below the word region of the previous line, aligned with the block division grid. If the height of the character filling box for the current line of characters is different from that of the previous line, the character filling box for the current line of characters may be joined below the word region of the previous line, aligned with the block division grid. After the adaptive alignment of all lines of characters is complete, a fully aligned word region can be obtained to obtain the image frame (which may be called the word image) containing all the characters in the original image. It should be noted that in this embodiment of the present application, the width of the aligned word region does not have to exceed the width of the original image, or may have a different width. This is not specifically limited.

[0307] In this embodiment of the present application, the adaptive word alignment method described above can be implemented by the word processing module shown in Figure 7.

[0308] 3. Word alignment side information encoding In the two steps described above, the width and height of the character box, the width and height of the character filling box, and the vertical and horizontal coordinates of the characters within a line can be obtained. This information helps the decoder to restore the word region to its correct position. Thus, the above information is used as word alignment side information to be encoded in order to obtain a bitstream of word alignment side information. It should be noted that the method for encoding the word alignment side information is not specifically limited in the embodiments of this application.

[0309] In this embodiment of the present application, the aforementioned method for encoding word alignment side information may be implemented by the word alignment side information encoding module shown in Figure 7.

[0310] 4. Standard Encoding The word and non-word images obtained in the steps described above can be encoded using standard video encoding methods. For example, the standard encoders that can be used may be encoders conforming to standards such as H.264, H.265, H.266, AVS3, and AV1. This is not specifically limited to the embodiments of this application.

[0311] In this embodiment of the present application, the aforementioned standard encoding method can be implemented by the standard video encoding module shown in Figure 7.

[0312] 5. Bitstream transmission The encoder transmits the bitstream to the decoder.

[0313] In this embodiment of the present application, the bitstream transmission method described above can be implemented by the transmission module shown in Figure 7.

[0314] 6. Word alignment side information decoding The decoder decodes the bitstream using a decoding method corresponding to the encoder in order to obtain word alignment side information, which may include the width and height of character boxes, the width and height of character filling boxes, and the vertical and horizontal coordinates of characters within a line.

[0315] In this embodiment of the present application, the aforementioned method for decoding word alignment side information may be implemented by the word alignment side information decoding module shown in Figure 7.

[0316] 7. Standard Decryption The decoder decodes the bitstream using the decoding method corresponding to the encoder in order to obtain the reconstructed word image and the reconstructed non-word image.

[0317] In this embodiment of the present application, the above-mentioned standard decoding method can be implemented by the standard video decoding module shown in Figure 7.

[0318] 8. Word Recovery Based on the word alignment side information, the text filling box is first obtained from the word image, and then the text box is extracted from the text filling box.

[0319] 9. Area joining The corresponding locations within the reconstructed non-word image are found based on the coordinate information in the word alignment side information, and the pixel values ​​in the character frame are used to replace the pixel values ​​of the found corresponding locations within the reconstructed non-word image to obtain the reconstructed image.

[0320] In this embodiment of the present application, the word recovery method described above can be implemented by the word reconstruction module shown in Figure 7.

[0321] Based on the encoding system 100 shown in Figure 10, the technical solutions of the embodiments of the aforementioned method will be described in detail below using several specific embodiments.

[0322] Embodiment 1 encoder side Figure 11a shows word region detection and segmentation. As shown in Figure 11a, the original frame is

number

[0323]

number

[0324] The pixel values ​​within the word area corresponding to each of the four text boxes are obtained, resulting in four images containing the text. The resolution of these images is lower than that of the original frame, and the size of the images corresponds to the size of the text boxes.

[0325] Next, the pixel values ​​within the word regions corresponding to each of the four character boxes in the original frame are filled with pre-set pixel values. For example, the pre-set pixel values ​​may be 0 or 1, or they may be the pixel values ​​of the background region within the character boxes other than the character pixels. This is not specifically limited. The image obtained through refilling does not contain any characters and is therefore sometimes referred to as a non-word image (or non-word region).

[0326] Figure 11b is a diagram of adaptive word alignment. As shown in Figure 11b, taking the original frame in Figure 11a, which is arranged from top to bottom and left to right (zigzag shape),

number

[0327] Character filling boxes containing characters from the same line are joined to the word area of ​​the line, aligned with the block division grid. For example,

number

[0328] In the process described above, in order to restore the characters within the aligned word area to their positions in the original frame, word alignment side information needs to be output and sent to the decoder. Word alignment side information is obtained by simply performing a data conversion on the character box position and includes the following elements (Figures 11a and 11b are used as examples). (1) The height H (i.e., 10) and width W (i.e., 11) of the text box, and the size K (i.e., 16) of the text filling box. (2) The vertical coordinates y of the upper left corner of the character line are 15, 15, 15, and 15. The vertical coordinates of characters in the same character line are the same. Therefore, the vertical coordinates of multiple characters within the same character line can be simplified to represent the same information, for example, the vertical coordinates of the first character in the character line (i.e., 15) and the number of characters (i.e., 4) are recorded. (3) The horizontal coordinates x of the upper left corner of the character line are 2, 14, 25, and 36. The difference between the horizontal coordinates of multiple characters is 12, 11, and 11, and it can be seen that the data is close to the width W of the character box. Therefore, the horizontal coordinates of multiple characters within the same character line can be simplified to represent the same information. For example, the horizontal coordinate of the first character of the character line (i.e., 2) and the difference between the horizontal coordinates (i.e., 12, 11, and 11) are recorded, or the result E obtained by subtracting the horizontal coordinate of the first character of the character line (i.e., 2) and the difference between the horizontal coordinates (i.e., 12, 11, and 11) by W is recorded (i.e., E1, E2, E3 = [12-11], [11-11], [12-11] = 1, 0, 0). By recording this data, the horizontal coordinate of the upper left corner of the character line can be reconstructed. For example, the horizontal coordinate of the second character = the horizontal coordinate of the first character + W + E1 = 2 + 11 + 1 = 14, and the rest can be inferred by analogy.

[0329] The word alignment side information includes H, W, K, the horizontal and vertical coordinates of the first character of each character line, and the number of characters in each line, E. H, W, K, and the number of characters in each line are encoded in binary mode. The value of E is close to 0, and the exponential Golomb coding scheme is used. For the horizontal and vertical coordinates of the first character of each character line, binary coding can first be performed on the horizontal and vertical coordinates of the first character of the first character line. For each line, the difference between the horizontal and vertical coordinates of the first character of the line and the horizontal and vertical coordinates of the first character of the previous line is calculated, and then exponential Golomb coding is performed on the coordinate difference.

[0330] It should be noted that the aforementioned method for encoding word alignment side information is merely an example. In this embodiment of the present application, word alignment side information may be encoded by other means, but is not specifically limited.

[0331] In this embodiment of the present application, non-word images and word images can be encoded and decoded, respectively, using standard encoding / decoding schemes. Further details are not described here.

[0332] Decoder side The decoder decodes the bitstream using the decoding method corresponding to the encoder in order to obtain word alignment side information.

[0333] Figure 11c is a diagram of word reconstruction. As shown in Figure 11c, based on the word alignment side information, K x K character filling boxes are extracted from the reconstructed aligned word region (reconstructed word image). Then, based on the word alignment side information, four W x H character boxes are extracted in a zigzag order from the four character filling boxes. Each of the four character boxes is:

number

[0334] In zigzag order, the pixel values ​​within the four character boxes are filled into their corresponding positions in the reconstructed non-word image based on the coordinates recorded in the alignment side information. For example,

number

number

number

number

[0335] Embodiment 2 The difference from Embodiment 1 is that in Embodiment 1, adaptive word alignment is performed in both the width and height directions, whereas in this embodiment, adaptive word alignment is performed only in the height direction.

[0336] Figure 12a shows word region detection and segmentation. As shown in Figure 12a, the original frame is

number

[0337] The pixel values ​​within the word area corresponding to the text box are obtained, resulting in an image containing four characters. The image resolution is lower than that of the original frame, and the image size corresponds to the size of the text box.

[0338] Next, the pixel values ​​within the word region corresponding to the text box in the original frame are filled with a preset pixel value. For example, the preset pixel value may be 0 or 1, or it may be the pixel value of the background region within the text box, excluding the text pixels. This is not specifically limited. The image obtained through refilling does not contain text and is therefore sometimes referred to as a non-word image (or non-word region).

[0339] Figure 12b is a diagram of adaptive word alignment. As shown in Figure 12b, the character box is filled into a character filling box with height K, starting from top to bottom. Since K is selected as 16 based on the height of the character box in Figure 12a, the size of the character filling box is 45 × 16. In this embodiment of the present application, the filling is performed on the lower side of the character box to reach the size of the character filling box. The pixel values ​​of the filling may be the pixel values ​​of the background area within the character box, or they may be preset pixel values ​​(e.g., 0 or 1). This is not specifically limited. Note that the height and width of the character filling box may be different and are set based on the size of the CU. For example, if the size of the CU is 16, K may be 1x 16 (i.e., 16) in this embodiment; if the size of the CU is 8, K may be 2x 8 (i.e., 16) in this embodiment; if the size of the CU is 32, K may be 1x 32 (i.e., 32) in this embodiment; or if the size of the CU is 64, K may be 1x 64 (i.e., 64) in this embodiment.

[0340] In the process described above, in order to restore the characters within the aligned word area to their positions in the original frame, word alignment side information needs to be output and sent to the decoder. Word alignment side information is obtained by simply performing a data conversion on the character box position and includes the following elements (Figures 12a and 12b are used as examples). (1) The height H (i.e., 10) and width W (i.e., 45) of the text box, and the height K (i.e., 16) of the text filling box. (2) Vertical coordinate y:15 of the upper left corner of the text line (3) Horizontal coordinate x:2 of the upper left corner of the text line

[0341] Word alignment side information includes H, W, K, and the horizontal and vertical coordinates of the first character of a character line. H, W, and K are encoded in binary mode. For the horizontal and vertical coordinates of the first character of a character line, binary encoding can first be performed on the horizontal and vertical coordinates of the first character of the first character line. For another line, the difference between the horizontal and vertical coordinates of the first character of the character line and the horizontal and vertical coordinates of the first character of the previous line is calculated, and then exponential Golomb encoding is performed on the coordinate difference.

[0342] It should be noted that the aforementioned method for encoding word alignment side information is merely an example. In this embodiment of the present application, word alignment side information may be encoded by other means, but is not specifically limited.

[0343] In this embodiment of the present application, non-word images and word images can be encoded and decoded, respectively, using standard encoding / decoding schemes. Further details are not described here.

[0344] Decoder side The decoder decodes the bitstream using the decoding method corresponding to the encoder in order to obtain word alignment side information.

[0345] Based on the word alignment side information, 45 × K character filling boxes are extracted from the reconstructed aligned word region (reconstructed word image). Then, based on the word alignment side information, W × H character boxes are extracted from the character filling boxes. The character boxes are,

number

[0346] The pixel values ​​within the text box are filled into the corresponding positions in the reconstructed non-word image based on the coordinates recorded in the alignment side information. For example, in a reconstructed non-word image,

number

[0347] Embodiment 3 The difference from Embodiment 2 is that the original frame contains multiple lines of text. In word region detection and splitting, multiple lines of text are detected in advance. Therefore, during splitting, multiple lines of text need to be further split to obtain multiple word regions, each word region containing one line of text.

[0348] Next, the aforementioned multiple word regions are encoded / decoded separately using the method of Embodiment 2. Further details will not be explained here.

[0349] Embodiment 4 The difference from Embodiment 1 is that the original frame contains multiple lines of characters. In word region detection and splitting, multiple lines of characters are detected in advance. Therefore, during splitting, multiple lines of characters need to be further split to obtain multiple lines of characters, and then each line of characters is split to obtain multiple word regions, each word region containing one character.

[0350] Next, the aforementioned multiple word regions are encoded / decoded separately using the method of Embodiment 1. Further details will not be explained here.

[0351] Figure 13 is an illustrative diagram of the structure of an encoding device 1300 according to one embodiment of the present application. As shown in Figure 13, the encoding device 1300 of this embodiment may be used on the encoder side 20. The encoding device 1300 may include an acquisition module 1301, a filling module 1302, and an encoding module 1303.

[0352] The acquisition module 1301 is configured to acquire a word region in a first image, where the word region contains at least one character. The filling module 1302 is configured to fill the word region to acquire a word filling region, where the height or width of the word filling region is n times a preset size, n ≥ 1, and to acquire a second image based on the word filling region. The encoding module 1303 is configured to encode the second image and word alignment side information to acquire a first bitstream, where the word alignment side information includes the height or width of the word filling region.

[0353] In possible embodiments, the filling module 1302 is specifically configured such that it obtains the height of a word filling area based on the height of the word area, the height of the word filling area is n times a preset size, and it fills the word area vertically to obtain the word filling area, and the height of the filling is the difference between the height of the word filling area and the height of the word area.

[0354] In possible embodiments, the filling module 1302 is specifically configured such that it obtains the width of a word filling area based on the width of the word area, the width of the word filling area is n times a preset size, and it fills the word area horizontally to obtain the word filling area, and the width of the filling is the difference between the width of the word filling area and the width of the word area.

[0355] In possible embodiments, a word region may contain one character, or a word region may contain one line of characters, and the line of characters may contain multiple characters, or a word region may contain one string of characters, and the string of characters may contain multiple characters.

[0356] In possible embodiments, the acquisition module 1301 is specifically configured to perform character recognition on a first image in order to acquire a word region.

[0357] In possible embodiments, the acquisition module 1301 is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region containing multiple character lines, and to perform line splitting on the intermediate word region for each character line to acquire a word region.

[0358] In possible embodiments, the acquisition module 1301 is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region containing multiple strings, and to perform string-by-string column splitting on the intermediate word region to acquire a word region.

[0359] In possible embodiments, the acquisition module 1301 is specifically configured to perform character recognition on a first image to acquire an intermediate word region, the intermediate word region comprising multiple character lines or multiple strings, and to perform character-by-character row splitting and column splitting on the intermediate word region to acquire a word region.

[0360] In possible embodiments, the filling module 1302 is specifically configured to use the word filling region as a second image if there is only one word filling region.

[0361] In possible embodiments, the filling module 1302 is specifically configured to join multiple word filling regions from top to bottom in order to obtain a second image, provided that multiple word filling regions exist and each word filling region contains one line of characters.

[0362] In possible embodiments, the filling module 1302 is specifically configured to join multiple word filling regions from left to right to obtain a second image, given that multiple word filling regions exist and each word filling region contains one string.

[0363] In possible embodiments, the filling module 1302 is configured such that, if there are multiple word filling areas and each word filling area contains one character, it joins the multiple word filling areas from left to right and from top to bottom in order to obtain a second image, and the multiple word filling areas in the same row have the same height or the multiple word filling areas in the same column have the same width.

[0364] In possible embodiments, the word alignment side information further includes the height and width of the word region, as well as the horizontal and vertical coordinates of the pixel at the upper-left corner of the word region in the first image.

[0365] In possible embodiments, the preset size is one of 8, 16, 32, and 64.

[0366] In possible embodiments, the filling module 1302 is further configured to fill pixel values ​​in word regions within the first image with preset pixel values ​​in order to obtain a third image, and the encoding module 1303 is further configured to encode the third image in order to obtain a second bitstream.

[0367] The apparatus in this embodiment may be used to carry out the technical solution of the method embodiment shown in Figure 8. The principle and technical effect of the apparatus embodiment are the same as those of the technical solution of the method embodiment shown in Figure 8. Further details are not described here.

[0368] Figure 14 is an illustrative diagram of the structure of a decoding device 1400 according to one embodiment of the present application. As shown in Figure 14, the decoding device 1400 of this embodiment may be used on the decoder side 30. The decoding device 1400 may include an acquisition module 1401, a decoding module 1402, and a reconstruction module 1403.

[0369] The acquisition module 1401 is configured to acquire a bitstream. The decoding module 1402 is configured to decode the bitstream to acquire a first image and word alignment side information, the word alignment side information including the height or width of a word filling region contained in the first image, the height or width of the word filling region being n times a preset size, and n ≥ 1. The reconstruction module 1403 is configured to acquire a word filling region based on the first image and the height or width of the word filling region, and to acquire a word region in the image to be reconstructed based on the word filling region.

[0370] In possible embodiments, the word alignment side information further includes the height and width of the word region, and the reconstruction module 1403 is specifically configured to extract corresponding pixel values ​​from the word filling region based on the height and width of the word region in order to obtain the word region.

[0371] In possible embodiments, the decoding module 1402 is further configured to decode the bitstream to obtain a second image, and the reconstruction module 1403 is further configured to obtain a reconstruction target image based on the word region and the second image.

[0372] In possible embodiments, the word alignment side information further includes the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and the reconstruction module 1403 is specifically configured to determine a replacement region in the second image based on the height and width of the word region and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and to fill in the pixel values ​​in the replacement region with the pixel values ​​in the word region in order to obtain the image to be reconstructed.

[0373] In possible embodiments, a word region may contain one character, or a word region may contain one line of characters, and the line of characters may contain multiple characters, or a word region may contain one string of characters, and the string of characters may contain multiple characters.

[0374] In possible embodiments, the preset size is one of 8, 16, 32, and 64.

[0375] The apparatus in this embodiment may be used to carry out the technical solution of the method embodiment shown in Figure 9. The principle and technical effect of the apparatus embodiment are the same as those of the technical solution of the method embodiment shown in Figure 9. Further details are not described here.

[0376] In one embodiment of the process, the steps in the above-described method embodiment can be implemented by using hardware integrated logic circuits in a processor or by using instructions in software form. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application may be directly provided to be performed and completed by a hardware encoding processor or by a combination of the hardware of the encoding processor and a software module. The software module may be located in a mature storage medium of the art, such as random-access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory, and the processor reads the information from memory and performs the steps in the aforementioned manner together with the processor hardware.

[0377] The memory in the above-described embodiment may be volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) used as an external cache. Many forms of RAM may be used, not as an example but as an example, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchlink dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM). Note that the memory of the systems and methods described herein includes, but is not limited to, these memories and any other suitable type of memory.

[0378] Those skilled in the art will recognize that the units and algorithmic steps in the examples described with reference to the embodiments disclosed herein may be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether the functions are implemented by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the embodiments should not be considered to exceed the scope of this application.

[0379] For the sake of brevity, it will be readily apparent to those skilled in the art that the detailed operating processes of the aforementioned systems, apparatus, and units can be described by referring to the corresponding processes in the method embodiments described above. Further details are not provided here.

[0380] It should be understood that in some embodiments provided in this application, the disclosed systems, apparatus, and methods may be implemented in other ways. For example, the embodiments of the apparatus described above are merely examples. For example, the division into units is merely a logical functional division, and other divisions may be used in actual embodiments. For example, multiple units or components may be combined or integrated to form another system, and some features may be ignored or not performed. In addition, the interconnections or direct connections or communication connections presented or described may be implemented through some interfaces. Indirect connections or communication connections between apparatus or units may be implemented in an electrical, mechanical, or other form.

[0381] Units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units. Specifically, components may be located in one place or distributed across multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0382] In addition, the functional units of the embodiments of this application may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit.

[0383] If the functionality is implemented in the form of a software function unit and sold or used as an independent product, the functionality may be stored on a computer-readable storage medium. Based on this understanding, the technical solutions of this application, either in essence, or contributing to the prior art, or a part of the technical solutions, may be implemented in the form of a software product. The software product is stored on a storage medium and includes several instructions for instructing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the method in the embodiments of this application. The storage medium includes any medium capable of storing program code, such as a USB flash drive, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0384] The foregoing description is merely a specific embodiment of the present application and is not intended to limit the scope of protection of this application. Any modifications or substitutions that are readily conceivable by a person skilled in the art within the scope of the art disclosed herein shall also fall within the scope of protection of this application. Accordingly, the scope of protection of this application shall be subject to the scope of protection of the claims. [Explanation of symbols]

[0385] 2 Output Third Layer Index 4. Second input index 10 Video Encoding Systems 12. Source device 13 Communication Channels 14 Destination device 16 Image Sources 17 Images, Image Data 18. Preprocessor, pre-processing unit 19 Preprocessed image data 20 Video Encoders 21 Encoded image data 22 Communication Interfaces 25 Training Engines 28 Communication Interfaces 30 video decoders 31 Decoded image data 32 Post-processors, Post-processing Units 33 Post-processed image data 34 Display Devices 40 Video Encoding Systems 41 Imaging devices 42 Antennas 43 processors 44 memory 45 Display Devices 46 Processing Circuit 70 coding systems 100 coding systems 201 Input terminal, input interface 203 Image Block 204 Residual Calculation Unit 205 Residual Block 206 Conversion Processing Unit 207 Conversion coefficient 208 Quantization Units 209 Quantization coefficient, Quantization residual coefficient 210 Inverse Quantization Unit 211 Inverse quantization coefficients, inverse quantization residual coefficients 212 Inverse Transform Processing Unit 213 Reconstructed residual blocks, transformed blocks 214 Reconfiguration Unit, Adder 215 Reconstructed Blocks 220 Loop Filter Unit 221 Reconstructed Blocks 230 Decoded image buffer 231 decrypted images 244 Interpretation Units 246 Interpretation Block 254 Intra Prediction Units 260 Mode Selection Unit 262 divided units 265 predicted blocks 266 Syntax Elements 270 Entropy Coding Units 272 Output terminal, output interface 304 Entropy Decoding Unit 309 Quantization coefficient 310 Inverse Quantization Unit 311 Inverse Quantization Coefficient 312 Inverse Transform Processing Unit 313 Reconstructed residual blocks, transformed blocks 314 Reconfiguration Unit, Adder 315 Reconfigured Blocks 320 Loop Filter Unit 321 filtered blocks 330 Decoded image buffer 331 decrypted images 332 Output terminal 344 Interpretation Units 354 Intra Prediction Units 360 Mode Applicable Unit 365 predicted blocks 400 video encoding devices 410 Inlet port, Input port 420 Receiver Unit 440 Transmitter Unit 450 exit ports, output ports 460 memory 470 Encoding Module 500 devices 502 Processors 504 memory 506 Codes and Data 508 Operating Systems 510 Applications 512 Bus 518 displays 800 processes 1300 encoding device 1301 Acquisition Module 1302 Filling Module 1303 Encoding Module 1400 Decoder 1401 Acquisition Module 1402 Decoding Module 1403 Reconfiguration Module

Claims

1. A word-containing image encoding method, A step of obtaining a word region in a first image, wherein the word region contains at least one character; A step of filling the aforementioned word area to obtain a word filling area, wherein the height or width of the word filling area is n times a preset size, and n ≥ 1. The steps include: obtaining a second image based on the word-filling region; A step of encoding the second image and word alignment side information to obtain a first bitstream, wherein the word alignment side information includes the height or width of the word filling area. Methods that include...

2. The step of filling the aforementioned word area to obtain a word filling area is: A step of obtaining the height of the word filling area based on the height of the word area, wherein the height of the word filling area is n times the preset size, A step of obtaining a word filling area by filling the word area vertically, wherein the height of the filling is the difference between the height of the word filling area and the height of the word area. The method according to claim 1, including the method described in claim 1.

3. The step of filling the aforementioned word area to obtain a word filling area is: A step of obtaining the width of the word filling area based on the width of the word area, wherein the width of the word filling area is n times the preset size, A step of obtaining a word filling area by filling the word area horizontally, wherein the width of the filling is the difference between the width of the word filling area and the width of the word area. The method according to claim 1 or 2, including the method described in claim 1 or 2.

4. The aforementioned word area contains one character, or The word area contains one line of characters, and the line of characters contains multiple characters, or The aforementioned word area contains one string, and the aforementioned string contains multiple characters. The method according to claim 1 or 2.

5. The step of obtaining the word region within the first image is: Steps to perform character recognition on the first image and obtain the word region. The method according to claim 1 or 2, including the method described in claim 1 or 2.

6. The step of obtaining the word region within the first image is: A step of performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region includes multiple character lines, The steps include: performing line splitting on each character line of the intermediate word area to obtain the word area; The method according to claim 1 or 2, including the method described in claim 1 or 2.

7. The step of obtaining the word region within the first image is: A step of performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region includes a plurality of character strings, The steps include: obtaining the word area by performing column splitting on the intermediate word area for each string; The method according to claim 1 or 2, including the method described in claim 1 or 2.

8. The step of obtaining the word region within the first image is: A step of performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region includes multiple character lines or multiple character strings, The steps include: obtaining the word region by performing row splitting and column splitting on each character of the intermediate word region; The method according to claim 1 or 2, including the method described in claim 1 or 2.

9. The step of obtaining a second image based on the word-filling region is: If only one word-filling region exists, the step of using the word-filling region as the second image. The method according to claim 1 or 2, including the method described in claim 1 or 2.

10. The step of obtaining a second image based on the word-filling region is: If there are multiple word-filling regions, and each of the multiple word-filling regions contains one line of text, the step of joining the multiple word-filling regions from top to bottom to obtain the second image. The method according to claim 1 or 2, including the method described in claim 1 or 2.

11. The step of obtaining a second image based on the word-filling region is: If there are multiple word-filling regions, and each of the multiple word-filling regions contains one string, the step of joining the multiple word-filling regions from left to right to obtain the second image. The method according to claim 1 or 2, including the method described in claim 1 or 2.

12. The step of obtaining a second image based on the word-filling region is: The step of obtaining the second image by joining the multiple word filling areas from left to right and from top to bottom, where multiple word filling areas in the same row have the same height, or multiple word filling areas in the same column have the same width. The method according to claim 1 or 2, including the method described in claim 1 or 2.

13. The method according to claim 1 or 2, wherein the word alignment side information further includes the height and width of the word region, and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region in the first image.

14. The method according to claim 1 or 2, wherein the preset size is one of 8, 16, 32, and 64.

15. After the step of obtaining the word region in the first image, the method, A step of obtaining a third image by filling the pixel values ​​in the word region of the first image with pre-set pixel values, The steps include: encoding the third image to obtain a second bitstream; The method according to claim 1 or 2, further comprising:

16. A method for decoding images containing words, Steps to obtain a bitstream, A step of decoding the bitstream to obtain a first image and word alignment side information, wherein the word alignment side information includes the height or width of a word filling region included in the first image, and the height or width of the word filling region is n times a preset size, and n ≥ 1. A step of obtaining the word filling area based on the first image and the height or width of the word filling area, The steps include obtaining word regions within the image to be reconstructed based on the word-filling region, and Methods that include...

17. The word alignment side information further includes the height and width of the word area, The step of obtaining a word region in the image to be reconstructed based on the word filling region is: The step of obtaining the word region by extracting corresponding pixel values ​​from the word filling region based on the height and width of the word region. The method according to claim 16, including the method described in claim 16.

18. The steps include decoding the bitstream to obtain a second image, A step of acquiring the image to be reconstructed based on the word region and the second image. The method according to claim 17, further comprising:

19. The word alignment side information further includes the horizontal and vertical coordinates of the pixel at the upper left corner of the word region. The step of obtaining the image to be reconstructed based on the word region and the second image is: A step of determining a replacement region in the second image based on the height and width of the word region and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, The steps include: filling the pixel values ​​in the replacement region with the pixel values ​​in the word region to obtain the image to be reconstructed; The method according to claim 18, including the method described in claim 18.

20. The aforementioned word area contains one character, or The word area contains one line of characters, and the line of characters contains multiple characters, or The aforementioned word area contains one string, and the aforementioned string contains multiple characters. The method according to claim 16 or 17.

21. The method according to claim 16 or 17, wherein the preset size is one of 8, 16, 32, and 64.

22. An encoding device, An acquisition module configured to acquire a word region within a first image, wherein the word region contains at least one character, A filling module configured to fill the aforementioned word area to obtain a word filling area, and to obtain a second image based on the word filling area, wherein the height or width of the word filling area is n times a preset size, and n ≥ 1, and the filling module An encoding module configured to encode the second image and word alignment side information to obtain a first bitstream, wherein the word alignment side information includes the height or width of the word filling area, and A device equipped with the following features.

23. The aforementioned filling module is The method involves obtaining the height of the word filling area based on the height of the word area, wherein the height of the word filling area is n times the preset size. The method involves filling the word area vertically to obtain the word filling area, wherein the height of the filling is the difference between the height of the word filling area and the height of the word area. The apparatus according to claim 22, further configured to perform the following:

24. The aforementioned filling module is The width of the word filling area is obtained based on the width of the word area, wherein the width of the word filling area is n times the preset size. The method involves filling the word area horizontally to obtain the word filling area, wherein the width of the filling is the difference between the width of the word filling area and the width of the word area. The apparatus according to claim 22 or 23, further configured to perform the following:

25. The aforementioned word area contains one character, or The word area contains one line of characters, and the line of characters contains multiple characters, or The aforementioned word area contains one string, and the aforementioned string contains multiple characters. The apparatus according to claim 22 or 23.

26. The apparatus according to claim 22 or 23, wherein the acquisition module is further configured to perform character recognition on the first image to acquire the word region.

27. The acquisition module described above is Performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region includes multiple character lines, The intermediate word area is split into lines for each character line to obtain the word area. The apparatus according to claim 22 or 23, further configured to perform the following:

28. The acquisition module described above is Performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region contains multiple character strings, The intermediate word area is split into columns for each string to obtain the word area. The apparatus according to claim 22 or 23, further configured to perform the following:

29. The acquisition module described above is The process involves performing character recognition on the first image to obtain an intermediate word region, wherein the intermediate word region includes multiple character lines or multiple character strings. The intermediate word area is obtained by performing row splitting and column splitting for each character. The apparatus according to claim 22 or 23, further configured to perform the following:

30. The apparatus according to claim 22 or 23, wherein the filling module is further configured to use the word filling region as the second image when only one word filling region is present.

31. The apparatus according to claim 22 or 23, wherein the filling module is further configured to join the multiple word filling regions from top to bottom to obtain the second image, if there are multiple word filling regions and each of the multiple word filling regions contains one line of characters.

32. The apparatus according to claim 22 or 23, wherein the filling module is further configured to obtain the second image by joining the multiple word filling regions from left to right, if there are multiple word filling regions and each of the multiple word filling regions contains one string.

33. The apparatus according to claim 22 or 23, wherein the filling module is further configured to obtain the second image by joining the multiple word filling areas from left to right and from top to bottom, where multiple word filling areas in the same row have the same height or multiple word filling areas in the same column have the same width.

34. The apparatus according to claim 22 or 23, wherein the word alignment side information further includes the height and width of the word region, and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region in the first image.

35. The apparatus according to claim 22 or 23, wherein the preset size is one of 8, 16, 32, and 64.

36. The filling module is further configured to fill the pixel values ​​in the word region of the first image with preset pixel values ​​to obtain a third image, The encoding module is further configured to encode the third image to obtain a second bitstream. The apparatus according to claim 22 or 23.

37. A decoding device, A capture module configured to acquire a bitstream, A decoding module configured to decode the bitstream to obtain a first image and word alignment side information, wherein the word alignment side information includes the height or width of a word filling region included in the first image, and the height or width of the word filling region is n times a preset size, and n ≥ 1, and the decoding module A reconstruction module configured to acquire the word filling region based on the first image and the height or width of the word filling region, and to acquire the word region in the image to be reconstructed based on the word filling region, A device equipped with the following features.

38. The word alignment side information further includes the height and width of the word area, The apparatus according to claim 37, wherein the reconstruction module is further configured to obtain the word region by extracting corresponding pixel values ​​from the word filling region based on the height and width of the word region.

39. The decoding module is further configured to decode the bitstream to obtain a second image, The reconstruction module is further configured to acquire the image to be reconstructed based on the word region and the second image. The apparatus according to claim 38.

40. The word alignment side information further includes the horizontal and vertical coordinates of the pixel at the upper left corner of the word region. The apparatus according to claim 39, wherein the reconstruction module is further configured to determine a replacement region in the second image based on the height and width of the word region and the horizontal and vertical coordinates of the pixel at the upper left corner of the word region, and to obtain the image to be reconstructed by filling the pixel values ​​in the replacement region with the pixel values ​​in the word region.

41. The aforementioned word area contains one character, or The word area contains one line of characters, and the line of characters contains multiple characters, or The aforementioned word area contains one string, and the aforementioned string contains multiple characters. The apparatus according to claim 37 or 38.

42. The apparatus according to claim 37 or 38, wherein the preset size is one of 8, 16, 32, and 64.

43. One or more processors, Memory configured to store one or more programs and Equipped with, When the one or more programs are executed by the one or more processors, the one or more processors become capable of carrying out the method described in claim 1. encoder.

44. One or more processors, Memory configured to store one or more programs and Equipped with, When the one or more programs are executed by the one or more processors, the one or more processors become capable of carrying out the method according to claim 16. Decoder.

45. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on the computer, the computer becomes capable of carrying out the method according to claim 1.

46. A computer program comprising instructions, wherein when the instructions are executed on a computer or processor, the computer or processor becomes capable of carrying out the method according to claim 1.

47. A method for storing a bitstream, The steps include receiving a bitstream with at least one receiver, A step of storing the bitstream in at least one memory, wherein the bitstream is generated by the method of claim 1, and Methods that include...

48. A device for storing a bitstream, At least one receiver configured to receive a bitstream, At least one memory configured to store the bitstream, wherein the bitstream is generated by the method described in claim 1, and A device equipped with the following features.

Images