Video enlargement apparatus and program

The video enlarging device addresses encoding artifacts in super-resolution processing by using parameter-based threshold adjustments and deblocking, achieving high-quality enlarged videos with reduced block distortion.

JP2024159663APending Publication Date: 2024-11-08NIPPON HOSO KYOKAI +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024071801
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-28
Filing Date
2024-04-25
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing video super-resolution technologies suffer from encoding artifacts such as block distortion when enlarging decoded videos, particularly in high compression scenarios.

Method used

A video enlarging device that performs super-resolution processing on decoded videos, utilizing parameter acquisition, frequency decomposition, alignment, and allocation units to generate high-quality enlarged images by adjusting threshold values based on quantization parameters and motion vectors, and optionally includes deblocking processing to reduce block noise.

Benefits of technology

Suppresses encoding artifacts, resulting in high-quality enlarged videos with improved image quality by minimizing block distortion and enhancing overall video clarity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159663000001_ABST
    Figure 2024159663000001_ABST
Patent Text Reader

Abstract

To improve image quality of an enlarged video obtained by enlarging a decoded video.SOLUTION: A video enlargement apparatus 1 includes: a parameter acquisition unit 11 for acquiring parameter information; a frequency decomposition unit 12 for performing frequency decomposition processing on each image of a decoded video to generate a frequency-decomposed image; a positioning unit 13 for generating positioning information indicating a displacement amount in small block units between a low-frequency component image of a reference image and an image to be enlarged; an allocation unit 14 for generating an initial image in which the image to be enlarged is a low-frequency component image, and allocating the high-frequency component image of the reference image to a high-frequency band of the initial image according to the positioning information; and a frequency reconfiguration unit 15 for performing frequency reconfiguration processing to an allocation image to generate an enlarged video, where the allocation unit 14 performs allocation processing only when a square error sum or an absolute value error sum of the small blocks of a positioning source and a positioning destination is equal to or less than a threshold, and changes the threshold according to the parameter information.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image enlargement device and a program for enlarging an image. [Background technology]

[0002] Video coding methods such as H.265 / HEVC (High Efficiency Video Coding) (hereinafter referred to as "HEVC") and H.266 / VVC (Versatile Video Coding) (hereinafter referred to as "VVC") achieve high efficiency by combining component technologies such as block division, orthogonal transform, quantization, entropy coding, intra-frame prediction, and inter-frame prediction.

[0003] On the other hand, as super-resolution techniques for converting an original image into a higher resolution image, generally known are single-frame super-resolution processing, which performs super-resolution processing using a single frame, and multi-frame super-resolution processing, which performs super-resolution processing using multiple frames. For example, Non-Patent Document 1 introduces an overview and trends of related technologies for super-resolution processing techniques, dividing them into single-frame super-resolution and multi-frame super-resolution. Patent Document 1 also discloses a technique for improving coding efficiency by using downsampling and super-resolution processing in a coding device that utilizes scalability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-182776 A [Non-patent literature]

[0005] [Non-Patent Document 1] Masatoshi Okutomi, Masayuki Tanaka, Hidenori Takeshima, Nobuyuki Matsumoto, "Latest Trends in Image Super-Resolution Processing Technology", Journal of the Institute of Electronics, Information and Communication Engineers, vol. 93, no. 8, pp. 693-698, Aug. 2010. Summary of the Invention [Problem to be solved by the invention]

[0006] Systems that perform super-resolution enlargement of images as a post-decoding process can display images such as 4K broadcasts in high quality on 8K monitors. In addition, it is possible to perform ultra-high compression encoding by enlarging images that have been reduced in size after decoding. However, encoding artifacts such as block distortion can occur in images that have been enlarged after decoding.

[0007] In view of the above circumstances, an object of the present invention is to provide an image enlargement device and a program capable of improving the image quality of an enlarged image obtained by enlarging a decoded image. [Means for solving the problem]

[0008] The gist and configuration of the present invention to achieve the above object is as follows.

[0009] (1) A video enlargement device that enlarges a decoded image that has been encoded and then decoded by super-resolution processing, the video enlargement device comprising: a parameter acquisition unit that acquires parameter information when encoding and decoding processes are performed; a frequency decomposition unit that performs frequency decomposition processing on each image of the decoded image to generate a frequency resolved image; a positioning unit that generates positioning information indicating a positional deviation amount on a small block basis between a low-frequency component image of one or more reference images including an image to be enlarged in the decoded image and the enlarged image; an allocation unit that generates an initial image in which the image to be enlarged is a low-frequency component image, and allocates a high-frequency component image of the reference image to a high-frequency band of the initial image in accordance with the positioning information to generate an allocated image; and a frequency reconstruction unit that performs frequency reconstruction processing on the allocated image to generate an enlarged image, wherein the allocation unit performs the allocation processing only if the sum of squared errors or the sum of absolute error between the small blocks of the source and destination of the alignment is equal to or less than a threshold, and changes the threshold in accordance with the parameter information.

[0010] (2) The video enlargement device described in (1), wherein the parameter information includes QP information indicating a quantization parameter for each divided block, which is a processing unit for encoding and decoding, or motion vector information indicating a motion vector in inter-frame prediction, and the allocation unit reduces the threshold value as the value of the quantization parameter indicated by the QP information or the magnitude of the motion vector indicated by the motion vector information becomes larger.

[0011] (3) The image enlargement device according to (1) or (2), wherein the allocation unit reduces the threshold value as the distance between the source picture and the destination picture increases.

[0012] (4) The image enlargement device according to any one of (1) to (3), wherein the position matching unit determines the amount of positional shift by performing block matching between the low-frequency component image and the enlarged image in small block units.

[0013] (5) A video enlargement device as described in any of (1) to (4), wherein the parameter information includes motion vector information indicating a motion vector in inter-frame prediction for each divided block, which is a processing unit for encoding and decoding, and the alignment unit divides the enlarged image into small blocks, and, when the small blocks and the divided blocks are the same, calculates the amount of positional shift by scaling the motion vector of the divided block corresponding to the small block according to the size of the low-frequency component image, and, when the small blocks and the divided blocks are different, calculates the amount of positional shift by scaling an average vector of the motion vectors of all divided blocks belonging to the small block or a motion vector of a divided block belonging to a reference point of the small block according to the size of the low-frequency component image.

[0014] (6) An image enlargement device described in any of (1) to (5), wherein the size of the small block is the same as the size of the divided block, and the allocation unit performs the allocation process only when the converted value per pixel of the squared error sum or the absolute error sum is equal to or less than a threshold value.

[0015] (7) A video enlargement device as described in any of (1) to (6), further comprising a pre-processing unit that performs a deblocking process on each image of the decoded video only when the value of the quantization parameter indicated by the QP information is equal to or greater than a first threshold value or the magnitude of the motion vector indicated by the motion vector information is equal to or greater than a second threshold value, and the frequency decomposition unit performs a frequency decomposition process on the image processed by the pre-processing unit to generate a frequency decomposition image.

[0016] (8) A program for causing a computer to function as the image enlargement device according to any one of (1) to (7). Effect of the Invention

[0017] According to the present invention, it is possible to suppress coding artifacts and generate an enlarged image with high image quality. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an example of the configuration of an image enlargement device according to a first embodiment. [Diagram 2] FIG. 2 illustrates an example of a random access reference structure. [Diagram 3] FIG. 2 is a diagram showing a processing state when single-frame super-resolution processing is performed by the image enlargement device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing a processing state when a multi-frame super-resolution process is performed by the image enlargement device according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of the configuration of an image enlargement device according to a second embodiment. [Figure 6] FIG. 11 is a block diagram showing an example of the configuration of an image enlargement device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0020] First embodiment 1 is a block diagram showing an example of the configuration of a video enlargement device according to an embodiment 1. The video enlargement device 1 is a device that performs enlargement processing (super-resolution processing) on ​​a decoded video that is decoded after encoding, as a post-processing of the video encoding.

[0021] The decoding device 10 is a device that inputs coded data from a coding device (not shown) that performs a predictive process on an original video to generate coded data, and decodes the coded data. As the predictive process, inter-frame prediction (also called inter-prediction or inter-screen prediction) and intra-frame prediction (also called intra-prediction or intra-screen prediction) are used. Inter-frame prediction is a predictive coding method that generates a predicted image from a reference frame that precedes or follows in time, and codes the difference between the original image and the predicted image. Intra-frame prediction is a predictive coding method that generates a predicted image from pixels of adjacent blocks that have already been coded for a block to be coded within a frame, and codes the difference between the original image and the predicted image.

[0022] The video enlargement device 1 acquires from the decoding device 10 a decoded video of an original video, and video processing parameter information indicating parameters used in encoding and decoding of the original video. The encoding and decoding processes are performed by a general video encoding method such as HEVC or VVC. The video processing parameter information includes divided block information in each image of the video, QP (Quantization parameter) information for each divided block, and motion vector information (inter-frame prediction information) for each divided block. Furthermore, the video processing parameters may include random access reference structure information.

[0023] "Divided block information" is information that indicates the shape and size of divided blocks, which are the processing units when encoding and decoding processes are performed. In HEVC and VVC, "divided blocks" are called CUs (Coding Units). In VVC, CUs and PUs (Prediction Units) are basically the same size. Also, while the shape of a CU in HEVC is square, in VVC the shape of a CU can be not only square but also rectangular.

[0024] "QP information" is information indicating a quantization parameter (QP). The quantization parameter has a predetermined correspondence relationship with the quantization step. For example, if the quantization parameter increases by 6, the quantization step doubles.

[0025] The "motion vector information" is information indicating a motion vector (motion vector) of an object between a target frame and a reference frame when performing inter-frame prediction. The motion vector is calculated by, for example, a block matching method, but is not limited to this.

[0026] "Random access reference structure information" is information indicating the position of a reference frame in inter-frame prediction, in other words, information indicating which frame the prediction target frame refers to for inter-frame prediction. Figure 2 is a diagram showing an example of a random access reference structure. In the random access mode of HEVC and VVC, inter-frame prediction is performed using B pictures (bidirectionally predictively coded pictures) in a hierarchical structure as shown in Figure 2.

[0027] Returning to Fig. 1, a description will be given of the configuration of the image enlargement device 1. In the example shown in Fig. 1, the image enlargement device 1 includes a parameter acquisition unit 11, a frequency decomposition unit 12, a position alignment unit 13, an allocation unit 14, and a frequency reconstruction unit 15.

[0028] The parameter acquisition unit 11 acquires, from the decoding device 10, video processing parameter information obtained when the encoding process and the decoding process are performed. Alternatively, the parameter acquisition unit 11 may extract and acquire the video processing parameter information from the encoded data of the original video. The parameter acquisition unit 11 outputs the divided block information, QP information, motion vector information, and random access reference structure information to the allocation unit 14. The parameter acquisition unit 11 may also output the random access reference structure information to the alignment unit 13, but this is not essential, and therefore in FIG. 1, the arrow from the parameter acquisition unit 11 pointing to the alignment unit 13 is indicated by a dashed line.

[0029] The frequency decomposition unit 12 performs frequency decomposition processing on each image (also referred to as a picture or a frame) of the decoded video input from the decoding device 10 to generate a frequency-resolved image. In this embodiment, one-level frequency decomposition processing is performed, but multiple levels of frequency decomposition processing may be performed. The frequency decomposition unit 12 performs, for example, wavelet decomposition processing as the frequency decomposition processing. For example, the frequency decomposition image obtained by performing first-level wavelet decomposition processing on image I is composed of a low-frequency band component image (low-frequency component image) LL(I) and high-frequency band component images (high-frequency component images) LH(I), HL(I), and HH(I). Of the generated frequency decomposition images, the frequency decomposition unit 12 outputs the low-frequency component image to the positioning unit 13 and outputs the high-frequency component image to the allocation unit 14.

[0030] The alignment unit 13 performs alignment processing (registration processing) in small block units (e.g., 16×16 pixels) between a low-frequency component image of one or more reference images including an enlarged image (image to be enlarged, original picture to be aligned) in the decoded video and the enlarged image, and generates alignment information (alignment vector) indicating the amount of misalignment. The amount of misalignment is obtained by block matching in small block units, etc. The alignment unit 13 performs the above processing by treating each image of the decoded video as the enlarged image. The alignment unit 13 outputs the generated alignment information to the allocation unit 14.

[0031] The allocation unit 14 generates an initial image in which the enlarged image is a low-frequency component image. The high-frequency component image of the initial image is either an image with zero components (zero matrix mode) or an image obtained by enlarging the high-frequency component image of the enlarged image (high spatial frequency enlargement mode). Any enlargement method can be used in the high spatial frequency enlargement mode, and nearest neighbor interpolation may be used, for example. Two modes, the zero matrix mode and the high spatial frequency enlargement mode, may be prepared so that the user can select which mode to use in advance.

[0032] Next, the allocation unit 14 performs allocation processing to allocate high-frequency component images of one or more reference images including the image to be enlarged to a high-frequency band (super-resolution frequency band exceeding the maximum frequency of the image to be enlarged) of the initial image according to the alignment information input from the alignment unit 13, and generates an allocated image. The allocation is performed according to the alignment information of the same phase position in the low-frequency component image of the reference image. However, the allocation unit 14 calculates the sum of squared difference (SSD) or sum of absolute difference (SAD) of the small blocks of the source and destination of alignment, and performs the allocation processing only if the calculated value is equal to or less than the allocation threshold.

[0033] Furthermore, the allocation unit 14 may correct the allocation threshold value. Three examples of the method of correcting the allocation threshold value will be described below.

[0034] <<First allocation threshold correction method>> The allocation unit 14 changes the allocation threshold according to the magnitude of the QP information and / or the motion vector information of the small block position of the allocation source. For example, the allocation unit 14 reduces the allocation threshold as the value of the quantization parameter indicated by the QP information and the magnitude of the motion vector indicated by the motion vector information are larger, and increases the allocation threshold as the value of the quantization parameter indicated by the QP information and the magnitude of the motion vector indicated by the motion vector information are smaller. It is desirable to correct (weight) the magnitude of the motion vector according to the position of the reference frame indicated by the random access reference structure information. For example, when the distance between the prediction target frame and the reference frame is one frame interval (the reference frame is adjacent to the prediction target frame), the magnitude of the motion vector is not corrected, and when the distance between the prediction target frame and the reference frame is n frame interval, the magnitude of the motion vector is corrected to 1 / n.

[0035] The size of the small blocks used by the positioning unit 13 and the allocation unit 14 is arbitrary, but may be the same as the size of the divided blocks indicated by the video processing parameter information. When the size of the small blocks is matched to the divided blocks of the video processing parameters, the QP information and motion vector information at the position of the small block from which the allocation originates are the QP information and motion vector information used in encoding and decoding the divided block corresponding to the small block, which simplifies the processing.

[0036] When the size of a small block is variable like a divided block, the allocation unit 14 calculates a conversion value per pixel of the sum of squared errors or the sum of absolute errors of the source and destination small blocks, and performs allocation processing only if the conversion value is less than or equal to the allocation threshold value.

[0037] Table 1 shows an example of the allocation threshold correction. The correction coefficient is a coefficient by which the default allocation threshold value is multiplied. For example, if the default allocation threshold value is Th, the quantization parameter value is 42, and the motion vector size is 15 pixels, then according to Table 1 the corrected allocation threshold Th1 will be Th x 0.8 x 0.9. Note that the corrected value is rounded off to the nearest whole number. Since the performance of the quantization parameter, motion vector, etc. is largely determined by the encoding method adopted by the encoding device, an appropriate value for this allocation threshold may be determined in advance by a preliminary experiment.

[0038] [Table 1]

[0039] <<Second allocation threshold correction method>> In the second allocation threshold correction method, the allocation unit 14 corrects the allocation threshold according to the distance between the source picture and the destination picture. When the destination picture is determined based on the random access reference structure, the allocation unit 14 calculates the corrected allocation threshold Th2 by equation (1) using the random access reference structures of the pictures to which the source and destination small blocks belong. Th2 = Th * (a * 1 / |CurPOC - RefPOC|) (1)

[0040] In formula (1), Th is the default value of the allocation threshold, and a is a weighting coefficient. CurPOC is the Picture Order Count (count value of the display order of the picture) of the source picture, and RefPOC is the Picture Order Count of the destination picture. In formula (1), the default value Th is corrected by multiplying the inverse of the difference between CurPOC and RefPOC by the weighting coefficient a, so that the allocation threshold Th2 becomes smaller as the distance between the source and destination pictures becomes larger. In general, the correlation between pictures tends to be lower as the distance between the source and destination pictures becomes larger. Therefore, by correcting the allocation threshold using formula (1), it is possible to avoid performing allocation processing as much as possible between small blocks with a large inter-picture distance.

[0041] <<Third allocation threshold correction method>> In the third allocation threshold correction method, the allocation unit 14 sets the weighted average of Th1 obtained by the first allocation threshold correction method and Th2 obtained by the second allocation threshold correction method as the final allocation threshold. Which of the first to third allocation threshold correction methods is to be used may be set in advance in consideration of the speed and accuracy of the allocation process.

[0042] The frequency reconstruction unit 15 generates an enlarged video consisting of enlarged images obtained by enlarging the enlarged image by performing frequency reconstruction on each of the allocated images input from the allocation unit 14. When the frequency decomposition unit 12 performs wavelet decomposition as the frequency decomposition process, the frequency reconstruction unit 15 performs wavelet reconstruction using the same wavelet. The frequency reconstruction unit 15 outputs the generated enlarged video to the outside of the video enlargement device 1.

[0043] Next, with reference to FIG. 3, a processing example of the image enlargement device 1 when performing single-frame super-resolution processing will be described.

[0044] In the example shown in Figure 3, the frequency decomposition unit 12 performs first-order wavelet decomposition on the enlarged image α, and generates a frequency decomposition image α' consisting of a low-frequency component image LL(α) and high-frequency component images LH(α), HL(α), and HH(α).

[0045] The position matching unit 13 performs position matching processing between the enlarged image α and the low-frequency component image LL(α) in the frequency decomposition image α' in units of small blocks.

[0046] The allocation unit 14 generates an initial image in which the low frequency component image is the enlarged image α and the high frequency component images are images LH(αE), HL(αE), and HH(αE). In the zero matrix mode, the images LH(αE), HL(αE), and HH(αE) are zero matrices of the same size as the enlarged image α, and in the high spatial frequency enlargement mode, the high frequency component images LH(α), HL(α), and HH(α) of the enlarged image α are enlarged twice in the horizontal and vertical directions to the same size as the enlarged image α. Next, the allocation unit 14 performs allocation processing from the high frequency component images LH(α), HL(α), and HH(α) of the enlarged image α to the high frequency component images LH(αE), HL(αE), and HH(αE) of the initial image according to the alignment information of the same phase position in the low frequency component image LL(α) of the enlarged image α, and generates an allocation image β.

[0047] The frequency reconstruction unit 15 performs first-order wavelet reconstruction on the allocated image β to generate an enlarged image γ by enlarging the enlarged image α.

[0048] Next, with reference to FIG. 4, a processing example of the image enlargement device 1 when performing multi-frame super-resolution processing will be described.

[0049] The images before and after the enlarged image used in the multi-frame super-resolution process can be ±p images before and after the enlarged image. In addition, when the positioning unit 13 acquires random access reference structure information from the parameter acquisition unit 11, the images can be the reference picture lists (list 0 and list 1) of the enlarged image indicated by the random access reference structure information.

[0050] In the example shown in FIG. 4, the frequency decomposition unit 12 performs first-order wavelet decomposition on the enlarged image a to generate a frequency decomposition image a' consisting of a low-frequency component image LL(a) and high-frequency component images LH(a), HL(a), and HH(a). The frequency decomposition unit 12 also performs first-order wavelet decomposition on one or more reference images before and after the enlarged image. For convenience of explanation, FIG. 4 describes super-resolution processing using one reference image b. The frequency decomposition unit 12 performs first-order wavelet decomposition on the reference image b to generate a frequency decomposition image b' consisting of a low-frequency component image LL(b) and high-frequency component images LH(b), HL(b), and HH(b).

[0051] The position matching unit 13 performs position matching processing between the enlarged image a and the low-frequency component image LL(b) in the frequency decomposition image b' in units of small blocks.

[0052] The allocation unit 14 generates an initial image with the low frequency component image as the enlarged image a and the high frequency component images as LH(aE), HL(aE), and HH(aE). In the zero matrix mode, the images LH(aE), HL(aE), and HH(aE) are zero matrices of the same size as the enlarged image a, and in the high spatial frequency enlargement mode, the high frequency component images LH(a), HL(a), and HH(a) of the enlarged image a are enlarged twice in the horizontal and vertical directions to the same size as the enlarged image a. Next, the allocation unit 14 performs allocation processing from the high frequency component images LH(b), HL(b), and HH(b) of the reference image b to the high frequency component images LH(aE), HL(aE), and HH(aE) of the initial image according to the alignment information of the same phase position in the low frequency component image LL(b) of the reference image b, to generate an allocation image c.

[0053] Although not shown in Fig. 4, the positioning unit 13 performs positioning processing in small block units between the enlarged image a and the low-frequency component image LL(a) in the frequency decomposition image a', as in Fig. 3. In addition, the allocation unit 14 performs allocation processing from the high-frequency component images LH(a), HL(a), and HH(a) of the enlarged image a to the high-frequency component images LH(aE), HL(aE), and HH(aE) of the initial image according to the positioning information of the same phase position in the low-frequency component image LL(α) of the enlarged image a.

[0054] The frequency reconstruction unit 15 performs first-order wavelet reconstruction on the allocated image c to generate an enlarged image d by enlarging the enlarged image a.

[0055] As described above, the allocation unit 14 of the video enlargement device 1 performs allocation processing only when the sum of squared errors or the sum of absolute errors between the source and destination small blocks is equal to or less than the allocation threshold, and changes the allocation threshold according to the value of the quantization parameter or the magnitude of the motion vector at the source small block position. If the value of the quantization parameter is large or the magnitude of the motion vector is large, block distortion may occur and degrade the decoded image, so by changing the allocation threshold, the accuracy of allocation can be improved. Therefore, the video enlargement device 1 can improve the image quality of the enlarged image.

[0056] <Second embodiment> Next, an image enlargement device according to a second embodiment will be described.

[0057] Fig. 5 is a block diagram showing a configuration example of an image enlargement device 2 according to the second embodiment. In the example shown in Fig. 5, the image enlargement device 2 includes a parameter acquisition unit 11, a frequency decomposition unit 12, a positioning unit 13, an allocation unit 14, a frequency reconstruction unit 15, and a pre-processing unit 16. The image enlargement device 2 according to the second embodiment differs from the image enlargement device 1 according to the first embodiment in that it further includes a pre-processing unit 16. Since the other configurations are the same as those of the first embodiment, the same reference numbers are used and the description is omitted as appropriate.

[0058] The pre-processing unit 16 acquires, as video processing parameter information, divided block information, QP information and motion vector information for each divided block, and random access reference structure information from the parameter acquisition unit 11. The pre-processing unit 16 performs frequency decomposition processing on each image of the decoded video, performs deblocking processing, and then performs frequency reconstruction processing to generate a pre-processed image. The pre-processing unit 16 performs deblocking processing only when the value of the quantization parameter indicated by the QP information is equal to or greater than a predetermined threshold Thq, or the magnitude of the motion vector indicated by the motion vector information is equal to or greater than a predetermined threshold Thv. Here, it is desirable to correct (weight) the magnitude of the motion vector according to the position of the reference frame indicated by the random access reference structure information. The pre-processing unit 16 outputs the generated pre-processed image to the frequency decomposition unit 12.

[0059] For example, the pre-processing unit 16 performs a third-order wavelet decomposition process. If the value of the quantization parameter is equal to or greater than a predetermined threshold Thq, or if the magnitude of the motion vector is equal to or greater than a predetermined threshold Thv, the pre-processing unit 16 outputs a first-order high frequency component image LH 1 ,HL 1 ,H.H. 1 The function y=ax (e.g. a=0.1) is applied to the elements at the CU boundary positions of the CU in order to obtain the second-order high-frequency component image LH 2 ,HL 2 ,H.H. 2 The function y=bx (e.g. b=0.3) is applied to the elements at the CU boundary positions of the CU in order to obtain the third-order high frequency component image LH 3 ,HL 3 ,H.H. 3 A function of y=cx (for example, c=0.5) is applied to the elements at the CU boundary positions of the CU in question. Finally, the pre-processing unit 16 performs third-order wavelet reconstruction processing.

[0060] The frequency decomposition unit 12 of the video enlargement device 1 performs processing after frequency decomposition on the decoded image input from the decoding device 10, while the frequency decomposition unit 12 of the video enlargement device 2 performs processing after frequency decomposition on the preprocessed image. The video enlargement device 2 performs enlargement processing on an image in which block noise has been reduced by the preprocessing unit 16, so it is possible to further improve the image quality of the enlarged image compared to the video enlargement device 1. In addition, when the value of the quantization parameter is large or the magnitude of the motion vector is large, block distortion may occur and the decoded image may be deteriorated, so that it is possible to appropriately remove block noise by performing deblocking processing only when the value of the quantization parameter is equal to or greater than a first threshold value or the magnitude of the motion vector is equal to or greater than a second threshold value.

[0061] <Third embodiment> Next, an image enlargement device according to a third embodiment will be described.

[0062] Fig. 6 is a block diagram showing a configuration example of an image enlargement device 3 according to the third embodiment. In the example shown in Fig. 6, the image enlargement device 3 includes a parameter acquisition unit 11, a frequency decomposition unit 12, a position alignment unit 13a, an allocation unit 14, a frequency reconstruction unit 15, and a preprocessing unit 16. The image enlargement device 3 according to the third embodiment is different from the image enlargement device 1 according to the first embodiment in that it includes a position alignment unit 13a instead of the position alignment unit 13. Since the other configurations are the same as those of the first embodiment, the same reference numbers are used and the description is omitted as appropriate.

[0063] The parameter acquisition unit 11 outputs motion vector information for each divided block, which is a processing unit when performing encoding and decoding, to the alignment unit 13a. Inter-picture prediction is performed between similar divided blocks with small differences, so it can be used for alignment. When performing inter-picture prediction, the reference image is determined based on a random access reference structure. The frequency decomposition unit 12 outputs information indicating the number of layers during frequency decomposition to the alignment unit 13a. The number of layers during frequency decomposition may be set in advance in the alignment unit 13a.

[0064] The positioning unit 13a regards each image of the decoded video as an enlarged image, divides the enlarged image into small blocks, and obtains positioning information (positioning vector) indicating the amount of positional deviation for each small block using motion vector information. The positioning unit 13a outputs the generated positioning information to the allocation unit 14.

[0065] Specifically, when a small block and a divided block are the same, the positioning unit 13a obtains the amount of positional deviation by scaling the motion vector of the divided block corresponding to the small block according to the number of layers of the reference image (the size of the low-frequency component image), and sets the amount of positional deviation as the positioning information of the small block. N Scale to.

[0066] When the small block and the divided block are different, the positioning unit 13a calculates the amount of positional deviation by scaling a vector (first motion vector) obtained by averaging the motion vectors of all divided blocks belonging to the small block, or a motion vector (second motion vector) of a divided block belonging to a reference point of the small block (e.g., the center position of the small block) according to the size of the low-frequency component image of the reference image, and sets the calculated amount as positioning information for the small block. The positioning unit 13a may be set in advance as to which of the first motion vector and the second motion vector to use.

[0067] In addition to the above-mentioned alignment process (first alignment process), the alignment unit 13a may perform the same alignment process (second alignment process) as the alignment unit 13 of the first embodiment, and the multiple alignment vectors obtained by both processes may be used as alignment vectors for the small blocks. In this case, the small blocks used in the first alignment process and the second alignment process do not have to be the same. Also, if the small blocks are the same as the divided blocks, this is preferable because the partition information can be used as is. On the other hand, even if the small blocks and the divided blocks are different, the overhead of the partition information is not so large.

[0068] <Program> A computer capable of executing program instructions may be used to function as each of the above-mentioned image enlargement devices 1 to 3. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0069] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types. The processor reads a program from the storage unit and executes it to control each of the above components and perform various arithmetic processing. At least a part of the processing contents may be realized by hardware. The input unit is an input interface that receives a user's input operation and acquires information based on the user's operation, and is a pointing device, a keyboard, a mouse, or the like. The output unit is an output interface that outputs information, and is a display, a speaker, or the like. The communication interface is an interface for communicating with an external device, and is, for example, a LAN (Local Area Network) interface.

[0070] The program may be recorded on a computer-readable recording medium. By using such a recording medium, the program can be installed on the computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

[0071] For example, a program for functioning as the video enlargement device 1 causes a computer to execute the following steps: acquiring parameter information when encoding and decoding processes are performed; performing frequency decomposition processing on each image of the decoded video to generate a frequency resolved image; generating alignment information indicating the amount of positional deviation on a small block basis between the enlarged image and a low-frequency component image of one or more reference images including the image to be enlarged in the decoded video; generating an initial image in which the image to be enlarged is a low-frequency component image, and allocating the high-frequency component image of the reference image to the high-frequency band of the initial image according to the alignment information to generate an allocated image; and performing reconstruction processing on the allocated image to generate an enlarged video. In the allocation step, the allocation process is performed only if the sum of squared errors or the sum of absolute errors of the small blocks to be aligned and the small blocks to be aligned are equal to or less than a threshold value, and the threshold value is changed according to the parameter information.

[0072] Furthermore, the image enlargement devices 1-3 may be configured with one or more semiconductor chips, and the semiconductor chip may be equipped with a CPU that executes a program describing the processing contents for realizing each function of the image enlargement devices 1-3.

[0073] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to integrate multiple building blocks shown in the configuration diagram of the embodiment, or to divide one building block.

[0074] For example, the image enlargement device 3 according to the third embodiment may include a pre-processing unit 16 between the parameter acquisition unit 11 and the frequency decomposition unit 12, similarly to the second embodiment. [Explanation of symbols]

[0075] 1,2,3 Image Enlarger 10 Decryption device 11 Parameter acquisition section 12 Frequency decomposition section 13,13a Alignment section 14 Allocation section 15 Frequency reconstruction unit 16 Pretreatment section

Claims

1. A video enlargement device that enlarges a decoded video by super-resolution processing after encoding, comprising: A parameter acquisition unit that acquires parameter information when encoding and decoding are performed; a frequency decomposition unit that performs frequency decomposition processing on each image of the decoded video to generate a frequency resolved image; a positioning unit that generates positioning information indicating a positional deviation amount in small block units between the enlarged image and a low-frequency component image of one or more reference images including the enlarged image in the decoded video; an allocating unit that generates an initial image in which the enlarged image is a low-frequency component image, and allocates a high-frequency component image of the reference image to a high-frequency band of the initial image in accordance with the positioning information, thereby generating an allocated image; a frequency reconstruction unit that performs a frequency reconstruction process on the allocated image to generate an enlarged image, The allocation unit performs allocation processing only when the sum of squared errors or the sum of absolute errors of the small blocks of the source and destination of alignment is equal to or less than a threshold, and changes the threshold in accordance with the parameter information.

2. The parameter information includes QP information indicating a quantization parameter for each divided block, which is a processing unit for encoding and decoding, or motion vector information indicating a motion vector in inter-frame prediction, The image enlargement device according to claim 1 , wherein the allocating unit reduces the threshold value as the value of the quantization parameter indicated by the QP information or the magnitude of the motion vector indicated by the motion vector information increases.

3. The image enlargement device according to claim 1 , wherein the allocating unit reduces the threshold value as the distance between the source picture and the destination picture increases.

4. 3. The image enlargement device according to claim 1, wherein the position matching section determines the amount of positional deviation by performing block matching between the low-frequency component image and the enlarged image in units of small blocks.

5. the parameter information includes motion vector information indicating a motion vector in inter-frame prediction for each divided block which is a processing unit for encoding and decoding; The image enlargement device of claim 1 or 2, wherein the alignment unit divides the enlarged image into small blocks, and when the small block and the divided block are the same, calculates the amount of positional shift by scaling the motion vector of the divided block corresponding to the small block according to the size of the low-frequency component image, and when the small block and the divided block are different, calculates the amount of positional shift by scaling the average vector of the motion vectors of all divided blocks belonging to the small block or the motion vector of the divided block belonging to the reference point of the small block according to the size of the low-frequency component image.

6. The size of the small block is the same as the size of the divided block, 3 . The image enlargement device according to claim 1 , wherein the allocating section performs the allocating process only when a converted value per pixel of the sum of squared errors or the sum of absolute errors is equal to or smaller than a threshold value.

7. A pre-processing unit that performs a deblocking process on each image of the decoded video only when a value of a quantization parameter indicated by the QP information is equal to or greater than a first threshold value or a magnitude of a motion vector indicated by the motion vector information is equal to or greater than a second threshold value, The image enlargement device according to claim 1 , wherein the frequency decomposition section performs frequency decomposition processing on the image processed by the preprocessing section to generate a frequency resolved image.

8. A program for causing a computer to function as the image enlargement device according to claim 1.

Citation Information

Patent Citations

  • Coder, decoder, moving image coding method, and moving image decoding method

    JP2009182776A