Methods for producing high-resolution images
Through the methods of encoding, decoding and super-resolution operations on images, the problems of large bandwidth consumption, large operation volume and high cost in traditional super-resolution processing are solved, and efficient and low-cost high-resolution image generation is achieved.
Patent Information
- Application Number
- CN202410020235.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-04
- Filing Date
- 2024-01-05
- Publication Date
- 2025-04-08
Smart Images
Figure CN119788883A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an imaging technology, and more particularly to a method for generating high-resolution images. Background Art
[0002] In traditional methods, the processor performs super-resolution calculations on low-resolution images to generate high-resolution versions of the images. However, due to the large amount of image data, the bandwidth consumed is large and the amount of operations is large. A high-specification processor is required to perform a large number of calculations, resulting in higher costs and more energy consumption. Therefore, how to design to solve the above problems is an important issue in this field. Summary of the invention
[0003] The embodiment of the present invention includes a method for generating an image. The method includes the following operations: performing an encoding operation on a first image to generate streaming data; performing a first decoding operation on the streaming data to generate first quantized data; performing a super-resolution operation on the first quantized data to generate second quantized data; and performing a second decoding operation on the second quantized data to generate a second image. The second image is a high-resolution version of the first image, the amount of data of the streaming data is smaller than the amount of data of the first quantized data, and the amount of data of the first quantized data is smaller than the amount of data of the first image.
[0004] In some embodiments, performing the encoding operation includes: performing at least a first encoding operation and a second encoding operation on the first image in sequence to generate the stream data, wherein the first decoding operation is an inverse operation of the second encoding operation, and the second decoding operation is an inverse operation of the first encoding operation.
[0005] In some embodiments, the ratio of the data volume of the first quantized data to the data volume of the second quantized data is equal to the ratio of the data volume of the first image to the data volume of the second image.
[0006] In some embodiments, the data volume of the first quantized data is smaller than the data volume of the second quantized data, and the data volume of the second quantized data is smaller than the data volume of the second image.
[0007] In some embodiments, the method further includes: performing training based on a portion of the first encoding operation and the super-resolution operation to generate a plurality of weights, wherein performing the super-resolution operation includes: performing the super-resolution operation on the first quantized data based on the weights to generate the second quantized data.
[0008] In some embodiments, performing training to generate the weights includes: sequentially performing a conversion operation and a quantization operation on a first training data to generate a first training quantized data; and performing the super-resolution operation on the first training quantized data to generate a second training quantized data, wherein the portion of the first encoding operation includes the conversion operation and the quantization operation.
[0009] In some embodiments, training to generate the weights further includes: sequentially performing a dequantization operation and a deconversion operation on the second training quantized data to generate a second training data; comparing a high-resolution version of the first training data with the second training data to generate a loss function; and adjusting the super-resolution operation according to the loss function to determine the weights, wherein the dequantization operation and the deconversion operation are respectively the inverse operations of the quantization operation and the conversion operation.
[0010] In some embodiments, the second decoding operation includes the inverse quantization operation and the inverse transform operation.
[0011] In some embodiments, the data amount of the first training quantized data is smaller than the data amount of the first training data, the data amount of the first training quantized data is smaller than the data amount of the second training quantized data, and the data amount of the second training quantized data is smaller than the data amount of the second training data.
[0012] In some embodiments, the method further includes: performing a prediction operation on the first training data to generate a first prediction data; performing the conversion operation after subtracting the first training data from the first prediction data; and adding the output data of the inverse conversion operation to a prediction value corresponding to the prediction operation to generate the second training data. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 is a schematic diagram of a method for generating an image according to some embodiments of the present disclosure.
[0014] Figure 2 A schematic diagram of a method for generating weights according to some embodiments of the present disclosure.
[0015] Figure 3 A schematic diagram of a method for generating weights according to some embodiments of the present disclosure.
[0016] Description of reference numerals:
[0017] 100, 200, 300: Method
[0018] M11, M12: Images
[0019] OP11, OP111, OP112: Encoded Operation
[0020] OP12, OP13: Decoding operation
[0021] SRC1: Super-resolution calculation
[0022] SD11: Streaming Data
[0023] QD11, QD111, QD12: Quantitative data
[0024] W1: Weight
[0025] TD21, TD22, TH21: training data
[0026] OP21: Conversion Operation
[0027] OP22: Quantization Operation
[0028] OP23: Dequantization Operation
[0029] OP24: Deconversion Operation
[0030] OP25: Comparison Operation
[0031] TM21, TM22: Intermediate data
[0032] TQ21, TQ22: Training Quantized Data
[0033] L2: Loss function
[0034] OP31: Predictive Operation
[0035] OP32, OP33: Calculation operation
[0036] PD31: Forecast data
[0037] OD3: Output Data
[0038] PV3: Predicted value DETAILED DESCRIPTION
[0039] In this document, although the terms "first", "second", etc. are used to describe different components, the terms are only used to distinguish components or operations described with the same technical terms. Unless the context clearly indicates otherwise, the terms do not specifically refer to or imply an order or sequence, nor are they used to limit the present disclosure.
[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and the present disclosure, and will not be interpreted as an idealized or overly formal meaning unless explicitly defined as such herein.
[0041] The terms used herein are only for the purpose of describing specific embodiments and are not restrictive. As used herein, unless the content clearly indicates, the singular forms "one", "an" and "the" are intended to include multiple forms, including "at least one". "Or" means "and / or". As used herein, the term "and / or" includes any and all combinations of one or more related listed items. It should also be understood that when used in this specification, the terms "include" and / or "comprise" specify the presence and / or parts of the features, regions, entireties, steps, operations, elements, but do not exclude the presence or addition of one or more other features, regions, entireties, steps, operations, elements, parts and / or their combinations.
[0042] The following will disclose multiple embodiments of the present disclosure with the accompanying drawings. For the purpose of clarity, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present disclosure. In other words, in some embodiments of the present disclosure, these practical details are not necessary.
[0043] Figure 1 FIG. 1 is a schematic diagram of a method 100 for generating an image according to some embodiments of the present disclosure. The method 100 can generate an image M12 based on the image M11, wherein the image M12 is a high-resolution version of the image M11. Figure 1 As shown, method 100 may include encoding operation OP11, decoding operations OP12, OP13 and super-resolution operation SRCl. In some embodiments, each of encoding operation OP11, decoding operations OP12, OP13 and super-resolution operation SRCl may be executed by a memory and a processor.
[0044] In the encoding operation OP11 , the processor compresses the image M11 , for example, changes the data format of the image M11 , to generate stream data SD11 . The stream data SD11 is the compressed data of the image M11 , and the data volume of the stream data SD11 is smaller than that of the image M11 .
[0045] In the encoding operation OP11, the processor sequentially performs encoding operations OP111 and OP112 on the image M11. In the encoding operation OP111, the processor compresses the image M11 to generate quantized data QD111. In the encoding operation OP112, the processor compresses the quantized data QD111 to generate stream data SD11. The data volume of the quantized data QD111 is smaller than the data volume of the image M11, and the data volume of the stream data SD11 is smaller than the data volume of the quantized data QD111.
[0046] In some embodiments, the encoding operation OP112 may be an entropy encoding, such as Huffman encoding, but the present disclosure is not limited thereto. On the other hand, the encoding operation OP111 may be a compression operation other than entropy encoding. For example, the encoding operation OP111 may include a quantization operation and a conversion operation for processing the data of the image M11 through a quantization matrix and a conversion matrix, but the present disclosure is not limited thereto.
[0047] In some embodiments, the super-resolution operation SRC1 can be implemented by an artificial intelligence of a real super-resolution generative adversarial network (Real-ESRGAN), but the present disclosure is not limited thereto. In various embodiments, the super-resolution operation SRC1 can be implemented by various types of artificial intelligence.
[0048] In the decoding operation OP12, the processor decompresses the stream data SD11 into the quantized data QD11. In some embodiments, the decoding operation OP12 may be an entropy decoding, such as Huffman decoding, but the disclosure is not limited thereto. In some embodiments, the decoding operation OP12 is an inverse operation of the encoding operation OP112. In various embodiments, the quantized data QD11 may be the same as the quantized data QD111, or may not be completely the same as the quantized data QD111.
[0049] After the decoding operation OP12, the processor performs a super-resolution operation SRC1 on the quantized data QD11 to generate the quantized data QD12. Then, in the decoding operation OP13, the processor decompresses the quantized data QD12 to generate the image M12. The data amount of the quantized data QD12 is greater than the data amount of the quantized data QD11, and the data amount of the image M12 is greater than the data amount of the quantized data QD12. In some embodiments, the decoding operation OP13 is an inverse operation of the encoding operation OP111. For example, the decoding operation OP13 may include an inverse quantization operation and an inverse transformation operation for processing the quantized data QD12 by means of an inverse matrix of the above-mentioned quantization matrix and an inverse matrix of the above-mentioned transformation matrix, but the present disclosure is not limited thereto.
[0050] In some embodiments, the ratio of the data volume of the quantized data QD11 to the data volume of the quantized data QD12 is determined by the resolution magnification of the super-resolution operation SRC1. For example, when the resolution magnification of the super-resolution operation SRC1 is four times, the number of pixels of the image M12 is four times the number of pixels of the image M11. Correspondingly, the data volume of the image M12 is four times the data volume of the image M11, and the data volume of the quantized data QD12 is four times the data volume of the quantized data QD11. In other words, in some embodiments, the ratio of the data volume of the quantized data QD11 to the data volume of the quantized data QD12 is equal to the ratio of the data volume of the image M11 to the data volume of the image M12.
[0051] In some methods, the processor directly performs super-resolution operations on the image to generate a high-resolution version of the image. However, due to the large amount of image data, the bandwidth consumed is large and the amount of operations is large. It also requires a higher-specification processor to perform a large amount of calculations, resulting in higher costs and more energy consumption.
[0052] Compared to the above method, in the embodiment of the present invention, the processor performs super-resolution operation on the quantized data QD11 instead of the image M11 to generate the quantized data QD12, and then generates a high-resolution version of the image M11, that is, the image M12, based on the quantized data QD12. Since the data volume of the quantized data QD11 is smaller than that of the image M11, the bandwidth consumed is smaller and the amount of operation is smaller, and a processor with lower specifications can be used to perform a small amount of operation to generate a high-resolution image M12, which reduces the cost and saves energy.
[0053] like Figure 1As shown, the super-resolution operation SCR1 can be performed based on a plurality of weights W1. For example, the processor performs the super-resolution operation SCR1 on the quantized data QD11 based on the weights W1 to generate the quantized data QD12. In some embodiments, the processor performs training based on a portion of the encoding operation OP111 and the super-resolution operation SCR1 to generate the weights W1. The specific details of the operation of generating the weights W1 are described below with respect to Figure 2 and Figure 3 The embodiments are further described in the examples.
[0054] Figure 2 2 is a schematic diagram of a method 200 for generating weights W1 according to some embodiments of the present disclosure. The method 200 can generate a loss function L2 according to the training data TD21 and TH21, and adjust the weight value in the super-resolution operation SCR1 according to the loss function L2 to determine the weight W1. Figure 2 As shown, the method 200 includes a conversion operation OP21, a quantization operation OP22, a super-resolution operation SCR1, an inverse quantization operation OP23, an inverse conversion operation OP24, and a comparison operation OP25. In some embodiments, each of the conversion operation OP21, the quantization operation OP22, the super-resolution operation SCR1, the inverse quantization operation OP23, the inverse conversion operation OP24, and the comparison operation OP25 can be executed by a memory and a processor.
[0055] In the conversion operation OP21, the processor processes the training data TD21 through the conversion matrix to generate the intermediate data TM21. In the quantization operation OP22, the processor processes the intermediate data TM21 through the quantization matrix to generate the training quantization data TQ21, wherein the data amount of the training quantization data TQ21 is smaller than the data amount of the training data TD21. Next, the processor performs a super-resolution operation SRC1 on the training quantization data TQ21 to generate the training quantization data TQ22, wherein the data amount of the training quantization data TQ21 is smaller than the data amount of the training quantization data TQ22.
[0056] In the inverse quantization operation OP23, the processor processes the training quantized data TQ22 through the inverse quantization matrix to generate intermediate data TM22. In the inverse transform operation OP24, the processor processes the intermediate data TM22 through the inverse transform matrix to generate training data TD22, wherein the data amount of the training quantized data TQ22 is less than the data amount of the training data TD22. In some embodiments, the inverse quantization operation OP23 and the inverse transform operation OP24 are inverse operations of the quantization operation OP22 and the transform operation OP21, respectively.
[0057] In the comparison operation OP25, the processor compares the training data TD22 and the training data TH21, and generates a loss function L2 according to the difference between the training data TD22 and the training data TH21. In some embodiments, the training data TH21 is a high-resolution version of the training data TD21. In some embodiments, the comparison operation OP25 can be performed by a discriminator in the processor.
[0058] Next, the processor adjusts the super-resolution operation SRC1 according to the loss function L2 to determine the weight W1. For example, when the value of the loss function L2 is not the minimum value, the processor continues to adjust the weight value in the super-resolution operation SRC1 and repeatedly executes the method 200 until the value of the loss function L2 is the minimum value, and the weight value of the super-resolution operation SRC1 corresponding to the minimum value of the loss function L2 is selected as the weight W1.
[0059] Please refer to Figure 1 and Figure 2 The encoding operation OP111 may include a quantization operation OP22 and a conversion operation OP21. For example, the processor may perform a conversion operation OP21 and a quantization operation OP22 on the image M11 in sequence during the encoding operation OP111 to generate quantization data QD111.
[0060] On the other hand, the decoding operation OP13 may include an inverse quantization operation OP23 and an inverse transformation operation OP24. For example, the processor may perform an inverse quantization operation OP23, an inverse transformation operation OP24 and other decompression operations on the quantized data QD12 in the decoding operation OP13 to generate the image M12.
[0061] In some embodiments, the conversion operation OP21 may be a discrete cosine transform (DCT), a discrete wavelet transform (DWT), or a fast Fourier transform (FFT), but the present disclosure is not limited thereto. In other embodiments, the conversion operation OP21 may also be other types of conversion.
[0062] Figure 3 Schematic diagram of a method 300 for generating weight W1 according to some embodiments of the present disclosure. Figure 2 and Figure 3 Method 300 is a variation of method 200, and some operations of method 300 follow the same numbering scheme as method 200. For the sake of brevity, the discussion will focus on the differences between method 300 and method 200 rather than the similarities.
[0063] Compared with method 200, method 300 further includes a prediction operation OP31 and calculation operations OP32 and OP33. In the prediction operation OP31, the processor predicts the training data TD21 to generate prediction data PD31. In the calculation operation OP32, the processor subtracts the training data TD21 from the prediction data PD31 and then performs a conversion operation OP21.
[0064] Next, similar to the method 200, the processor performs a conversion operation OP21 to generate intermediate data TM21, and performs a quantization operation OP22 to generate training quantized data TQ21. Next, the processor performs a super-resolution operation SRC1 on the training quantized data TQ21 to generate training quantized data TQ22.
[0065] After the super-resolution operation SRC1, the processor performs a dequantization operation OP23 on the training quantized data TQ22 to generate intermediate data TM22, and performs a conversion operation OP24 on the intermediate data TM22 to generate output data OD3.
[0066] After the inverse conversion operation OP24, the processor performs a calculation operation OP33. In the calculation operation OP33, the processor adds the output data OD3 of the inverse conversion operation OP24 and the prediction value PV3 of the corresponding prediction operation OP31 to generate training data TD22.
[0067] Next, the processor performs a comparison operation on the training data TD22 and the training data TH21 to generate a loss function L2 , and performs a weight value in the super-resolution operation SRC1 according to the loss function L2 to determine a weight W1 .
[0068] In some embodiments, the prediction operation OP31 may be implemented by intra prediction or inter prediction, but the present disclosure is not limited thereto. In other embodiments, the prediction operation OP31 may also include other prediction methods.
[0069] In some embodiments, the processor may also generate the weight W1 by a portion of the operations of methods 200 and 300. In other embodiments, the processor may also generate the weight by a method other than methods 200 and 300.
[0070] Although the present disclosure has been disclosed as above by way of embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make some changes and modifications without departing from the concept and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be determined by the claims.
Claims
1. A method for generating an image, characterized in that: Include: Performing an encoding operation on a first image to generate a stream of data; Performing a first decoding operation on the stream data to generate first quantized data; Performing a super-resolution operation on the first quantized data to generate second quantized data; as well as performing a second decoding operation on the second quantized data to generate a second image, wherein the second image is a high resolution version of the first image, The data volume of the stream data is smaller than the data volume of the first quantized data, and The data volume of the first quantized data is smaller than the data volume of the first image.
2. The method of claim 1, wherein performing the encoding operation comprises: performing at least a first encoding operation and a second encoding operation on the first image in sequence to generate the stream data, wherein the first decoding operation is the inverse operation of the second encoding operation, and The second decoding operation is the inverse operation of the first encoding operation. 3 . The method of claim 1 , wherein a ratio of a data volume of the first quantized data to a data volume of the second quantized data is equal to a ratio of a data volume of the first image to a data volume of the second image.
4. The method of claim 1, wherein The data amount of the first quantized data is smaller than the data amount of the second quantized data, and The data volume of the second quantized data is smaller than the data volume of the second image.
5. The method of claim 1, further comprising: training based on a portion of the encoding operation and the super-resolution operation to generate a plurality of weights, The super-resolution operation includes: The super-resolution operation is performed on the first quantized data according to the weights to generate the second quantized data.
6. The method of claim 5, wherein performing training to generate the weights comprises: performing a conversion operation and a quantization operation on a first training data in sequence to generate a first training quantized data; and performing the super-resolution operation on the first training quantized data to generate a second training quantized data, The part of the encoding operation includes the conversion operation and the quantization operation.
7. The method of claim 6, wherein training to generate the weights further comprises: performing a dequantization operation and a deconversion operation on the second training quantized data in sequence to generate a second training data; comparing the high resolution version of the first training data with the second training data to generate a loss function; and The super-resolution operation is adjusted according to the loss function to determine the weights, The inverse quantization operation and the inverse transformation operation are respectively inverse operations of the quantization operation and the transformation operation.
8. The method of claim 7, wherein the second decoding operation comprises the inverse quantization operation and the inverse transform operation.
9. The method of claim 7, wherein The data volume of the first training quantized data is smaller than the data volume of the first training data. The data amount of the first training quantized data is smaller than the data amount of the second training quantized data, and The data amount of the second training quantized data is smaller than the data amount of the second training data.
10. The method of claim 7, further comprising: Performing a prediction operation on the first training data to generate first prediction data; The conversion operation is performed after subtracting the first training data from the first prediction data; and The output data of the inverse conversion operation is added to a prediction value corresponding to the prediction operation to generate the second training data.