Super-resolution apparatus and method

Abstract

An ultra-resolution device and method. The device includes an input interface and a shared layer operator group. An image and a scaling signal are received by the input interface, where the scaling signal is used to indicate that a two-times scaling operation or a four-times scaling operation is to be performed on the image. When the scaling signal indicates that the four-times scaling operation is to be performed, the shared layer operator group performs a plurality of convolution operations based on a first number of input channels and a first number of output channels. When the scaling signal indicates that the two-times scaling operation is to be performed, the shared layer operator group performs the plurality of convolution operations based on a second number of input channels and a second number of output channels.

Description

Technical Field

[0001] This invention relates to a super-resolution apparatus and method. More specifically, it relates to a super-resolution apparatus and method capable of sharing a shared layer operator set. Background Technology

[0002] In existing technologies, when using convolutional neural networks (CNNs) with hardware circuitry for image superresolution scaling, feature extraction of the image is first performed by the CNN model. However, since a single CNN model can only perform one magnification, different CNN models need to be trained for feature extraction at different magnifications. For example, when the hardware device simultaneously provides magnification of 2x or 4x, two different sets of hardware circuits are needed to perform the corresponding feature extraction separately.

[0003] Therefore, if multiple convolutional neural network models are provided to correspond to different magnification ratios, additional hardware circuits corresponding to different magnification ratios must be manufactured, resulting in a significant increase in circuit area.

[0004] Therefore, how to share the feature extraction circuits used at different amplification rates in order to save circuit area is a goal that the industry urgently needs to strive for. Summary of the Invention

[0005] One object of the present invention is to provide a super-resolution apparatus. The super-resolution apparatus includes an input interface and a shared layer operator group. The input interface is used to receive an image and a magnification signal, wherein the magnification signal indicates that a 2x magnification operation or a 4x magnification operation should be performed on the image. The shared layer operator group includes a plurality of multiply-accumulate operators corresponding to a first number, which are used to perform a plurality of convolution operations, wherein the first number is twice the initial number of output channels. When the magnification signal indicates that the 4x magnification operation should be performed, the shared layer operator group performs the plurality of convolution operations based on a first number of input channels and a first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2. When the magnification signal indicates that the 2x magnification operation should be performed, the shared layer operator group performs the plurality of convolution operations based on a second number of input channels and a second number of output channels. The first number of input channels is twice the second number of input channels, and the first number of output channels is twice the second number of output channels.

[0006] Another object of the present invention is to provide a super-resolution method. This super-resolution method is used in a super-resolution apparatus. The super-resolution apparatus includes an input interface and a shared layer operator group, the shared layer operator group including a plurality of multiply-accumulate operators corresponding to a first number, the plurality of multiply-accumulate operators being used to perform a plurality of convolution operations, wherein the first number is twice the initial number of output channels. The super-resolution method includes the following steps: receiving an image and a magnification signal, wherein the magnification signal is used to indicate performing a 2x magnification operation or a 4x magnification operation on the image; when the magnification signal indicates performing the 4x magnification operation, the shared layer operator group performs the plurality of convolution operations based on a first number of input channels and a first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2; and when the magnification signal indicates performing the 2x magnification operation, the shared layer operator group performs the plurality of convolution operations based on a second number of input channels and a second number of output channels; wherein the first number of input channels is twice the second number of input channels, and the first number of output channels is twice the second number of output channels.

[0007] The super-resolution technology (including at least an apparatus and a method) provided by this invention restricts the correspondence between the number of input channels and the number of output channels for different amplification operations (e.g., 2x amplification and 4x amplification), so that the number of multiply-accumulate arithmetic units required for the 2x amplification and 4x amplification operations is the same. Therefore, the super-resolution technology provided by this invention can still share the same feature extraction circuit when performing amplification operations at different magnification rates, thereby saving circuit area and solving the disadvantage of the prior art that requires additional hardware circuits to be designed for different magnification rates.

[0008] The following detailed description of the technical features and embodiments of the present invention, in conjunction with the accompanying drawings, enables those skilled in the art to understand the technical features of the claimed invention. Attached Figure Description

[0009] Figure 1A This is a schematic diagram depicting the architecture of the super-resolution device according to the first embodiment;

[0010] Figure 1B This is a schematic diagram depicting the architecture of a super-resolution device in some implementations;

[0011] Figure 2 This is a schematic diagram depicting a multiply-accumulate-address operator according to the first embodiment;

[0012] Figure 3A This is a schematic diagram depicting the four-fold magnification operation corresponding to the first embodiment;

[0013] Figure 3B This is an operational schematic diagram depicting the four-fold magnification operation corresponding to the first embodiment;

[0014] Figure 4A This is a schematic diagram depicting the 2x magnification operation corresponding to the first embodiment;

[0015] Figure 4B This is an operational schematic diagram depicting the 2x magnification operation corresponding to the first embodiment;

[0016] Figure 5A This is a schematic diagram depicting the architecture of a super-resolution device in certain implementations;

[0017] Figure 5B These are schematic diagrams depicting the architecture of a super-resolution device in certain embodiments; and

[0018] Figure 6 This is a partial flowchart depicting the super-resolution method of the second embodiment.

[0019] Symbol explanation: 1: Super-resolution device 11: Input Interface 13: Shared Layer Arithmetic Unit Group 17: Output Interface M1, M2, ..., Mn: Multiply-Accumulate-Address Operator V: Pixel value W: Weight 100: Input Image 102: Input Image CH1, CH2: Convolution kernels V1, V2: numerical values A1, A2: Addition operations P1, P2, P3, P4: pixels 12: First-level arithmetic unit 14: Shared line buffer 15: Shared Layer Adder Group 600: Object Detection Model Training Methods S601, S603, S605: Steps Detailed Implementation

[0020] The following description, through embodiments, explains the super-resolution apparatus and method provided by the present invention. However, these embodiments are not intended to limit the implementation of the invention to any of the environments, applications, or methods described herein. Therefore, the description of the embodiments is for illustrative purposes only and is not intended to limit the scope of the invention. It should be understood that in the following embodiments and figures, components not directly related to the present invention have been omitted and are not shown, and the dimensions of each component and the dimensional ratios between components are merely illustrative and not intended to limit the scope of the invention.

[0021] The first embodiment of the present invention is a super-resolution device 1, the schematic diagram of which is depicted in... Figure 1A The super-resolution device 1 includes an input interface 11 and a shared layer operator group 13. The input interface 11 is an interface capable of receiving and transmitting data, or other interfaces capable of receiving and transmitting data known to those skilled in the art. The shared layer operator group 13 includes multiple multiply-accumulate operators M1, M2, ..., Mn (hereinafter referred to as the first number), which are used to perform multiple convolution operations corresponding to feature extraction. The details of performing these multiple convolution operations will be described later.

[0022] In some implementations, such as Figure 1B As shown, the super-resolution device 1 may further include an output interface 17, which is electrically connected to the shared layer processor group 13. The output interface 17 can convert the feature information output by the shared layer processor group 13 into a corresponding output image according to the selected magnification (e.g., 2x or 4x magnification).

[0023] It should be noted that, in order to allow the computing devices used for the 2x amplification operation and the 4x amplification operation to be shared, it is necessary to limit the number of multiply-accumulate operators included in the shared layer arithmetic unit group. In this embodiment, the first number is twice the initial number of output channels.

[0024] For example, when the initial output channel number of the corresponding image is 2, the shared layer operator group 13 contains 4 multiply-accumulate operators M1, M2, M3 and M4. The 2x magnification operation and the 4x magnification operation can use the multiply-accumulate operators M1, M2, M3 and M4 in the shared layer operator group 13 to complete the feature extraction operation.

[0025] It should be noted that the super-resolution device 1 also includes other necessary components or operations for performing super-resolution operations, such as pixel shuffle operations. However, the focus of this invention is on hardware-shared operations related to feature extraction, so the following paragraphs will only describe the implementation details related to this invention.

[0026] In this embodiment, the input interface 11 is used to receive an image and a magnification signal, wherein the magnification signal indicates whether to perform a 2x or 4x magnification operation on the image. In some embodiments, the super-resolution device 1 may also include a multiplexer (MUX) to select the 2x or 4x magnification operation. Furthermore, the output interface 17 can convert the feature information output by the shared layer processor group 13 into a corresponding output image based on the magnification ratio performed by the multiplexer (e.g., 2x or 4x magnification).

[0027] It should be noted that during the hardware product design, the super-resolution device 1 needs to define a maximum image resolution output size (e.g., 3840*2160). Therefore, the maximum acceptable image resolution input size of the super-resolution device 1 varies depending on the magnification. For example, when the maximum resolution output size of the super-resolution device 1 is 3840*2160, the maximum acceptable image input resolution for the super-resolution device 1 performing a 2x magnification operation is 1920*1080 (i.e., half the width and half the length). When performing a 4x magnification operation, the maximum acceptable image input resolution for the super-resolution device 1 is 960*540 (i.e., one-quarter the width and one-quarter the length).

[0028] It should be understood that when the output resolution is 3840*2160, the maximum input corresponding to a four-fold magnification is 960*540, and the input corresponding to a two-fold magnification is 1920*1080. Therefore, the difference in the amount of data that needs to be calculated for both is four times (i.e., the length and width are each twice). Therefore, in order to maintain the same frame rate (e.g., 60 frames per second) when the super-resolution device 1 performs two-fold and four-fold magnification operations, the computational load required for the two-fold magnification operation is four times that for the four-fold magnification operation when having the same maximum output. The aforementioned conditions are a key principle in the design of the hardware circuit of this invention; therefore, the shared layer arithmetic unit group 13 proposed in this invention will satisfy the aforementioned conditions.

[0029] In this embodiment, for a 2x or 4x amplification operation, the shared layer operator group 13 performs the multiple convolution operations based on different numbers of input and output channels. Specifically, when the amplification signal indicates the execution of the 4x amplification operation, the shared layer operator group 13 performs the multiple convolution operations based on a first number of input channels and a first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2.

[0030] Furthermore, when the amplification signal indicates the execution of the double amplification operation, the shared layer arithmetic unit 13 performs the multiple convolution operations based on the second number of input channels and the second number of output channels. The first number of input channels is twice the second number of input channels, and the first number of output channels is twice the second number of output channels.

[0031] In some implementations, the initial number of output channels corresponding to the shared layer operator group 13 needs to be an even number. Therefore, before the shared layer operator group 13 performs the multiple convolution operations, the super-resolution device 1 further performs a first-layer convolution operation on the image to make the initial number of output channels a multiple of 2. Specifically, please refer to... Figure 5A The super-resolution device 1 further includes a first-layer arithmetic unit 12, which is electrically connected to the input interface 11 and the shared layer arithmetic unit group 13. The first-layer arithmetic unit 12 performs a first-layer convolution operation, wherein the first-layer convolution operation corresponds to the initial number of output channels (i.e., the number of output channels generated for the image after the first-layer convolution operation is the initial number of output channels), and the initial number of output channels is an even positive integer.

[0032] In some implementations, such as Figure 5B As shown, the super-resolution device 1 may further include an output interface 17, which is electrically connected to the shared layer processor group 13. The output interface 17 can convert the feature information output by the shared layer processor group 13 into a corresponding output image according to the selected magnification (e.g., 2x or 4x magnification).

[0033] In some implementations, the plurality of multiply-accumulate operators M1, M2, ..., Mn perform the plurality of convolution operations based on convolution kernels, which correspond to width and length values, and each of the plurality of convolution operations is performed by one of the plurality of multiply-accumulate operators M1, M2, ..., Mn.

[0034] In some implementations, each of the convolution operations performed by the plurality of multiply-accumulate operators includes performing N*M multiplication operations and (N*M)-1 addition operations, where N is the width value corresponding to the convolution kernel, M is the length value corresponding to the convolution kernel, and N and M are positive integers.

[0035] For easier understanding, please refer to Figure 2 . Figure 2 This example illustrates the multiplication and addition operations performed by the multiply-accumulate operator M1 when using a convolution kernel with a width and length of 3*3. Here, V and W correspond to the pixel values ​​of the image and the weights of the convolution kernel, respectively. Figure 2It can be seen that when the convolution kernel is 3*3, the multiply-accumulate operator M1 needs to perform 9 multiplication operations and 8 addition operations.

[0036] It should be noted that in this embodiment, each multiply-accumulate operator M1, M2, ..., Mn performs the same number of multiplication and addition operations. Furthermore, since the number of multiplication and addition operations corresponding to the same convolution kernel is the same, each multiply-accumulate operator can perform the aforementioned multiplication and addition operations through a fixed number of hardware components (e.g., 9 multipliers and 8 adders). Those skilled in the art should be able to understand the implementation methods of each multiply-accumulate operator based on the foregoing, and therefore, further details are omitted.

[0037] For ease of understanding, the following paragraphs will detail specific examples of how the shared layer operator group 13 performs the multiple convolution operations in the 2x and 4x magnification operations. It should be noted that in the following specific examples, the shared layer operator group 13 will perform the multiple convolution operations corresponding to the 2x and 4x magnification operations based on different numbers of input and output channels.

[0038] First, please refer to Figure 3A and Figure 3B The example shown is a specific example of a four-fold magnification operation, but it is not intended to limit the scope of the invention. In this example, the convolution kernel is set to 3x3, and both the number of input channels and the number of output channels are set to 2. Figure 3A As shown, the shared layer arithmetic unit 13 will perform a convolution operation on the input image 100 using convolution kernels with two input channels (i.e., convolution kernel CH1 and convolution kernel CH2) to generate values ​​V1 and V2 corresponding to the two output channels.

[0039] like Figure 3B As shown, the shared layer operator group 13 performs the aforementioned convolution operation using four multiply-accumulate operators M1, M2, M3, and M4. Specifically, multiply-accumulate operators M1 and M2 calculate the value V1, and multiply-accumulate operators M3 and M4 calculate the value V2. In some embodiments, the values ​​V1 and V2 are further processed by other operations (e.g., bias, activation function, etc.) to generate pixel P1.

[0040] Next, please refer to Figure 4A and Figure 4B The example shown is a specific example of a 2x magnification operation, but it is not intended to limit the scope of the invention. In this example, the convolution kernel is also set to 3x3, but the number of input channels and the number of output channels are both set to 1 (i.e., the number of input channels and the number of output channels are half of those in the aforementioned 4x operation). Figure 4AAs shown, the shared layer operator group 13 will perform a convolution operation on the input image 102 through a convolution kernel (i.e., convolution kernel CH1) with one input channel to produce four pixels P1, P2, P3 and P4.

[0041] like Figure 4B As shown, the shared layer operator group 13 performs the convolution operation described above using the same four multiply-accumulate operators M1, M2, M3, and M4. Specifically, pixels P1, P2, P3, and P4 are calculated by the multiply-accumulate operators M1, M2, M3, and M4, respectively. In some embodiments, pixels P1, P2, P3, and P4 are further generated through other operations (e.g., bias, activation function, etc.).

[0042] As can be seen from the aforementioned specific examples, this invention limits the number of input channels and output channels corresponding to the 2x and 4x amplification operations (i.e., the number of input channels corresponding to the 4x amplification operation is twice the number of input channels corresponding to the 2x amplification operation, and the number of output channels corresponding to the 4x amplification operation is twice the number of output channels corresponding to the 2x amplification operation), so that the number of multiply-accumulate arithmetic units required for the 2x and 4x amplification operations is the same. Therefore, they can share the same hardware circuit and meet the same frame rate (i.e., the amount of computation required for the 2x amplification operation is 4 times that for the 4x amplification operation).

[0043] It should be noted that the foregoing examples are merely illustrative and are not intended to limit the scope of the present invention. It should be understood that in other embodiments, when the number of input channels or output channels increases, because the shared layer arithmetic unit group 13 has the aforementioned limiting conditions (i.e., the number of input channels corresponding to the quadruple amplification operation is twice the number of input channels corresponding to the doubly amplification operation, and the number of output channels corresponding to the quadruple amplification operation is twice the number of output channels corresponding to the doubly amplification operation), the number of multiply-accumulate arithmetic units required for the doubly amplification operation and the quadruple amplification operation will remain the same, and therefore the same hardware circuitry can be shared. Those skilled in the art should be able to understand the implementation methods with different numbers of input channels and output channels based on the foregoing content, and therefore will not be elaborated further.

[0044] In some implementations, please refer to Figure 5A The super-resolution device 1 further includes a shared layer adder group 15 electrically connected to a shared layer arithmetic unit group 13. The shared layer adder group 15 is used to perform multiple addition operations. The super-resolution device 1 generates multiple pixels corresponding to the image through the shared layer arithmetic unit group 13 and the shared layer adder group 15. For example, the shared layer adder group 15 is used to perform... Figure 3B The addition operations A1 and A2 in the middle, or Figure 4BThe addition operation of P3 and P4 is calculated in addition (i.e., the addition operation of two additional sets of bias values ​​and excitation functions).

[0045] It should be noted that the number of addition operations performed by the shared layer adder group 15 is related to the number of input channels and the number of output channels. Specifically, when the amplification signal indicates the execution of the quadruple amplification operation, the shared layer adder group 15 performs the plurality of addition operations corresponding to the second number to generate the plurality of pixels corresponding to the image. When the amplification signal indicates the execution of the 2x amplification operation, the shared layer adder group 15 performs the plurality of addition operations corresponding to the third number to generate the plurality of pixels corresponding to the image. The second number is generated based on the first number of input channels and the first number of output channels, and the second number is the same as the third number (that is, since the additional number of addition operations is the same, the 2x amplification operation and the quadruple amplification operation can share the shared layer adder group 15).

[0046] In some implementations, please refer to Figure 5A The super-resolution device 1 further includes a shared line buffer 14. The shared line buffer 14 includes a plurality of temporary storage blocks corresponding to a fourth number, wherein the shared layer arithmetic unit 13 performs the plurality of convolution operations corresponding to the four-fold magnification operation and the two-fold magnification operation based on the plurality of temporary storage blocks, and the fourth number is generated based on the first number of input channels and the first number of output channels.

[0047] It should be noted that since the number of multiply-accumulate operators required for the 2x and 4x magnification operations is the same, the number of temporary storage blocks required is also the same. Accordingly, the super-resolution device 1 can use the shared line buffer 14 to perform these multiple convolution operations through address allocation operations (e.g., pointing a pointer to the location of the temporary storage block).

[0048] As can be seen from the above description, the super-resolution device 1 provided by the present invention limits the correspondence between the number of input channels and the number of output channels for different amplification operations (e.g., 2x amplification operation and 4x amplification operation), so that the number of multiply-accumulate arithmetic units required for the 2x amplification operation and the 4x amplification operation is the same. Therefore, the super-resolution device 1 provided by the present invention can still share the same feature extraction circuit when performing amplification operations at different magnification ratios, thereby saving circuit area and solving the disadvantage of the prior art that requires additional design of hardware circuits corresponding to different magnification ratios.

[0049] The second embodiment of the present invention is a super-resolution method, the flowchart of which is depicted in Figure 6The super-resolution method 600 is applicable to a super-resolution device, such as the super-resolution device 1 described in the first embodiment. This super-resolution device includes an input interface and a shared layer operator group, which includes a plurality of multiply-accumulate operators corresponding to a first number, used to perform a plurality of convolution operations, wherein the first number is twice the initial number of output channels, for example, the input interface 11 and the shared layer operator group 13 described in the first embodiment. The super-resolution method 600 performs the plurality of convolution operations using the shared layer operator group through steps S601 to S605.

[0050] In some embodiments, the super-resolution device may further include an output interface, such as the output interface 17 described in the first embodiment. The output interface is electrically connected to the shared layer processor group. The output interface can convert the feature information output by the shared layer processor group into a corresponding output image according to the selected magnification (e.g., 2x or 4x magnification).

[0051] In step S601, the super-resolution device receives an image and a magnification signal, wherein the magnification signal indicates whether to perform a 2x or 4x magnification operation on the image. In step S603, when the magnification signal indicates the 4x magnification operation, the shared layer arithmetic unit group performs the plurality of convolution operations based on a first number of input channels and a first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2.

[0052] In step S605, when the amplification signal indicates the execution of the double amplification operation, the shared layer arithmetic unit group performs the multiple convolution operations based on the second number of input channels and the second number of output channels. The first number of output channels is twice the second number of output channels, and the first number of channels is twice the second number of channels.

[0053] In some embodiments, the super-resolution device further includes a first-layer arithmetic unit electrically connected to the input interface and the shared layer arithmetic unit group, wherein the super-resolution method 600 further includes the following steps: the first-layer arithmetic unit performs a first-layer convolution operation, wherein the first-layer convolution operation corresponds to the initial number of output channels, and the initial number of output channels is an even positive integer.

[0054] In some embodiments, where the image is mapped to an input resolution and the super-resolution device corresponds to a maximum output resolution, the super-resolution method 600 further includes the following steps: when the magnification signal indicates the execution of the quadruple magnification operation, the input resolution corresponding to the image is at most one-quarter of the width and one-quarter of the length of the maximum output resolution; and when the magnification signal indicates the execution of the double magnification operation, the input resolution corresponding to the image is at most one-half the width and one-half the length of the maximum output resolution.

[0055] In some embodiments, the super-resolution apparatus further includes a shared layer adder group electrically connected to the shared layer arithmetic unit group, wherein the super-resolution method 600 further includes the following steps: performing a plurality of addition operations by the shared layer adder group; wherein the shared layer arithmetic unit group and the shared layer adder group are used to generate a plurality of pixels corresponding to the image.

[0056] In some embodiments, the super-resolution method 600 further includes the following steps: when the amplification signal indicates the execution of the four-fold amplification operation, the shared layer adder group performs the plurality of addition operations corresponding to a second number to generate the plurality of pixels corresponding to the image.

[0057] In some embodiments, the super-resolution method 600 further includes the following steps: when the amplification signal indicates the execution of the 2x amplification operation, the shared layer adder group performs the plurality of addition operations corresponding to a third number to generate the plurality of pixels corresponding to the image.

[0058] In some implementations, the second quantity is generated based on the first number of input channels and the first number of output channels, and the second quantity is the same as the third quantity.

[0059] In some implementations, the plurality of multiply-accumulate operators perform the plurality of convolution operations based on convolution kernels, the convolution kernels corresponding to width and length values, and each of the plurality of convolution operations is performed by one of the plurality of multiply-accumulate operators.

[0060] In some implementations, each of the convolution operations performed by the plurality of multiply-accumulate operators includes the following operations: performing N*M multiplication operations and (N*M)-1 addition operations, where N is the width value corresponding to the convolution kernel, M is the length value corresponding to the convolution kernel, and N and M are positive integers.

[0061] In some embodiments, the super-resolution device further includes a shared line buffer containing a plurality of temporary storage blocks corresponding to a fourth number, wherein the super-resolution method 600 further includes the following steps: the shared layer operator performs the plurality of convolution operations corresponding to the quadruple magnification operation and the double magnification operation based on the plurality of temporary storage blocks, wherein the fourth number is generated based on the first number of input channels and the first number of output channels.

[0062] In addition to the steps described above, the second embodiment can also perform all the operations and steps of the super-resolution device 1 described in the first embodiment, have the same function, and achieve the same technical effect. Those skilled in the art to which this invention pertains can directly understand how the second embodiment performs these multiple operations and steps based on the first embodiment described above, has the same function, and achieves the same technical effect, so it will not be described in detail here.

[0063] It should be noted that in the specification and claims of this invention, certain terms (including the number of input channels, the number of output channels, and quantity) are preceded by "first," "second," "third," or "fourth." These "first," "second," "third," or "fourth" are used only to distinguish different terms. For example, "first" and "second" in "first number of input channels" and "second number of input channels" are used only to indicate different numbers of input channels used in different stages.

[0064] In summary, the super-resolution technology (including at least an apparatus and a method) provided by this invention restricts the correspondence between the number of input channels and the number of output channels for different amplification operations (e.g., 2x amplification and 4x amplification), ensuring that the number of multiply-accumulate arithmetic units required for the 2x and 4x amplification operations is the same. Therefore, the super-resolution technology provided by this invention can still share the same feature extraction circuit during amplification operations at different magnification rates, thereby saving circuit area and overcoming the disadvantage of prior art requiring additional hardware circuits to be designed for different amplification rates.

[0065] Although preferred and feasible embodiments of the present invention have been disclosed above, these embodiments are not intended to limit the present invention. Those skilled in the art can make changes to the technical features of the present invention based on the explicit or implicit content of the present invention. All such changes fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be determined by the scope defined in the claims of this application.

Claims

1. A super-resolution device, characterized in that, The super-resolution device includes: An input interface for receiving images and magnification signals, wherein the magnification signals indicate whether to perform a 2x or 4x magnification operation on the image; and A shared layer operator group includes a plurality of multiply-accumulate operators corresponding to a first number, the plurality of multiply-accumulate operators being used to perform a plurality of convolution operations, wherein the first number is twice the number of initial output channels; When the amplification signal indicates the execution of the quadruple amplification operation, the shared layer arithmetic unit group performs the plurality of convolution operations based on the first number of input channels and the first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2; When the amplification signal indicates the execution of the double amplification operation, the shared layer arithmetic unit group performs the plurality of convolution operations based on the second number of input channels and the second number of output channels; Wherein, the number of the first input channels is twice the number of the second input channels, and the number of the first output channels is twice the number of the second output channels; The shared layer arithmetic unit group contains multiple multiply-accumulate arithmetic units that are shared when performing the quadruple amplification operation and the double amplification operation.

2. The super-resolution device as described in claim 1, characterized in that, The super-resolution device further includes: The first layer arithmetic unit is electrically connected to the input interface and the shared layer arithmetic unit group, and is used to perform a first layer convolution operation, wherein the first layer convolution operation corresponds to the initial number of output channels, and the initial number of output channels is an even positive integer.

3. The super-resolution device as described in claim 1, characterized in that, The image is mapped to the input resolution, and the super-resolution device corresponds to the maximum output resolution. The super-resolution device also includes the following operations: When the magnification signal indicates that the four-fold magnification operation is performed, the input resolution corresponding to the image is at most one-quarter of the width and one-quarter of the length of the maximum output resolution; as well as When the magnification signal indicates that the 2x magnification operation is performed, the input resolution corresponding to the image is at most half the width and half the length of the maximum output resolution.

4. The super-resolution device as described in claim 1, characterized in that, The super-resolution device further includes: A shared layer adder group is electrically connected to the shared layer arithmetic unit group and is used to perform multiple addition operations; The shared layer arithmetic unit group and the shared layer adder group are used to generate multiple pixels corresponding to the image.

5. The super-resolution device as described in claim 4, characterized in that, The super-resolution device also includes the following operations: When the amplification signal indicates the execution of the four-fold amplification operation, the shared layer adder group performs the plurality of addition operations corresponding to the second number to generate the plurality of pixels corresponding to the image.

6. The super-resolution device as described in claim 5, characterized in that, The super-resolution device also includes the following operations: When the amplification signal indicates the execution of the 2x amplification operation, the shared layer adder group performs up to a third number of the addition operations to generate the plurality of pixels corresponding to the image.

7. The super-resolution device as claimed in claim 1, characterized in that, The plurality of multiply-accumulate operators perform the plurality of convolution operations based on convolution kernels, wherein the convolution kernels correspond to width and length values, and each of the plurality of convolution operations is performed by one of the plurality of multiply-accumulate operators.

8. The super-resolution device as described in claim 7, characterized in that, Each of the convolution operations performed by the plurality of multiply-accumulate operators includes the following operations: Execute N M multiplication operations and (N) M-1 addition operations, where N is the width value corresponding to the convolution kernel, M is the length value corresponding to the convolution kernel, and N and M are positive integers.

9. The super-resolution device as claimed in claim 1, characterized in that, The super-resolution device further includes: A shared line buffer contains multiple temporary storage blocks corresponding to a fourth number, wherein the shared layer arithmetic unit performs the multiple convolution operations corresponding to the quadruple amplification operation and the double amplification operation based on the multiple temporary storage blocks, and the fourth number is generated based on the first number of input channels and the first number of output channels.

10. A super-resolution method for a super-resolution device, characterized in that, The super-resolution device includes an input interface and a shared layer operator group. The shared layer operator group includes a plurality of multiply-accumulate operators corresponding to a first number, which are used to perform a plurality of convolution operations. The first number is twice the initial number of output channels. The super-resolution method includes the following steps: Receive an image and a magnification signal, wherein the magnification signal is used to indicate whether to perform a 2x or 4x magnification operation on the image; When the amplification signal indicates the execution of the quadruple amplification operation, the shared layer arithmetic unit group performs the plurality of convolution operations based on the first number of input channels and the first number of output channels, wherein the first number of input channels and the first number of output channels are at least even positive integers greater than or equal to 2; as well as When the amplification signal indicates the execution of the double amplification operation, the shared layer arithmetic unit group performs the plurality of convolution operations based on the second number of input channels and the second number of output channels; Wherein, the number of the first input channels is twice the number of the second input channels, and the number of the first output channels is twice the number of the second output channels; The shared layer arithmetic unit group contains multiple multiply-accumulate arithmetic units that are shared when performing the quadruple amplification operation and the double amplification operation.

Images