Apparatus and method for bilinear upsampling operation
The bilinear upsampling operation device using a depth-wise convolutional accelerator addresses the inefficiency of NPUs by applying preset filters through a depth-wise convolutional algorithm, enhancing performance in deep learning applications.
Patent Information
- Application Number
- KR1020230191076
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-12-26
- Filing Date
- 2023-12-26
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2043-12-26
AI Technical Summary
Conventional neural processing units (NPUs) struggle to efficiently perform bilinear upsampling operations, which are computationally intensive and often relegated to CPUs, leading to performance degradation in deep learning applications like image segmentation.
A bilinear upsampling operation device utilizing a depth-wise convolutional accelerator, comprising a memory and processor, applies preset filters through a depth-wise convolutional acceleration algorithm to input pixel data, generating output pixel data and an upsampled image.
This approach significantly reduces execution time by replacing bilinear upsampling with depth-wise convolution, enabling efficient processing within the NPU.
Smart Images

Figure 112023145361964-PAT00002_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to an arithmetic apparatus and method for performing a bilinear upsampling operation. Background Technology
[0002] Recently, a lot of hardware has been developed to accelerate convolutional neural networks, and processors known as Neural Processing Units (NPUs) belong to the category of convolutional neural network accelerators. Conventional NPUs were designed for the purpose of accelerating general 3D convolutions, but with the recent development of Convolutional Neural Networks (CNNs) based on depth-wise convolutions, methods to accelerate depth-wise convolutions are also being researched and developed.
[0003] Depth-wise convolution is widely used in convolutional neural networks. In depth-wise convolution, 3D data is provided as input, which consists of 2D input images with distinct features, each of which is referred to as a channel. Then, a KxK filter is applied to the 2D input image for each channel to obtain an output image with filtering applied to each channel. In depth-wise convolution, the filters applied to each channel have different characteristics. Figure 1 is an example of depth-wise convolution, showing an example where an input image with 3 channels is transformed into an output image with 3 channels through depth-wise convolution.
[0004] However, if there are operations that an accelerator utilizing a convolutional neural network operating on this principle cannot handle, it is common practice to have those operations performed by other processors, such as a CPU. One of the operations that an accelerator cannot process is bilinear upsampling, and since bilinear upsampling requires a large amount of computation, having it performed on a CPU results in significant performance degradation.
[0005] Bilinear upsampling is an operation that smoothly transforms a small screen into a large screen and is actively used in deep learning computer vision applications such as image segmentation. For example, DeepLabV3+, a deep learning model for image segmentation, performs bilinear interpolation twice in the latter part of the network. Bilinear upsampling requires a large amount of computation because it must apply bilinear interpolation between input pixels to calculate a single output pixel.
[0006] Therefore, a technology is required to enable these bilinear upsampling operations to be processed by a neural processing unit. Prior art literature
[0007] Korean Published Patent No. 10-2022-0155737 (Title of Invention: Method and Apparatus for Generating Super-Resolution Images Using a Low-Lightweight Convolutional Neural Network Structure) The problem to be solved
[0008] The present invention has the technical objective of providing a computational device and method that perform bilinear upsampling operations using a depth-dependent convolutional accelerator to solve the aforementioned problems.
[0009] However, the technical problems that this embodiment aims to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem
[0010] As a technical means for solving the technical problem described above, a bilinear upsampling operation device according to one embodiment of the present invention comprises a memory storing an operation program and a processor executing said operation program, wherein the operation program receives input pixel data for each of a plurality of channels of an input image, calculates output pixel data for each channel by applying a plurality of preset filters for upsampling by N (N is a natural number greater than or equal to 2) times to the input pixel data of each channel, provides a bilinear upsampled output image using said output pixel data of each channel, and the plurality of filters are combinations of weights applied to the input pixel data.
[0011] In addition, a bilinear upsampling operation method according to another embodiment of the present invention comprises: receiving input pixel data for each of a plurality of channels of an input image; applying a plurality of preset filters for N (N is a natural number greater than or equal to 2) times upsampling to the input pixel data of each channel to produce output pixel data of each channel; and providing a bilinearly upsampled output image using the output pixel data of each channel, wherein the plurality of filters are combinations of weights applied to the input pixel data. Effects of the invention
[0012] According to the means for solving the problem of the present invention described above, the execution time can be significantly reduced by replacing the bilinear upsampling operation with a depth-wise convolution operation. Brief explanation of the drawing
[0013] Figure 1 is an example of a conventional depth-based convolution operation. FIG. 2 is a conceptual diagram of a bilinear upsampling operation device according to one embodiment of the present invention. FIGS. 3 to 8 are exemplary diagrams for explaining the bilinear upsampling operation of an arithmetic device. FIG. 9 is a flowchart illustrating a bilinear upsampling operation method according to an embodiment of the present invention. Specific details for implementing the invention
[0014] The present invention will be described in detail below with reference to the attached drawings. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings. In order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and the size, form, and shape of each component shown in the drawings may be varied in various ways. Identical or similar parts throughout the specification are denoted by identical or similar reference numerals.
[0015] In the following description, suffixes such as "module" and "part" for components are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.
[0016] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.
[0017] Terms indicating ordinal numbers, such as first, second, etc., used in this specification are used solely for the purpose of distinguishing one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be named the second component, and similarly, the second component may be named the first component.
[0018] FIG. 2 is a conceptual diagram schematically illustrating a bilinear upsampling operation device according to one embodiment of the present invention.
[0019] Referring to FIG. 2, a bilinear upsampling operation device (100) according to an embodiment of the present invention will be described. The bilinear upsampling operation device (100) applies a plurality of preset filters to input pixel data for each channel of an input image using a depth-wise convolutional acceleration algorithm to produce bilinear upsampled output pixel data, and provides a bilinear upsampled output image using the output pixel data of each channel. Here, the bilinear upsampling operation device (100) may be a depth-wise convolutional accelerator.
[0020] To perform such an operation, the bilinear upsampling operation unit (100) includes memory (110) and a processor (120).
[0021] Memory (110) stores a computational program, and memory (110) should be interpreted as a general term for non-volatile storage devices that retain stored information even when power is not supplied, and volatile storage devices that require power to retain stored information. Memory (110) can perform the function of temporarily or permanently storing data processed by the processor (120). Memory (110) may include magnetic storage media or flash storage media in addition to volatile storage devices that require power to retain stored information, but the scope of the present invention is not limited thereto.
[0022] And, the processor (120) executes a computation program stored in memory (110) to apply a plurality of preset filters to input pixel data for each channel of the input image and provides a bilinear upsampled output image. In this embodiment, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.
[0023] Additionally, the processor (120) executes a computation program stored in memory (110) and provides the function of controlling the hardware of the bilinear upsampling computation device (100) according to the execution of the program. That is, the processor (120) can perform hardware control functions such as necessary file systems, memory allocation, networks, basic libraries, timers, device control (display, media, input devices, 3D, etc.), and other utilities as the program is executed.
[0024] Next, the bilinear upsampling operation of the computation program is explained.
[0025] The computation program receives an upsampling multiplier and input pixel data for each of the multiple channels of the input image, and generates padding data for each channel by applying padding operations to the input pixel data for each channel. Then, using a depth-wise convolutional acceleration algorithm, it applies multiple preset filters for N-fold upsampling (where N is a natural number greater than or equal to 2) to the padding data of each channel to produce output pixel data for each channel. Subsequently, an output image upsampled by N times is provided using the output pixel data of each channel.
[0026] Here, N-fold upsampling means that the width and height of each channel increase by N times, and the total number of pixels is N 2 It means a multiple increase. In addition, multiple filters are combinations of weights applied to input pixel data, have a size of 2x2, and N 2 It is prepared as a dog.
[0027] The computation program applies different filters at each depth set in the depth-wise convolutional acceleration algorithm, applies the same filter to each channel at each depth, and produces one output pixel for each channel at each depth. The depth of the depth-wise convolutional acceleration algorithm is N, which corresponds to the number of filters. 2 It is formed into dogs.
[0028] FIGS. 3 to 8 are exemplary diagrams for explaining bilinear upsampling operations. The bilinear upsampling operation of the computing device (100) will be explained by example with reference to FIGS. 3 to 8.
[0029] As illustrated in FIG. 3, the operation of upsampling a 3-channel input image by 2x is explained as an example. The input image and the filter are 2x2 in size, and four filters are applied to the input image. As shown in FIG. 4, the operation program receives input pixel data (10) for each channel and applies padding operations to the input pixel data (10) of each channel to generate padding data (20) for each channel.
[0030] Then, a plurality of preset filters (30) for 2x upsampling are applied to the padding data (20) of each channel through a depth-wise convolution acceleration algorithm to produce output pixel data (40) of each channel.
[0031] Referring to FIGS. 5 and FIGS. 6, an upsampling operation for a first position (21) of padding data (20) is described. Referring to FIGS. 5, the first position (21) of padding data (20) is the original input pixel data (10). When four filters (30) are applied to the first position (21), output pixels included in the first position (41) of output pixel data (40) can be produced as shown in FIGS. 6.
[0032] Specifically, by applying the first filter (30-1) to the first position (21) of the padding data (20), the first output pixel of the first position (41) of the output pixel data (40) can be produced, and by applying the second filter (30-2) to the first position (21) of the padding data (20), the second output pixel of the first position (41) of the output pixel data (40) can be produced. Additionally, by applying the third filter (30-3) to the first position (21) of the padding data (20), the third output pixel of the first position (41) of the output pixel data (40) can be produced, and by applying the fourth filter (30-4) to the first position (21) of the padding data (20), the fourth output pixel of the first position (41) of the output pixel data (40) can be produced.
[0033] Here, the first to fourth filters (30-1, 30-2, 30-3, 30-4) are filters that produce output pixels at positions 1, 2, 3, and 4, respectively, and are set at the first to fourth depths of the depth-wise convolution acceleration algorithm so that when padding data (20) passes through the first to fourth depths, output pixels included at the first position (41) of the output pixel data (40) can be produced.
[0034] Next, with reference to FIGS. 7 and FIGS. 8, an operation to calculate the pixels included in the second position (42) excluding the first position (41) in the output pixel data (40) will be described.
[0035] If only the input pixel data (10) is used, the number of pixels used to calculate the output pixel included in the second position (42) of the output pixel data (40) is less than 4, so the output pixel included in the second position (42) of the output pixel data (40) cannot be calculated. For this reason, the operation program generates padding data (20) to calculate the output pixel included in the second position (42) of the output pixel data (40).
[0036] By applying a filter (30) to the second position (22) of the padding data (20) shown in FIG. 7 (a), an output pixel included in the virtual position (50) of FIG. 8 can be produced. Here, since the 4th pixel of the virtual position (50) is identical to the 5th output pixel of the second position (42) of the output pixel data (40), the 5th output pixel of the output pixel data (40) can be produced by applying a fourth filter (30-4) to the second position (22) of the padding data (20).
[0037] With such operation, the 6th output pixel of the output pixel data (40) can be produced by applying the third filter (30-3) to the third position (23) of the padding data (20) shown in FIG. 7 (b), the 7th output pixel of the output pixel data (40) can be produced by applying the second filter (30-2) to the fourth position (24) of the padding data (20) shown in FIG. 7 (c), and the 8th output pixel of the output pixel data (40) can be produced by applying the first filter (30-1) to the fifth position (25) of the padding data (20) shown in FIG. 7 (d). Other output pixels included in the second position (42) of the output pixel data (40) can also be produced in a corresponding manner.
[0038] The computation program provides a 2x upsampled output image using the output pixel data (40) for each channel calculated in this way.
[0039] Additionally, the communication module (130) may include a device comprising hardware and software necessary to perform data communication with an external device and to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device. Furthermore, the database (140) may store various data for the operation of a computation program. For example, multiple filter information set for each multiple may be stored.
[0040] FIG. 9 is a flowchart illustrating a bilinear upsampling operation method according to an embodiment of the present invention. With reference to FIG. 2 and FIG. 9, an operation method (S100) using a bilinear upsampling operation device (100) according to an embodiment of the present invention will be described.
[0041] A bilinear upsampling operation device (100) receives input pixel data for each of a plurality of channels of an input image (step S110), and applies a plurality of preset filters for N (N is a natural number greater than or equal to 2) times upsampling to the input pixel data of each channel to produce output pixel data for each channel (step S120). Subsequently, a bilinear upsampled output image is provided using the output pixel data of each channel (step S130). Here, the plurality of filters are combinations of weights applied to the input pixel data, have a size of 2x2, and N 2 It is prepared as a dog.
[0042] Next, the process of calculating the output pixel data of each channel (step S120) is explained in detail.
[0043] The process of calculating output pixel data for each channel (step S120) performs a bilinear upsampling operation based on a depth-wise convolution acceleration algorithm, applies a padding operation to the input pixel data for each of the multiple channels of the input image to generate padding data for each channel, and applies multiple filters to each padding data to calculate output pixel data for each channel.
[0044] The process of producing output pixel data for each channel (step S120) applies different filters at each depth set in the depth-wise convolutional acceleration algorithm, applies the same filter to each channel at each depth, and produces one output pixel for each channel at each depth. The depth of the depth-wise convolutional acceleration algorithm is N, which corresponds to the number of filters. 2 It is formed into dogs.
[0045] The present invention may also be embodied in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and non-removable media, implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data.
[0046] Additionally, although the method and system of the present invention have been described in relation to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.
[0047] A person skilled in the art to which the present invention pertains will understand that, based on the foregoing description, modifications can be easily made to other specific forms without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included within the scope of the present invention.
[0048] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols
[0049] 100: Bilinear Upsampling Unit 110: Memory 120: Processor
Claims
Claim 1 A computing device for performing a bilinear upsampling operation comprises a memory storing a computing program and a processor executing said computing program, wherein the computing program receives input pixel data for each of a plurality of channels of an input image, calculates output pixel data for each channel by applying a plurality of preset filters for N (N is a natural number greater than or equal to 2) times upsampling to the input pixel data of each channel, and provides a bilinear upsampled output image using said output pixel data of each channel, wherein the plurality of filters are combinations of weights applied to the input pixel data, and N 2 A bilinear upsampling computing device comprising a plurality of input pixel data for each of the input image channels, wherein the computing device is a depth-wise convolutional accelerator and the computing program performs the bilinear upsampling operation based on a depth-wise convolutional accelerator algorithm, wherein the padding operation is applied to the input pixel data for each of the plurality of channels of the input image to generate padding data for each channel, and the plurality of filters are applied to the padding data to produce output pixel data for each channel. Claim 2 delete Claim 3 delete Claim 4 In claim 1, the operation program is a bilinear upsampling operation device that applies the same filter to each channel to produce one output pixel per channel. Claim 5 In claim 1, the operation program is a bilinear upsampling operation device that applies different filters at each depth set in the depth-wise convolution acceleration algorithm. Claim 6 A bilinear upsampling operation method performed by a bilinear upsampling operation device comprises: receiving input pixel data for each of a plurality of channels of an input image; calculating output pixel data for each channel by applying a plurality of preset filters for N (N is a natural number greater than or equal to 2) times upsampling to the input pixel data of each channel; and providing a bilinearly upsampled output image using the output pixel data of each channel, wherein the plurality of filters are combinations of weights applied to the input pixel data, and N 2 A bilinear upsampling operation method comprising: a method wherein the computing device is a depth-wise convolutional accelerator, and the step of producing the output pixel data comprises performing the bilinear upsampling operation based on a depth-wise convolutional accelerator algorithm, wherein a padding operation is applied to the input pixel data for each of the plurality of channels of the input image to generate padding data for each channel, and the plurality of filters are applied to each padding data to produce output pixel data for each channel. Claim 7 delete Claim 8 delete Claim 9 In claim 6, the step of producing the output pixel data is a bilinear upsampling operation method that produces one output pixel per channel by applying the same filter to each channel. Claim 10 In claim 6, the step of producing the output pixel data is a bilinear upsampling operation method that applies different filters at each depth set in the depth-wise convolution acceleration algorithm. Claim 11 A non-transient computer-readable recording medium having a computer program recorded thereon for a processor to perform a bilinear upsampling operation method according to any one of paragraphs 6, 9 and 10.
Citation Information
Patent Citations
Method and apparatus for image processing
CN114202657A
Adaptive spatial variant interpolation for image upscaling
US20080036792A1
Method for Filtering Data with Arbitrary Kernel Filters
US20080219580A1
Apparatus and Method for Parallel Polyphase Image Interpolation
US20180108113A1
Deep neural network operation method and apparatus
US20210397953A1
