Electronic device for performing convolution operation for acquiring output image by up-scaling resolution of input image, and operation method of electronic device

Abstract

An electronic device comprises a memory and a processor, wherein the electronic device may: acquire an input image comprising a plurality of pixels; acquire a plurality of interleaving parameters in which a plurality of parameters included in each of a plurality of convolution filters for performing a convolution operation are interleaved; acquire a plurality of interleaving receptive pixels in which a plurality of receptive pixels included in receptive fields respectively corresponding to the plurality of convolution filters are interleaved; and acquire an output image on the basis of a result of a multiplication operation included in a convolution operation for an interleaving parameter having a non-zero value from among the plurality of interleaving parameters and an interleaving receptive pixel having a non-zero value from among the plurality of interleaving receptive pixels.

Description

An electronic device that performs a convolution operation to obtain an output image by upscaling the resolution of an input image, and a method of operation of the electronic device

[0001] The present disclosure relates to an electronic device and a method of operating the electronic device. Specifically, it relates to an electronic device and a method of operating the electronic device for performing a convolution operation to obtain an output image by up-scaling the resolution of an input image.

[0002] Recently, with technological advancements, electronic devices offering high-resolution displays are being provided. However, since the majority of content on the market is produced at a resolution lower than that of these devices, technology utilizing artificial intelligence to improve image quality is being employed as a method to fully utilize the high-resolution displays of these devices.

[0003] Among the technologies that improve image quality using artificial intelligence, a technology that generates and provides content with improved image quality by performing convolution operations is being utilized.

[0004] At this time, to implement artificial intelligence, technology is being utilized to design and use high-performance parallel computing hardware that includes arithmetic units such as multipliers and accumulators.

[0005] One embodiment of the present disclosure provides an electronic device for performing a convolution operation to obtain an output image by up-scaling the resolution of an input image. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor including a processing circuit. By having at least one processor execute the program or at least one instruction stored in memory individually or collectively, the electronic device may obtain an input image comprising a plurality of pixels. The electronic device may obtain a plurality of interleaving parameters in which a plurality of parameters included in each of a plurality of convolution filters for performing a convolution operation are interleaving. The electronic device may obtain a plurality of interleaving receptive pixels in which a plurality of receptive pixels included in receptive fields corresponding to each of a plurality of convolution filters among a plurality of pixels are interleaving. The electronic device can acquire an output image based on the result of a multiplication included in a convolution operation for at least one interleaving parameter having a non-zero value among a plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among a plurality of interleaving accepting pixels.

[0006] In one embodiment of the present disclosure, a method of operation of an electronic device for performing a convolution operation to upscale the resolution of an input image may be provided. The method of operation of the electronic device may include the step of acquiring an input image comprising a plurality of pixels. The method of operation of the electronic device may include the step of acquiring a plurality of interleaving parameters, wherein a plurality of parameters included in each of a plurality of convolution filters for performing a convolution operation are interleaving. The method of operation of the electronic device may include the step of acquiring a plurality of interleaving accepting pixels, wherein a plurality of accepting pixels included in a receptive field corresponding to each of the plurality of convolution filters among the plurality of pixels are interleaving. The method of operation of the electronic device may include the step of acquiring an output image based on the result of a multiplication included in a convolution operation for at least one interleaving parameter having a non-zero value among the plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among the plurality of interleaving accepting pixels.

[0007] In one embodiment of the present disclosure, a computer-readable recording medium may be provided on which a program for performing at least one of the embodiments of the method of operating the disclosed electronic device is recorded on a computer.

[0008] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0009] The present disclosure may be understood by the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0010] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a diagram illustrating the operation of an electronic device that performs a convolution operation on interleaved parameters and interleaved receiving pixels according to one embodiment of the present disclosure.

[0014] FIG. 5 is a flowchart for explaining the operation of reading a plurality of receiving pixels in an interleaved state from a memory according to one embodiment of the present disclosure and acquiring a plurality of interleaved receiving pixels.

[0015] FIG. 6 is a diagram illustrating a method of interleaving parameters and receiving pixels according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating the scheduling of a multiplier operation so as not to perform a multiplication operation as at least one of the interleaving parameter or the interleaving accepting pixel has a value of "0", according to one embodiment of the present disclosure.

[0017] FIG. 8 is a flowchart illustrating an operation to obtain the result of a product operation of a representative parameter and a plurality of interleaving accepting pixels according to one embodiment of the present disclosure as the result of a product operation of each of two interleaving sub-parameters and a plurality of interleaving accepting pixels.

[0018] FIG. 9 is a diagram illustrating an operation to obtain the result of a product operation of a representative parameter and a plurality of interleaving accepting pixels according to one embodiment of the present disclosure as the result of a product operation of each of two interleaving sub-parameters and a plurality of interleaving accepting pixels.

[0019] FIG. 10 is a flowchart illustrating the scheduling of an accumulator's operation to accumulate the result of a product operation included in a convolution operation for interleaved parameters and interleaved receiving pixels, according to one embodiment of the present disclosure.

[0020] FIG. 11 is a flowchart for explaining the operation of an electronic device that performs a multiplication operation included in a convolution operation for interleaved parameters and interleaved receiving pixels according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating the operation of an electronic device that obtains an output image by upscaling an input image including a plurality of channels through a convolution operation according to one embodiment of the present disclosure.

[0022] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0023] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Accordingly, "A or B" may mean "A, B, or both" unless clearly indicated otherwise by the context.

[0024] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0025] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0026] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification.

[0027] Throughout this disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as used in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or as a combination of hardware and software.

[0028] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0029] In addition, when a component is described in the present disclosure as being “connected” or “connected” to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0030] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0031] All functions or operations described in this document may be processed by a single processor or a combination of multiple processors.

[0032] Functions related to artificial intelligence according to the present disclosure are operated through processors and memory. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0033] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the electronic device itself in which the artificial intelligence model according to the present disclosure is used, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0034] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a Deep Neural Network (DNN), such as a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the examples mentioned above.

[0035] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0036] Embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0037] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0038] Referring to FIG. 1, in one embodiment of the present disclosure, an electronic device (100) can acquire an input image (200). The electronic device (100) can acquire an output image (300) by up-scaling the resolution of the acquired input image (200). The electronic device (100) can provide the acquired output image (300) to a user using the electronic device (100).

[0039] In one embodiment of the present disclosure, the electronic device (100) may be implemented as an electronic device of various shapes, such as a television, a mobile device, a smartphone, a laptop computer, a desktop, a tablet PC, and a wearable device.

[0040] However, the present disclosure is not limited thereto, and the electronic device (100) may be implemented as an electronic device of the form of a desktop, set-top box, or server device. The electronic device (100) may receive an input image (200) from an external electronic device (e.g., an image generating device) connected through an input / output interface and provide the acquired output image (300) to an external electronic device (e.g., an image display device). Additionally, the electronic device (100) may receive an input image (200) from an external electronic device or an external server through a communication interface and provide the acquired output image (300) to an external electronic device or an external server.

[0041] However, the present disclosure is not limited thereto, and the input image (200) may be generated through an electronic device (100), and the electronic device (100) may obtain an output image (300) by upscaling the resolution of the generated input image (200).

[0042] In one embodiment of the present disclosure, the input image (200) may be an image comprising a plurality of pixels. The input image (200) may be an image having a first resolution composed of a plurality of pixels. The input image (200) may be a color or black-and-white image. However, the present disclosure is not limited thereto, and the input image (200) may be an image corresponding to a single frame of a video composed of a plurality of frames.

[0043] In this case, the electronic device (100) may obtain an output image (300) which is a video composed of images with upscaled resolution.

[0044] In one embodiment of the present disclosure, the output image (300) may be an image comprising a plurality of pixels. The output image (300) may be an image having a second resolution composed of a plurality of pixels. In this case, the second resolution may be a higher resolution than the first resolution. The number of a plurality of pixels included in the output image (300) may be greater than the number of a plurality of pixels included in the input image (200).

[0045] In one embodiment of the present disclosure, the electronic device (100) may include an upscaling module (111) for upscaling the resolution of an input image (200). An upscaling module (111) composed of instructions or program code regarding an operation or function for upscaling the resolution of an image may be stored in a memory (110) included in the electronic device (100). The electronic device (100) may upscale the resolution of the input image (200) using the upscaling module (111).

[0046] In one embodiment of the present disclosure, the upscaling module (111) may include an artificial intelligence model that is pre-trained to receive an input image (200) of a first resolution as input, upscale the input image (200), and infer an output image (300) of a second resolution. At this time, the artificial intelligence model included in the upscaling module (111) may be a Deep Neural Network (DNN) and may be an artificial intelligence model that includes a layer performing convolution operations. The artificial intelligence model included in the upscaling module (111) may include a Convolutional Neural Network (CNN) or a Generative Adversarial Network (GAN), etc. However, the artificial intelligence model in the present disclosure is not limited to the examples described above.

[0047] In one embodiment of the present disclosure, the convolution operation in the upscaling module (111) may be performed using a plurality of convolution filters. In this case, the "convolution filter" may be a filter for extracting features such as boundaries, patterns, and textures of an input image (200) through the convolution operation. The convolution filter may include at least one parameter that has been pre-learned to extract features of the input image (200).

[0048] At least one parameter included in each of the plurality of convolutional filters may have different values ​​depending on the difference in features that each of the plurality of convolutional filters is to extract. The plurality of convolutional filters may be different convolutional filters for extracting different features of the input image (200).

[0049] In one embodiment of the present disclosure, the number of parameters included in the convolutional filter may vary depending on the size of the convolutional filter. Hereinafter, for convenience of explanation, the convolutional filter is described as including multiple parameters. However, it is obvious that when the size of the convolutional filter is 1*1, the convolutional filter may include only one parameter.

[0050] In one embodiment of the present disclosure, an electronic device (100) may perform a convolution operation by multiplying each of a plurality of convolution filters and accumulating the result of the multiplication operation for a plurality of receptive pixels included in a receptive field corresponding to each of a plurality of convolution filters among a plurality of pixels included in an input image (200). At this time, "receptive field" may refer to an area among a plurality of pixels of the input image (200) that corresponds to the size of the convolution filter. In one embodiment of the present disclosure, when the convolution filter has a size of 3*3, the receptive field may refer to a pixel area having a size of 3*3. Additionally, "pixel" may include feature information of the input image (200) as unit information of the input image (200).

[0051] In one embodiment of the present disclosure, the electronic device (100) may include a multiplier, an accumulator, or a multiply-accumulate (MAC) unit designed in hardware using a hardware description language, such as Verilog or VHDL (VHSIC Hardware Description Language), to perform multiplication and accumulation operations. Additionally, the electronic device (100) may include an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or a hardware accelerator device designed to include a multiplier, an accumulator, or a multiply-accumulate (MAC).

[0052] In one embodiment of the present disclosure, the electronic device (100) may schedule the operation of a plurality of multipliers to perform a convolution operation on an input image (200). Additionally, the electronic device (100) may schedule the operation of a plurality of accumulators to perform a convolution operation on an input image (200). When the multipliers and accumulators operate as MAC operators, the electronic device (100) may schedule the operation of a plurality of MAC operators.

[0053] In one embodiment of the present disclosure, when performing a multiplication operation, if the value of a parameter included in a convolution filter is "0" or if there is a pixel among a plurality of receiving pixels that has a value of "0", the result of the multiplication operation may be "0". Accordingly, when the value of a parameter is "0" or the value of a receiving pixel is "0", the electronic device (100) may schedule the multiplier not to perform the corresponding multiplication operation. Accordingly, the workload of a plurality of multipliers can be reduced when performing a convolution operation on an input image (200).

[0054] In one embodiment of the present disclosure, a plurality of multipliers can perform convolution operations on an input image (200) in parallel. Accordingly, the more the number of multipliers, the shorter the time required for the operation of upscaling the resolution of the input image (200) to obtain an output image (300).

[0055] However, as the number of multiple multipliers increases, the physical space required to arrange the multiple multipliers may increase. In addition, the power required to drive the multiple multipliers may increase.

[0056] Accordingly, the electronic device (100) of the present disclosure may include fewer multipliers than the number of multipliers arranged to perform conventional convolution operations, in order to reduce the space required to arrange multiple multipliers and to reduce power consumption. In addition, the electronic device (100) of the present disclosure aims to prevent performance degradation, such as delays in the operation of upscaling the resolution of an input image (200) to obtain an output image (300), while reducing the number of multiple multipliers.

[0057] In one embodiment of the present disclosure, the electronic device (100) may perform interleaving on a plurality of parameters included in each of a plurality of convolutional filters. The electronic device (100) may perform interleaving on each of a plurality of receiving pixels. In this case, "interleaving" means rearranging the order of data. Specifically, interleaving is rearranging the order of columns of data, and through interleaving, even if specific information such as noise is included in the data, the specific information can be rearranged so that it is distributed, thereby preventing the information from being concentrated in a specific part of the data.

[0058] In one embodiment of the present disclosure, the electronic device (100) may obtain a plurality of interleaved parameters in which a plurality of parameters included in each of a plurality of convolutional filters are interleaved. In one embodiment of the present disclosure, at least one parameter having a value of "0" may be evenly distributed and not concentrated in a specific region within the plurality of interleaved parameters.

[0059] In one embodiment of the present disclosure, an electronic device (100) may acquire a plurality of interleaved receiving pixels, each of which is interleaved. In one embodiment of the present disclosure, at least one pixel having a value of "0" may be evenly distributed and not concentrated in a specific area within the plurality of interleaved receiving pixels.

[0060] In one embodiment of the present disclosure, the electronic device (100) may schedule the operation of a plurality of multipliers to perform a multiplication operation of at least one interleaving parameter having a non-zero value among a plurality of interleaving parameters and at least one interleaving receiving pixel having a non-zero value among a plurality of interleaving receiving pixels. That is, the electronic device (100) may schedule the operation of a plurality of multipliers to perform a multiplication operation when both the interleaving parameter and the receiving pixel to be multiplied have non-zero values.

[0061] On the other hand, the electronic device (100) can schedule the operation of multiple multipliers so as not to perform a multiplication operation of at least one interleaving parameter having a value of "0" among multiple interleaving parameters and at least one interleaving accepting pixel having a value of "0" among multiple interleaving accepting pixels. That is, the electronic device (100) can schedule the operation of multiple multipliers so as not to perform a multiplication operation when at least one of the interleaving parameter or accepting pixel to be multiplied has a value of "0".

[0062] In one embodiment of the present disclosure, as a convolution operation is performed using a plurality of interleaving parameters and a plurality of interleaving accepting pixels in which the value of "0" is evenly distributed through interleaving, the frequency at which a plurality of multipliers are scheduled not to perform a specific multiplication operation may be increased.

[0063] Specifically, before interleaving, the distribution of parameters having a value of "0" included in multiple parameters and the distribution of receiving pixels having a value of "0" included in multiple receiving pixels are dense, so there may be many cases where both the parameter and the receiving pixel subject to the multiplication operation have a value of "0". On the other hand, the distribution of parameters having a value of "0" included in multiple interleaving parameters and the distribution of receiving pixels having a value of "0" included in multiple interleaving receiving pixels are uniformly distributed, so there may be many cases where only one of the parameter and the receiving pixel subject to the multiplication operation has a value of "0".

[0064] Accordingly, when performing a convolution operation, the number of multiplication operations that do not use a multiplier may increase. Therefore, even if the electronic device (100) of the present disclosure includes fewer multipliers than the number of multipliers arranged to perform a conventional convolution operation, it is possible to prevent performance degradation, such as delaying the operation of upscaling the resolution of the input image (200) to obtain an output image (300).

[0065] In addition, in one embodiment of the present disclosure, when the sizes of a plurality of convolutional filters are the same, the electronic device (100) can perform a multiplication operation on an input image (200) using a plurality of parameters each included in a plurality of convolutional filters for the same receiving pixel.

[0066] In one embodiment of the present disclosure, an electronic device (100) may obtain a plurality of parameter pairs, each consisting of two parameters included in different convolutional filters. The electronic device (100) may obtain the difference in values ​​of the two parameters included in each of the plurality of parameter pairs. The electronic device (100) may identify parameter pairs in order of increasing difference in value, starting from the parameter pair with the smallest difference in value. At this time, the electronic device (100) may identify parameter pairs with increasing difference in value, starting from the parameter pair with the smallest difference in value, up to a predetermined first reference number.

[0067] At this time, the "first reference number" is a number pre-set to perform a multiplication operation using a representative parameter. When the difference between the number of multiple multipliers included in the electronic device (100) of the present disclosure and the number of multiple multipliers arranged to perform a conventional convolution operation is called the "second reference number," the first reference number may increase as the second reference number increases.

[0068] In one embodiment of the present disclosure, an electronic device (100) may obtain a representative parameter based on two parameters each included in at least one identified parameter pair. The "representative parameter" is a parameter set to represent the corresponding parameter pair, and is not limited to any one of the average value, minimum value, maximum value, etc. of the two parameters.

[0069] In one embodiment of the present disclosure, the electronic device (100) may use the result of a product operation of at least one representative parameter and an accepting pixel corresponding to each of at least one pair of parameters as the result of a product operation of two parameters and an accepting pixel included in each of at least one pair of parameters.

[0070] In one embodiment of the present disclosure, since the difference between the values ​​of two parameters included in each of at least one parameter pair is small, even if the result of the multiplication operation obtained using the representative embodiment is used commonly, it may not significantly affect the accuracy of the convolution operation.

[0071] Accordingly, since the result of two multiplication operations can be obtained using a single multiplier, the electronic device (100) of the present disclosure can prevent performance degradation of the operation of upscaling the resolution of an input image (200) to obtain an output image (300) even if it includes fewer multipliers than the number of multipliers arranged to perform a conventional convolution operation.

[0072] Hereinafter, the configuration and operation method of the electronic device (100) of the present disclosure will be described in FIGS. 2 to FIGS. 12.

[0073] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0074] Referring to FIGS. 1 and FIGS. 2, in one embodiment of the present disclosure, an electronic device (100) may include a display (120), a memory (110), at least one processor (130), an input / output interface (140), and a communication interface (150).

[0075] However, not all components illustrated in FIG. 2 are essential components. The electronic device (100) may be implemented by more components than those illustrated in FIG. 2, or by fewer components. In one embodiment of the present disclosure, the electronic device (100) may not include a display (120).

[0076] A display (120), memory (110), at least one processor (130), input / output interface (140), and communication interface (150) included in an electronic device (100) can each be electrically connected to one another.

[0077] In one embodiment of the present disclosure, the display (120) may include any one of a liquid crystal display, a plasma display, an organic light emitting diode display, or an inorganic light emitting diode display. However, the present disclosure is not limited thereto, and the display (120) may include other types of displays capable of displaying the output image (300) described in FIG. 1.

[0078] In one embodiment of the present disclosure, at least one processor (130) controls a display (120) to display an output image (300), so that the electronic device (100) can provide the output image (300) to a user of the electronic device (100).

[0079] In one embodiment of the present disclosure, the memory (110) may store instructions, data structures, and program code that can be read by at least one processor (130). In one embodiment of the present disclosure, the memory (110) may be one or more. Operations performed by the electronic device (100) may be implemented by at least one processor (130) executing the instructions or code of a program stored in the memory (110).

[0080] In one embodiment of the present disclosure, the memory (110) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Mask ROM, Flash ROM, etc.), a hard disk drive (HDD), or a solid-state drive (SSD).

[0081] In one embodiment of the present disclosure, the memory (110) may not exist separately and may be configured to be included in at least one processor (130).

[0082] In one embodiment of the present disclosure, instructions or program code for performing functions or operations of an electronic device (100) may be stored in the memory (110). The instructions, algorithms, data structures, program code, and application programs stored in the memory (110) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, assembler, etc.

[0083] In one embodiment of the present disclosure, various types of modules that can be used to perform the operation of an electronic device (100) may be stored in the memory (110).

[0084] In one embodiment of the present disclosure, the memory (110) may store a memory control module (113), an interleaving module (114), an upscaling module (111), a parameter identification module (115), and a scheduling module (116). However, not all of the modules illustrated in FIG. 2 are required. The memory (110) may store more modules than those illustrated in FIG. 2, or fewer modules.

[0085] In one embodiment of the present disclosure, a 'module' included in the memory (110) may mean a unit that processes a function or operation performed by at least one processor (130). The 'module' included in the memory (110) may be implemented as software such as instructions, algorithms, data structures, or program code.

[0086] In one embodiment of the present disclosure, the memory control module (113) may be composed of instructions or program code regarding the operation or function of storing data in the memory (110) or reading data stored in the memory (110).

[0087] In one embodiment of the present disclosure, by having at least one processor (130) execute instructions or program code of a memory control module (113), the electronic device (100) can store a plurality of pixels of an acquired input image (200) in memory (110). Additionally, the electronic device (100) can store a plurality of parameters for performing a convolution operation in an artificial intelligence model (112) included in an upscaling module (111), which are acquired through an input / output interface (140) or a communication interface (150), in memory (110).

[0088] In one embodiment of the present disclosure, by having at least one processor (130) execute instructions or program code of a memory control module (113), the electronic device (100) can read a plurality of pixels of an input image stored in memory (110) to perform a convolution operation in an artificial intelligence model (112). Additionally, the electronic device (100) can read a plurality of parameters stored in memory (110) to perform a convolution operation in an artificial intelligence model (112).

[0089] In one embodiment of the present disclosure, the interleaving module (114) may be composed of instructions or program code regarding an operation or function of performing interleaving on data. In this case, the interleaving module (114) may include a method of performing interleaving of data regularly according to a predetermined rule or a method of performing interleaving of data randomly, and is not limited to either one.

[0090] In one embodiment of the present disclosure, by having at least one processor (130) execute instructions or program code of an interleaving module (114), the electronic device (100) can store a plurality of pixels of an input image that has undergone interleaving or a plurality of parameters that have undergone interleaving in memory (110).

[0091] However, the present disclosure is not limited thereto, and by having at least one processor (130) execute instructions or program code of an interleaving module (114), the electronic device (100) can read a plurality of pixels of an input image or a plurality of parameters from a memory (110), and can read an interleaved input image or a plurality of interleaved parameters that have been interleaved and read.

[0092] However, although the memory control module (113) and the interleaving module (114) are shown as separate modules in FIG. 2, the present disclosure is not limited thereto. It goes without saying that the operation of the memory control module (113) and the operation of the interleaving module (114) may be performed together in a single module.

[0093] In one embodiment of the present disclosure, the upscaling module (111) may be composed of instructions or program code regarding an operation or function of upscaling the resolution of an input image (200) to obtain an output image (300).

[0094] In one embodiment of the present disclosure, the upscaling module (111) may include an artificial intelligence model (112). The artificial intelligence model (112) may be an artificial intelligence model that includes a layer for performing convolution operations. The artificial intelligence model (112) included in the upscaling module (111) may be a super resolution model, which is an artificial intelligence model that has been pre-trained to infer an output image (300) by upscaling the resolution of an input image (200) provided as input data using convolution operations.

[0095] In one embodiment of the present disclosure, the artificial intelligence model (112) stored in memory (110) may be implemented using various known deep neural network architectures and algorithms suitable for super-resolution, or through variations of various known deep neural network architectures and algorithms. For example, the artificial intelligence model (112) may be implemented through a Super-Resolution Convolutional Neural Network (SRCNN), Enhanced Deep Super-Resolution (EDSR), Super-Resolution Generative Adversarial Network (SRGAN), and variations thereof, but is not limited to the examples described above.

[0096] In one embodiment of the present disclosure, the artificial intelligence model (112) may include an architecture of layers that perform convolution operations and a plurality of convolution filters.

[0097] In one embodiment of the present disclosure, the artificial intelligence model (112) included in the upscaling module (111) is pre-trained in an external electronic device or an external server device, and the electronic device (100) may use the artificial intelligence model (112) by performing transfer learning or fine-turning using the pre-trained model. However, the present disclosure is not limited thereto, and the artificial intelligence model (112) included in the upscaling module (111) may be trained in the electronic device (100) based on a training input image and an upscaled training output image.

[0098] In one embodiment of the present disclosure, by having at least one processor (130) execute instructions or program code of an upscaling module (111), the electronic device (100) can extract various features from an input image (200) and infer a plurality of pixel information included in an output image (300) from the extracted features to perform upscaling of the resolution.

[0099] In one embodiment of the present disclosure, the parameter identification module (115) may be composed of instructions or program code regarding an operation or function of obtaining a plurality of parameter pairs consisting of two interleaving parameters each included in any two of the plurality of convolutional filters based on a plurality of interleaving parameters interleaved through the interleaving module (114).

[0100] In one embodiment of the present disclosure, the parameter identification module (115) may be composed of instructions or program code regarding an operation or function of obtaining the difference in value between two interleaving parameters included in each of a plurality of parameter pairs.

[0101] In one embodiment of the present disclosure, the parameter identification module (115) may be composed of instructions or program code regarding an operation or function for identifying at least one parameter pair, which is pre-set as a first reference number, in order of smallest difference between the values ​​of two interleaving parameters among a plurality of parameter pairs.

[0102] In one embodiment of the present disclosure, the parameter identification module (115) may be composed of instructions or program code regarding an operation or function of obtaining at least one representative parameter based on two interleaving parameters each included in at least one identified parameter pair.

[0103] In one embodiment of the present disclosure, by having at least one processor (130) execute instructions or program code of a parameter identification module (115), the electronic device (100) can obtain the difference in value between two interleaving parameters included in each of a plurality of parameter pairs. The electronic device (100) can identify at least one parameter pair, which is a first reference number of pre-set values, in order of smallest difference in value between two interleaving parameters among the plurality of parameter pairs. The electronic device (100) can obtain at least one representative parameter based on the two interleaving parameters included in each of the identified at least one parameter pair.

[0104] In one embodiment of the present disclosure, at least one processor (130) may include a plurality of multipliers and a plurality of accumulators capable of performing multiplication and accumulation operations. Additionally, at least one processor (130) may include a plurality of MAC arithmetic units capable of performing multiplication and accumulation operations. Hereinafter, for convenience of explanation, at least one processor (130) is described as including a plurality of multipliers and a plurality of accumulators.

[0105] In one embodiment of the present disclosure, the scheduling module (116) may be composed of instructions or program code for controlling the operation of a plurality of multipliers and a plurality of accumulators. The scheduling module (116) may be composed of instructions or program code for controlling the operation of a plurality of multipliers and a plurality of accumulators to perform a convolution operation between a plurality of interleaving parameters and a plurality of interleaving accepting pixels performed through an artificial intelligence model (112).

[0106] In one embodiment of the present disclosure, at least one processor (130) can control the operation of a plurality of multipliers and a plurality of accumulators to perform a convolution operation between a plurality of interleaving parameters and a plurality of interleaving accepting pixels by executing instructions or program code of a scheduling module (116).

[0107] However, the present disclosure is not limited thereto, and at least one of the memory control module (113), interleaving module (114), upscaling module (111), parameter identification module (115) or scheduling module (116) may be designed as hardware using a hardware description language, such as Verilog or VHDL (VHSIC Hardware Description Language), and implemented as an ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or hardware accelerator device.

[0108] In this case, it goes without saying that the operation of the module designed and implemented in hardware may take place in each hardware block.

[0109] In one embodiment of the present disclosure, at least one processor (130) may be configured to control a series of processes to operate an electronic device (100) according to the embodiments described below, and may be composed of one or more processors.

[0110] In one embodiment of the present disclosure, at least one processor (130) may be composed of at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), and a Neural Processing Unit or an AI-dedicated processor designed with a hardware structure specialized for the learning and processing of an artificial intelligence model (AI), but is not limited thereto.

[0111] In one embodiment of the present disclosure, at least one processor (130) may be composed of a circuit such as a System on Chip (SoC) or an Integrated Circuit (IC). At least one processor (130) may include a processing circuit.

[0112] In one embodiment of the present disclosure, at least one processor (130) can execute various types of modules stored in memory (110). At least one processor (130) can execute at least one instruction constituting the various types of modules stored in memory (110) individually or collectively. By executing a program or at least one instruction stored in memory (110), at least one processor (130) can process data according to a predefined operation rule.

[0113] In one embodiment of the present disclosure, at least one processor (130) may include a plurality of processors. In one embodiment of the present disclosure, at least one module among a plurality of modules in memory (110) may be executed by any one of the plurality of processors. The remaining modules among the plurality of modules stored in memory (110) may be executed by another processor among the plurality of processors.

[0114] In one embodiment of the present disclosure, at least one processor (130) may acquire an input image (200) from an external electronic device, etc., through an input / output interface (140). Additionally, at least one processor (130) may provide an output image (300) to an external electronic device including a display through the input / output interface (140).

[0115] In one embodiment of the present disclosure, the input / output interface (140) may perform input / output operations with an external electronic device using at least one of an input / output method including an HDMI port (High-Definition Multimedia Interface port), DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-mentioned input / output methods.

[0116] In one embodiment of the present disclosure, the communication interface (150) can perform data communication with an external server or an external electronic device under the control of at least one processor (130).

[0117] The communication interface (150) can perform data communication with an external server or an external electronic device using at least one of the data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0118] In one embodiment of the present disclosure, at least one processor (130) may acquire an input image (200) from an external electronic device or an external server through a communication interface (150). Additionally, at least one processor (130) may provide an output image (300) to an external electronic device or an external server including a display through the communication interface (150).

[0119] In one embodiment of the present disclosure, at least one processor (130) may receive an artificial intelligence model that has been pre-trained to perform upscaling of an input image (200) from an external server or external electronic device through a communication interface (150). Specifically, at least one processor (130) may receive an architecture of an artificial intelligence model that has been pre-trained to perform upscaling of an input image (200) and a plurality of parameters each included in a plurality of convolutional filters from an external server or external electronic device through a communication interface (150).

[0120] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0121] Referring to FIGS. 1, 2 and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S100) of acquiring an input image (200) including a plurality of pixels.

[0122] In step S100, the electronic device (100) may acquire an input image (200) through an input / output interface (140) or a communication interface (150). However, the present disclosure is not limited thereto, and the electronic device (100) may further include a camera and may acquire an image captured through the camera as the input image (200).

[0123] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S200) of obtaining a plurality of interleaved parameters, wherein a plurality of parameters included in each of a plurality of convolutional filters for performing a convolutional operation are interleaved.

[0124] In step S200, the electronic device (100) can obtain a plurality of interleaved parameters from the memory (110), wherein a plurality of parameters are interleaved. The electronic device (100) can obtain a plurality of interleaved parameters stored in the memory (110) in an interleaved state. Additionally, the electronic device (100) can read the plurality of parameters stored in the memory (110) in an interleaved state. The operation of step S200 will be described later in FIG. 4.

[0125] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S300) of obtaining a plurality of interleaved receiving pixels, wherein a plurality of receiving pixels included in a receiving region corresponding to each of a plurality of convolutional filters among a plurality of pixels are interleaved.

[0126] In step S300, the electronic device (100) can obtain a plurality of interleaved accepting pixels from the memory (110), wherein a plurality of accepting pixels are interleaved. The electronic device (100) can obtain a plurality of interleaved accepting pixels stored in the memory (110) in an interleaved state. Additionally, the electronic device (100) can read a plurality of accepting pixels stored in the memory (110) in an interleaved state. The operation of step S300 will be described later in FIG. 4.

[0127] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include a step (S400) of obtaining an output image based on the result of a multiplication operation included in a convolution operation for at least one interleaving parameter having a non-zero value among a plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among a plurality of interleaving accepting pixels. In one embodiment of the present disclosure, step S400 may include a step of performing a multiplication operation for each of the at least one interleaving parameter having a non-zero value and at least one interleaving accepting pixel corresponding to the at least one interleaving parameter. Step S400 may include a step of obtaining an output image by accumulating the result of the multiplication operation.

[0128] In step S400, the electronic device (100) can obtain an output image based on the result of a product-operation included in a convolution operation for at least one interleaving parameter having a non-zero value among a plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among a plurality of interleaving accepting pixels.

[0129] The operation of step S400 will be described below in Fig. 4.

[0130] FIG. 4 is a diagram illustrating the operation of an electronic device that performs a convolution operation on interleaved parameters and interleaved receiving pixels according to an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating the operation of reading a plurality of receiving pixels in an interleaved state from a memory and acquiring a plurality of interleaved receiving pixels according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating the method of interleaving parameters and receiving pixels according to an embodiment of the present disclosure.

[0131] Referring to FIGS. 1, 2 and 4, in one embodiment of the present disclosure, FIG. 4 illustrates the operation of an electronic device (100) in one of the convolutional layers included in an artificial intelligence model (112) included in an upscaling module (111) for convenience of explanation.

[0132] In one embodiment of the present disclosure, an electronic device (100) may store an acquired input image (200) in a memory (110). At this time, the input image (200) may include a plurality of channels. FIG. 4 illustrates one image channel (400) among the plurality of channels of the input image (200) stored in the memory (110) for convenience of explanation. FIG. 4 also illustrates, for convenience of explanation, that a convolution operation by each of a plurality of convolution filters (420) is performed on a plurality of receiving pixels included in a specific receiving area of ​​one image channel (400). The operation of FIG. 4 may also be applied in a case where, in one image channel (400), each of the plurality of convolution filters (420) moves with a predetermined stride and a convolution operation is performed on a plurality of interleaving receiving pixels included in a corresponding receiving area. It may also be applied in a case where a convolution operation is performed in another image channel of the input image (200).

[0133] Additionally, a plurality of convolutional filters (420) are stored in the memory (110). Although FIG. 4 illustrates that each of the plurality of convolutional filters (420) has a size of 3*3, the present disclosure is not limited thereto, and the plurality of convolutional filters (420) may have other sizes. At this time, the memory (110) may include a plurality of parameters corresponding to each of the plurality of convolutional filters (420).

[0134] In one embodiment of the present disclosure, the electronic device (100) may perform a convolution operation to upscale the resolution of an input image (200). In one embodiment of the present disclosure, the electronic device (100) may read a plurality of parameters from a memory (110) through a memory control module (113) and provide them to a plurality of multipliers (440).

[0135] In one embodiment of the present disclosure, a method of operating an electronic device (100) may include the step of reading a plurality of parameters in an interleaved state from a memory (110) and transmitting the interleaved plurality of interleaved parameters to a plurality of multipliers (440).

[0136] The electronic device (100) can read multiple parameters in an interleaved state from memory (110) through an interleaving module (114) and transmit the interleaved multiple parameters to multiple multipliers (440).

[0137] However, the present disclosure is not limited thereto, and a plurality of interleaved parameters may be stored in the memory (110), wherein a plurality of parameters are interleaved. In this case, the electronic device (100) may read the plurality of interleaved parameters from the memory (110) and transmit them to a plurality of multipliers (440).

[0138] In one embodiment of the present disclosure, an electronic device (100) may read a plurality of receiving pixels (410) from a memory (110) in one image channel (400) through a memory control module (113) and provide them to a plurality of multipliers (440). At this time, the plurality of receiving pixels (410) may be pixels having a 3*3 resolution included in a receiving area corresponding to the size of a plurality of convolutional filters (420).

[0139] Referring to FIGS. 3, 4, and 5, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S310) of reading a plurality of receiving pixels (410) from a memory (110) in an interleaved state and acquiring a plurality of interleaved receiving pixels. At this time, step S310 may be performed after step S200.

[0140] Referring again to FIG. 4, in one embodiment of the present disclosure, an electronic device (100) can read a plurality of receiving pixels in an interleaved state from a memory (110) through an interleaving module (114) and transmit the interleaved plurality of interleaved receiving pixels to a plurality of multipliers (440).

[0141] However, the present disclosure is not limited thereto, and a plurality of interleaved accepting pixels may be stored in the memory (110), wherein a plurality of accepting pixels are interleaved. In this case, the electronic device (100) may read a plurality of interleaved accepting pixels from the memory (110) and transmit them to a plurality of multipliers (440).

[0142] Referring to FIG. 6, in one embodiment of the present disclosure, FIG. 6 illustrates an interleaving method performed in an interleaving module (114). An arrangement of data (600) before interleaving can be changed into an arrangement of data (610) in an interleaved state by crossing it according to a set rule or randomly. However, the interleaving method illustrated in FIG. 6 is merely one embodiment, and the electronic device (100) of the present disclosure can use various interleaving methods.

[0143] Referring again to FIG. 4, in one embodiment of the present disclosure, a plurality of interleaving parameters may include a plurality of first interleaving parameters (431) in which a plurality of first parameters are interleaved, which are included in a first convolutional filter (430) of any one of the plurality of convolutional filters (420). The plurality of first interleaving parameters (431) may be in which nine first parameters included in the first convolutional filter (430) of the plurality of convolutional filters (420) are interleaved.

[0144] In one embodiment of the present disclosure, a plurality of interleaving receiving pixels may include a plurality of first interleaving receiving pixels (411) in which a plurality of first receiving pixels are interleaved and included in a receiving region corresponding to a first convolutional filter (430). A plurality of first interleaving receiving pixels (411) may be in which nine first receiving pixels included in the receiving region are interleaved.

[0145] In one embodiment of the present disclosure, a convolution operation by each of a plurality of convolution filters (420) may be performed on a plurality of interleaving receiving pixels included in the same receiving region. The plurality of interleaving receiving pixels provided to the plurality of convolution filters (420) in the plurality of multipliers (440) may be identical to each other.

[0146] In one embodiment of the present disclosure, the multiplication operation of a plurality of first interleaving parameters (431) and a plurality of first interleaving accepting pixels (411) may be performed by a first multiplier (441) included in a plurality of multipliers (440). The electronic device (100) may schedule the operation of the plurality of multipliers (440) through a scheduling module (116) so that the multiplication operation of the plurality of first interleaving parameters (431) and a plurality of first interleaving accepting pixels (411) is performed by a first multiplier (441) included in the plurality of multipliers (440).

[0147] In one embodiment of the present disclosure, a multiplication operation of a plurality of first interleaving parameters (431) and a plurality of first interleaving accepting pixels (411) may be performed serially in a first multiplier (441). A multiplication operation of each of the nine first interleaving parameters and nine first interleaving accepting pixels corresponding to each of the nine first interleaving parameters may be performed serially in the first multiplier (441).

[0148] In one embodiment of the present disclosure, the step (S400) of obtaining an output image (300) may include the step of obtaining an output image by performing a multiplication operation of a plurality of first interleaving parameters and a plurality of first interleaving accepting pixels serially in one of a plurality of multipliers for performing a multiplication operation.

[0149] In one embodiment of the present disclosure, the electronic device (100) can control the operation of the first multiplier (441) so as not to perform a multiplication operation through a scheduling module (116) when one of the plurality of first interleaving parameters (431) sequentially provided to the first multiplier (441) has a value of "0" or when one of the plurality of first interleaving accepting pixels (411) has a value of "0". The electronic device (100) can control the operation of the first multiplier (441) to perform the next operation through the scheduling module (116).

[0150] In one embodiment of the present disclosure, a plurality of accumulators (450) can accumulate the results of multiplication operations performed in a plurality of multipliers (440) to obtain an output in a convolutional layer. FIG. 4 is illustrated as the output obtained from the plurality of accumulators (450) being an output image (300), but the present disclosure is not limited thereto.

[0151] In one embodiment of the present disclosure, the output obtained from a plurality of accumulators (450) may be provided as an input to another convolutional layer or another type of layer (e.g., a pooling layer, an activation function layer, a fully connected layer, a normalization layer, etc.) included in the artificial intelligence model (112) included in the upscaling module (111). Hereinafter, for convenience of explanation, the output obtained from the plurality of accumulators (450) will be described as an output image (300).

[0152] In one embodiment of the present disclosure, each of the plurality of accumulators (450) can obtain an output image (300) by accumulating the result of a multiplication operation performed in each of the corresponding plurality of multipliers (440).

[0153] In one embodiment of the present disclosure, a plurality of accumulators (450) may include a first accumulator (451). The result of a multiplication operation performed sequentially in a first multiplier (441) is added in the first accumulator (451) to represent the features of a plurality of receiving pixels (410) extracted by a first convolutional filter (430).

[0154] In one embodiment of the present disclosure, the electronic device (100) can schedule the operation of a plurality of accumulators (450) through a scheduling module (116) so that the result of a multiplication operation performed in a first multiplier (441) is performed in a first accumulator (451) included in a plurality of accumulators (450).

[0155] In one embodiment of the present disclosure, the electronic device (100) may provide information to the first accumulator (451) that a multiplication operation was not performed in the corresponding turn through the scheduling module (116) when a multiplication operation was not performed in the first multiplier (441). The electronic device (100) may provide a value of "0" to the first accumulator (451) in the corresponding turn through the scheduling module (116).

[0156] In one embodiment of the present disclosure, as a convolution operation is performed using a plurality of first interleaving parameters (431) and a plurality of first interleaving accepting pixels (411) in which the value of "0" is evenly distributed through interleaving, the frequency at which the first multiplier (441) is scheduled not to perform a multiplication operation may be increased. Accordingly, the time and workload required for the electronic device (100) to perform a convolution operation through a plurality of multipliers (440) and a plurality of accumulators (450) can be reduced.

[0157] In one embodiment of the present disclosure, it is obvious that the convolution operation of each of the plurality of interleaving parameters and each of the plurality of interleaving accepting pixels can be performed in the same manner as described above.

[0158] FIG. 7 is a diagram illustrating the scheduling of a multiplier operation so as not to perform a multiplication operation as at least one of the interleaving parameter or the interleaving accepting pixel has a value of "0", according to one embodiment of the present disclosure.

[0159] Referring to FIGS. 1, FIGS. 2, FIGS. 4 and FIGS. 7, in one embodiment of the present disclosure, FIG. 7 illustrates a plurality of interleaving accepting pixels, a plurality of interleaving parameters, a plurality of multipliers (730), a plurality of accumulators (740), and a scheduling module (116).

[0160] In one embodiment of the present disclosure, a plurality of interleaving receiving pixels may include a plurality of first interleaving receiving pixels (700), a plurality of second interleaving receiving pixels (710), and a plurality of third interleaving receiving pixels (720). A plurality of interleaving parameters may include a plurality of first interleaving parameters (701), a plurality of second interleaving parameters (711), and a plurality of third interleaving parameters (721).

[0161] In one embodiment of the present disclosure, a plurality of interleaving receiving pixels may be receiving pixels included in the same receiving area. A plurality of first interleaving receiving pixels (700), a plurality of second interleaving receiving pixels (710), and a plurality of third interleaving receiving pixels (720) may have the same value and be interleaved in the same arrangement.

[0162] In one embodiment of the present disclosure, a plurality of interleaving parameters may each be parameters included in different convolutional filters that are interleaved. Each of the plurality of first interleaving acceptance parameters (701), the plurality of second interleaving acceptance parameters (711), and the plurality of third interleaving acceptance parameters (721) may have different values.

[0163] In one embodiment of the present disclosure, a plurality of multipliers (730) may include a first multiplier (731) and a second multiplier (732). In one embodiment of the present disclosure, the number of multipliers included in the plurality of multipliers (730) may be reduced compared to the prior art. A plurality of accumulators (740) may include a first accumulator (741), a second accumulator (742), and a third accumulator (743).

[0164] In one embodiment of the present disclosure, the electronic device (100) can obtain features of an input image (200) extracted using a first convolutional filter by accumulating the result of a multiplication operation performed using a plurality of first interleaving acceptance parameters (701) through a first accumulator (741). The electronic device (100) can obtain features of an input image (200) extracted using a second convolutional filter by accumulating the result of a multiplication operation performed using a plurality of second interleaving acceptance parameters (711) through a second accumulator (742). The electronic device (100) can obtain features of an input image (200) extracted using a third convolutional filter by accumulating the result of a multiplication operation performed using a plurality of third interleaving acceptance parameters (721) through a third accumulator (743).

[0165] In one embodiment of the present disclosure, a multiplication operation of a plurality of first interleaving accepting pixels (700) and a plurality of first interleaving accepting parameters (701) may be performed sequentially in a first multiplier (731). In one embodiment of the present disclosure, the result of the multiplication operation performed in the first multiplier (731) may be accumulated in a first accumulator (741).

[0166] In one embodiment of the present disclosure, the multiplication operation of a plurality of second interleaving accepting pixels (710) and a plurality of second interleaving accepting parameters (711) may be performed sequentially in a second multiplier (732). In one embodiment of the present disclosure, the result of the multiplication operation performed in the first multiplier (731) may be accumulated in a second accumulator (742).

[0167] At this time, FIG. 7 illustrates that the multiplication operation of the corresponding operation sequence is not performed in the second multiplier (732) as it is identified that any one of the plurality of second interleaving receiving pixels (710) has a value of "0" or that any one of the plurality of second interleaving receiving parameters (711) has a value of "0".

[0168] In this case, the multiplication operation of one of the third interleaving receiving pixels (720) and one of the third interleaving receiving parameters (721) can be performed in the second multiplier (732) at the corresponding operation sequence.

[0169] The electronic device (100) of the present disclosure can be scheduled so that, through a scheduling module (116), a second interleaving accepting pixel among a plurality of second interleaving accepting pixels (710) has a value of "0", or a second interleaving accepting parameter among a plurality of second interleaving accepting parameters (711) has a value of "0", and thus the multiplication operation between the second interleaving accepting pixel and the second interleaving accepting parameter is not performed in the second multiplier (732) at the corresponding operation sequence. The electronic device (100) can be scheduled so that a multiplication operation between the third interleaving accepting pixel and the third interleaving accepting parameter is performed in the second multiplier (732) at the corresponding operation sequence.

[0170] The electronic device (100) of the present disclosure can schedule the operation of a plurality of accumulators (740) through a scheduling module (116) such that the result of a multiplication operation performed in a second multiplier (732) at a corresponding operation sequence is accumulated by a third accumulator (743). The electronic device (100) can schedule the operation of a plurality of accumulators (740) through a scheduling module (116) such that a value of "0" is accumulated in the second accumulator (742) at a corresponding operation sequence as the result of the multiplication operation.

[0171] By doing so, the electronic device (100) of the present disclosure can prevent performance degradation of the operation of upscaling the resolution of an input image (200) to obtain an output image (300) by scheduling only the multiplication operation in which the result of the multiplication operation does not become "0" through the plurality of multiplication units (730) even if the number of multiplication units included in the plurality of multiplication units (730) is reduced.

[0172] FIG. 8 is a flowchart illustrating an operation to obtain the result of a product operation between a representative parameter and a plurality of interleaving accepting pixels according to an embodiment of the present disclosure as the result of a product operation between each of two interleaving sub-parameters and a plurality of interleaving accepting pixels. FIG. 9 is a diagram illustrating an operation to obtain the result of a product operation between a representative parameter and a plurality of interleaving accepting pixels according to an embodiment of the present disclosure as the result of a product operation between each of two interleaving sub-parameters and a plurality of interleaving accepting pixels. Hereinafter, steps and configurations identical to those described in FIG. 3 and FIG. 7 are given the same reference numerals, and redundant descriptions are omitted.

[0173] Referring to FIGS. 2, FIGS. 3 and FIGS. 8, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S320) of obtaining a plurality of parameter pairs consisting of two interleaving parameters each included in any two of the plurality of convolutional filters based on a plurality of interleaving parameters.

[0174] At this time, step S320 can be performed after step S300.

[0175] In step S320, the electronic device (100) can obtain a plurality of parameter pairs consisting of two interleaving parameters each included in any two of the plurality of convolutional filters based on a plurality of interleaving parameters.

[0176] In one embodiment of the present disclosure, the method of operating an electronic device (100) may include the step (S330) of identifying at least one parameter pair in order of increasing difference in value from the smallest difference in value of two interleaving parameters included in each of the plurality of parameter pairs, in order of increasing difference in value.

[0177] In step S330, the electronic device (100) can identify at least one parameter pair, which is a preset first reference number, in order from the smallest difference in the values ​​of the two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in values.

[0178] In one embodiment of the present disclosure, if the first reference number is 5, the electronic device (100) obtains the difference in values ​​of two interleaving parameters included in each of the plurality of parameter pairs and can identify 5 parameter pairs in order starting from the one with the smallest difference in value. At this time, there may also be parameter pairs with a difference of "0". In addition, if there are parameter pairs with the same difference in value, making it difficult to identify 5 in order starting from the one with the smallest difference in value, 5 may be identified randomly among them.

[0179] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S340) of obtaining the result of a product operation of a representative parameter and a plurality of interleaving accepting pixels based on two interleaving parameters each included in at least one pair of parameters as the result of a product operation of each of the two interleaving parameters and a plurality of interleaving accepting pixels.

[0180] In one embodiment of the present disclosure, at least one parameter pair in step S340 may mean a parameter pair identified by a first reference number in step S330.

[0181] In step S340, the electronic device (100) may obtain the result of a product operation of a representative parameter based on two interleaving parameters each included in at least one parameter pair and a plurality of interleaving accepting pixels as the result of a product operation of each of the two interleaving parameters and a plurality of interleaving accepting pixels. Step S340 will be described later in FIG. 9.

[0182] After step S340, step S400 of obtaining an output image (300) based on the result of the multiplication operation may be performed.

[0183] Referring to FIG. 7 and FIG. 9, in one embodiment of the present disclosure, FIG. 9 illustrates a plurality of interleaving receiving pixels, a plurality of interleaving parameters, a plurality of multipliers (730), a plurality of accumulators (740), a parameter identification module (115), and a scheduling module (116).

[0184] In one embodiment of the present disclosure, a plurality of interleaving receiving pixels may include a plurality of first interleaving receiving pixels (700), a plurality of second interleaving receiving pixels (710), and a plurality of third interleaving receiving pixels (720). A plurality of interleaving parameters may include a plurality of first interleaving parameters (701), a plurality of second interleaving parameters (711), and a plurality of third interleaving parameters (721).

[0185] In one embodiment of the present disclosure, a plurality of multipliers (730) may include a first multiplier (731) and a second multiplier (732). A plurality of accumulators (740) may include a first accumulator (741), a second accumulator (742), and a third accumulator (743).

[0186] In one embodiment of the present disclosure, the electronic device (100) can obtain a plurality of parameter pairs consisting of two interleaving parameters each included in any two of the plurality of convolutional filters through a parameter identification module (115).

[0187] Specifically, the electronic device (100) can obtain a first parameter pair consisting of a plurality of first interleaving parameters (701) and a plurality of second interleaving parameters (711), a second parameter pair consisting of a plurality of first interleaving parameters (701) and a plurality of third interleaving parameters (721), and a third parameter pair consisting of a plurality of second interleaving parameters (711) and a plurality of third interleaving parameters (721).

[0188] However, this is merely an example, and the electronic device (100) can obtain multiple parameter pairs consisting of two other interleaving parameters included in multiple interleaving parameters.

[0189] In one embodiment of the present disclosure, the electronic device (100) can identify at least one parameter pair through a parameter identification module (115) in order from the smallest difference in value between two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in value, up to a predetermined first reference number.

[0190] Specifically, the electronic device (100) can obtain the difference in the value of two interleaving parameters provided to each of the multiple multipliers (730) at a specific operation order among the two interleaving parameters included in each of the multiple parameter pairs.

[0191] Accordingly, the electronic device (100) can obtain a first difference between the value of one interleaving parameter among the first interleaving parameters (701) included in the first parameter pair and one interleaving parameter among the second interleaving parameters (711), each provided to a plurality of multipliers (730) in a specific operation sequence.

[0192] The electronic device (100) can obtain a first difference between the value of one interleaving parameter among the first interleaving parameters (701) included in the second parameter pair and one interleaving parameter among the third interleaving parameters (721), each provided to a plurality of multipliers (730) in a specific operation sequence.

[0193] The electronic device (100) can obtain a third difference between the value of one interleaving parameter of the second interleaving parameters (711) included in the third parameter pair and one interleaving parameter of the third interleaving parameters (721), each provided to a plurality of multipliers (730) in a specific operation sequence.

[0194] In one embodiment of the present disclosure, the electronic device (100) can identify at least one parameter pair, which is pre-set, through a parameter identification module (115), in order from the smallest difference in value between two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in value.

[0195] In one embodiment of the present disclosure, the third difference may be the smallest value. In one embodiment of the present disclosure, the order of smallest difference between the first difference and the second difference may be greater than the first reference number. Although not shown in FIG. 9, the order of smallest difference between the values ​​of two interleaving parameters included in another parameter pair among a plurality of parameter pairs may be less than the first reference number.

[0196] In one embodiment of the present disclosure, the electronic device (100) can identify a third parameter pair. In this case, the third parameter pair may be identified as having a value difference that is equal to or within a first reference number.

[0197] In one embodiment of the present disclosure, the electronic device (100) may obtain a representative parameter based on one interleaving parameter of the second interleaving parameters (711) included in the identified third parameter pair and one interleaving parameter of the third interleaving parameters (721). The representative parameter may be obtained as any one of the average value of one interleaving parameter of the second interleaving parameters (711) and one interleaving parameter of the third interleaving parameters (721), the maximum value of the two interleaving parameters, or the minimum value, but is not limited thereto.

[0198] In one embodiment of the present disclosure, a multiplication operation by a third parameter pair may be performed in a second multiplier (732). Specifically, at a specific operation sequence in which a representative parameter is obtained, the representative parameter may be provided to the second multiplier (732), and a plurality of second interleaving accepting pixels (710) may be provided to the second multiplier (732) to perform a multiplication operation. At this time, a plurality of third interleaving accepting pixels (720) may not be provided to the second multiplier (732).

[0199] In one embodiment of the present disclosure, the result of a multiplication operation performed in the second multiplier (732) may be accumulated in the second accumulator (742) and the third accumulator (743), respectively. The electronic device (100) may schedule the operation of a plurality of multipliers (730) and a plurality of accumulators (740) through a scheduling module (116) so that the result of a multiplication operation performed in the second multiplier (732) is accumulated in the second accumulator (742) and the third accumulator (743), respectively.

[0200] By doing so, the electronic device (100) of the present disclosure can prevent performance degradation of the operation of upscaling the resolution of an input image (200) to obtain an output image (300) by scheduling the multiplication operation to be performed in one multiplier using a representative parameter and providing the result of the multiplication operation to two accumulators that each accumulate the result of the multiplication operation of the parameter pair, even if the number of multipliers included in the plurality of multipliers (730) is reduced.

[0201] FIG. 10 is a flowchart illustrating the scheduling of an accumulator's operation to accumulate the result of a product operation included in a convolution operation for interleaved parameters and interleaved receiving pixels, according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 and FIG. 8 are given the same reference numerals, and redundant descriptions are omitted.

[0202] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S350) of scheduling the operation of a plurality of accumulators to acquire an output image by accumulating the result of a multiplication operation such that the result of a multiplication operation of a representative parameter based on at least one pair of parameters and one interleaving accepting pixel can be accumulated as the result of a multiplication operation of each of the two interleaving parameters included in the pair of at least one parameters and at least one interleaving accepting pixel.

[0203] In one embodiment of the present disclosure, step S350 may be performed after step S340.

[0204] Referring to FIGS. 9 and 10, in step S350, the electronic device (100) can schedule the operation of a plurality of accumulators (740) to acquire an output image (300) by accumulating the result of a multiplication operation such that the result of a multiplication operation of a representative parameter based on at least one pair of parameters and one interleaving accepting pixel can be accumulated as the result of a multiplication operation of each of the two interleaving parameters included in the pair of at least one parameters and at least one interleaving accepting pixel through a scheduling module (116).

[0205] After step S350, step S400 of obtaining an output image (300) based on the accumulated result may be performed.

[0206] FIG. 11 is a flowchart illustrating the operation of an electronic device that performs a multiplication operation included in a convolution operation on interleaved parameters and interleaved receiving pixels according to one embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3 and FIG. 8, and redundant descriptions are omitted.

[0207] Referring to FIGS. 1, FIGS. 2, FIGS. 3, FIGS. 7, FIGS. 9 and FIGS. 11, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S360) of identifying whether the value of any one of a plurality of interleaving accepting pixels is "0".

[0208] Step S360 can be performed after Step S300.

[0209] In step S360, the electronic device (100) can identify whether the value of any one of the multiple interleaving receiving pixels is "0".

[0210] In one embodiment of the present disclosure, as the value of one of the receiving pixels is determined to be "0" in step S360, the method of operation of the electronic device (100) may include a step (S380) of scheduling the operation of a plurality of multipliers (730) so that a multiplication operation is not performed on the one receiving pixel determined to be "0".

[0211] In step S380, the electronic device (100) can schedule the operation of multiple multipliers (730) to perform the next sequence of operations without performing a multiplication operation for one receiving pixel whose value is determined to be "0".

[0212] In one embodiment of the present disclosure, as it is determined in step S360 that the value of any one of the plurality of interleaving accepting pixels is not "0", the method of operation of the electronic device (100) may include a step (S370) of identifying whether the value of any one of the plurality of interleaving parameters corresponding to one of the interleaving accepting pixels is "0". At this time, step S370 may be a step performed for one of the interleaving accepting pixels that is determined not to have a value of "0" in step S360.

[0213] In step S360, the electronic device (100) can determine whether the value of one of the interleaving parameters corresponding to one of the interleaving parameters is "0", as the value of one of the interleaving parameters is determined not to be "0".

[0214] In one embodiment of the present disclosure, as the value of one interleaving parameter is determined to be "0" in step S370, the method of operation of the electronic device (100) may include a step (S380) of scheduling the operation of a plurality of multipliers (730) so that a multiplication operation for the one interleaving parameter determined to be "0" is not performed.

[0215] In step S380, the electronic device (100) may schedule the operation of a plurality of multipliers (730) to perform the next sequence of operations without performing a multiplication operation for one interleaving parameter whose value is determined to be "0". At this time, in step S380, a multiplication operation may not be performed between any one interleaving accepting pixel whose value magnitude was determined in steps S360 and S370, respectively, and any one interleaving parameter corresponding thereto.

[0216] In one embodiment of the present disclosure, as it is determined in step S370 that the value of any one of the interleaving parameters is not "0", the method of operation of the electronic device (100) may include the step of obtaining a difference in a plurality of values ​​corresponding to each of the plurality of parameter pairs containing any one interleaving parameter. At this time, the difference in values ​​obtained in the step may be the difference in value between one interleaving parameter and another interleaving parameter included in each of the plurality of parameter pairs containing the one interleaving parameter determined not to have a value of "0" in step S370.

[0217] In one embodiment of the present disclosure, the electronic device (100) can obtain a difference of a plurality of values ​​corresponding to each of a plurality of parameter pairs including one of a plurality of interleaving parameters, as it is determined that the value of one of a plurality of interleaving parameters is not "0".

[0218] In one embodiment of the present disclosure, the method of operating an electronic device (100) may include a step (S390) of identifying whether the order in which the difference between the values ​​of at least one of a plurality of parameter pairs is small is equal to or smaller than a first reference number. The plurality of parameter pairs determined in step S390 may be a plurality of parameter pairs including one interleaving parameter determined not to have a value of "0" in step S370.

[0219] In one embodiment of the present disclosure, the operation of obtaining the difference of a plurality of values ​​corresponding to each of a plurality of parameter pairs including any one interleaving parameter, and the operation of identifying whether the order of the difference of values ​​of at least one parameter pair among the plurality of parameter pairs is equal to or smaller than a first reference number, may be performed in a single step.

[0220] In step S390, the electronic device (100) can identify whether the order in which the difference between the values ​​of at least one of the plurality of parameter pairs is smaller is equal to or smaller than the first reference number.

[0221] In one embodiment of the present disclosure, in step S390, as it is identified that there is at least one pair of parameters in order of having a smaller difference in value is equal to or smaller than a first reference number, the electronic device (100) can perform the operation of step S340.

[0222] In one embodiment of the present disclosure, in step S390, as it is identified that there is no pair of parameters identified as having a smaller difference in value than or equal to the first reference number, the electronic device (100) can perform the operation of step S400.

[0223] FIG. 12 is a diagram illustrating the operation of an electronic device that obtains an output image by upscaling an input image including a plurality of channels through a convolution operation, according to one embodiment of the present disclosure. Hereinafter, configurations identical to those described in FIG. 4 and FIG. 7 are given the same reference numerals, and redundant descriptions are omitted.

[0224] Referring to FIG. 12, in one embodiment of the present disclosure, FIG. 12 illustrates an acquired input image (200) and a plurality of image channels included in the input image (200). When the number of the plurality of image channels is N, the operation of FIG. 12 may be repeated N times, or may be performed by N parallel-connected plurality of multipliers (1220) and plurality of accumulators (1240). Hereinafter, for convenience of explanation, a convolution operation in one of the plurality of image channels (210) will be described. In this case, N may be a natural number greater than or equal to 1.

[0225] In one embodiment of the present disclosure, a convolution operation in one image channel (210) can be performed on a plurality of receiving pixels each included in a plurality of corresponding receiving regions, with a plurality of convolution filters moving in one image channel (210).

[0226] In one embodiment of the present disclosure, a plurality of multipliers (1220) may be designed to perform a multiplication operation for each of a plurality of convolutional filters and each of a plurality of receiving pixels in parallel. A plurality of multipliers (1220) may be designed to perform an operation for each of a plurality of convolutional filters and each of a plurality of receiving pixels in parallel.

[0227] In one embodiment of the present disclosure, when the number of convolutional filters is M and the number of reception regions is K, the plurality of multipliers (1220) may be designed to perform M*K operations in parallel. In this case, M and K may be natural numbers greater than or equal to 1.

[0228] In one embodiment of the present disclosure, FIG. 12 illustrates that a multiplication operation for K of a plurality of receiving regions included in one image channel (210) is performed at once through a plurality of multipliers (1220) designed in parallel.

[0229] However, the present disclosure is not limited thereto, and it is understood that a multiplication operation for some of the multiple receiving regions included in one image channel (210) may be performed at once through a plurality of multipliers (1220) designed in parallel, and a multiplication operation for the remaining receiving regions included in one image channel (210) may be performed in the next sequence through the plurality of multipliers (1220). In this case, the number of multiple receiving regions included in one image channel (210) may be L, which is greater than K, and the number of some receiving regions may be K. At this time, L may be a natural number greater than K.

[0230] In one embodiment of the present disclosure, the number of a plurality of multipliers (1220) may be less than M*K. The number of a plurality of multipliers (1220) may be less than M*K by a second reference number.

[0231] In one embodiment of the present disclosure, the electronic device (100) may provide a plurality of interleaved receiving pixels, in which a plurality of receiving pixels included in a first receiving region (1200) are interleaved, to a plurality of multipliers (1220). The electronic device (100) may provide a plurality of interleaved parameters, in which a plurality of parameters included in a plurality of convolutional filters, for example, a first convolutional filter (1201) or a second convolutional filter (1211), are interleaved, to a plurality of multipliers (1220).

[0232] In one embodiment of the present disclosure, the electronic device (100) may be scheduled to sequentially perform multiplication operations of a plurality of interleaving accepting pixels and a plurality of interleaving parameters, such as a plurality of multipliers (1220), e.g., a first multiplier (1221).

[0233] In one embodiment of the present disclosure, an electronic device (100) may be scheduled to accumulate a plurality of accumulators (1240), such as a first accumulator (1241), to accumulate a multiplication operation performed in a plurality of multipliers (1220). The electronic device (100) may obtain an output image (300) based on output data (1250) obtained from a plurality of accumulators (1240). At this time, the output data (1250) obtained from a plurality of accumulators (1240) may include feature information extracted from one image channel (210) based on a plurality of convolutional filters.

[0234] In one embodiment of the present disclosure, the electronic device (100) may schedule a plurality of multipliers (1220) so as not to perform the multiplication operation when a plurality of interleaving accepting pixels or a plurality of interleaving parameters contain data with a value of "0" when performing the convolution operation described above.

[0235] Additionally, in one embodiment of the present disclosure, the electronic device (100) may schedule a plurality of multipliers (1220) and a plurality of accumulators (1240) to perform the convolution operation described above, wherein for pairs of parameters among a plurality of interleaving parameters that have a small difference in value, a multiplication operation is performed using a representative parameter, and the result is provided to an accumulator corresponding to each pair of parameters.

[0236] Through this, the electronic device (100) of the present disclosure can prevent performance degradation of the operation of upscaling the resolution of the input image (200) through convolutional operation to obtain an output image (300), even if the number of multipliers included in the plurality of multipliers (1220) is reduced.

[0237] To solve the technical problem described above, one embodiment of the present disclosure provides an electronic device that performs a convolution operation to obtain an output image by upscaling the resolution of an input image. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor including a processing circuit. By having at least one processor execute the program or at least one instruction stored in memory individually or collectively, the electronic device may obtain an input image containing a plurality of pixels. The electronic device may obtain a plurality of interleaving parameters in which a plurality of parameters included in each of a plurality of convolution filters for performing a convolution operation are interleaving. The electronic device may obtain a plurality of interleaving receptive pixels in which a plurality of receptive pixels included in receptive fields corresponding to each of the plurality of convolution filters among the plurality of pixels are interleaving. The electronic device can acquire an output image based on the result of a multiplication included in a convolution operation for at least one interleaving parameter having a non-zero value among a plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among a plurality of interleaving accepting pixels.

[0238] In one embodiment of the present disclosure, an electronic device may obtain a plurality of interleaving parameters from a memory. An electronic device may obtain a plurality of interleaving accepting pixels from a memory. The manner in which a plurality of accepting pixels are interleaved into a plurality of interleaving accepting pixels may be the same as the manner in which a plurality of parameters are interleaved into a plurality of interleaving parameters.

[0239] In one embodiment of the present disclosure, an electronic device may read a plurality of receiving pixels included in a memory in an interleaved state and obtain a plurality of interleaved receiving pixels. The method of interleaving a plurality of receiving pixels into a plurality of interleaved receiving pixels may be the same as the method of interleaving a plurality of parameters into a plurality of interleaved parameters.

[0240] In one embodiment of the present disclosure, an electronic device may include a plurality of multipliers for performing a multiplication operation. A plurality of interleaving parameters may include a plurality of first interleaving parameters in which a plurality of first parameters included in a first convolutional filter of any one of a plurality of convolutional filters are interleaved. A plurality of interleaving accepting pixels may include a plurality of first interleaving accepting pixels in which a plurality of first accepting pixels included in an accepting region corresponding to a first convolutional filter are interleaved. The electronic device may perform a multiplication operation of a plurality of first interleaving parameters and a plurality of first interleaving accepting pixels serially in any one of the plurality of multipliers.

[0241] In one embodiment of the present disclosure, the electronic device can schedule the operation of a plurality of multipliers so that a multiplication operation using a plurality of interleaving parameters and a plurality of interleaving accepting pixels is performed using a plurality of multipliers.

[0242] In one embodiment of the present disclosure, the electronic device may schedule the operation of a plurality of multipliers such that a multiplication operation is not performed using at least one interleaving parameter having a value of "0" among a plurality of interleaving parameters or at least one interleaving accepting pixel having a value of "0" among a plurality of interleaving accepting pixels.

[0243] In one embodiment of the present disclosure, each of the plurality of convolutional filters may have the same size. Based on the plurality of interleaving parameters, the electronic device may obtain a plurality of parameter pairs, each consisting of two interleaving parameters included in any two of the plurality of convolutional filters. The electronic device may identify at least one parameter pair among the plurality of parameter pairs in order from the smallest difference in value between the two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in value. The electronic device may obtain the result of a product operation between a representative parameter based on the two interleaving parameters included in each of the at least one parameter pair and a plurality of interleaving accepting pixels as the result of a product operation between each of the two interleaving parameters and a plurality of interleaving accepting pixels.

[0244] In one embodiment of the present disclosure, the number of a plurality of multipliers may be smaller than the number of a plurality of convolutional filters by a predetermined second reference number. As the second reference number increases, the first reference number may also increase.

[0245] In one embodiment of the present disclosure, an electronic device can identify whether the value of any one of a plurality of interleaving receiving pixels is "0". As the electronic device identifies that the value of one interleaving receiving pixel is not "0", it can identify whether the value of any one of a plurality of interleaving parameters corresponding to one of the interleaving receiving pixels is "0". As the electronic device identifies that the value of one interleaving parameter is not "0", it can obtain the difference between a plurality of values ​​corresponding to a plurality of parameter pairs containing one interleaving parameter. As the order in which the difference between the values ​​of at least one parameter pair among a plurality of parameter pairs is smaller is equal to or smaller than a first reference number, the electronic device can obtain the result of a product operation between a representative parameter based on at least one parameter pair and one interleaving receiving pixel as the result of a product operation between each of the two interleaving parameters included in at least one parameter pair and at least one interleaving receiving pixel.

[0246] In one embodiment of the present disclosure, an electronic device may include a plurality of accumulators for acquiring an output image by accumulating the result of a multiplication operation. The electronic device may schedule the operation of the plurality of accumulators so that the result of a multiplication operation between a representative parameter based on at least one pair of parameters and one interleaving accepting pixel can be accumulated as the result of a multiplication operation between each of the two interleaving parameters included in the pair of at least one parameter and at least one interleaving accepting pixel.

[0247] To solve the technical problem described above, one embodiment of the present disclosure may provide a method of operation for an electronic device that performs a convolution operation to upscale the resolution of an input image. The method of operation for the electronic device may include the step of acquiring an input image comprising a plurality of pixels. The method of operation for the electronic device may include the step of acquiring a plurality of interleaving parameters, wherein a plurality of parameters included in each of a plurality of convolution filters for performing a convolution operation are interleaving. The method of operation for the electronic device may include the step of acquiring a plurality of interleaving accepting pixels, wherein a plurality of accepting pixels included in a receptive field corresponding to each of the plurality of convolution filters among the plurality of pixels are interleaving. The method of operation for the electronic device may include the step of acquiring an output image based on the result of a multiplication included in the convolution operation for at least one interleaving parameter having a non-zero value among the plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among the plurality of interleaving accepting pixels.

[0248] In one embodiment of the present disclosure, the step of acquiring a plurality of interleaving parameters may include the step of acquiring a plurality of interleaving parameters from memory. The step of acquiring a plurality of interleaving accepting pixels may include the step of acquiring a plurality of interleaving accepting pixels from memory. The method in which a plurality of accepting pixels are interleaved into a plurality of interleaving accepting pixels may be the same as the method in which a plurality of parameters are interleaved into a plurality of interleaving parameters.

[0249] In one embodiment of the present disclosure, the step of acquiring a plurality of interleaving accepting pixels may include the step of acquiring a plurality of interleaving accepting pixels by reading a plurality of accepting pixels from memory in an interleaving state. The method of interleaving a plurality of accepting pixels into a plurality of interleaving accepting pixels may be the same as the method of interleaving a plurality of parameters into a plurality of interleaving parameters.

[0250] In one embodiment of the present disclosure, a plurality of interleaving parameters may include a plurality of first interleaving parameters in which a plurality of first parameters included in a first convolutional filter of any one of a plurality of convolutional filters are interleaved. A plurality of interleaving receiving pixels may include a plurality of first interleaving receiving pixels in which a plurality of first receiving pixels included in a receiving region corresponding to a first convolutional filter are interleaved. The step of acquiring an output image may include the step of acquiring the output image by performing a multiplication operation of a plurality of first interleaving parameters and a plurality of first interleaving receiving pixels serially in any one of a plurality of multipliers for performing a multiplication operation.

[0251] In one embodiment of the present disclosure, the step of acquiring an output image may include the step of scheduling the operation of a plurality of multipliers so that a multiplication operation is not performed using at least one interleaving parameter having a value of "0" among a plurality of interleaving parameters or at least one interleaving accepting pixel having a value of "0" among a plurality of interleaving accepting pixels.

[0252] In one embodiment of the present disclosure, a method of operating an electronic device may include the step of obtaining a plurality of parameter pairs, each consisting of two interleaving parameters included in any two convolutional filters among a plurality of interleaving sub-parameters, based on a plurality of interleaving parameters. The method of operating an electronic device may include the step of identifying at least one parameter pair, which is a predetermined first reference number, in order from the smallest difference in value between the two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in value. The method of operating an electronic device may include the step of obtaining the result of a product operation between a representative parameter based on two interleaving parameters included in each of at least one parameter pair and a plurality of interleaving accepting pixels as the result of a product operation between each of the two interleaving parameters and a plurality of interleaving accepting pixels.

[0253] In one embodiment of the present disclosure, the step of acquiring an output image may include identifying whether the value of any one of the interleaving accepting pixels is "0". The step of acquiring an output image may include identifying whether the value of any one of the interleaving parameters corresponding to the interleaving accepting pixel is "0", as the value of one of the interleaving accepting pixels is identified as not being "0". The step of acquiring an output image may include acquiring the difference between a plurality of values ​​corresponding to each of the plurality of parameter pairs containing one interleaving parameter, as the value of one interleaving parameter is identified as not being "0". The step of acquiring an output image may include acquiring the result of a product operation between a representative parameter based on at least one parameter pair and one interleaving accepting pixel as the result of a product operation between each of the two interleaving parameters included in at least one parameter pair and at least one interleaving accepting pixel, as the order in which the difference between the values ​​of at least one parameter pair among the plurality of parameter pairs is smaller is equal to or smaller than a first reference number.

[0254] In one embodiment of the present disclosure, the step of acquiring an output image may include the step of scheduling the operation of a plurality of accumulators to acquire an output image by accumulating the result of a multiplication operation such that the result of a multiplication operation of a representative parameter based on at least one pair of parameters and one interleaving accepting pixel can be accumulated as the result of a multiplication operation of each of the two interleaving parameters included in at least one pair of parameters and at least one interleaving accepting pixel.

[0255] In order to solve the aforementioned technical problem, a computer-readable recording medium may be provided on which a program for performing at least one method of an embodiment of the method of operating an electronic device disclosed in the present disclosure is recorded on a computer.

[0256] A program executed by an electronic device described in this disclosure may be implemented by hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0257] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively.

[0258] Software can be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). Computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The recording medium is readable by a computer, stored in memory, and can be executed by a processor.

[0259] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0260] In addition, the program according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0261] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically through a manufacturer of an electronic device or an electronic market (e.g., Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer of the electronic device, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0262] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or module are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. An electronic device (100) that performs a convolution operation to obtain an output image by up-scaling the resolution of an input image, Memory (110) where a program or at least one instruction is stored; and It includes at least one processor (130) including a processing circuit, and By having the above at least one processor (130) execute the above program or the above at least one instruction stored in the memory (110) individually or collectively, the electronic device (100) Acquire the input image including multiple pixels, Obtaining multiple interleaving parameters in which multiple parameters included in each of the multiple convolution filters for performing the above convolution operation are interleaving, and A plurality of interleaved receptive pixels are obtained, wherein a plurality of receptive pixels included in a receptive field corresponding to each of the plurality of convolutional filters among the plurality of pixels are interleaved, and An electronic device (100) that acquires an output image based on the result of a multiplication included in the convolution operation for at least one interleaving parameter having a non-zero value among the plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among the plurality of interleaving accepting pixels.

2. In Paragraph 1, The above electronic device (100) is, From the memory (110), the plurality of interleaving parameters are obtained, and From the memory (110), the plurality of interleaving accepting pixels are obtained, The method in which the plurality of receiving pixels are interleaved into the plurality of interleaved receiving pixels is the same as the method in which the plurality of parameters are interleaved into the plurality of interleaved parameters, electronic device (100).

3. In either Paragraph 1 or Paragraph 2, The above electronic device (100) is, From the memory (110), the plurality of receiving pixels are read in an interleaved state to obtain the plurality of interleaved receiving pixels, and The method in which the plurality of receiving pixels are interleaved into the plurality of interleaved receiving pixels is the same as the method in which the plurality of parameters are interleaved into the plurality of interleaved parameters, electronic device (100).

4. In Paragraph 1, The above electronic device (100) further includes a plurality of multipliers for performing multiplication operations, and The plurality of interleaving parameters includes a plurality of first interleaving parameters in which a plurality of first parameters included in a first convolutional filter among the plurality of convolutional filters are interleaved. The plurality of interleaving receiving pixels includes a plurality of first interleaving receiving pixels in which a plurality of first receiving pixels included in a receiving region corresponding to the first convolutional filter are interleaved. The electronic device (100) acquires the output image by performing a multiplication operation of the plurality of first interleaving parameters and the plurality of first interleaving accepting pixels serially in one of the plurality of multipliers.

5. In Paragraph 4, The above electronic device (100) is, An electronic device (100) that schedules the operation of the plurality of multipliers so that a multiplication operation using the plurality of interleaving parameters and the plurality of interleaving accepting pixels is performed using the plurality of multipliers.

6. In either Paragraph 4 or Paragraph 5, The above electronic device (100) is, An electronic device (100) for scheduling the operation of the plurality of multipliers so that a multiplication operation is not performed using at least one interleaving parameter having a value of "0" among the plurality of interleaving parameters or at least one interleaving accepting pixel having a value of "0" among the plurality of interleaving accepting pixels.

7. In Paragraph 4, Each of the above plurality of convolutional filters has the same size, and The above electronic device (100) is, Based on the above plurality of interleaving parameters, a plurality of parameter pairs consisting of two interleaving parameters each included in any two of the above plurality of convolutional filters are obtained, and Among the plurality of parameter pairs, at least one parameter pair is identified in order from the smallest difference in the values ​​of the two interleaving parameters included in each of the plurality of parameter pairs to the largest difference in values, up to a predetermined first reference number, and An electronic device (100) that obtains the result of a product operation of a representative parameter based on two interleaving parameters each included in at least one pair of parameters and a plurality of interleaving accepting pixels as the result of a product operation of each of the two interleaving parameters and the plurality of interleaving accepting pixels.

8. In Paragraph 7, The number of the plurality of multipliers is smaller than the number of the plurality of convolutional filters by a preset second reference number, and An electronic device (100) in which the number of first standards increases as the number of second standards increases.

9. In either Paragraph 7 or Paragraph 8, The above electronic device (100) is, Identify whether the value of any one of the aforementioned multiple interleaving accepting pixels is "0", and As the value of the one interleaving accepting pixel is identified as not being "0", the value of any one interleaving parameter corresponding to the one interleaving accepting pixel among the plurality of interleaving parameters is identified as being "0", and As the value of the above-mentioned one interleaving parameter is identified as not being "0", the difference between a plurality of values ​​corresponding to a plurality of parameter pairs including the above-mentioned one interleaving parameter is obtained, and An electronic device (100) that obtains the result of a product operation between a representative parameter based on the at least one parameter pair and the one interleaving accepting pixel, as the result of a product operation between each of the two interleaving parameters included in the at least one parameter pair and the at least one interleaving accepting pixel, depending on whether the order of the difference between the values ​​of at least one parameter pair among the plurality of parameter pairs is equal to or smaller than the first reference number.

10. In Paragraph 9, The above electronic device (100) is, It further includes a plurality of accumulators for accumulating the result of the above multiplication operation to obtain the output image, and The above electronic device (100) is, An electronic device (100) for scheduling the operation of a plurality of accumulators such that the result of the multiplication operation of a representative parameter based on the at least one pair of parameters and the one interleaving accepting pixel can be accumulated as the result of the multiplication operation of each of the two interleaving parameters included in the at least one pair of parameters and the at least one interleaving accepting pixel.

11. A method of operation of an electronic device (100) for performing a convolution operation to upscale the resolution of an input image, A step of acquiring the input image including a plurality of pixels (S100); A step (S200) of obtaining a plurality of interleaving parameters in which a plurality of parameters included in each of the plurality of convolution filters for performing the above convolution operation are interleaving; A step (S300) of obtaining a plurality of interleaved receptive pixels, wherein a plurality of receptive pixels included in a receptive field corresponding to each of the plurality of convolutional filters among the plurality of pixels are interleaved; A method of operation of an electronic device (100) comprising the step (S400) of obtaining an output image based on the result of a multiplication included in the convolution operation for at least one interleaving parameter having a non-zero value among the plurality of interleaving parameters and at least one interleaving accepting pixel having a non-zero value among the plurality of interleaving accepting pixels.

12. In Paragraph 11, In the step (S200) of acquiring the plurality of interleaving parameters, the plurality of interleaving parameters are acquired from memory, and In the step (S300) of acquiring the plurality of interleaving accepting pixels, the plurality of interleaving accepting pixels are acquired from the memory, and The method of operation of an electronic device (100) in which the plurality of receiving pixels are interleaved with the plurality of interleaved receiving pixels is the same as the method in which the plurality of parameters are interleaved with the plurality of interleaved parameters.

13. In either Article 11 or Article 12, The step (S300) of acquiring the plurality of interleaving accepting pixels is, The method includes the step of reading the plurality of receiving pixels from memory in an interleaved state and obtaining the plurality of interleaved receiving pixels. The method of operation of an electronic device (100) in which the plurality of receiving pixels are interleaved with the plurality of interleaved receiving pixels is the same as the method in which the plurality of parameters are interleaved with the plurality of interleaved parameters.

14. In Paragraph 11, The plurality of interleaving parameters includes a plurality of first interleaving parameters in which a plurality of first parameters included in a first convolutional filter among the plurality of convolutional filters are interleaved. The plurality of interleaving receiving pixels includes a plurality of first interleaving receiving pixels in which a plurality of first receiving pixels included in a receiving region corresponding to the first convolutional filter are interleaved. The step of acquiring the above output image (S400) is, A method of operation of an electronic device (100) comprising the step of obtaining the output image by performing a multiplication operation of the plurality of first interleaving parameters and the plurality of first interleaving accepting pixels serially in one of the plurality of multipliers for performing a multiplication operation.

15. A computer-readable recording medium having a program recorded thereon for performing the method of operation described in any one of claims 11 through 14 on a computer.

Images