Electronic device and control method thereof

Abstract

An electronic device and a control method thereof are provided. The electronic device includes a memory for storing at least one instruction, and a processor configured to execute the at least one instruction, wherein the processor is configured to perform a convolution operation on an input image and obtain intermediate feature data related to the image. The intermediate feature data is convolved with a first kernel in a channel direction to obtain first data. The first data is then convolved with a second kernel in a spatial direction to obtain second data. Values of one or more weights included in the first kernel and the second kernel are set based on the second data, and the values of the weights can be adjusted based on positions of the weights.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device obtaining an image having no moire by performing a convolution operation on feature data related to the image using a plurality of kernels, and a control method thereof. BACKGROUND

[0002] In recent years, artificial intelligence systems are applied to various fields. Unlike an intelligent system that performs various functions based on rules applied in advance, an artificial intelligence system is a system in which a machine trains itself, determines, and becomes intelligent. Accordingly, as the artificial intelligence system is used, the recognition rate is improved, and the preference of a user can be more accurately understood, and thus, existing intelligent systems are being gradually replaced by artificial intelligence systems. A neural network is a representative technology of such an artificial intelligence system.

[0003] A neural network is a learning algorithm obtained by modeling the characteristics of a biological neuron through a mathematical expression. The neural network can generate a mapping between input data and output data through the above-described learning algorithm, and the ability to generate the mapping can be the learning ability of the neural network. A convolutional neural network in the neural network is mainly used to analyze a visual image.

[0004] In a convolutional neural network or the like, an inverse convolution operation (or process) needs to be performed in order to generate an output image having a size larger than that of an input image by upsampling the input image. However, when the inverse convolution operation is performed, the degree of overlap of a kernel can be different at each position of an output image in a case where a size value of the kernel is not divided by a size value of a step applied to the inverse convolution operation. When the degree of overlap of the kernel becomes different at each position of the output image, artifacts can be uniformly generated in the image in a checkerboard shape.

[0005] In addition, there is a problem in that the processing amount of the existing inverse convolution operation occupies a considerable portion of the entire processing amount of a network. SUMMARY

[0006] An electronic device is provided herein, including a memory for storing at least one instruction; and a processor configured to execute the at least one instruction, wherein the processor is configured to execute the at least one instruction to perform a first convolution operation on an input image and obtain intermediate feature data as a result of the first convolution operation, obtain first data by performing a second convolution operation on the intermediate feature data using a plurality of first kernels in a channel direction, wherein the plurality of first kernels includes first weights, obtain second data by performing a third convolution operation on the first data using a second kernel in a spatial direction, wherein the second kernel includes second weights, set a first value of the first weights or set a second value of the second weights based on the second data, adjust the first value of the first weights based on a first position of the first weights, and adjust the second value of the second weights based on a second position of the second weights.

[0007] In some embodiments of the electronic device, one of a height and a width of the plurality of first kernels has a first parameter, and the other of the height and the width has a second parameter, wherein the first parameter is 1, and the second parameter is a predetermined integer value other than 1, wherein the processor is further configured to normalize the first value of the first weights based on the first position of the first weights in the plurality of first kernels, and normalize the second value of the second weights based on the second position of the second weights in the second kernel.

[0008] In some embodiments of the electronic device, the processor is further configured to adjust the first value of the first weights to be the same in each of the plurality of first kernels.

[0009] In some embodiments of the electronic device, the processor is further configured to adjust the second value of the second weights by applying a reliability map including a weight function to the second kernel.

[0010] In some embodiments of the electronic device, the weight function includes a function having a value gradually changing from a center of the reliability map.

[0011] In some embodiments of the electronic device, the processor is further configured to decompose the second weights of the second kernel into a plurality of groups, and normalize each of the plurality of groups based on the position of the second weights in the second kernel.

[0012] In some embodiments of the electronic device, the processor is further configured to identify a number of the plurality of groups and a number of weights included in each of the plurality of groups based on a parameter value of the second kernel and a size of a step applied for the third convolution operation.

[0013] In some embodiments of the electronic device, the processor is further configured to adjust a second value of the second weight for a first group of the plurality of groups to make a sum of the second weight included in the first group of the plurality of groups consistent.

[0014] In some embodiments of the electronic device, the processor is further configured to: obtain second data by performing a third convolution operation on the first data using the plurality of groups, and obtain the output image by rearranging the second data.

[0015] In some embodiments, the electronic device further includes a display, and the processor is further configured to control the display to display the output image, wherein the output image has a first size that is greater than a second size of the input image.

[0016] Also provided herein is a method for controlling an electronic device, the method including: performing a first convolution operation on an input image, and obtaining intermediate feature data as a result of the first convolution operation; obtaining first data by performing a second convolution operation on the intermediate feature data using a plurality of first kernels in a channel direction, wherein the plurality of first kernels include first weights; obtaining second data by performing a third convolution operation on the first data using a second kernel in a spatial direction, wherein the second kernel includes second weights; setting a first value of the first weights or a second value of the second weights based on the second data; adjusting the first value of the first weights based on a first position of the first weights; and adjusting the second value of the second weights based on a second position of the second weights.

[0017] According to embodiments of the disclosure, there is provided a memory for storing at least one instruction, and a processor configured to execute the at least one instruction, wherein the processor is configured to perform a convolution operation on an input image and obtain intermediate feature data related to the image, perform a convolution operation on the intermediate feature data using a first kernel in a channel direction to obtain first data, and perform a convolution operation on the obtained first data using a second kernel in a spatial direction to obtain second data. A value of one or more weights included in the first kernel and the second kernel is set based on the obtained second data, and the set value of the weights is adjusted based on a position of the weights.

[0018] According to another embodiment of the disclosure, a method for controlling an electronic device is provided, the method including performing a convolution operation on an input image and obtaining intermediate feature data related to the image, obtaining first data by performing a convolution operation on the intermediate feature data using a first kernel in a channel direction, obtaining second data by performing a convolution operation on the obtained first data using a second kernel in a spatial direction, setting values of one or more weights included in the first kernel and the second kernel based on the obtained second data, and adjusting the set values of the weights based on positions of the weights.

[0019] Advantages of the Invention

[0020] According to embodiments of the disclosure, an electronic device can prevent a checkerboard effect from being generated by performing a convolution operation on data related to an image using a plurality of kernels, generate a high-quality image when resizing the image, and reduce a processing amount and a size of a memory. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1A is a view for describing a process of obtaining second data by performing a convolution operation on an input image according to an embodiment;

[0022] Figure 1B is a view for describing a process of obtaining second data by performing a convolution operation on an input image according to an embodiment;

[0023] Figure 1C is a view for describing a process of obtaining second data by performing a convolution operation on an input image according to an embodiment;

[0024] Figure 2A is a block diagram simply illustrating a configuration of an electronic device according to an embodiment;

[0025] Figure 2B is a block diagram specifically illustrating a configuration of an electronic device according to an embodiment;

[0026] Figure 3 is a view for describing a process of performing a deconvolution operation according to an embodiment;

[0027] Figure 4 is a view for describing a process of performing a convolution operation on intermediate feature data using a first kernel in a channel direction according to an embodiment;

[0028] Figure 5 is a view for describing a process of adjusting values of weights included in a second kernel according to an embodiment;

[0029] Figure 6is a view for describing a process of decomposing weights included in a second kernel into a plurality of groups according to an embodiment;

[0030] Figure 7 is a view illustrating generation of an image with a checkerboard effect and an image without a checkerboard effect according to an embodiment; and

[0031] Figure 8 is a flowchart for describing a method for controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0032] The disclosure is made to solve the above problems, and an object of the disclosure is to provide an electronic device that performs a convolution operation on data related to an image using a plurality of kernels and adjusts a value of a weight included in each kernel based on a result value of the performance, and a control method thereof.

[0033] Hereinafter, various embodiments of the disclosure will be described with reference to the accompanying drawings. It should be noted that the technology disclosed in the disclosure is not intended to limit the scope of the disclosure to a specific embodiment, but should be interpreted to include all modifications, equivalents, or alternatives of the embodiments of the disclosure. With regard to the explanation of the drawings, like reference numerals can be used for like elements.

[0034] In the disclosure, terms such as "consisting of," "may consist of," "comprising" or "may include" indicate the presence of a feature (for example, a component such as a number, a function, an operation, or a part) and do not exclude the presence of additional features.

[0035] In the disclosure, expressions such as "A or B," "at least one of A [and / or] B," or "one or more of A [and / or] B" include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" includes any one of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0036] The expressions "first," "second," and the like used in the disclosure can represent various elements regardless of order and / or importance, and can be used to distinguish one element from another element, and do not limit the elements.

[0037] If it is described that a certain element (e.g., a first element) is "operatively or communicatively coupled / coupled to" or "connected to" another element (e.g., a second element), it should be understood that the certain element can be connected to the other element directly or through yet another element (e.g., a third element). On the other hand, if it is described that a certain element (e.g., a first element) is "directly coupled to" or "directly connected to" another element (e.g., a second element), it can be understood that there is no element (e.g., a third element) between the certain element and the other element.

[0038] Further, the expression "configured to" used in the present disclosure can be used interchangeably with other expressions such as "suitable for", "having the capacity to", "designed to", "adapted to", "made to", and "capable of", depending on the circumstances. In addition, the expression "configured to" does not necessarily mean that the device is "specifically designed to" in terms of hardware. Rather, in some cases, the expression "a device configured to" can mean that the device "is capable of" performing an operation with another device or component. For example, the phrase "a unit or a processor configured (or set) to perform A, B, and C" can mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in a memory device.

[0039] The electronic device according to various embodiments of the present disclosure can include at least one of, for example, a smartphone, a tablet PC, a mobile phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a portable multimedia player (PMP), a medical device, a camera, or a wearable device. In the present disclosure, the term "user" can refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0040] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0041] Figure 1A 、 Figure 1B and Figure 1C are views for describing a process of obtaining second data by performing a convolution operation on an input image according to an embodiment of the present disclosure. As described above, the convolution operation is a process of obtaining a feature map by applying a filter to an input image. Figure 1AAs shown, an image 10 with parameters of height h and width w can be input into an electronic device 100. The electronic device 100 can feed the input image 10 into a convolutional neural network (CNN) to extract features from the input image 10 and obtain intermediate feature data 30 related to the image based on the extracted features. The intermediate feature data 30 can be a feature map obtained based on the extracted features from the input image 10, or it can be in vector or matrix form, but this is merely an example. Figure 1A As shown, the intermediate feature data 30 can have height h and width w parameters in the same way as the input image 10, and can have channel parameters d.

[0042] like Figure 1B As shown, the electronic device 100 can obtain first data by performing a convolution operation 40 on intermediate feature data 30 using first kernels 50-1, 50-2, 50-3, ..., and 50-N in the channel direction, and obtain second data 90 by performing a convolution operation 50 on the obtained first data using a second kernel 60 in the spatial direction. The channel direction may correspond to the input depth. In some embodiments, for example, a channel may correspond to a color (or pattern). One of the height and width of each of the first kernels 50-1, 50-2, ..., and 50-N in the channel direction may have a parameter 1, while the other may have a parameter with a predetermined integer value other than 1, and the channel parameter d may be the same as the channel parameter of the intermediate feature data 30. Using the second kernel 60 in the spatial direction, convolution can be performed in the spatial direction for each channel of the first data.

[0043] According to embodiments of this disclosure, Figure 1B The diagram shows first kernels 50-1, 50-2, ..., and 50-N, in which the height has a parameter of 1 and the width has a predetermined value W other than 1. VK The parameters are the same as those of the intermediate feature data 30, and the channel parameters are identical. Therefore, the first kernel operates on depth d. Figure 1B As shown, the operation performed between the first kernels 50-1, 50-2, ..., 50-N in the channel direction and the intermediate feature data 30 can be referred to as vertical convolution. In another embodiment, the height is utilized by a parameter W other than 1. VK A convolution performed by a kernel with a predetermined width of 1 and channel parameters identical to those of the intermediate feature data 30 can be termed a horizontal convolution. (Refer to...) Figure 4 Describe in detail the convolution operations between the first kernels 50-1, 50-2, ..., and 50-N and the intermediate feature data 30.

[0044] The electronic device 100 can normalize the first kernels 50-1, 50-2,..., and 50-N based on positions of the weights included in the first kernels 50-1, 50-2,..., and 50-N. Specifically, the electronic device 100 can adjust values of the weights to have the same sum of the weights included in each of the first kernels 50-1, 50-2,..., and 50-N. Generally, in a case where a deconvolution operation is performed between input data and a kernel, a rapid change in values of the weights included in the kernel can cause a checkerboard effect in output data. For example, the deconvolution can be used to enlarge an image or reduce blurring. In particular, when adjacent weight values are rapidly changed in a high-frequency region (e.g., a region having high pixel values) of the input data, the checkerboard effect can be generated in a region of the output data corresponding to the high-frequency region. Accordingly, to prevent generation of the checkerboard effect, the electronic device 100 can normalize the first kernels 50-1, 50-2,..., and 50-N to have the same sum of the weights included in the first kernels 50-1, 50-2,..., and 50-N. The reason for the generation of the checkerboard effect and the normalization process will be described in detail. Figure 3 and Figure 5 The reason for the generation of the checkerboard effect and the normalization process will be described in detail.

[0045] The electronic device 100 can adjust values of the weights included in the second kernels 60 by applying the reliability map 70 including a weight function to the second kernels 60. The weight function can include a function in which values are gradually changed from the center of the reliability map 70. In an embodiment, the weight function can include at least one of a linear function, a Gaussian function, a Laplacian function, and a spline function, but this is merely an embodiment, and the weight function can include various functions. In a case where the reliability map 70 is applied to the second kernels 60, the values of the weights included in the second kernels 60 are not rapidly changed, and thus generation of the checkerboard effect in the second data 90 can be prevented. Specifically, generation of the checkerboard effect in a region of the second data 90 corresponding to a high-frequency region (e.g., a region having high pixel values) of the input data can be prevented.

[0046] In addition, the electronic device 100 can decompose the weight of the second kernel 60 into a plurality of groups 80-1, 80-2, 80-3,..., and 80-N and normalize each of the plurality of decomposed groups 80-1, 80-2,..., and 80-N based on a position of the weight included in the second kernel 60. The decomposition of a filter function such as a kernel can also be referred to as factorization of a convolution kernel. Specifically, the electronic device 100 can determine the number of the plurality of groups 80-1, 80-2,..., and 80-N and the number of weights included in the plurality of groups 80-1, 80-2,..., and 80-N based on the parameter value of the second kernel 60 and the size of the stride applied to the convolution operation. In addition, the electronic device 100 can adjust the values of the weights to make the sum of the weights included in each of the plurality of groups 80-1, 80-2,..., and 80-N consistent. The process of decomposing the second kernel 60 and setting the sum of the weights to be consistent will be described in detail with reference to Figure 6 A detailed description of the process of decomposing the second kernel 60 and setting the sum of the weights to be consistent.

[0047] The electronic device 100 can obtain the second data 90 by performing a convolution operation on the plurality of groups 80-1, 80-2,..., and 80-N in the spatial direction and the first data, and obtain the output image 95 by rearranging the obtained second data 90. The convolution operation performed on the plurality of groups 80-1, 80-2,..., and 80-N in the spatial direction with respect to each channel of the first data can be referred to as a depthwise convolution. The process of performing the depthwise convolution will be described in detail with reference to Figure 4 and Figure 5 A detailed description of the process of performing the depthwise convolution.

[0048] In addition, the electronic device 100 can obtain the output image 95 without the chessboard effect, which is larger in size than the input image 10, and display the obtained output image 95 on the display 130.

[0049] Figure 2A The configuration of the electronic device 100 according to an embodiment of the disclosure is simply illustrated. As Figure 2A As illustrated, the electronic device 100 can include the memory 110 and the processor 120. However, there is no limitation on the above-described configuration, and some configurations can be added or omitted according to the type of the electronic device 100.

[0050] The memory 110 can store instructions or data related to at least one of the other elements of the electronic device 100. In particular, the memory 110 can be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The processor 120 can access the memory 110, and the processor 120 can perform reading, recording, editing, deleting, or updating of data. The term "memory" in the present disclosure can include the memory 110, a ROM (not shown) or a RAM (not shown) in the processor 120, or a memory card (not shown) (e.g., a micro SD card or a memory stick) installed on the electronic device 100. In addition, the memory 110 can store programs or data for configuring various screens displayed on a display area of the display 130.

[0051] In addition, the memory 110 can store a program for executing an artificial intelligence agent. The artificial intelligence agent is a customized program for providing various services to the electronic device 100. In addition, the memory 110 can store an artificial intelligence model trained to extract data of an input image.

[0052] The processor 120 can be electrically connected to the memory 110 and control general operations and functions of the electronic device 100 by executing at least one instruction.

[0053] In particular, the processor 120 can perform a convolution operation related to an input image and obtain intermediate feature data related to the image. In an embodiment of the disclosure, the processor 120 can input the image to a convolutional neural network (CNN) and extract intermediate feature data or a feature map. It is well known to extract feature data of an input image through a CNN, and thus will be omitted.

[0054] The processor 120 can obtain first data by performing a convolution operation (vertical direction convolution or horizontal direction convolution) on the obtained intermediate feature data related to the image by using a first kernel in a channel direction, and obtain second data by performing a convolution operation (depth direction convolution) on the obtained first data by using a second kernel in a spatial direction.

[0055] In addition, the processor 120 can set values of one or more weights included in the first kernel and the second kernel based on the obtained second data. In an embodiment, the processor 120 can set the weight values included in the first kernel and the second kernel using a learning algorithm including error backpropagation or gradient descent. In particular, the processor 120 can obtain an output image by rearranging the obtained second data, and compare and analyze the output image and an image obtained by upscaling the input image. The processor 120 can set the weight values of the first kernel and the second kernel based on the analysis result.

[0056] The processor 120 can normalize each first kernel based on positions of weights included in the first kernel. Specifically, the number of weights applied to each pixel included in first data obtained by performing a convolution operation with the first kernel in a channel direction can differ from each other, and when the weights applied to one pixel are not normalized, the sum of the weights applied to each pixel of the first data can not be consistent. Accordingly, in an embodiment, the processor 120 can adjust the values of the weights to make the sum of the weights included in each first kernel consistent.

[0057] In addition, the processor 120 can adjust the values of the weights included in the second kernel by applying a reliability map including a weight function to the second kernel. Specifically, the processor 120 can adjust the values of the weights included in the second kernel by multiplying the second kernel by the reliability map. The weight function included in the reliability map can include at least one of a linear function, a Gaussian function, a Laplacian function, and a spline function, but this is merely an embodiment, and the weight function can include various functions.

[0058] The processor 120 can decompose the weights of the second kernel into a plurality of groups, and normalize each of the plurality of decomposed groups based on positions of the weights included in the second kernel. Specifically, the processor 120 can determine the number of the plurality of groups and the number of the weights included in the plurality of groups based on a parameter value (or size) of the second kernel and a size of a stride applied to the convolution operation. In addition, the processor 120 can adjust the values of the weights to make the sum of the weights included in the plurality of decomposed groups consistent.

[0059] Further, the processor 120 can obtain second data by performing a convolution operation on the plurality of groups in a spatial direction with the first data, and obtain an output image by obtaining the second data. The size of the output image can be greater than the size of the input image, and no checkerboard effect can occur. The processor 120 can control the display 130 to display the output image.

[0060] In describing the present disclosure, the processor 120 can be constituted by one or more processors. The memory 110 and the processor 120 operate functions related to artificial intelligence according to the present disclosure. The one or more processors 120 perform control to process input data according to a predefined action rule or an artificial intelligence model stored in the memory 110. The predefined action rule or the artificial intelligence model is formed by training. Formed by training herein means that a predefined action rule or an artificial intelligence model having desired characteristics is formed by applying a training algorithm to a plurality of pieces of learning data. Such training can be performed in a device that exhibits artificial intelligence according to the present disclosure, or by a separate server or system.

[0061] The functions related to artificial intelligence according to the disclosure are operated by the processor and the memory. The processor can be composed of one or more processors. The one or more processors can be a general-purpose processor such as a CPU, an AP, or a digital signal processor (DSP), a graphics-specialized processor such as a GPU or a VPU, or an artificial intelligence processor such as an NPU. The one or more processors perform control to process the input data according to a predefined action rule or an artificial intelligence model stored in the memory. In addition, if the one or more processors are artificial intelligence-specialized processors, the artificial intelligence-specialized processors can be designed to have a hardware structure specialized in processing a specific artificial intelligence model.

[0062] The predefined action rule or the artificial intelligence model is formed by training. The meaning of being formed by training herein means that a predefined action rule or an artificial intelligence model set to perform a desired feature is formed by training a basic artificial intelligence model according to a training algorithm using a plurality of pieces of learning data. Such training can be performed in the device that exhibits artificial intelligence according to the disclosure, or by a separate server or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to these examples.

[0063] The artificial intelligence model can be composed of a plurality of neural network layers. The plurality of neural network layers respectively have a plurality of weight values, and perform neural network processing through processing between a processing result of a previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers can be optimized by a training result of the artificial intelligence model. For example, the plurality of weights can be updated to reduce or minimize a loss value or a cost value obtained by the artificial intelligence model during training processing. The artificial neural network can include a deep neural network (DNN), and for example, include a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, but is not limited to these examples.

[0064] Figure 2B is a block diagram specifically illustrating a configuration of the electronic device 100 according to an embodiment of the disclosure. As shown in Figure 2B the electronic device 100 can include a memory 110, a processor 120, a display 130, a camera 140, and a communication unit 150. The communication unit 150 can include a network interface card for communication with a network and / or a radio transceiver for wireless communication. The memory 110 and the processor 120 have been described in Figure 2A , and thus a repetitive description will be omitted.

[0065] The display 130 can display various information under the control of the processor 120. Specifically, the processor 120 can control the display 130 to display output data obtained by rearranging the second data.

[0066] The display 130 can be implemented as a touch screen having a touch panel. However, there is no limitation on the above-described embodiment, and the display 130 can be variously implemented according to the type of the electronic device 100.

[0067] The camera 140 can image a user. Specifically, the captured image of the user can be included in a UI displayed when the user is recognized. The camera 140 can be disposed on at least one of the front or the back of the electronic device 100. The camera 140 can be disposed in the electronic device 100, but this is merely an embodiment, and the camera 140 can also be disposed outside the electronic device 100 and connected to the electronic device 100 in a wired or wireless manner.

[0068] The communication unit 150 can perform communication with an external device through various communication methods. The communication connection between the communication unit 150 and the external device can include communication via a third device (for example, a relay device, a hub, an access point, a server, or a gateway).

[0069] The communication unit 150 can include various communication modules for performing communication with an external device. As an example, the communication unit 150 can include a wireless communication module, and for example, can include a cellular communication module using at least one of LTE, LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunication System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). In another example, the wireless communication module can include at least one of WiFi (Wireless-Fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Near Field Communication (NFC), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN), for example. In addition, the communication unit 150 can include a wired communication module, and for example, can include at least one of a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Recommended Standard 232 (RS-232), power line communication, or a Plain Old Telephone Service (POTS). A network through which wireless communication or wired communication is performed can include at least one of a telecommunication network (for example, a computer network (for example, LAN or WAN)), the Internet, or a telephone network.

[0070] The processor 120 can include or be defined as one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), and an ARM processor. In addition, the processor 120 can be implemented as a system on chip (SoC) or a large scale integration (LSI) having an embedded processing algorithm, or can be implemented in the form of a field programmable gate array (FPGA). The processor 120 can perform various functions by executing computer executable instructions stored in the memory 110. In addition, the processor 120 can include at least one of a graphic processing unit (GPU), a neural processing unit (NPU), and a vision processing unit (VPU) as a separate AI-dedicated processor in order to perform an artificial intelligence function.

[0071] Figure 3 is a view for describing a process of performing a deconvolution operation and a reason for generating a checkerboard effect. That is, Figure 3 is a view for describing a case where a checkerboard effect can be generated when a deconvolution operation is immediately performed to change the size of image-related intermediate feature data obtained from an input image.

[0072] In Figure 3 , for convenience of description, it is assumed that the input data 310, the kernel 320, and the output data 330 are in a one-dimensional manner. In addition, it is assumed that the size of the input data 310 is 5, the size of the kernel 320 applied to the input data 310 is 5, the size of the step is 1, and the size of the output data 330 is 9.

[0073] Referring to Figure 3 , each of values I0*W0, I0*W1, I0*W2, I0*W3, and I0*W4 obtained by multiplying the pixel value I0 of the input data 310 by the weight values W0, W1, W2, W3, and W4 included in the kernel can be mapped to each of the first to fifth pixels 331, 332, 333, 334, and 335 of the output data 330.

[0074] In addition, each of values I1*W0, I1*W1, I1*W2, I1*W3, and I1*W4 obtained by multiplying the pixel value I1 of the input data 310 by the weight values W0, W1, W2, W3, and W4 included in the kernel 320 can be mapped to each of the second to sixth pixels 332, 333, 334, 335, and 336 of the output data 330.

[0075] In addition, each of values I2*W0, I2*W1, I2*W2, I2*W3, and I2*W4 obtained by multiplying the pixel value I2 of the input data 310 by the weight values W0, W1, W2, W3, and W4 included in the kernel 320 can be mapped to each of the third to seventh pixels 333, 334, 335, 336, and 337 of the output data 330.

[0076] In addition, each of values I3*W0, I3*W1, I3*W2, I3*W3, and I3*W4 obtained by multiplying the pixel value I3 of the input data 310 by the weight values W0, W1, W2, W3, and W4 included in the kernel 320 can be mapped to each of the fourth to eighth pixels 334, 335, 336, 337, and 338 of the output data 330.

[0077] In addition, each of values I4*W0, I4*W1, I4*W2, I4*W3, and I4*W4 obtained by multiplying the pixel value I4 of the input data by the weight values W0, W1, W2, W3, and W4 included in the kernel 320 can be mapped to each of the fifth to ninth pixels 335, 336, 337, 338, and 339 of the output data 330.

[0078] Accordingly, the value O0 of the first pixel 331 of the output data 330 is I0*W0, the value O1 of the second pixel 332 is I0*W1+I1*W0, the value O2 of the third pixel 333 is I0*W2+I1*W1+I2*W0, the value O3 of the fourth pixel 334 is I0*W3+I1*W2+I2*W1+I3*W0, and the value O4 of the fifth pixel 335 is I0*W4+I1*W3+I2*W2+I3*W1+I4*W0.

[0079] From the perspective of the input data 310, each of the plurality of weight values (for example, W0, W1, W2, W3, and W4) is multiplied by one pixel value (for example, I0) of the input data 310, and the values 340 obtained by multiplying the plurality of weights are mapped to the plurality of pixels (for example, 331 to 335) of the output data, and thus the deconvolution operation corresponds to the dispersion operation.

[0080] When the weight values (e.g., W0, W1, W2, W3, and W4) included in the kernel rapidly change, a checkerboard effect can occur in the output data. In particular, when the adjacent weight values rapidly change in a high-frequency region (a region having high pixel values) of the input data 310, the checkerboard effect can occur in a region of the output data corresponding to the high-frequency region. In addition, from the perspective of the output data 330, one pixel value (e.g., O4) of the output data 330 is determined according to a value obtained by adding values 350 obtained by multiplying each of a plurality of pixel values (e.g., I0, I1, I2, I3, and I4) of the input data 310 by each of a plurality of weight values (e.g., W0, W1, W2, W3, and W4). Accordingly, the deconvolution operation corresponds to the aggregation operation.

[0081] The weight applied to each pixel included in the output data 330 is not the same. For example, referring to Figure 3 one weight W0 is applied to the first pixel 331, two weights W0 and W1 are applied to the second pixel 332, three weights W0, W1, and W2 are applied to the third pixel 333, four weights W0, W1, W2, and W3 are applied to the fourth pixel 334, and five weights W0, W1, W2, W3, and W4 are applied to the fifth pixel 335. As described above, when the number of weights applied to each pixel included in the output data 330 is different from each other and the weights applied to one pixel are not normalized, the sum of the weights applied to each pixel of the output data 330 can not be consistent.

[0082] For example, when the sum of the four weights W0, W1, W2, and W3 applied to the fourth pixel 334 is inconsistent with the sum of the five weights W0, W1, W2, W3, and W4 applied to the fifth pixel, a checkerboard effect can occur in the output data when the deconvolution operation is performed. In some cases, the number of applicable weights depends on the location of the pixel being obtained (see 331, 339 in Figure 3 By adjusting the sum of the weights according to which pixel is being obtained, the variation in the output image caused by the filter weights themselves can be reduced. When image processing is performed on a small region in an image, the number of weights applied to a pixel can vary at the edges of the small region. Repeated occurrences of the varying number of weights throughout the processing of the entire image can result in a checkerboard pattern.

[0083] Figure 4 is a view for describing a process of performing a convolution operation on the intermediate feature data 30 using a first kernel pair in a channel direction according to an embodiment of the disclosure. As Figure 4As illustrated, the electronic device 100 can perform a convolution operation on the intermediate feature data 30 and the first kernel 50-1 in the channel direction. The channel parameter of the first kernel 50-1 in the channel direction can be the same as that of the intermediate feature data 30 (e.g., d). As for the parameter of the first kernel 50-1, one of the height and the width can have a parameter of 1, and the other can have a parameter of a predetermined integer value other than 1. Figure 4 The first kernel 50-1 in which the height has a parameter of 1 and the width has a parameter of a predetermined integer value other than 1 is illustrated, but this is merely an embodiment, and the first kernel can have a parameter in which the width has a parameter of 1 and the height has a parameter of a predetermined integer value other than 1.

[0084] Figure 4 Only one first kernel 50-1 is illustrated, but the electronic device 100 can obtain the first data 400 by performing a convolution operation on the intermediate feature data 30 using N first kernels. The electronic device 100 can compress the intermediate feature data 30 into one channel by performing a convolution operation using the first kernel in the channel direction. As Figure 4 As illustrated, the channel parameter of the first data 400 can be N because the electronic device 100 performs a convolution operation using N first kernels.

[0085] All pixels included in the intermediate feature data 30 can include the same pixel value (e.g., 1). The value of each pixel included in the first data 400 can be expressed as the sum of the weights applied to each pixel. In the case where the weights applied to one pixel are not normalized, the sum of the weights applied to each pixel is not consistent, and thus, the first data 400 can include a checkerboard effect having a certain pattern. Accordingly, the electronic device 100 can normalize the first kernel 50-1 based on the positions of the weights included in the first kernel 50-1. In an example, the electronic device 100 can adjust the values of the weights so that the sum of the weights included in each first kernel is consistent. In addition, the electronic device 100 can adjust the weights so that the value of the pixel of the first data 400 is the same as that of the pixel of the intermediate feature data 30 (e.g., 1), and the sum of the weights applied to each pixel of the first data 400 becomes 1.

[0086] Figure 5 is a view for describing a process of adjusting the values of the weights included in the second kernel 60 according to an embodiment of the disclosure. As Figure 5 As illustrated, the electronic device 100 can apply (501) the reliability map 70 including the weight function to the second kernel 60. The electronic device 100 can decompose the weights of the second kernel 60 into a plurality of groups, and normalize each of the plurality of decomposed groups based on the positions of the weights included in the second kernel 60.

[0087] The electronic device 100 can set values of one or more weights included in the second kernel 60 used in the convolution operation. At this time, the values of the weights included in the second kernel 60 can be set according to learning and updating of a neural network including a convolution layer that performs the convolution operation, but are not limited thereto.

[0088] The electronic device 100 according to an embodiment of the disclosure can adjust the values of the one or more weights included in the second kernel 60 by applying (e.g., performing multiplication) the reliability map 70 to the second kernel 60. The reliability map 70 according to an embodiment of the disclosure can include a weight function, and the weight function can be a function that decreases values from the center of the reliability map 70. That is, the reliability is high when approaching the center of the reliability map 70. The weight function can include at least one of a linear function, a Gaussian function, a Laplacian function, and a spline function, but this is merely an embodiment. Figure 5 The reliability map 70 shown in FIG. 7A can be a graph representing a Gaussian function.

[0089] According to an embodiment of the disclosure, in a case where the reliability map 70 is applied to the second kernel 60, the values of the one or more weights included in the second kernel 60 can not be rapidly changed. In a case where the values of the weights are rapidly changed, a checkerboard effect can be generated in a high-frequency region of second data obtained by convolution performed with the second kernel. Accordingly, the electronic device 100 can set the values of the weights not to be rapidly changed by applying (e.g., performing multiplication) the reliability map 70 to the second kernel 60.

[0090] The electronic device 100 can decompose the weights included in the second kernel 60 into a plurality of groups 80-1, 80-2,..., and 80-N based on positions in the second kernel 60. This will be described with reference to FIG. 8. Figure 6 A method for decomposing the weights included in the second kernel 60 into a plurality of groups will be described in detail.

[0091] The electronic device 100 can normalize each of the plurality of decomposed groups 80-1, 80-2,..., and 80-N. In an example, the electronic device 100 can perform normalization to make the weights included in the first group 80-1 and the second group 80-2 have the same sum (e.g., have the same sum '1'). In a case where the sum of the weights included in each of the groups 80-1, 80-2,..., and 80-N is not consistent, second data obtained by convolution operation with the plurality of groups 80-1, 80-2,..., and 80-N can include a checkerboard effect.

[0092] The electronic device 100 can obtain second data by performing a convolution operation on the plurality of groups 80-1, 80-2,..., and 80-N in the spatial direction and the first data. The convolution operation performed between the first data and the plurality of groups 80-1, 80-2,..., and 80-N can be referred to as a depthwise convolution. In an embodiment, the electronic device 100 can perform a convolution operation on the first data with only the first group 80-1 in the spatial direction, not in the channel direction. As Figure 5 indicated, the second kernel 60 is decomposed into N groups, and thus the electronic device 100 can obtain second data by performing a convolution operation on the N groups in the spatial direction and the first data.

[0093] The electronic device 100 can obtain an output image without a checkerboard effect having a size greater than that of the input image by rearranging the obtained second data. In addition, the electronic device 100 can display the output image on the display 130.

[0094] As Figure 4 and Figure 5 indicated, in a case in which the electronic device 100 performs a convolution on the intermediate feature data with the first kernel in the channel direction and with the second kernel in the spatial direction, the amount of processing can be significantly reduced compared to a case in which the existing deconvolution operation is performed on the intermediate feature data at one time.

[0095] The ratio of the reduction in the amount of processing can be confirmed in detail through the following mathematical expression (1). In the mathematical expression (1), the expression in the denominator is used to calculate the amount of processing when the deconvolution operation is performed on the intermediate feature data at one time, and the expression in the numerator is used to calculate the amount of processing when the convolution operation is performed with the first kernel and the second kernel.

[0096] [Mathematical Expression 1]

[0097]

[0098] In a case in which the channel parameter d of the intermediate feature data is 64, the width parameter of the first kernel is 3, and the height and width parameters of each decomposed group of the second kernel are 3, the value 0.349 is derived when each value is substituted in the expression (1). That is, when the output image is output by performing the convolution operation according to the embodiment of the disclosure, the amount of processing can be reduced by about 65% compared to a case in which the existing deconvolution operation is performed.

[0099] Figure 6 is a view for describing a process of decomposing the weight included in the second kernel into a plurality of groups according to the embodiment of the disclosure. That is, Figure 6is a view for describing a process of determining the number of groups and the number of weights included in the groups based on the parameter value (or size) of the second kernel and the size of the stride applied to the convolution operation by the electronic device 100.

[0100] In Figure 6 , a method for decomposing the weights included in the second kernel 610 into a plurality of groups in the case where the size (tap) of the second kernel 610 is 11x11 and the size of the stride is 4 will be described. Figure 6 The coordinates 630 shown are coordinates representing the second data, in which the horizontal coordinate w represents the position of a pixel included in the second data in the horizontal direction, and the vertical coordinate h represents the position of a pixel included in the second data in the vertical direction.

[0101] Assuming that the second kernel 610 according to the embodiment is represented as a two-dimensional matrix (11x11 matrix), the index shown in the weights 622 shown in the upper part of the coordinates 630 represents the horizontal position j of the weight in the second kernel 610. In addition, the index shown in the weights 621 shown in the left part of the coordinates represents the vertical position i of the weight in the kernel.

[0102] Further, by considering the size of the stride (for example, an interval of four pixels) and the positions of the pixels included in the second data, the weights 621 and 622 shown in the upper part and the left part of the coordinates are shown to correspond to the positions of the pixels to which the weights are applied.

[0103] For example, regarding the weights applied to the first pixel 631 included in the second data, the horizontal position j is 1, 5, and 9, and the vertical position i is 1, 5, and 9. When the horizontal position and the vertical position of the weights are combined, the weights applied to the first pixel 631 are W 1,1 (611), W 1,5 (615), W 1,9 (619), W 5,1 (651), W 5,5 (655), W 5,9 (659), W 9,1 (691), W 9,5 (695), and W 9,9 (699).

[0104] In addition, regarding the weights applied to the second pixel 632 included in the second data, the horizontal position j is 3 and 7, and the vertical position i is 3 and 7. When the horizontal position and the vertical position of the weights are combined, the weights applied to the second pixel 632 are W 3,3 , W 3,7 , W 7,3 , and W 7,7 .

[0105] Furthermore, regarding the weights applied to the third pixel 633 included in the second data, the horizontal position j is 0, 4, and 8, and the vertical position i is 0, 4, and 8. When the horizontal and vertical positions of the weights are combined, the weights applied to the third pixel 633 are W included in the second kernel 610. 0,0 W 0,4 W 0,8 W 4,0 W 4,4 W 4,8 W 8,0 W 8,4 and W 8,8 .

[0106] In other words, the electronic device 100 can decompose the weights applied to each pixel included in the second data into multiple groups. In an embodiment, the electronic device 100 can use the group of nine weights applied to the first pixel 631 as the first group, and the first group can be represented as matrix A. 0,0 ,like Figure 6 As shown. Additionally, the electronic device 100 can use a group of four weights applied to the second pixel 632 as a second group, and this second group can be represented as matrix A. 2,2 The electronic device 100 can use a group of nine weights applied to the third pixel 633 as a third group, and the third group can be represented as A. 3,3 .

[0107] Including Figure 6 In the weights shown in the second kernel 610, weights indicated by the same color (or pattern) can represent weights included in the same group (applied to the same pixel).

[0108] When representing the weights grouped together using a matrix, the size of the matrix (size(A)) i，j )) can be represented by the mathematical expression 2 shown below.

[0109] [Mathematical Expression 2]

[0110] Size(A (i，j) )=[M, N]=[floor(((tap-1)-(c+i)) / s)+floor((c+i) / s)+1, floor(((tap-1)-(c+i)) / s)+floor((c+i) / s)+1]

[0111] In mathematical expression 2, floor represents rounding down, s represents the step size, and c can be represented by mathematical expression 3 as shown below.

[0112] [Mathematical Expression 3]

[0113]

[0114] Referring to mathematical expressions 2 and 3, the number of groups is determined based on the size of the kernel (tap) and the size of the stride (s), and the number of weights included in each of the groups can also be determined based on the size of the kernel (tap) and the size of the stride (s).

[0115] In addition, the index of the element included in the matrix A can be represented by mathematical expression 4 shown below.

[0116] [mathematical expression 4]

[0117]

[0118] In mathematical expression 4, t M，i can be represented by mathematical expression 5 shown below, and t N，j can be represented by mathematical expression 6.

[0119] [mathematical expression 5]

[0120] t M，i = (t+1) % s + (M-1)xs

[0121] [mathematical expression 6]

[0122] t N，j = (t+1) % s + (M-1)xs

[0123] In mathematical expressions 5 and 6, % denotes a remainder. For example, (t+1) % s denotes a remainder obtained by dividing (t+1) by s.

[0124] For example, in the case where the size of the kernel (tap) is 11 and the size of the stride (s) is 4, when the calculation is performed by applying these to mathematical expressions 1 to 5, the matrix A 0，0 has a size of 3x3 (M=3, N=3), and the index of the first element of the matrix A 0，0 is W 9，9 .

[0125] Regarding each matrix, the electronic device 100 according to an embodiment can normalize the sum of the element values (weight values) included in each matrix. In an embodiment, the electronic device 100 can adjust the weight values to make the sum of the weights included in each matrix consistent (e.g., make the sum "1").

[0126] Figure 7 is a view illustrating a chessboard effect image and a chessboard effect-free image generated according to an embodiment of the disclosure. As Figure 7As illustrated, the electronic device 100 can obtain intermediate feature data by inputting the input image 710 to the CNN, and obtain second data by performing convolution on the intermediate feature data with first kernels in a channel direction and performing convolution on the result value of the performance with second kernels in a spatial direction. The electronic device 100 can obtain an output image by rearranging the second data. In a case where normalization is not performed on the first kernels and a reliability map is not applied to the second kernels and normalization thereof is not performed, the electronic device 100 can obtain an output image 720 that produces a checkerboard effect. However, in a case where normalization is performed on the first kernels and a reliability map is applied to the second kernels and normalization thereof is performed, the electronic device 100 can obtain a checkerboard effect-free output image 730.

[0127] Figure 8 FIG. 10 is a flowchart for describing a method for controlling an electronic device 100 according to an embodiment of the disclosure.

[0128] First, the electronic device 100 can perform a convolution operation on an input image and obtain intermediate feature data related to the image (S810). Specifically, the electronic device 100 can extract features by inputting the input image to a CNN, and obtain intermediate feature data based on the extracted features. Obtaining intermediate feature data by inputting an input image to a CNN is a well-known technology, and thus will be omitted.

[0129] The electronic device 100 can obtain first data by performing a convolution operation on the intermediate feature data with first kernels in a channel direction, and obtain second data by performing a convolution operation on the obtained first data with second kernels in a spatial direction (S820). The channel parameters of the first kernels in the channel direction can be the same as the channel parameters of the intermediate feature data. One of the height and width of each of the first kernels can have a parameter of 1, and the other can have a parameter of a predetermined integer value other than 1.

[0130] The electronic device 100 can set one or more weight values included in the first and second kernels based on the obtained second data (S830). According to an embodiment of the disclosure, the electronic device 100 can set the weight values included in the first and second kernels using a learning algorithm including error backpropagation or gradient descent.

[0131] In addition, the electronic device 100 can compare and analyze the obtained output image and the enlarged input image, and set weight values applied to each kernel of the convolution based on the analysis result.

[0132] The electronic device 100 can adjust the values of the set weights based on the positions of the weights (S840). According to an embodiment of the disclosure, the electronic device 100 can perform normalization to make the sum of the weights included in each first kernel consistent. In addition, the electronic device 100 can apply (e.g., multiply) a reliability map to the second kernel so that the values of the weights included in the second kernel do not change rapidly. The electronic device 100 can decompose the weights included in the second kernel into a plurality of groups based on the positions of the weights and perform normalization to make the sum of the weights included in each of the plurality of groups consistent.

[0133] As described above, according to an embodiment of the disclosure, the electronic device can prevent the generation of a checkerboard effect by performing a convolution operation on data related to an image using a plurality of kernels, generate a high-quality image when adjusting the size of the image, and reduce the amount of processing and the size of a memory.

[0134] In the disclosure, the term "unit" or "module" can include a unit implemented with hardware, software, or firmware, and can be interchangeably used with, for example, a term, such as logic, logical block, component, or circuit. The unit or module can be a minimum unit of a component that is integrally formed or a minimum unit of a component that performs one or more functions. For example, a module can be implemented as an application-specific integrated circuit (ASIC).

[0135] Various embodiments of the disclosure can be implemented as software including instructions stored in a machine (e.g., computer)-readable storage medium. The machine here is a device that invokes the instructions stored in the storage medium and operates according to the invoked instructions, and can include an electronic device (e.g., the electronic device 100) according to the disclosed embodiments. In the case where a processor executes the instructions, the processor can directly perform a function corresponding to the instructions or perform a function corresponding to the instructions using other elements under the control of the processor. The instructions can include code generated by a compiler or executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is tangible and does not include a signal, and it does not distinguish whether data is semi-permanently or temporarily stored in the storage medium. For example, the "non-transitory storage medium" can include a buffer that temporarily stores data.

[0136] In an embodiment, a method according to various embodiments of the disclosure can be provided to be included in a computer program product. The computer program product can be exchanged between a seller and a buyer as a commercial product. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or online through an application store (e.g., Play StoreTM). In the case of online distribution, at least a portion of the computer program product (e.g., an application) can be temporarily stored or temporarily generated in a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0137] Each element (e.g., a module or a program) according to various embodiments can consist of a single entity or a plurality of entities, and some of the above-described sub-elements can be omitted or other sub-elements can be further included in various embodiments. Alternatively or additionally, some elements (e.g., a module or a program) can be integrated into one entity to perform the same or similar functions performed by each separate element before the integration. According to various embodiments, the operations performed by a module, a program, or other elements can be executed sequentially, in parallel, repeatedly, or heuristically, or different orders can be performed, at least some operations can be omitted, or different operations can be added.

Claims

1. An electronic device comprising: Memory for storing at least one instruction; as well as The processor is configured to execute the at least one instruction. The processor is configured to execute the at least one instruction to perform the following operations: Perform a first convolution operation on the input image and obtain intermediate feature data as the result of the first convolution operation. First data is obtained by performing a second convolution operation on intermediate feature data using multiple first kernels along the channel direction, wherein the multiple first kernels include first weights. In this configuration, one of the height and width of the plurality of first kernels has a first parameter, and the other of the height and width has a second parameter, wherein the first parameter is 1, and the second parameter is a predetermined integer value other than 1. Second data is obtained by performing a third convolution operation on the first data using a second kernel in the spatial direction, wherein the second kernel includes second weights. Based on the second data, set a first value for the first weight or a second value for the second weight. Adjust the first value of the first weight so that the same sum is included in each of the plurality of first cores. Based on the reliability graph, which includes a weighting function that gradually changes from the center to the periphery, the second value of the second weight is adjusted. The second weight of the second kernel is decomposed into multiple groups, and Based on the position of the second weight in the second kernel, each of the plurality of groups is normalized.

2. The electronic device according to claim 1, wherein, The processor is also configured to adjust a second value of the second weight by applying a reliability graph, including a weighting function, to the second core.

3. The electronic device according to claim 1, wherein, The processor is also configured to identify the number of the plurality of groups and the number of weights included in each of the plurality of groups based on the parameter values ​​of the second core and the stride size of the third convolution operation.

4. The electronic device according to claim 1, wherein, The processor is also configured to adjust a second value of the second weight for a first group of the plurality of groups so that the sum of the second weights included in the first group of the plurality of groups is consistent.

5. The electronic device according to claim 1, wherein, The processor is also configured to perform the following operations: The second data is obtained by performing a third convolution operation on the first data using the multiple groups. The output image is obtained by rearranging the second data.

6. The electronic device according to claim 5, further comprising: monitor, The processor is further configured to control the display to show an output image, wherein the output image has a first size that is larger than the second size of the input image.

7. A method for controlling an electronic device, the method comprising: Perform a first convolution operation on the input image and obtain intermediate feature data as the result of the first convolution operation; First data is obtained by performing a second convolution operation on intermediate feature data using multiple first kernels in the channel direction, wherein the multiple first kernels include first weights; In this configuration, one of the height and width of the plurality of first kernels has a first parameter, and the other of the height and width has a second parameter, wherein the first parameter is 1, and the second parameter is a predetermined integer value other than 1. Second data is obtained by performing a third convolution operation on the first data using a second kernel in the spatial direction, wherein the second kernel includes second weights; Based on the second data, set the first value of the first weight or the second value of the second weight; The first value of the first weight is adjusted so that the same sum is included in each of the plurality of first cores; Based on the reliability graph, which includes a weighting function that gradually changes from the center to the periphery, the second value of the second weight is adjusted. Decompose the second weight of the second kernel into multiple groups; and Based on the position of the second weight in the second kernel, each of the plurality of groups is normalized.

8. The method according to claim 7, wherein, The step of adjusting the second value of the second weight also includes: applying a reliability graph including the weighting function to the second kernel to adjust the second value of the second weight.

Images