Electronic device and control method

Abstract

This electronic device comprises: a memory for storing at least one instruction; and a processor connected to the memory so as to control the electronic device, wherein the processor, by executing the at least one instruction, acquires a plurality of images including an object, acquires first feature information of the object in a spatial domain on the basis of the plurality of acquired images, transforms the acquired first feature information into second feature information of a frequency domain, acquires a plurality of pieces of second feature information by duplicating the second feature information, and acquires a plurality of pieces of third feature information having different resolutions by applying a filter to each of the plurality of pieces of second feature information.

Description

Electronic devices and control methods

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an operation of an electronic device and a control method thereof for solving an aliasing problem.

[0002] 3D rendering is an image processing method that creates and provides an image at a specific point in time based on modeled virtual object information.

[0003] In the 3D rendering process, if the resolution of the signal or image is not high, there is an aliasing problem, which is when distortion or incorrect information that does not exist in the original signal or image appears.

[0004] To address aliasing, methods such as supersampling and pre-filtering have existed. However, supersampling suffers from the problem of long training times, and pre-filtering, due to its reliance on multi-layer perception (MLP), also suffers from the problem of long training times.

[0005] Therefore, as a solution to the aliasing problem and the problem of long learning time, the present disclosure requests a method to reduce the aliasing problem by adding a learning process in the frequency domain before implementing 3D rendering.

[0006] An electronic device according to one embodiment of the present disclosure may acquire a plurality of images including an object, acquire first feature information of the object in a spatial domain based on the acquired plurality of images, convert the acquired first feature information into second feature information in a frequency domain, duplicate the second feature information to acquire a plurality of second feature information, and apply a filter to each of the plurality of second feature information to acquire a plurality of third feature information having different resolutions.

[0007] The above processor can obtain second feature information transformed into a frequency domain using at least one of a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a fast Fourier transform (FFT).

[0008] The processor can obtain frequency band-specific feature information by performing a discrete wavelet transform (DWT) on the obtained first feature information, and can obtain second feature information converted into a frequency domain by applying a discrete cosine transform (DCT) or a fast Fourier transform (FFT) to the obtained frequency band-specific feature information.

[0009] The processor can obtain the plurality of third feature information by applying LPF to the second feature information converted into the frequency domain, if the processor can obtain a filtering coefficient based on the second feature information converted into the frequency domain.

[0010] The processor, when it is not possible to obtain a filtering coefficient based on the second feature information converted into the frequency domain, may obtain feature information decomposed by resolution by performing a Discrete Wavelet Transform (DWT) on the first feature information, obtain second feature information converted into the frequency domain data by performing a Discrete Cosine Transform (DCT) on the feature information decomposed by resolution, obtain filtering coefficients based on the obtained second feature information, and obtain the plurality of third feature information by applying a LPF.

[0011] The above processor can filter the replicated second feature information with a LPF according to a conversion method for converting into a frequency domain.

[0012] The above processor may repeat the process of filtering the replicated second feature information multiple times for each resolution when filtering the replicated second feature information with an LPF.

[0013] The processor can obtain a plurality of fourth feature information based on the plurality of third feature information, activation function, and mask value, and convert the obtained plurality of fourth feature information into a spatial domain to obtain a plurality of images with different resolutions.

[0014] A method for controlling an electronic device according to one embodiment of the present disclosure comprises: a step of obtaining first characteristic information of the object in a spatial domain based on the obtained plurality of images; a step of converting the obtained first information into second characteristic information in a frequency domain; a step of obtaining a plurality of pieces of second characteristic information by replicating the second characteristic information; and a step of obtaining a plurality of pieces of third characteristic information having different resolutions by applying a filter to each of the plurality of pieces of second characteristic information.

[0015] It may include a step of obtaining second feature information transformed into a frequency domain using at least one of a Discrete Wavelet Transform (DWT), a Discrete Cosine Transform (DCT), or a Fast Fourier Transformation (FFT).

[0016] The method may include a step of obtaining feature information for each frequency band by performing a Discrete Wavelet Transform (DWT) on the obtained first feature information; and a step of obtaining second feature information converted into a frequency domain by applying a Discrete Cosine Transform (DCT) or Fast Fourier Transformation (FFT) to the obtained feature information for each frequency band.

[0017] In a case where a filtering coefficient can be obtained based on the second feature information converted into the frequency domain, a step of obtaining the plurality of third feature information by applying a LPF to the second feature information converted into the frequency domain may be included.

[0018] If it is not possible to obtain a filtering coefficient based on the second feature information converted into the frequency domain, the method may include: a step of obtaining feature information decomposed by resolution by performing a Discrete Wavelet Transform (DWT) on the first feature information; a step of obtaining second feature information converted into the frequency domain data by performing a Discrete Cosine Transform (DCT) on the feature information decomposed by resolution; and a step of obtaining filtering coefficients based on the obtained second feature information and applying a LPF to obtain the plurality of third feature information.

[0019] It may include a step of filtering the replicated second feature information with a LPF according to a conversion method for converting into a frequency domain.

[0020] When filtering the above-described replicated second feature information with LPF, the method may include a step of repeating the process of filtering the replicated feature information multiple times for each resolution.

[0021] The method may include: a step of obtaining a plurality of fourth feature information based on the plurality of third feature information, an activation function, and a mask value; and a step of converting the obtained plurality of fourth feature information into a spatial domain to obtain a plurality of images with different resolutions.

[0022]

[0023] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0024] FIG. 2 is a drawing for explaining a process of obtaining characteristic information of an object according to one embodiment of the present disclosure.

[0025] FIG. 3 is a diagram for explaining a process of converting feature information of an object acquired in a spatial domain into a frequency domain according to one embodiment of the present disclosure.

[0026] FIG. 4 is a diagram for explaining a process of replicating feature information converted into a frequency domain according to one embodiment of the present disclosure.

[0027] FIGS. 5 to 7 are drawings for explaining a process of applying a LPF (Low Pass Filter) to a plurality of replicated feature information according to one embodiment of the present disclosure.

[0028] FIG. 8 is a flowchart illustrating a method for acquiring multiple images with different resolutions based on multiple third feature information according to one embodiment of the present disclosure.

[0029] FIG. 9 is a drawing for explaining a method for obtaining multiple images with different resolutions of multiple third feature information according to one embodiment of the present disclosure.

[0030] FIG. 10 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0031] FIG. 11 is a flowchart for explaining a method for controlling an electronic device according to one embodiment of the present disclosure.

[0032]

[0033] Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. It should be understood that the contents described herein are not intended to limit the scope of the present invention to specific embodiments, but rather include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, the same or similar reference numerals may be used for similar components.

[0034] Additionally, the terms "first," "second," and the like used herein are used to distinguish various components from each other, regardless of order or importance. Therefore, these terms do not limit the order or importance of the components. For example, the first component could be renamed the second component, and similarly, the second component could be renamed the first component, without departing from the scope of the rights set forth in this document.

[0035] Additionally, when it is stated herein that one component (e.g., a first component) is operatively or communicatively coupled or connected to another component (e.g., a second component), it should be understood that this includes all cases where the components are directly connected or indirectly connected through another component (e.g., a third component). Conversely, when it is stated that a component (e.g., a first component) is "directly coupled" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between the component and the other component.

[0036] The terms used in this disclosure are used to describe certain embodiments and may not be intended to limit the scope of other embodiments. In addition, although singular expressions may be used in this disclosure for convenience of explanation, this may be interpreted to include plural expressions unless the context clearly indicates otherwise. In addition, the terms used in this disclosure may have the same meaning as generally understood by a person of ordinary skill in the relevant technical field. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted as having the same or similar meaning in the context of the related technology, and shall not be interpreted in an idealized or overly formal meaning unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude the embodiments of this disclosure.

[0037] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0038] FIG. 1 is a block diagram illustrating a configuration of an electronic device (100) according to at least one embodiment of the present disclosure. The electronic device (100) includes a memory (110) and a processor (120). Meanwhile, the configuration of the electronic device (100) illustrated in FIG. 1 is merely an example, and it is to be understood that some configurations may be added depending on the type of the electronic device (100). The electronic device (100) according to one embodiment of the present disclosure may be implemented as a server, but this is merely an example, and may be implemented as a server in conjunction with other electronic devices (e.g., a television, a digital video disk (DVD) player, a media box (e.g., Samsung HomeSync). TM , Apple TV™, or Google TV™), game consoles (e.g., Xbox™, PlayStation™), camcorders, VR headsets, AR headsets, 3D scanners, smart glasses, or electronic picture frames, etc.

[0039] The memory (110) can store an operating system (OS) for controlling the overall operation of components of the electronic device (100) and instructions or data related to components of the electronic device (100).

[0040] The memory (110) can store multiple images. The memory (110) can store multiple images containing an object to obtain characteristic information about the object. Although not illustrated in FIG. 1, the electronic device (100) may include a camera, and may store images acquired by the camera in the memory (110).

[0041] Meanwhile, for the convenience of explanation, the feature information of the object acquired in the spatial domain is defined as first feature information (220), the feature information obtained by converting the first feature information (220) into the frequency domain is defined as second feature information (310), the feature information obtained by applying a filter to multiple replicated second feature information (310-1, 310-2 … 310-n) is defined as third feature information (510-1, 510-2 … 510-n), the feature information obtained by applying an activation function and mask value (910-1 … 910-n) to the third feature information (510-1, 510-2 … 510-n) is defined as fourth feature information (920-1 … 910-n), and the feature information obtained by converting the fourth feature information (920-1 … 910-n) back into the spatial domain is defined as multiple images (930-1 … 930-n).

[0042] The memory (110) can store data required to process third feature information (510-1, 510-2 ... 510-n) of an object having a different resolution in the frequency domain when the first feature information (220) of the object is converted from the spatial domain to the frequency domain. The memory (110) can store data for an image acquisition module (121), an image feature information acquisition module (122), a frequency domain conversion module (123), a second feature information duplication module (124), a filtering module (125), a mask application module (126), an activation function application module (127), and a plurality of image information acquisition modules (128) to acquire the third feature information (510-1, 510-2 ... 510-n) of the object. When an event for acquiring the third feature information (510-1, 510-2…510-n) of an object occurs, the processor (120) can load the image acquisition module (121), the image feature information acquisition module (122), the frequency domain conversion module (123), the second feature information duplication module (124), the filtering module (125), the mask application module (126), the activation function application module (127), and the plurality of image information acquisition modules (128) stored in the memory (110), as illustrated in FIG. 1.

[0043] Meanwhile, the memory (110) may include nonvolatile memory capable of maintaining stored information even when power is cut off, and volatile memory requiring continuous power supply to maintain the stored information. Data for performing various operations, such as a module for acquiring multiple image data containing an object or first characteristic information of an object, and a module for converting characteristic information of an object from a spatial domain to a frequency domain, may be stored in the nonvolatile memory.

[0044] The memory (110) may be implemented in various forms, such as volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0045] The processor (120) is electrically connected to the memory (110) and can control the overall functions and operations of the electronic device (100).

[0046] The processor (120) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The processor (120) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0047] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0048] In particular, the processor (120) can acquire a plurality of images including an object by executing at least one instruction, acquire first feature information (220) of the object in a spatial domain based on the acquired plurality of images, convert the acquired first feature information (220) into second feature information (310) in a frequency domain, duplicate the second feature information (310) to acquire a plurality of second feature information (310-1, 310-2 ... 310-n), and apply a filter to each of the plurality of second feature information (310-1, 310-2 ... 310-n) to acquire a plurality of third feature information (510-1 ... 510-n) having different resolutions.

[0049] Hereinafter, the operation of the electronic device (100) controlled by the processor (120) will be described in more detail with reference to FIGS. 2 to 11.

[0050] First, the image acquisition module (121) can acquire multiple images containing objects. The image acquisition module (121) can acquire multiple images containing objects via a camera or can acquire multiple images containing objects from an external device. In this case, each of the multiple images can be an image containing one or more objects.

[0051] The object feature information acquisition module (122) can acquire first feature information (220) of an object as illustrated in FIG. 2.

[0052] The first feature information (220) may be information representing geometric information of an object (e.g., vertex, edge, face, color of the object, scale of the object (ratio of size of the object for each axis). Here, the feature information may be called a feature map, feature value, feature matrix, characteristic distribution, etc.

[0053] The object feature information acquisition module (122) can acquire first feature information (220) of an object by combining feature information (210-1, 210-2, 210-3) on each axis of the spatial domain.

[0054] For convenience of explanation, the first characteristic information (220) of the object in FIG. 2 is depicted as a rectangle, but the first characteristic information (220) is a vector matrix structure (i.e., a matrix structure having vectors as elements).

[0055] The object feature information acquisition module (122) can acquire the first feature information (220) of the object using a vector matrix decomposition method.

[0056] Vector matrix decomposition is a method used in data compression, which decomposes the original matrix into several elements to make data processing easier. There are methods for decomposing vectors and matrices, such as singular value decomposition (SVD), eigen decomposition, and QR decomposition. Singular value decomposition (SVD) is a method for decomposing an arbitrary m * n matrix A into the product of three matrices. Eigen decomposition is a method for decomposing a matrix into eigenvalues ​​and eigenvectors, and is mainly used for symmetric matrices. QR decomposition is a method for decomposing a given matrix into two matrices Q and R. Q is an orthogonal matrix, whose column vectors are orthogonal to each other and whose size is 1. R is an upper triangular matrix, in which all elements below the main diagonal are 0. QR decomposition can use the Gram-Schmidt regularization method or the Householder transform method. The Gram-Schmidt normalization method is a method to obtain Q and R by orthogonalizing and normalizing the column vectors of a given matrix. The Householder transformation is a method to decompose a matrix into an orthogonal matrix and an upper triangular matrix by reflecting the given vectors in the direction of a new vector.

[0057] As an embodiment of acquiring first feature information (220), the object feature information acquisition module (122) can acquire first feature information (220) of an object in a spatial domain in a grid manner. When acquiring first feature information (220) in a grid manner, data can be managed by dividing a three-dimensional space into uniform grids. In other words, data can be stored and queried by dividing the space into grids of uniform size, and each grid cell can have a fixed size. The resolution of the grid is already defined and can be set to a fixed size.

[0058] In another embodiment of obtaining the first feature information (220), the object feature information obtaining module (122) can obtain the first feature information (220) using another method such as the Octree method, the Tri method (Triangular Mesh), Implicit Neural Representations, or Point-based Methods. The Octree method is a tree-structured data structure that efficiently divides and represents 3D space. The object feature information obtaining module (122) can obtain the first feature information (220) by repeatedly dividing the 3D space into eight volumes of equal size using the Octree method. The Tri method (Triangular Mesh) is a method of expressing the 3D space as a mesh composed of triangles. A 3D surface can be represented using points and triangles. Each triangle is defined by a connection of three points, and these can be used to directly represent the space. The object feature information obtaining module (122) can obtain the first feature information (220) using the Tri method (Triangular Mesh).

[0059] The frequency domain conversion module (123) can convert the feature information (220) of an object in the spatial domain into feature information (310) of an object in the frequency domain. This will be described later with reference to FIG. 3.

[0060] The frequency domain transformation module (123) can transform the first feature information (220) into the second feature information (310) by using at least one of the following transformation methods: Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), or Fast Fourier Transformation (FFT), as illustrated in FIG. 3.

[0061] The frequency domain transformation module (123) can transform the first feature information (220) obtained in the spatial domain through the Discrete Wavelet Transform (DWT) method to obtain feature information for each frequency band, and then apply Discrete Cosine Transform (DCT) or Fast Fourier Transformation (FFT) to the obtained feature information for each frequency band to convert it into second feature information.

[0062] In one embodiment, the frequency domain conversion module (123) can apply a DWT conversion method (Discrete Wavelet Transform) to the first feature information (220) to divide it into a high-frequency filter and a low-frequency filter, and convert it into the frequency domain. A signal passing through the low-frequency filter can be decomposed into rough information, and a signal passing through the high-frequency filter can be decomposed into detailed information. The frequency domain conversion module (123) can divide the signal passing through the low-frequency filter into two parts again using a high-frequency filter and a low-frequency filter, and repeatedly decompose the low-frequency components to analyze the rough structure of the signal in increasingly more detail.

[0063] In another embodiment, the frequency domain transformation module (123) can transform into the frequency domain using the DCT transformation method (Discrete Cosine Transform). When the first feature information (220) is transformed into the second feature information (310) using the DCT transformation method (Discrete Cosine Transform), energy is concentrated on low-frequency components, so data can be compressed. When image data is input, the image can generally be expressed as a matrix of pixel values. The image can usually be divided into blocks of 8x8 or 16x16 size and processed. The frequency domain transformation module (123) can perform DCT transformation on each block, and among the frequency components obtained as a result of the DCT transformation, the low-frequency components mainly represent rough information of the block, and the high-frequency components can represent detailed information of the block. The frequency components obtained as a result of the DCT transformation can be quantized using an appropriate quantization table to perform further compression.

[0064] In another embodiment, the frequency domain transformation module (123) can transform the first feature information (220) into the frequency domain using the FFT transformation method (Fast Fourier Transform). The FFT transformation method (Fast Fourier Transform) is a type of divide-and-conquer algorithm that can recursively perform calculations while reducing the signal size by half. The frequency domain transformation module (123) can obtain the magnitude and phase information of the signal transformed into the frequency domain using the FFT transformation method on the first feature information (220), i.e., the second feature information (310).

[0065] In another embodiment, the frequency domain conversion module (123) can convert the first feature information (220) into a frequency domain using a DWT conversion method and a DCT conversion method. The frequency domain conversion module (123) can apply the DWT conversion method to the first feature information (220) to extract multiple resolution components, and apply the DCT conversion method to some of the extracted components to convert them into a frequency domain.

[0066] Specifically, the frequency domain transformation module (123) can extract low-frequency components and high-frequency components (i.e., low-resolution data and multiple high-resolution detailed components) by using the Discrete Wavelet Transform (DWT) transformation method on the first feature information (220). The frequency domain transformation module (123) can obtain data of each resolution level converted into the frequency domain by applying the Discrete Wavelet Transform (DWT) transformation method to the data of each resolution level obtained by the Discrete Wavelet Transform (DWT) transformation method. The frequency domain transformation module (123) can maintain data to be preserved at each resolution level by using the Discrete Wavelet Transform (DWT) transformation method and the Discrete Cosine Transform (DCT) transformation method.

[0067] In another embodiment, the frequency domain transformation module (123) may obtain high-frequency components and low-frequency components at each resolution level using the Discrete Wavelet Transform (DWT) transformation method for the first feature information (220), and apply FFT (Fast Fourier Transformation) to each of the obtained low-frequency components and high-frequency components to transform them into the frequency domain.

[0068] In another embodiment, the frequency domain conversion module (123) may convert the first feature information (220) into a frequency domain using an FFT conversion method, and then additionally use a DCT conversion method for the main frequency information.

[0069] The case where the frequency domain transformation module (123) transforms the second feature information (310) into the frequency domain using the transformation methods of Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), and Fast Fourier Transformation (FFT) has been described above. However, this is only an example, and the frequency domain transformation module (123) can transform the first feature information (220) into the second feature information (310) using the transformation methods of Short-Time Fourier Transform (STFT), Continuous Wavelet Transform (CWT), and Fractional Fourier Transform (FrFT).

[0070] The second feature information replication module (124) can replicate the second feature information (310) into a plurality of second feature information (310-1, 310-2…310-n), as illustrated in FIG. 4.

[0071] The filtering module (125) can filter a plurality of second feature information (310-1, 310-2…310-n) to obtain a plurality of third feature information (510-1…510-n). This will be described with reference to FIG. 5.

[0072] The filtering module (125) can apply a Scale 0 LPF (520-1) to the feature information (310-1) to obtain the feature information (510-1) with the highest resolution. Since the Scale 0 LPF (520-1) is applied to the highest resolution scale of the data, the finest details of the data can be preserved. The filtering module (125) can obtain the highest resolution feature information (510-1) containing fine and subtle details by applying the Scale 0 LPF (520-1) to the feature information (310-1).

[0073] In contrast, the filtering module (125) can apply a Scale 3 LPF (520-n) to the feature information (310-n) to obtain the feature information (510-n) with the lowest resolution. Since the Scale 3 LPF (520-n) is applied to the lowest resolution scale of the data, the overall structure of the data can be preserved. The filtering module (125) can obtain the feature information (510-n) with the lowest resolution by applying the Scale 3 LPF (520-n) to the feature information (310-n).

[0074] Meanwhile, the filters applied to each of the plurality of second characteristic information (310-1, 310-2…310-n) are described above on the assumption that they are different filters, but they may be the same filter.

[0075] The filtering module (125) can perform filtering using the filtering coefficients obtained according to the conversion method used in the frequency domain conversion module (123). The method for obtaining the filtering coefficients will be described below with reference to FIG. 6.

[0076] The filtering module (125) can determine whether a filtering coefficient can be obtained based on the second feature information (310) converted into the frequency domain (S610).

[0077] If the filtering module (125) can obtain filtering coefficients based on the second feature information (310) converted into the frequency domain (S610-Y), a LPF (Low Pass Filter) can be applied to a plurality of second feature information (310-1, 310-2...310-n) (S620). If the filtering module (125) can obtain filtering coefficients, the filtering module (125) can apply the LPF based on the obtained filtering coefficients without applying the Discrete Wavelet Transform method in the process of converting into the frequency domain.

[0078] If the filtering module (125) cannot obtain a filtering coefficient based on the second feature information (310) converted to the frequency domain (S610-N), the filtering coefficient can be obtained by applying DWT to the first feature information (220) and then applying DCT (S630).

[0079] For example, when the first feature information (220) is converted into the frequency domain, low-frequency components and high-frequency components may not be accurately separated, or most of the energy may be concentrated in the low-frequency components, making it difficult to obtain filtering coefficients.

[0080] If the filtering module (125) cannot obtain a filtering coefficient, the filtering module (125) can obtain feature information decomposed by resolution (low-frequency component and high-frequency component feature information by resolution) according to the DWT conversion method, and then apply DCT to a specific resolution to extract a filtering coefficient. The filtering module (125) can apply LPF to a plurality of second feature information (310-1, 310-2 ... 310-n) based on the obtained filtering coefficient (S620).

[0081] The filtering module (125) can filter the replicated second feature information (310-1…310-n) according to a conversion method for converting the first feature information (220) into the frequency domain.

[0082] In one embodiment, the filtering module (125) may filter a plurality of second feature information (310-1…310-n) with a Left-top Gaussian Kernel when the first feature information (220) is converted into the second feature information (310) through a DCT (Discrete Cosine Transform) transformation method. The Left-top Gaussian Kernel method is one of the Gaussian filters, and may be used to adjust the frequency component of the second feature information (310) when the first feature information (220) is converted into the second feature information (310) in the frequency domain through a DCT (Discrete Cosine Transform) transformation method.

[0083] In another embodiment, the filtering module (125) may filter the plurality of second feature information (310-1...310-n) with center Gaussian when the first feature information (220) is converted into the second feature information (310) through the FFT (Fast Fourier Transform) conversion method. When the center Gaussian filter is used, the frequency components around the center may be emphasized in the frequency domain. Since the absolute value of the frequency in the frequency domain is higher as it gets closer to the origin, the frequency components around the origin may be emphasized when the center Gaussian filter is applied.

[0084] In another embodiment, the filtering module (125) may obtain feature information decomposed by resolution using the Discrete Wavelet Transform (DWT) method on the first feature information (220), and may convert the feature information decomposed by resolution into second feature information (310) using the Discrete Cosine Transform (DCT) method. In this case, the filtering module (125) may filter using a Left-top Gaussian Kernel.

[0085] When filtering replicated feature information (310-1…310-n) by resolution, it is necessary to apply a low-pass filter (LPF) based on an accurate standard deviation. The method for determining the standard deviation is described below.

[0086] In one embodiment, the filtering module (125) may perform filtering by presetting the standard deviation to a specific value. However, since it is difficult to calculate an exact standard deviation, the filtering module (125) may set the standard deviation to N / 2n. N is the dimension of the DCT (Discrete Cosine Transform) domain (i.e., the number of samples of the DCT-transformed signal), and n is a reduction factor for reducing the resolution. If the standard deviation is large, a filter including a wide frequency band may be generated, and if the standard deviation is small, a filter including a narrow frequency band may be generated. The coefficients of the LPF may be generated based on the standard deviation. In the case of a Gaussian LPF, the larger the standard deviation, the larger the filter size, and at this time, the coefficients of the filter may give more weight to surrounding values.

[0087] In another embodiment, the filtering module (125) can filter by setting the standard deviation as a learnable parameter. If the standard deviation is a learnable parameter, the size and shape of the filter applied to the plurality of second feature information (310-1, 310-2... 310-n) can be adjusted. The filtering module (125) can set the standard deviation using the gradient descent method by setting the standard deviation as a learnable parameter. The filtering module (125) can define a loss function, calculate the gradient for the standard deviation of the LPF (Low Pass Filter) using the backpropagation algorithm, and then adjust the standard deviation of the LPF (Low Pass Filter) using the calculated gradient to obtain the standard deviation. The filtering module (125) can set a standard deviation that can perform precise filtering on the second feature information (310-1…310-n) by starting from an initially set value and adjusting it to fit the pattern of the second feature information (310-1…310-n).

[0088] In another embodiment, when the filtering module (125) filters the replicated second feature information with a LPF (Low Pass Filter) as illustrated in FIG. 7, the process of filtering the replicated feature information by resolution can be repeated multiple times to set a standard deviation.

[0089] The filtering module (125) can set the standard deviation for each resolution and then update the standard deviation. In Phase 1, the filtering module (125) can set the standard deviation value for each resolution to an arbitrary value (710-1, 710-2…710-n). When the filtering module (125) filters a plurality of second feature information (310-1…310-n) in Phase 1, less smoothing is applied, so that high-resolution detailed information can remain.

[0090] In Phase 2, the filtering module (125) can filter after slightly increasing the standard deviation. The filtering module (125) can filter the plurality of second feature information (310-1…310-n) based on the increased standard deviation (720-1, 720-2, 720-3, 720-4). When the filtering module (125) filters the plurality of second feature information (310-1…310-n) in Phase 2, more high-frequency components can be removed and low-frequency components can be maintained compared to Phase 1.

[0091] By repeating the process of Phase 1 and Phase 2 n times step by step, the filtering module (125) can filter by updating the standard deviation. As a result of repeating up to Phase n, the filtering module (125) can filter using a large standard deviation. In addition, when a LPF (Low Pass Filter) is applied using a large standard deviation to a plurality of second feature information (310-1…310-n), most high-frequency components are removed, and low-resolution information can remain.

[0092] The activation function application module (127) can obtain a plurality of fourth feature information (920-1…920-n) based on a plurality of third feature information (510-1…510-n), and the plurality of image information acquisition modules (128) can obtain a plurality of image information (930-1…930-n) converted into a spatial domain based on a plurality of fourth feature information (920-1…920-n). This will be described later with reference to FIG. 8.

[0093] The activation function application module (127) can obtain a plurality of fourth feature information (920-1 … 920-n) based on a plurality of third feature information (510-1 … 510-n), activation function, and mask values ​​(910-1 … 910-n) (S810).

[0094] In the process of applying the mask values ​​(910-1 … 910-n) to multiple third feature information (510-1 … 510-n), the method of obtaining the mask values ​​(910-1 … 910-n) will be described later with reference to FIG. 9.

[0095] The mask application module (126) can apply a mask value (910-1 ... 910-n) to a plurality of third feature information (510-1 ... 510-n). By applying the mask value (910-1 ... 910-n) to a plurality of third feature information (510-1 ... 510-n), a specific portion of the data can be emphasized or suppressed.

[0096] In one embodiment, the mask application module (126) may obtain continuous mask values ​​(910-1 ... 910-n) and apply the mask values ​​to a plurality of third feature information (510-1 ... 510-n). The mask values ​​(910-1 ... 910-n) may be expressed as continuous values ​​between 0 and 1. When the mask application module (126) applies the mask values ​​(910-1 ... 910-n) to a plurality of third feature information (510-1 ... 510-n), the closer the mask values ​​(910-1 ... 910-n) are to 0, the less emphasized the area to which the mask values ​​(910-1 ... 910-n) are applied may be. The closer the mask value (910-1 … 910-n) is to 1, the more the area to which the mask value (910-1 … 910-n) is applied can be emphasized. For example, when the mask value (910-1 … 910-n) is 0.8, the area to which the mask value (910-1 … 910-n) is applied can be maintained by 80%, and when the mask value (910-1 … 910-n) is 0.2, the area to which the mask value (910-1 … 910-n) is applied can be maintained by only 20%.

[0097] In another embodiment, the mask application module (126) may obtain a binary mask value (910-1 ... 910-n) and apply the mask value to a plurality of third feature information (510-1 ... 510-n). That is, the mask value (910-1 ... 910-n) may be set to 0 or 1. For example, when the mask value (910-1 ... 910-n) is 1, the mask application module (126) may apply the mask value (910-1 ... 910-n) to maintain the area to which the mask value is applied.

[0098] The mask application module (126) can suppress the area to which the mask value is applied by applying the mask value (910-1 ... 910-n) when the mask value (910-1 ... 910-n) is 0.

[0099] The mask application module (126) can apply the mask values ​​(910-1 ... 910-n) based on a specific threshold value when the mask values ​​(910-1 ... 910-n) are binary. The mask application module (126) can apply the mask values ​​by setting pixels larger than the threshold value to 1 and pixels smaller than the threshold value to 0.

[0100] In another embodiment, the mask application module (126) can apply mask values ​​(910-1 ... 910-n) for each resolution based on a plurality of third feature information (510-1 ... 510-n) having different resolutions. The feature information of the lowest resolution includes the overall and general characteristics of the object, and the mask application module (126) can set the mask values ​​(910-1 ... 910-n) applied at the lowest resolution. When the mask values ​​(910-1 ... 910-n) applied at the lowest resolution are set, the mask values ​​(910-1 ... 910-n) applied at the next lower resolution can be set based on the mask values ​​(910-1 ... 910-n) set first. The mask application module (126) can apply mask values ​​(910-1…910-n) to multiple third feature information (510-1…510-n) by hierarchically processing resolutions from low resolution to high resolution.

[0101] The mask application module (126) can apply mask values ​​(910-1…910-n) by considering a low-resolution mask (910-n) and a high-resolution mask (910-1). The low-resolution mask contains less detailed information of the data, and the high-resolution mask can contain detailed information. Since the low-resolution image contains important information, the high-resolution image (930-1) can be learned based on the low-resolution mask (920-n) value. At this time, the mask value can be hierarchically updated at each stage where the resolution increases.

[0102] The activation function application module (127) can apply an activation function to a plurality of third feature information (510-1 ... 510-n) to which mask values ​​(910-1 ... 910-n) are applied. By applying the activation function to the third feature information (510-1 ... 510-n), a plurality of fourth feature information (920-1 ... 920-n) can be obtained.

[0103] The activation function application module (127) can obtain multiple fourth feature information (920-1…920-n) by using the sigmoid function among the activation functions. The sigmoid function nonlinearly transforms the input value and outputs the output, and the resulting value can be a value between 0 and 1. The sigmoid function can be frequently used as an activation function in binary classification problems.

[0104] However, this is only an example, and the activation function application module (127) can obtain the fourth feature information (920-1…920-n) by using the ReLU function, Tanh (Hyperbolic Tangent) function, Softmax function, etc. among the activation functions.

[0105] The spatial domain conversion module (128) can convert a plurality of fourth feature information (920-1 … 920-n) from the frequency domain to the spatial domain to obtain a plurality of image information (930-1 … 930-n) (S820).

[0106] The spatial domain transformation module (128) can transform a plurality of fourth feature information (920-1…920-n) into a spatial domain using an inverse Fourier transform (IFT) method or an inverse discrete cosine transform (IDCT) method.

[0107] The inverse Fourier transform (IFT) is a fundamental method for transforming signals from the frequency domain to the spatial domain, allowing the original signal to be reconstructed from its frequency components. The inverse discrete cosine transform (IDCT) is the reverse process of the discrete cosine transform (DCT), allowing the frequency components to be restored back to the spatial domain of the signal, allowing the original signal to be reconstructed.

[0108] The spatial domain transformation module (128) can acquire multiple low-resolution images and multiple high-resolution images.

[0109] FIG. 10 is a flowchart for explaining the operation of an electronic device (100) according to one embodiment of the present disclosure.

[0110] First, the electronic device (100) can acquire multiple images containing an object (S1010). At this time, the multiple images containing the object can be acquired through a camera or from memory (110). Alternatively, the multiple images containing the object can be acquired from a separate external device.

[0111] The electronic device (100) can acquire first characteristic information of an object in a spatial domain based on the acquired multiple images (S1011). Here, the "object" can refer to any visually expressible entity included in the frame. In other words, the object can be an object such as a desk, a chair, or a telephone, as well as a visually expressible concept.

[0112] The electronic device (100) can convert the acquired feature information (220) into a frequency domain (S1012). At this time, the electronic device (100) can convert the acquired first feature information (220) into a frequency domain according to the method described above in FIG. 3.

[0113] The electronic device (100) can replicate the acquired second characteristic information (310) into a plurality of second characteristic information (310-1 ... 310-n) (S1013). The case where the electronic device (100) replicates the acquired second characteristic information (310) into a plurality of second characteristic information (310-1 ... 310-n) identical to the second characteristic information (310) has been described above, but the plurality of second characteristic information (310-1 ... 310-n) may not be identical.

[0114] The electronic device (100) can obtain a plurality of third feature information (510-1 ... 510-n) having different resolutions by applying a filter to each of the replicated second feature information (310-1 ... 310-n) (S1014). Specifically, the electronic device (100) can obtain third feature information (510-1 ... 510-n) implemented in high resolution from third feature information implemented in low resolution by applying a filter to each of the replicated second feature information (310-1 ... 310-n).

[0115] The electronic device (100) can obtain a plurality of fourth feature information (920-1 ... 920-n) synthesized based on the mask values ​​(910-1 ... 910-n) and the activation function from the acquired plurality of third feature information (510-1 ... 510-n) (S1015). The electronic device (100) can apply the mask values ​​(910-1 ... 910-n) to the acquired plurality of third feature information (510-1 ... 510-n). At this time, the mask values ​​(910-1 ... 910-n) can emphasize or suppress frequency components. The electronic device (100) can obtain a plurality of fourth feature information (920-1 ... 920-n) synthesized by inputting the third feature information (310-1 ... 310-n) to which the weights (1010-1 ... 1010-n) are applied to the activation function.

[0116] The electronic device (100) can convert the acquired plurality of fourth feature information (920-1 ... 920-n) into a spatial domain to acquire a plurality of images (930-1 ... 930-n) (S1016). Specifically, the electronic device (100) can convert the plurality of fourth feature information (920-1 ... 920-n) expressed in a frequency domain into a spatial domain to implement 3D modeling.

[0117] The electronic device (100) can obtain a 3D image with reduced aliasing problems by obtaining multiple images (930-1…930-n) of various resolutions.

[0118] FIG. 11 is a flowchart illustrating a method for controlling an electronic device (100) according to one embodiment of the present disclosure. Meanwhile, any description overlapping with FIG. 10 will be omitted.

[0119] The electronic device (100) can acquire multiple images including an object (S1110). The electronic device (100) can acquire multiple images including an object from a camera or an external device.

[0120] The electronic device (100) can acquire first characteristic information (220) of the object in the spatial domain based on the acquired multiple images (S1120). The first characteristic information (220) of the object includes geometric information of the object, etc., and can be acquired through a vector matrix decomposition method.

[0121] The electronic device (100) can convert the acquired first characteristic information (220) into second characteristic information (310) in the frequency domain (S1130). Since the method for converting into the frequency domain has been described above in FIG. 3, a detailed description thereof will be omitted.

[0122] The electronic device (100) can replicate the second characteristic information (310) into a plurality of second characteristic information (310-1 ... 310-n) (S1140). By replicating the second characteristic information (310) into a plurality of second characteristic information (310-1 ... 310-n), a low pass filter (LPF) can be applied.

[0123] The electronic device (100) can obtain a plurality of third feature information (510-1 ... 510-n) having different resolutions by applying a filter to each of the replicated second feature information (310-1 ... 310-n) (S1150). That is, when applying a LPF (Low Pass Filter) to the plurality of replicated second feature information (310-1 ... 310-n), a method corresponding to a method of converting into a frequency domain can be selected. When applying a LPF (Low Pass Filter), a detailed description will be omitted since the method of obtaining the size of the LPF (Low Pass Filter) has been described above with reference to FIGS. 5 and 6.

[0124] In addition, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user (20) devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0125] The methods according to various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored from the storage medium and operate according to the called commands, and may include a server device or an electronic device according to the disclosed embodiments.

[0126] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory readable recording medium. Here, the term "non-transitory readable recording medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0127] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0128] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, memory that stores at least one instruction; and A processor connected to the memory and controlling the electronic device; The processor, by executing the at least one instruction, Acquire multiple images containing objects, Obtaining first feature information of the object in the spatial domain based on the plurality of images obtained above, Obtaining multiple pieces of second feature information by replicating second feature information in the frequency domain converted based on the first feature information obtained above, An electronic device that obtains a plurality of third characteristic pieces of information having different resolutions by applying a filter to each of the plurality of second characteristic pieces of information.

2. In paragraph 1, The above processor, An electronic device that obtains second feature information transformed into a frequency domain using at least one of a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a fast Fourier transform (FFT).

3. In paragraph 2, The above processor, The first feature information obtained above is converted into Discrete Wavelet Transform (DWT) to obtain feature information for each frequency band, An electronic device that obtains second feature information converted into a frequency domain by applying a discrete cosine transform (DCT) or a fast Fourier transform (FFT) to the feature information for each frequency band obtained above.

4. In paragraph 1, The above processor, If it is possible to obtain a filtering coefficient based on the second feature information converted into the frequency domain, An electronic device that obtains the plurality of third feature information by applying a discrete cosine transform (DCT) to the second feature information converted into the frequency domain.

5. In paragraph 1, The above processor, If the filtering coefficient cannot be obtained based on the second feature information converted to the frequency domain, The first feature information is subjected to a discrete wavelet transform (DWT) to obtain feature information decomposed by resolution, The feature information decomposed by the above resolution is converted into the second feature information and filtering coefficients converted into the frequency domain data by discrete cosine transform (DCT), An electronic device that obtains the plurality of third characteristic pieces of information by applying a LPF to the plurality of second characteristic pieces of information.

6. In paragraph 1, The above processor, An electronic device that filters the replicated second feature information with a LPF according to a conversion method for converting it into a frequency domain.

7. In paragraph 6, The above processor, When filtering the above replicated second feature information with LPF, An electronic device that repeats the process of filtering the plurality of duplicated second feature information multiple times according to resolution.

8. In paragraph 1, The above processor, Obtaining a plurality of fourth feature information based on the plurality of third feature information, activation function and mask value, An electronic device that converts the plurality of fourth feature information acquired above into a spatial domain to acquire a plurality of images with different resolutions.

9. In the method of controlling an electronic device, A step of acquiring a plurality of images including an object; A step of obtaining first feature information of the object in a spatial domain based on the plurality of acquired images; A step of obtaining a plurality of second feature information by replicating second feature information of a frequency domain converted based on the first feature information obtained above; and A control method comprising: a step of obtaining a plurality of third feature information having different resolutions by applying a filter to each of the plurality of second feature information.

10. In paragraph 9, A control method comprising: a step of obtaining second feature information transformed into a frequency domain using at least one of a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a fast Fourier transform (FFT).

11. In paragraph 10, A step of obtaining feature information for each frequency band by performing a discrete wavelet transform (DWT) on the first feature information obtained above; and A control method comprising: a step of obtaining second feature information converted into a frequency domain by applying a discrete cosine transform (DCT) or a fast Fourier transform (FFT) to the feature information for each frequency band obtained above; 12. In paragraph 9, If it is possible to obtain a filtering coefficient based on the second feature information converted into the frequency domain, A control method comprising: a step of obtaining the plurality of third feature information by applying a LPF to the second feature information converted into the frequency domain; 13. In paragraph 9, If the filtering coefficient cannot be obtained based on the second feature information converted to the frequency domain, A step of obtaining feature information decomposed by resolution by performing a discrete wavelet transform (DWT) on the first feature information; A step of obtaining second feature information converted into frequency domain data by converting the feature information decomposed by the above resolution using a discrete cosine transform (DCT); and A control method comprising: a step of obtaining a filtering coefficient based on the obtained second feature information and applying an LPF to obtain the plurality of third feature information.

14. In paragraph 9, A control method comprising a step of filtering the replicated second feature information with a LPF according to a conversion method for converting into a frequency domain.

15. In paragraph 14, When filtering the above replicated second feature information with LPF, A control method comprising: a step of repeating a process of filtering the replicated second feature information by resolution multiple times;

Images