Method and electronic device for correcting distortion of image

Abstract

Provided are a method and an electronic device, the method comprising the steps of: obtaining a first image; generating a projection image in which perspective distortion of the first image is corrected, on the basis of the first image; generating a recognition map which divides the first image into first multiple types of regions according to a plurality of objects included in the first image; generating first mesh information for correcting the first image on the basis of the projection image and the recognition map, wherein the first mesh information includes movement information on a plurality of pixel positions; and generating an output image by correcting the first image on the basis of the first mesh information.

Description

Method and electronic device for correcting distortion of an image

[0001] The present disclosure relates to an image processing method. More specifically, the present disclosure relates to a method and an electronic device for correcting distortion of an image.

[0002] Electronic devices can acquire images from light collected through a lens. Electronic devices can identify the light collected through the lens using an image sensor. Electronic devices can process the electrical signals from the image sensor to generate images.

[0003] During the process of creating an image through a lens, image distortion can occur due to the optical characteristics of the lens. For example, image distortion can include lens distortion and perspective distortion. Lens distortion is related to the refractive action of the lens, and the type of distortion can vary depending on the type of lens. Perspective distortion can include distortion caused by the distance between an object and the camera or the object's position relative to the camera's field of view.

[0004] The above information is provided solely as background information to aid in understanding the present invention. No determination is made, and no assertion is made, as to whether any of the above is applicable as prior art in connection with the disclosure.

[0005] In one embodiment of the present disclosure, a method for correcting distortion of an image by an electronic device is provided. The method includes acquiring a first image. The method may include generating a projection image in which perspective distortion of the first image is corrected based on the first image. The method may include generating a perception map that divides the first image into a plurality of regions of a first plurality of types based on a plurality of objects included in the first image. The method may include generating first mesh information for correcting the first image based on the projection image and the perception map. The first mesh information may include movement information for a plurality of pixel locations. The method may include generating an output image by correcting the first image based on the first mesh information. The movement information may be determined to a different degree depending on the first plurality of types.

[0006] In one embodiment of the present disclosure, an electronic device for correcting image distortion is provided. The electronic device may include at least one processor including a processing circuit, and a memory including one or more storage media storing at least one instruction. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to acquire a first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate a projection image in which perspective distortion of the first image is corrected based on the first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate a perception map that divides the first image into a plurality of regions of a first plurality of types, based on a plurality of objects included in the first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate first mesh information for correcting the first image based on the projection image and the perception map. The first mesh information may include movement information of a plurality of pixel locations. The method can generate an output image by correcting a first image based on first mesh information. The movement information can be determined to different degrees depending on the first plurality of types.

[0007] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing the method on a computer is provided.

[0008] FIG. 1 is a diagram schematically illustrating an electronic device for processing image distortion according to one embodiment of the present disclosure.

[0009] FIG. 2 is a flowchart of a method for correcting image distortion according to one embodiment of the present disclosure.

[0010] FIG. 3 is a drawing for explaining lens distortion according to one embodiment of the present disclosure.

[0011] FIG. 4 is a drawing for explaining perspective distortion according to one embodiment of the present disclosure.

[0012] FIG. 5 is a block diagram illustrating an electronic device for correcting image distortion according to one embodiment of the present disclosure.

[0013] FIG. 6 is a drawing for explaining a projection image according to one embodiment of the present disclosure.

[0014] FIG. 7 is a diagram for explaining map information according to one embodiment of the present disclosure.

[0015] FIG. 8 is a drawing for explaining mesh information according to one embodiment of the present disclosure.

[0016] FIG. 9 is a diagram for explaining a process of correcting an image using mesh information according to one embodiment of the present disclosure.

[0017] FIG. 10 is a diagram for explaining a process of processing an image using inpainting according to one embodiment of the present disclosure.

[0018] FIG. 11 is a drawing for explaining an inpainting technique according to one embodiment of the present disclosure.

[0019] FIG. 12 is a diagram for explaining a process of displaying and storing an image by an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 13 is a diagram for explaining a process of processing continuously acquired images according to one embodiment of the present disclosure.

[0021] FIG. 14 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0022] FIG. 15 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0023] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0024] In the present disclosure, the expression "a and / or b" may refer to "a", "b", "a and b". "a and / or b" may mean "at least one of a or b".

[0025] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings can be understood through the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0026] In this disclosure, singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms containing ordinal numbers, such as "first" or "second," used in this disclosure may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0027] When a part of this disclosure is said to "include" a component, this does not exclude other components, but rather may include other components, unless otherwise specifically stated. In this disclosure, terms such as "part" and "module" refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0028] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of", depending on the context. The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Alternatively, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0029] When a component is referred to as being "connected" or "connected" to another component in this disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0030] Similar parts are designated by similar reference numerals throughout the specification. The dimensions of each component do not entirely reflect the actual size. Identical or corresponding components in each drawing are given the same reference number.

[0031] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to inform those skilled in the art of the scope of the disclosure. One embodiment of the present disclosure may be defined in accordance with the claims.

[0032] In the present disclosure, each block of the flowchart drawings and combinations of the flowchart drawings can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, and the instructions executed by the processor of the computer or other programmable data processing equipment can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing equipment to implement functions in a specific manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing equipment.

[0033] In the present disclosure, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment of the present disclosure, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0034] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment of the present disclosure, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided by a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In one embodiment of the present disclosure, the function of '~bu' can be performed by one or more processors.

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. Also, similar parts are designated by similar reference numerals throughout the specification. Furthermore, the reference numerals used in each drawing are only for the purpose of describing each drawing, and different reference numerals used in different drawings do not indicate different elements. The present disclosure will be described in detail below with reference to the attached drawings.

[0036] The artificial intelligence-related functions according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. The one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. If the one or more processors are artificial intelligence-only processors, the artificial intelligence-only processors may be designed with a hardware structure for processing artificial intelligence models.

[0037] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0038] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0039] FIG. 1 is a diagram schematically illustrating an electronic device for processing image distortion according to one embodiment of the present disclosure.

[0040] In one embodiment of the present disclosure, an electronic device (100) can provide a service for providing a distortion-corrected output image to a user (110). The method for correcting image distortion of the present disclosure processes distortion contained in an image captured by a user (110), thereby providing an output image with the distortion corrected to the user (110).

[0041] Referring to FIG. 1, a user (110) can capture an image using an electronic device (100). The electronic device (100) can obtain an input image (130).

[0042] In one embodiment of the present disclosure, the input image (130) may be acquired using a camera of the electronic device (100). The input image (130) may be a raw image in which light refracted through a lens of the electronic device (100) is identified by an image sensor. The input image (130) may be an image in which the raw image is processed by an image signal processor (ISP).

[0043] In one embodiment of the present disclosure, the input image (130) may be an image stored in the memory of the electronic device (100). The electronic device (100) may obtain the input image (130) stored in the memory.

[0044] The input image (130) may include lens distortion and / or perspective distortion. For example, the input image (130) may represent objects located in the outer periphery as being larger than they actually are. Lens distortion and perspective distortion of the image are described in detail with reference to FIGS. 3 and 4.

[0045] The electronic device (100) can generate an output image (140) in which the input image (130) is corrected. For example, the electronic device (100) can generate an output image (140) in which lens distortion and / or perspective distortion is corrected.

[0046] In one embodiment of the present disclosure, the electronic device (100) can generate an output image (140) by moving pixels included in an input image (130). The electronic device (100) can determine the degree of movement of pixels differently depending on an object included in the input image (130). For example, the electronic device (100) can determine the movement of pixel positions to reduce distortion for a human face, hair, or body. The electronic device (100) can determine that pixels do not move (e.g., so that the position is fixed) for objects that are easy for a user (110) to recognize when their shape changes, such as a straight line, a triangle, a square, or a circle. By determining the degree of movement of pixels differently depending on the object, the electronic device (100) can minimize image defects that occur while correcting distortion included in the input image (130).

[0047] In one embodiment of the present disclosure, the electronic device (100) can display an output image (140) through a display. The electronic device (100) can display an output image (140) with perspective distortion corrected, rather than an input image (130) containing perspective distortion.

[0048] In one embodiment of the present disclosure, the electronic device (100) can store the output image (140) in memory. The electronic device (100) can display the input image (130) or the output image (140) through a display and store the output image (140) in memory. For example, the electronic device (100) can display the input image (130) through the display and perform correction for perspective distortion based on what was captured. For example, the electronic device (100) can perform correction for perspective distortion of the input image (130) in real time and display the output image (140) through the display.

[0049] In one embodiment of the present disclosure, the electronic device (100) may be various types of devices that correct image distortion. For example, the electronic device (100) may be implemented as various types and forms of electronic devices. The electronic device (100) may include, but is not limited to, devices capable of processing images, such as a smart TV, a smart phone, a tablet PC, a laptop PC, a glasses-type display, a head-mounted display (HMD), a set-top box, and a desktop PC. The electronic device (100) may include, but is not limited to, a camera, and may include a device that is connected to a device including a camera or capable of acquiring images through a memory.

[0050] In one embodiment of the present disclosure, the electronic device (100) may include a server. For example, the electronic device (100) may include a server that provides cloud computing services. The electronic device (100) may perform operations, programs, and / or functions at the request of another electronic device (e.g., a client device), and transmit the performed results to the other electronic device for display. For example, the electronic device (100) may perform an operation to correct an image of the present disclosure and display the corrected image through a display of the other electronic device.

[0051] In one embodiment of the present disclosure, the operations for correcting an image may be performed separately by multiple electronic devices. For example, among the operations of the present disclosure, "Operation A" may be performed by "electronic device A," and "Operation B" may be performed by "electronic device B."

[0052] The operations of the electronic device (100) to correct distortion of an input image will be described in more detail through the drawings and descriptions thereof described below.

[0053] FIG. 2 is a flowchart of a method for correcting image distortion according to one embodiment of the present disclosure.

[0054] Referring to FIG. 2, operations of the electronic device (100) to correct image distortion will be schematically described, and a detailed description of each operation will be described with reference to the drawings that follow.

[0055] In operation S210, the electronic device (100) may obtain an input image. In one embodiment of the present disclosure, the input image may include perspective distortion.

[0056] In operation S220, the electronic device (100) may generate a projection image in which perspective distortion of the input image is corrected based on the input image. In one embodiment of the present disclosure, the electronic device (100) may determine the projection image based on the lens through which the input image was captured. For example, the projection image may be determined differently depending on the type of the lens through which the image was captured (e.g., telephoto lens, wide-angle lens) or the specifications of the lens (e.g., focal length, angle of view). A projection image according to one embodiment of the present disclosure will be described in detail with reference to FIG. 6.

[0057] In operation S230, the electronic device (100) may generate a cognitive map that divides the input image into two or more types of areas based on objects included in the input image. For example, the cognitive map may include an area of ​​interest (AoI) including a human face in the input image and a maintenance area including a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse. A cognitive map according to an embodiment of the present disclosure will be described in detail with reference to FIG. 7.

[0058] In operation S240, the electronic device (100) may generate mesh information for correcting an input image based on the projection image and the recognition map. The mesh information may include movement information for a plurality of pixel locations. The mesh information may include movement information of pixel locations for obtaining a corrected output image from the input image. The movement information may be determined to different degrees depending on a plurality of types. Mesh information according to an embodiment of the present disclosure will be described in detail with reference to FIG. 8.

[0059] In operation S250, the electronic device (100) may generate an output image by correcting the input image based on the mesh information. The electronic device (100) may generate an output image in which the input image is corrected using the input image and the mesh information. For example, the electronic device (100) may generate an output image by changing the pixel positions of the input image using the movement information of the pixel positions included in the mesh information. An output image according to an embodiment of the present disclosure will be described in detail with reference to FIG. 9.

[0060] In one embodiment of the present disclosure, some more steps may be included.

[0061] FIG. 3 is a drawing for explaining lens distortion according to one embodiment of the present disclosure.

[0062] Referring to FIG. 3, distorted images (e.g., first, second, and third distorted images (320, 330, 340)) are illustrated depending on the type of lens.

[0063] Lens distortion can include distortion caused by light passing through a spherical lens and then entering a flat surface. The type of lens distortion can be determined by the shape of the lens. Lens distortion can include at least one of barrel distortion, pincushion distortion, or mustache distortion.

[0064] The first distorted image (320) may include barrel distortion. Barrel distortion may include distortion in which the image magnification decreases as the object moves away from the optical axis. Barrel distortion may occur in a wide-angle lens. The central region of the first distorted image (320) has a high magnification as if enlarged, and the outer regions of the first distorted image (320) have a low image magnification. An object located in the central region of the first distorted image (320) may appear larger than an object located in the outer regions of the first distorted image (320).

[0065] The first mesh information (325) may include information on the positional movement of pixels from the distortion-corrected image (310) to the first distorted image (320). The first mesh information (325) may include information on the positional movement of pixels where barrel distortion occurs.

[0066] The second distorted image (330) may include pincushion distortion. Pincushion distortion may include distortion in which the image magnification increases as the object moves away from the optical axis. Pincushion distortion may occur in a telephoto lens. The central region of the second distorted image (330) has a low magnification as if it were reduced, and the outer regions of the second distorted image (330) have a high image magnification. Objects located in the central region of the second distorted image (330) may appear smaller than objects located in the outer regions of the second distorted image (330).

[0067] The second mesh information (335) may include information on the positional movement of pixels from the distortion-corrected image (310) to the second distorted image (330). The second mesh information (335) may include information on the positional movement of pixels where pincushion distortion occurs.

[0068] The third distorted image (340) may include mustache distortion. Mustache distortion may include distortion in which the image magnification decreases as it moves away from the optical axis and then increases again after a certain distance. Mustache distortion may exhibit a form that seems to be a combination of barrel distortion and pincushion distortion. Mustache distortion may occur in an aspheric lens. The central area of ​​the third distorted image (340) has a high magnification as if it were enlarged, and the image magnification decreases as it moves toward the outer area of ​​the third distorted image (340) and then increases again. The size of objects in the third distorted image (340) may decrease and then increase again as it moves from the central area to the outer area.

[0069] The third mesh information (345) may include information on the positional movement of pixels from the distortion-corrected image (310) to the third distorted image (340). The third mesh information (345) may include information on the positional movement of pixels where mustache distortion occurs.

[0070] In one embodiment of the present disclosure, the electronic device (100) can correct distortion of an image by performing pixel warping. In the present disclosure, pixel warping may include an operation of mapping each pixel of an image to a new location. The electronic device (100) can correct distortion by shifting the coordinates of pixels of the image.

[0071] In one embodiment of the present disclosure, the electronic device (100) can correct lens distortion through an image signal processor.

[0072] In one embodiment of the present disclosure, the electronic device (100) may correct lens distortion using a processor other than an image signal processor. For example, the electronic device (100) may correct lens distortion using a central processing unit, regardless of whether the lens distortion is corrected using an image signal processor.

[0073] FIG. 4 is a drawing for explaining perspective distortion according to one embodiment of the present disclosure.

[0074] Referring to FIG. 4, light can be identified by an image sensor (400) by passing through a lens. In one embodiment of the present disclosure, the image sensor (400) may be an image sensor of an electronic device (100). The area identified by the image sensor (400) may be determined by the angle of view of the lens. For example, when light passes through a wide-angle lens having an angle of view of 120 degrees, the image sensor (400) may identify an area corresponding to the angle of view of 120 degrees.

[0075] Referring to FIG. 4, for convenience of explanation, the first object (410) and the second object (430) represent objects of the same size that are positioned at the same distance from the image sensor (400) along the optical axis, but at different angles relative to the optical axis. The first image (420) and the second image (440) represent images generated by the first object (410) and the second object (430), respectively. The sizes of the first image (420) and the second image (440) can be determined according to the positions of the first object (410) and the second object (430), respectively. For example, the sizes of the first object (410) and the second object (430) are the same, but the size of the second image (440) for the second object (430) located at the outer edge relative to the optical axis is identified as being larger than the size of the first image (420) for the first object (410) located at the center relative to the optical axis. Perspective distortion can involve distortion of the size of an image depending on the position of the object.

[0076] Perspective distortion can occur independently of lens distortion. Both telephoto and wide-angle lenses can experience perspective distortion. The degree of perspective distortion can vary based on the focal length and field of view of the lens. In one embodiment of the present disclosure, the electronic device (100) can correct an image based on the field of view and / or focal length of the lens through which the input image was captured. The electronic device (100) can determine the degree of correction differently depending on the field of view and / or focal length.

[0077] Perspective distortion can occur at different degrees of distortion depending on the distance of the object. The closer the object is to the image sensor (400), the greater the degree of distortion can be. The third object (450) is positioned at the same angle with respect to the optical axis as the second object (430), but is positioned farther from the image sensor (400). The size of the third image (460) for the third object (450) is smaller than the size of the second image (440). Therefore, the third object (450) experiences relatively less distortion compared to the second object (430) that is closer to the image sensor (400).

[0078] In one embodiment of the present disclosure, the electronic device (100) can generate a projection image based on depth information of an object. For example, the electronic device (100) can determine a degree of distortion correction according to distance using the depth information of the object. The electronic device (100) can determine depth information of an object included in an input image. The electronic device (100) can generate a projection image based on the depth information.

[0079] FIG. 5 is a block diagram illustrating an electronic device for correcting image distortion according to one embodiment of the present disclosure.

[0080] Referring to FIG. 5, the output image can be generated from the input image via the projection module (510), the segmentation module (520), the mesh generation module (530), and / or the image correction module (540).

[0081] In one embodiment of the present disclosure, the projection module (510) can generate a projection image from an input image. The projection image may be an image in which perspective distortion of the input image is corrected. For example, the projection image may be an image in which perspective distortion of the input image is minimized.

[0082] The projection module (510) can generate a projection image based on the type of lens through which the input image was captured, the specifications of the lens, and / or the depth information (or distance information) of the object. As described in FIG. 4, perspective distortion can be determined differently depending on the type of lens (e.g., telephoto lens, wide-angle lens, fish-eye lens, etc.) and / or the specifications of the lens (e.g., angle of view, focal length, etc.). For example, perspective distortion can be reduced as the angle of view of the lens is narrower. For example, perspective distortion can be reduced as the focal length of the lens is longer. The projection module (510) can determine correction parameters based on the type of lens, the specifications of the lens, and / or the depth information of the object. The projection module (510) can generate a projection image by applying the correction parameters to the input image.

[0083] In one embodiment of the present disclosure, the projection module (510) can generate a projected image with perspective distortion corrected. However, since the projected image is an image corrected to reduce perspective distortion, some awkward aspects may occur depending on the user's perception. For example, geometric structures such as straight lines, triangles, squares, or circles may be deformed due to perspective distortion correction. For example, straight lines may become bent, or circles may become compressed, becoming ellipses.

[0084] In one embodiment of the present disclosure, a distortion-corrected image that does not degrade quality according to the user's perception can be generated using a projection module (510), a segmentation module (520), a mesh generation module (530), and / or an image correction module (540).

[0085] In one embodiment of the present disclosure, the segmentation module (520) may generate a perception map based on an input image. The perception map may be divided into two or more regions based on objects included in the input image. For example, the perception map may include a region of interest including a human face included in the input image and a retention region including a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse.

[0086] In one embodiment of the present disclosure, the segmentation module (520) can recognize objects included in an input image. For example, the segmentation module (520) can recognize a human face or a geometric configuration included in the input image.

[0087] In one embodiment of the present disclosure, the segmentation module (520) can determine the type of the extracted regions based on the type of object contained in each of the extracted regions. If the object contained in the region is a human face, the segmentation module (520) can determine the region as a region of interest. If the object contained in the region has a geometric configuration, the segmentation module (520) can determine the region as a maintenance region. If the object contained in the region has a repeating pattern rather than a geometric configuration, the segmentation module (520) can determine the region as a low correction area.

[0088] In one embodiment of the present disclosure, the segmentation module (520) may generate a perception map that includes information about the types of extracted regions. For example, the perception map may include values ​​indicating the types of regions in the input image. For example, the perception map may include indices indicating the types of regions, such as regions of interest indicating a value of 0 and regions of maintenance indicating a value of 1.

[0089] In one embodiment of the present disclosure, the projection image generated by the projection module (510) and the perception map generated by the segmentation module (520) can be transmitted to the mesh generation module (530).

[0090] In one embodiment of the present disclosure, the mesh generation module (530) can generate mesh information based on a projection image and a perception map. The mesh information can include pixel position shift information for obtaining a corrected output image from an input image. The pixel position shift information can be determined on a pixel-by-pixel basis or on a block-by-block basis containing pixels.

[0091] In one embodiment of the present disclosure, the mesh generation module (530) can divide each of the input image and the projection image into a plurality of blocks. The mesh generation module (530) can determine a motion vector based on a first block of the input image and a second block of the projection image corresponding to the first block. The mesh generation module (530) can determine a motion vector for each block. The mesh generation module (530) can generate mesh information including the motion vector.

[0092] In one embodiment of the present disclosure, the mesh information may include movement information of pixel positions determined such that the movement of pixel positions included in the region of interest is greater than the movement of pixel positions included in the low-correction region. For example, the mesh generation module (530) may generate the mesh information such that the pixel positions included in the region of interest are corrected to be identical to the projection image. For example, the mesh generation module (530) may generate the mesh information such that the pixel positions included in the low-correction region are moved to a position intermediate between the projection image and the input image.

[0093] In one embodiment of the present disclosure, the mesh information may include movement information of pixel positions determined such that the movement of pixel positions included in the low-correction area is greater than the movement of pixel positions included in the maintenance area.

[0094] In one embodiment of the present disclosure, the mesh information may include pixel position shift information, such that pixels included in the maintenance area have fixed positions. For example, pixels included in the maintenance area may have the same positions in the input image and the output image.

[0095] In one embodiment of the present disclosure, the mesh generation module (530) can transmit mesh information to the image correction module (540). The image correction module (540) can use the input image and the mesh information to generate an output image in which the input image is corrected.

[0096] The image correction module (540) can generate an output image by adjusting the positions of pixels of an input image using mesh information. The image correction module (540) can generate an output image by changing the positions of pixels of an input image using movement information corresponding to areas included in the mesh information. The image correction module (540) can generate an output image such that the input image corresponding to the region of interest moves like a projection image. The image correction module (540) can generate an output image such that the position of the input image corresponding to the maintenance region is fixed. The image correction module (540) can generate an output image such that the positions of pixels of the input image corresponding to the low-compensation region move with reference to the region of interest and the maintenance region.

[0097] In the present disclosure, the operations or processes described as being performed by at least one of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540) can be understood as being performed by the processor of the electronic device (100). For example, the processor of the electronic device (100) can perform at least one of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540). However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, detailed components or modules included in at least one of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540) can be viewed as software units that are responsible for a defined function or role in an overall program for processing image distortion. In one embodiment of the present disclosure, instructions causing the processor to perform at least one of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540) may be stored in the memory. In one embodiment of the present disclosure, at least one of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540) may be implemented as separate hardware. In one embodiment of the present disclosure, each of the projection module (510), the segmentation module (520), the mesh generation module (530), or the image correction module (540) may include various processing circuits and / or executable program instructions.

[0098] FIG. 6 is a drawing for explaining a projection image according to one embodiment of the present disclosure.

[0099] Referring to FIG. 6, a projection image corresponding to an input image according to one embodiment of the present disclosure is illustrated.

[0100] In one embodiment of the present disclosure, the electronic device (100) can generate a projection image by correcting the input image in a direction that minimizes distortion. For example, if barrel distortion occurs, the electronic device (100) can generate a projection image by shifting the input image outward. If pincushion distortion occurs, the electronic device (100) can generate a projection image by correcting the input image to shift inward.

[0101] In one embodiment of the present disclosure, the electronic device (100) can generate a projection image based on the type of lens and / or the specifications of the lens. As the type of lens and / or the specifications of the lens change, the type and degree of distortion may change.

[0102] In one embodiment of the present disclosure, the electronic device (100) can determine a vanishing point of an input image based on at least one of the angle of view or focal length of a lens through which the input image was captured. The vanishing point may include a point where parallel lines in the real world converge when projected onto the image. The electronic device (100) can determine at least one vanishing point for the input image.

[0103] In one embodiment of the present disclosure, the electronic device (100) can generate a projection image by correcting an input image based on a vanishing point. The electronic device (100) can correct distortion based on the location of the vanishing point and the locations of each pixel included in the input image. When the location of the vanishing point is close to the pixel location of the input image, distortion may occur more severely than when the locations are far apart. The electronic device (100) can reduce distortion by adjusting the pixel locations of the input image so that they do not converge based on the vanishing point.

[0104] Referring to Figure 6, an input image with barrel distortion has a relatively low image magnification in the central region and a relatively high image magnification in the peripheral regions. Therefore, the projection image can be shifted so that the image magnifications contained in the input image are identical or similar. The projection image can be generated by expanding the central region of the input image and shifting the pixels overall toward the peripheral regions.

[0105] FIG. 7 is a diagram for explaining map information according to one embodiment of the present disclosure.

[0106] In one embodiment of the present disclosure, an electronic device (100) can generate a recognition map based on an input image. The electronic device (100) can perform segmentation on the input image and generate a recognition map using the segmentation results. Segmentation may refer to an operation of recognizing an object included in an image.

[0107] The cognitive map may be classified into multiple regions based on the objects contained in the input image. In one embodiment of the present disclosure, the cognitive map may include a region of interest, a maintenance region, and / or a low-correction region.

[0108] In one embodiment of the present disclosure, the region of interest may include a human face and / or an animal face. The region of interest may include an area with a high priority for correction against perspective distortion. When perspective distortion occurs, an area where the user feels awkward may be determined as the region of interest. Objects classified as the region of interest may be determined in advance. In the present disclosure, the term "Area of ​​Interest" may be replaced with "Region of Interest" (ROI), but is not limited thereto, and may be expressed by various alternative terms with the same meaning.

[0109] In one embodiment of the present disclosure, the maintenance area may include a geometric configuration including curves, straight lines, polygons, circles, and / or ellipses. The maintenance area may include an area where perspective distortion correction is not performed. Since perspective distortion correction is performed differently from the area of ​​interest, an area where the user feels awkward may be determined as the maintenance area. For example, if a structure or object with a linear shape that the user knows in advance becomes bent due to distortion correction, the user may feel awkward. Objects classified as the maintenance area may be determined in advance. Objects included in the maintenance area may include not only objects with meaning (e.g., trees, utility poles), but also boundaries between objects with meaning (e.g., a straight boundary between a road and a sidewalk) or patterns within objects with meaning (e.g., a straight pattern included in a tie). In the present disclosure, the maintenance area may be replaced with, but is not limited to, a heterogeneous area, and may be expressed by various alternative terms with the same meaning.

[0110] In one embodiment of the present disclosure, the low-correction area may include a repeating pattern other than a geometric configuration included in a monochrome pattern and / or a maintenance area. The low-correction area may include an area with a low priority for correction for perspective distortion. When perspective distortion occurs, an area where the user feels relatively less awkward may be determined as the low-correction area. For example, compared to a person's face being enlarged due to perspective distortion or a straight line being bent due to correction, the position shift of a monochrome pattern may feel less awkward to the user. Objects classified as low-correction areas may be determined in advance. In the present disclosure, the low-correction area may be replaced with, but is not limited to, a homogeneous area, and may be expressed by various alternative terms with the same meaning.

[0111] In one embodiment of the present disclosure, the electronic device (100) can recognize objects included in an input image. In one embodiment of the present disclosure, the electronic device (100) can recognize objects included in the input image using an AI model. For example, the electronic device (100) can recognize objects included in the input image using a U-Net model. The electronic device (100) can recognize objects using a trained AI model to recognize objects corresponding to a region of interest, a maintenance region, and / or a low-correction region. For example, the AI ​​model can recognize objects using training data that identifies a human face or a geometric configuration.

[0112] In one embodiment of the present disclosure, the electronic device (100) can detect geometric configurations by transforming an image into a new coordinate system. For example, the electronic device (100) can recognize geometric configurations, such as straight lines and circles, from an input image using the Hough transform.

[0113] In one embodiment of the present disclosure, the electronic device (100) can determine the location of an object and the type of the object. The electronic device (100) can determine the location of the object and the type of the object on a pixel basis.

[0114] In one embodiment of the present disclosure, the electronic device (100) can extract regions corresponding to each recognized object. For example, the electronic device (100) can extract a region corresponding to a human face from an input image. The electronic device (100) can determine the type of the extracted regions based on the type of object included in each of the extracted regions. For example, the extracted region corresponding to a human face can be determined as a region of interest. The electronic device (100) can generate a recognition map including information about the types of the extracted regions. The recognition map can include the type of region represented by each pixel of the input image.

[0115] In one embodiment of the present disclosure, the electronic device (100) may determine weights for extracted regions. In one embodiment of the present disclosure, the electronic device (100) may determine a high positive weight for a region of interest. The larger the region of interest, the higher the weight value. For example, a region of interest corresponding to a face close to the camera may have a large size and thus a large weight value. The electronic device (100) may determine a negative weight for a maintenance region. The electronic device (100) may determine a low positive weight for a low-correction region.

[0116] In one embodiment of the present disclosure, the electronic device (100) can determine movement information of pixel positions for distortion correction for an area indicating a large weight, and can determine movement information so that the pixel positions are fixed for an area indicating a negative weight. The electronic device (100) can determine movement information of pixel positions by prioritizing an area indicating a large weight based on a plurality of weight values. The electronic device (100) can determine that the pixel positions of an area indicating a negative weight are fixed, separately from the priority of an area indicating a positive weight.

[0117] FIG. 8 is a drawing for explaining mesh information according to one embodiment of the present disclosure.

[0118] In one embodiment of the present disclosure, the electronic device (100) can generate mesh information using a projection image and a perception map. The mesh information can include movement information of pixel positions for converting pixels of an input image into pixels of an output image.

[0119] In one embodiment of the present disclosure, the electronic device (100) can divide the input image and the projected image into multiple sizes. For example, the electronic device (100) can divide the input image and the projected image into blocks of a defined size (e.g., blocks of 16 x 16 pixels). For example, the electronic device (100) can divide the input image and the projected image into 64 equal parts, respectively.

[0120] In one embodiment of the present disclosure, the electronic device (100) may determine a motion vector between blocks of a projection image corresponding to blocks of an input image. For example, the motion vector may include a degree of change in the position of a first sample of a block of the projection image based on a first position of the block of the input image (e.g., a position of an upper left sample). A plurality of motion vectors may be determined. For example, the motion vector may include a degree of change in the positions of a plurality of corresponding samples of a block of the projection image based on positions of a plurality of samples of the block of the input image.

[0121] In one embodiment of the present disclosure, the electronic device (100) may determine mesh information based on the types of regions recognized in the cognitive map. The electronic device (100) may determine a motion vector corresponding to a region of interest as motion information of the mesh information. The electronic device (100) may not determine a motion vector corresponding to a maintenance region as motion information of the mesh information. The electronic device (100) may determine that the maintenance region does not move or moves to a degree smaller than the motion vector. The electronic device (100) may determine motion information of the mesh information corresponding to a low-compensation region by referring to other regions. For example, the electronic device (100) may not apply a motion vector to a low-compensation region existing between maintenance regions by referring to the motion vectors of surrounding maintenance regions. The electronic device (100) may apply a motion vector to a low-compensation region existing between regions of interest by referring to the motion vectors of surrounding low-compensation regions.

[0122] Referring to FIG. 8, the electronic device (100) may determine a motion vector for a human face corresponding to an area of ​​interest included in a recognition map using the motion information of the mesh information. The electronic device (100) may not determine a motion vector for a geometric configuration corresponding to a maintenance area included in the recognition map using the motion information of the mesh information. The mesh information may store motion information of pixel positions so that the positions are the same in the input image and the output image.

[0123] In one embodiment of the present disclosure, the electronic device (100) may determine mesh information based on the weight of the recognition map. The electronic device (100) may determine mesh information such that the higher the weight of the recognition map, the more the mesh information has the same movement information as the movement vector. Referring to FIG. 8, the larger the area of ​​the region of interest, the less the movement vector may be deformed and determined as movement information of the mesh information. For example, the mesh information of FIG. 8 may determine movement information such that strong correction for perspective distortion is performed for a person's face with a large area of ​​the region of interest, and relatively weak correction is performed for a person's face with a small area of ​​the region of interest.

[0124] FIG. 9 is a diagram for explaining a process of correcting an image using mesh information according to one embodiment of the present disclosure.

[0125] In one embodiment of the present disclosure, the electronic device (100) can generate an output image (or a corrected image) based on an input image and mesh information.

[0126] The electronic device (100) can generate an output image by applying movement information of mesh information to an input image. The electronic device (100) can generate an output image by applying a movement vector to the position of a pixel of the input image.

[0127] In one embodiment of the present disclosure, the electronic device (100) can determine an output image on a pixel-by-pixel basis. The electronic device (100) can identify a motion vector corresponding to each pixel of the input image from mesh information. The electronic device (100) can apply the motion vector to each pixel of the input image to adjust the position of each pixel to generate an output image.

[0128] In one embodiment of the present disclosure, the electronic device (100) can determine an output image on a block-by-block basis. The electronic device (100) can identify a motion vector corresponding to each block containing a plurality of pixels of the input image from mesh information. The electronic device (100) can apply the motion vector to each block of the input image to adjust the positions of the plurality of pixels contained in each block, thereby generating an output image.

[0129] In one embodiment of the present disclosure, the electronic device (100) can generate an output image by applying a matrix representing mesh information to an input image. The electronic device (100) can obtain a matrix representing movement information included in the mesh information. The electronic device (100) can generate an output image by calculating the matrix on the input image.

[0130] FIG. 10 is a diagram for explaining a process of processing an image using inpainting according to one embodiment of the present disclosure.

[0131] In one embodiment of the present disclosure, the electronic device (100) can generate an output image without applying mesh information to a portion of the input image. The electronic device (100) can determine pixels for a blank area using the input image without adjusting the position of the input image.

[0132] Referring to FIG. 10, a human face in an input image may experience perspective distortion. The electronic device (100) can correct the perspective distortion of the human face. The electronic device (100) can generate an output image in which the perspective distortion of the human face included in the input image is corrected. For illustration, the input image and the output image may show a background having a linear geometric configuration behind the human face.

[0133] The electronic device (100) can determine a geometric configuration as a maintenance area, fix the positions of pixels, and determine a human face as a region of interest, and adjust the positions of pixels. In this example, a first area (1010) of an input image may represent a face, but a second area (1020) of an output image may represent a background. In order for the electronic device (100) to generate an output image while fixing the positions of pixels corresponding to the maintenance area, pixels corresponding to the second area (1020) may be determined. In the present disclosure, the second area (1020) may be referred to as a blank area or an area for inpainting. Inpainting may include an operation of generating pixels of an image using surrounding pixels.

[0134] In one embodiment of the present disclosure, the electronic device (100) can generate values ​​of the second area (1020) using an AI model. The AI ​​model used by the electronic device (100) to perform inpainting is described in detail with reference to FIG. 11.

[0135] FIG. 11 is a drawing for explaining an inpainting technique according to one embodiment of the present disclosure.

[0136] In one embodiment of the present disclosure, the electronic device (100) can determine samples of blank areas of an input image using an AI model. Referring to FIG. 11, a process of training an AI model that determines samples of blank areas when blank areas exist in an image is illustrated. The AI ​​model can be trained by the electronic device (100). The AI ​​model can be trained through an external server and transmitted to the electronic device (100).

[0137] In one embodiment of the present disclosure, a training data set for training an AI model may include a training image (1110) including a blank area and an actual image (1120) of the blank area. An electronic device (100) or a server performing training may use the AI ​​model to generate a predicted image (1130) for the blank area of ​​the training image (1110). The electronic device (100) or the server performing training may determine parameters of the AI ​​model so that errors between the actual image (1120) and the predicted image (1130) are minimized.

[0138] The electronic device (100) can generate a predicted image for a blank area of ​​an input image using an AI model on which training has been performed. In one embodiment of the present disclosure, the electronic device (100) can generate an output image from the input image using mesh information. The electronic device (100) can identify whether a blank area exists in the output image (or a corrected input image). The electronic device (100) can determine pixels of the blank area. The electronic device (100) can generate pixels of the blank area using the AI ​​model for the blank area.

[0139] Referring to FIG. 10, the electronic device (100) can generate pixel values ​​for the second area (1020) using an AI model.

[0140] FIG. 12 is a diagram for explaining a process of displaying and storing an image by an electronic device according to one embodiment of the present disclosure.

[0141] Referring to FIG. 12, a first user interface (UI) (1210) may include an interface for displaying an image acquired through a camera of an electronic device (100). A second user interface (1220) may include an interface for displaying an image stored in the electronic device (100).

[0142] In one embodiment of the present disclosure, the electronic device (100) can display several tens of frames per second through the first user interface (1210) depending on the camera settings. For example, if the camera settings are 60 fps, the first user interface (1210) can display 60 images per second.

[0143] In one embodiment of the present disclosure, the electronic device (100) may perform the perspective distortion correction method described with reference to FIGS. 1 to 11 on only some images selected from among images displayed through the first user interface (1210). The electronic device (100) may not be able to adjust perspective distortion in real time for all images displayed on the first user interface (1210). The electronic device (100) may perform the perspective distortion correction method of FIGS. 1 to 11 on images captured at set intervals based on camera settings.

[0144] In one embodiment of the present disclosure, the electronic device (100) can perform perspective distortion correction on a captured image and store the image. Images displayed through the second user interface (1220) may be perspective distortion-corrected images.

[0145] In one embodiment of the present disclosure, the electronic device (100) can adjust perspective distortion using the movement estimation information described in FIG. 13 and display it on the first user interface (1210).

[0146] FIG. 13 is a diagram for explaining a process of processing continuously acquired images according to one embodiment of the present disclosure.

[0147] In one embodiment of the present disclosure, the electronic device (100) can acquire a new image after the input image. Referring to FIG. 13, the electronic device (100) can acquire the t-th captured input image after the t-1-th captured image.

[0148] In one embodiment of the present disclosure, the electronic device (100) can determine a motion vector between a new image and an input image. The electronic device (100) can determine the motion vector through the difference between the new image and the input image. For example, the electronic device (100) can obtain the motion vector through the difference between the t-1th captured image and the tth captured input image.

[0149] In one embodiment of the present disclosure, the electronic device (100) can generate mesh information for a new image by modifying mesh information using a motion vector. For example, mesh information for the t-th captured input image can be obtained by modifying mesh information for the t-1th captured image using a motion vector.

[0150] In one embodiment of the present disclosure, the electronic device (100) can generate an output image for a new image based on mesh information of the new image acquired using a motion vector. The electronic device (100) can correct perspective distortion using the motion vector, thereby reducing the amount of computation and correcting perspective distortion. The electronic device (100) can display a photo with the perspective distortion corrected in real time.

[0151] FIG. 14 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0152] In one embodiment of the present disclosure, an electronic device (100) may include a processor (1410) and a memory (1420).

[0153] The processor (1410) can control the overall operations of the electronic device (100). For example, the processor (1410) can control the overall operations of the electronic device (100) to correct image distortion by executing one or more instructions of a program stored in the memory (1420). There may be one or more processors (1410).

[0154] The processor (1410) may be configured as at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), an Application Processor, a Neural Processing Unit, or an artificial intelligence processor designed as a hardware structure for processing an artificial intelligence model, but is not limited thereto.

[0155] Meanwhile, although not illustrated in FIG. 14, the electronic device (100) may further include additional components to perform the operations described in the aforementioned embodiments. For example, the electronic device (100) may further include a display, a camera, a microphone, a speaker, an input / output interface, etc. The display may output a video signal to the screen of the electronic device (100) under the control of the processor (1410).

[0156] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment of the present disclosure, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first and second operations may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an AI-dedicated processor). Here, an AI-dedicated processor, which is an example of the second processor, may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0157] One or more processors (1410) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0158] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors.

[0159] At least one processor (1410) according to an embodiment of the present invention may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which are configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, without limitation, a single processor performing some of the recited functions, other processor(s) performing other of the recited functions, and even a single processor performing all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. The at least one processor may execute program instructions to achieve or perform the various functions.

[0160] The memory (1420) may store instructions, data structures, and program codes that can be read by the processor (1410). Operations performed by the processor (1410) may be implemented by executing instructions or codes of a program stored in the memory (1420).

[0161] The memory (1420) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, or an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0162] The memory (1420) may store one or more instructions and / or programs that cause the electronic device (100) to operate to correct image distortion. For example, the memory (1420) may store instructions and / or programs for implementing operations to correct image distortion. Meanwhile, the memory (1420) may further store instructions and / or programs for implementing functions of a prompt generation module (not shown).

[0163] In one embodiment of the present disclosure, the processor (1410) may be configured to obtain an input image by performing one or more instructions included in the memory (1420). The processor (1410) may be configured to generate a projection image in which perspective distortion of the input image is corrected based on the input image by performing one or more instructions included in the memory (1420). The processor (1410) may be configured to generate a perception map that divides the input image into two or more types of regions according to objects included in the input image by performing one or more instructions included in the memory (1420). The processor (1410) may be configured to generate mesh information for correcting the input image based on the projection image and the perception map by performing one or more instructions included in the memory (1420). The processor (1410) may be configured to generate an output image in which the input image is corrected using the input image and mesh information by performing one or more instructions contained in the memory (1420).

[0164] In one embodiment of the present disclosure, the electronic device (100) may include other components in addition to the processor (1410) and the memory (1420). Components that the electronic device (100) may include in one embodiment of the present disclosure are described in detail with reference to FIG. 15.

[0165] FIG. 15 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0166] As illustrated in FIG. 15, an electronic device (100) according to one embodiment of the present disclosure may further include a camera (1510), a sensor unit (1520), a communication interface (1530), and / or a user interface (1540) in addition to a processor (1410) and a memory (1420).

[0167] The processor (1410) controls the operation of the electronic device (100). The processor (1410) can control the camera (1510), the sensor unit (1520), the communication interface (1530), the user interface (1540), and the memory (1420) by executing programs stored in the memory (1420).

[0168] The memory (1420) may store programs for processing and controlling the processor (1410), and may also store input / output data (e.g., cognitive maps or mesh information). The memory (1420) may also store artificial intelligence models. For example, the memory (1420) may store artificial intelligence models for object recognition or artificial intelligence models for inpainting.

[0169] A camera (1510) may refer to a device that acquires at least one frame. Here, the at least one frame may be expressed as an image (still image or moving image) or a photograph.

[0170] The camera (1510) may be a wide-angle camera and / or a telephoto camera capable of capturing the front or rear of the electronic device (100). The camera (1510) may also be a miniature camera or a pinhole camera. In one embodiment of the present disclosure, the electronic device (100) may include multiple cameras (1510).

[0171] The camera (1510) may include an image sensor (1511) and / or an image signal processor (ISP) (1512). The image sensor (1511) may include a device that identifies light transmitted through a lens of the camera (1510). The image signal processor (1512) may refer to a processor that processes a signal identified by the image sensor (1511). The image signal processor (1512) may perform primary correction for distortion generated by the camera (1510).

[0172] The sensor unit (1520) may include, but is not limited to, a depth sensor (1521) and / or an infrared sensor (1522). The depth sensor (1521) may include a sensor that measures depth information of an object. The infrared sensor (1522) may include a sensor that measures numerical values ​​using infrared rays. The infrared sensor (1522) may measure depth information of an object using infrared rays.

[0173] The sensor unit (1520) can obtain depth information using a UWB sensor. The UWB sensor can measure the round-trip time of a UWB signal. The sensor unit (1520) can determine depth information based on the round-trip time of the UWB signal.

[0174] The communication interface (1530) may include one or more components that enable communication between an electronic device (100) and a server device (not shown), or an electronic device (100) and a mobile terminal (not shown). For example, the communication interface (1530) may include a short-range communication unit (1531), a long-range communication unit (1532), etc.

[0175] The short-range wireless communication unit (1531) may include a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit (NFC), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, or an Ant+ communication unit, but the present disclosure is not limited thereto.

[0176] The remote communication unit (1532) may include, for example, the Internet, a computer network (e.g., a LAN or WAN), or a mobile communication unit. The mobile communication unit may include, for example, a 3G module, a 4G module, a 5G module, an LTE module, an NB-IoT module, or an LTE-M module, but the present disclosure is not limited thereto.

[0177] The user interface (1540) may include an output interface (1541) and an input interface (1542). The output interface (1541) is for outputting an audio signal or a video signal and may include, for example, a display unit or an audio output unit.

[0178] When the display unit and the touchpad are configured as a touch screen in a layered structure, the display unit can be used as an input interface (1542) in addition to the output interface (1541). The display unit can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode, a flexible display, a 3D display, or an electrophoretic display. In addition, depending on the implementation form of the electronic device (100), the electronic device (100) may include two or more display units.

[0179] The audio output unit can output audio data received from the communication interface (1530) or stored in the memory (1420). In addition, the audio output unit can output audio signals related to functions performed in the electronic device (100). The audio output unit can include, for example, a speaker or a buzzer.

[0180] According to one embodiment of the present disclosure, the display unit can display an image captured by a camera (1510) of the electronic device (100) in real time or an image stored in a memory (1420).

[0181] The input interface (1542) is for receiving input from a user. The input interface (1542) may be at least one of a key pad, a dome switch, or a touch pad (e.g., a contact-type electrostatic capacitance type, a pressure-type resistive film type, an infrared sensing type, a surface ultrasonic conduction type, an integral tension measurement type, or a piezoelectric effect type), but the present disclosure is not limited thereto.

[0182] In one embodiment of the present disclosure, a method for correcting distortion of an image by an electronic device is provided. The method may include obtaining an input image. The method may include generating a projection image in which perspective distortion of the input image is corrected based on the input image. The method may include generating a perception map that divides the input image into two or more types of regions according to objects included in the input image. The method may include generating mesh information for correcting the input image based on the projection image and the perception map. The mesh information may include movement information of pixel positions for obtaining a corrected output image from the input image. The method may include generating an output image in which the input image is corrected using the input image and the mesh information. The mesh information may include movement information of pixel positions determined to different degrees according to regions divided in the perception map.

[0183] In one embodiment of the present disclosure, the step of generating a recognition map may include a step of recognizing objects included in an input image. The step of generating the recognition map may include a step of extracting regions corresponding to each of the recognized objects. The step of generating the recognition map may include a step of determining the type of the extracted regions based on the type of object included in each of the extracted regions. The step of generating the recognition map may include a step of generating a recognition map including information about the types of the extracted regions.

[0184] In one embodiment of the present disclosure, the cognitive map may include at least one of a region of interest, a maintenance region, or a low-correction region. The region of interest may include a region with a high priority for correction for perspective distortion. The maintenance region may include a region where no correction for perspective distortion is performed. The low-correction region may include a region with a low priority for correction for perspective distortion.

[0185] In one embodiment of the present disclosure, the step of determining the type of the extracted regions may include determining the first region as a region of interest if an object included in a first region among the extracted regions is a face. The step of determining the type of the extracted regions may include determining the second region as a maintenance region if an object included in a second region among the extracted regions has a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse. The step of determining the type of the extracted regions may include determining the third region as a low-correction region if an object included in a third region among the extracted regions has a configuration including a repetitive pattern.

[0186] In one embodiment of the present disclosure, the movement of pixel positions included in the region of interest may be greater than the movement of pixel positions included in the low-correction region. The movement of pixel positions included in the low-correction region may be greater than the movement of pixel positions included in the maintenance region. The mesh information may include movement information of pixel positions determined such that the movement of pixel positions included in the region of interest is greater than the movement of pixel positions included in the low-correction region, and movement information of pixel positions determined such that the movement of pixel positions included in the low-correction region is greater than the movement of pixel positions included in the maintenance region.

[0187] In one embodiment of the present disclosure, pixels included in the maintenance area may have fixed positions.

[0188] In one embodiment of the present disclosure, the step of generating a projection image may include a step of determining a vanishing point of the input image based on at least one of an angle of view or a focal length of a lens through which the input image was captured. The step of generating the projection image may include a step of generating the projection image by correcting the input image based on the vanishing point.

[0189] In one embodiment of the present disclosure, the method may include the step of dividing each of an input image and a projection image into a plurality of blocks. The method may include the step of determining a motion vector based on a first block of the input image and a second block of the projection image corresponding to the first block. The mesh information may include the motion vector.

[0190] In one embodiment of the present disclosure, the step of generating a corrected output image may include the step of correcting an input image using mesh information. The step of generating the corrected output image may include the step of identifying whether a blank area exists in the corrected input image. The step of generating the corrected output image may include the step of determining pixels in the blank area.

[0191] In one embodiment of the present disclosure, the method may include a step of determining depth information of an object included in an input image. In one embodiment of the present disclosure, the method may include a step of generating a projection image based on the depth information.

[0192] In one embodiment of the present disclosure, the method may include the step of acquiring a new image after an input image. The method may include the step of determining a motion vector between the new image and the input image. The method may include the step of generating mesh information for the new image by modifying the mesh information using the motion vector.

[0193] In one embodiment of the present disclosure, an electronic device for correcting image distortion is provided. The electronic device may include a memory storing one or more instructions; and at least one processor. The one or more instructions may be executed by the at least one processor, thereby allowing the electronic device to obtain an input image. The one or more instructions may be executed by the at least one processor, thereby allowing the electronic device to generate a projection image in which perspective distortion of the input image is corrected based on the input image. The one or more instructions may be executed by the at least one processor, thereby allowing the electronic device to generate a perception map that divides the input image into two or more types of regions based on objects included in the input image. The one or more instructions may be executed by the at least one processor, thereby allowing the electronic device to generate mesh information for correcting the input image based on the projection image and the perception map. The mesh information may include movement information of pixel positions for obtaining a corrected output image from the input image. The one or more instructions may be executed by the at least one processor, thereby allowing the electronic device to generate an output image in which the input image is corrected using the input image and the mesh information. The mesh information may include movement information of pixel locations determined to different degrees depending on the area distinguished in the cognitive map.

[0194] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby allowing an electronic device to recognize objects included in an input image. One or more instructions are executed by at least one processor, thereby allowing the electronic device to extract regions corresponding to each of the recognized objects. One or more instructions are executed by at least one processor, thereby allowing the electronic device to determine the type of the extracted regions based on the type of object included in each of the extracted regions. One or more instructions are executed by at least one processor, thereby allowing the electronic device to generate a recognition map including information about the types of the extracted regions.

[0195] In one embodiment of the present disclosure, the cognitive map may include at least one of a region of interest, a maintenance region, or a low-correction region. The region of interest may include a region with a high priority for correction for perspective distortion. The maintenance region may include a region where no correction for perspective distortion is performed. The low-correction region may include a region with a low priority for correction for perspective distortion.

[0196] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor so that the electronic device can determine the first region as a region of interest if an object included in the first region among the extracted regions is a face. One or more instructions are executed by at least one processor so that the electronic device can determine the second region as a maintenance region if an object included in the second region among the extracted regions has a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse. One or more instructions are executed by at least one processor so that the electronic device can determine the third region as a low-correction region if an object included in the third region among the extracted regions has a configuration including a repetitive pattern.

[0197] In one embodiment of the present disclosure, the movement of pixel positions included in the region of interest may be greater than the movement of pixel positions included in the low-correction region. The movement of pixel positions included in the low-correction region may be greater than the movement of pixel positions included in the maintenance region. The mesh information may include movement information of pixel positions determined such that the movement of pixel positions included in the region of interest is greater than the movement of pixel positions included in the low-correction region, and movement information of pixel positions determined such that the movement of pixel positions included in the low-correction region is greater than the movement of pixel positions included in the maintenance region.

[0198] In one embodiment of the present disclosure, pixels included in the maintenance area may have fixed positions.

[0199] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby enabling the electronic device to determine a vanishing point of an input image based on at least one of an angle of view or a focal length of a lens through which the input image was captured. One or more instructions are executed by at least one processor, thereby enabling the electronic device to generate a projection image by correcting the input image based on the vanishing point.

[0200] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby allowing an electronic device to divide each of an input image and a projection image into a plurality of blocks. One or more instructions are executed by at least one processor, thereby allowing the electronic device to determine a motion vector based on a first block of the input image and a second block of the projection image corresponding to the first block. The mesh information may include the motion vector.

[0201] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby enabling an electronic device to correct an input image using mesh information. One or more instructions are executed by at least one processor, thereby enabling the electronic device to identify whether a blank area exists in the corrected input image. One or more instructions are executed by at least one processor, thereby enabling the electronic device to determine pixels in the blank area.

[0202] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby enabling an electronic device to determine depth information of an object included in an input image. One or more instructions are executed by at least one processor, thereby enabling the electronic device to generate a projection image based on the depth information.

[0203] In one embodiment of the present disclosure, one or more instructions are executed by at least one processor, thereby enabling an electronic device to obtain a new image after an input image. One or more instructions are executed by at least one processor, thereby enabling the electronic device to determine a motion vector between the new image and the input image. One or more instructions are executed by at least one processor, thereby enabling the electronic device to generate mesh information for the new image by modifying the mesh information using the motion vector.

[0204] In one embodiment of the present disclosure, a method for correcting distortion of an image by an electronic device is provided. The method includes acquiring a first image. The method may include generating a projection image in which perspective distortion of the first image is corrected based on the first image. The method may include generating a perception map that divides the first image into a plurality of regions of a first plurality of types based on a plurality of objects included in the first image. The method may include generating first mesh information for correcting the first image based on the projection image and the perception map. The first mesh information may include movement information for a plurality of pixel locations. The method may include generating an output image by correcting the first image based on the first mesh information. The movement information may be determined to a different degree depending on the first plurality of types.

[0205] In one embodiment of the present disclosure, a plurality of regions may correspond to a plurality of objects. The step of generating a recognition map may include a step of recognizing a plurality of objects included in a first image. The step of generating the recognition map may include a step of extracting a plurality of regions. The step of generating the recognition map may include a step of determining a first plurality of types based on a second plurality of types of the plurality of objects. The step of generating the recognition map may include a step of generating a recognition map based on the first plurality of types.

[0206] In one embodiment of the present disclosure, the cognitive map may include at least one of an Area of ​​Interest with a high priority for correction for perspective distortion, a maintenance area where no correction for perspective distortion is performed, or a low correction area with a low priority for correction for perspective distortion.

[0207] In one embodiment of the present disclosure, the step of determining the first plurality of types may include the step of determining a first region among the plurality of regions as a region of interest based on whether a first object in the first region includes a face. The step of determining the first plurality of types may include the step of determining a second region among the plurality of regions as a maintenance region based on whether a second object in the second region includes a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse. The step of determining the first plurality of types may include the step of determining a third region among the plurality of regions as a low-correction region based on whether a third object in the third region includes a repetitive pattern.

[0208] In one embodiment of the present disclosure, the first mesh information may include first movement information and second movement information. The step of generating the first mesh information may include the step of determining the first movement information such that the movement of the first pixel position included in the region of interest is greater than the movement of the second pixel position included in the low-correction region. The step of generating the first mesh information may include the step of determining the second movement information such that the movement of the third pixel position included in the low-correction region is greater than the movement of the fourth pixel position included in the maintenance region.

[0209] In one embodiment of the present disclosure, the step of generating a projection image may include a step of determining a vanishing point of the first image based on at least one of a field of view or a focal length of a lens through which the first image was captured. The step of generating the projection image may include a step of generating the projection image by correcting the first image based on the vanishing point.

[0210] In one embodiment of the present disclosure, the first mesh information may include a motion vector. The step of generating the first mesh information may include the step of dividing each of the first image and the projection image into a plurality of blocks. The step of generating the first mesh information may include the step of determining the motion vector based on a first block of the first image and a second block of the projection image corresponding to the first block.

[0211] In one embodiment of the present disclosure, the step of generating an output image may include a step of generating a second image by correcting the first image based on the first mesh information. The step of generating the output image may include a step of identifying whether the second image includes a blank area. The step of generating the output image may include a step of determining pixels of the blank area.

[0212] In one embodiment of the present disclosure, the step of generating a projection image may include a step of determining depth information of an object included in a first image. The step of generating the projection image may include a step of generating the projection image based on the depth information.

[0213] In one embodiment of the present disclosure, the method may include the step of acquiring a third image after the first image. The method may include the step of determining a motion vector between the third image and the first image. The method may include the step of generating second mesh information for the third image based on the first mesh information modified using the motion vector.

[0214] In one embodiment of the present disclosure, an electronic device for correcting image distortion is provided. The electronic device may include at least one processor including a processing circuit, and a memory including one or more storage media storing at least one instruction. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to acquire a first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate a projection image in which perspective distortion of the first image is corrected based on the first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate a perception map that divides the first image into a plurality of regions of a first plurality of types, based on a plurality of objects included in the first image. The at least one instruction may be individually or collectively executed by the at least one processor, thereby allowing the electronic device to generate first mesh information for correcting the first image based on the projection image and the perception map. The first mesh information may include movement information of a plurality of pixel locations. The method can generate an output image by correcting a first image based on first mesh information. The movement information can be determined to different degrees depending on the first plurality of types.

[0215] In one embodiment of the present disclosure, a plurality of regions may correspond to a plurality of objects. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing an electronic device to recognize a plurality of objects included in a first image. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing the electronic device to extract a plurality of regions. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing the electronic device to determine a first plurality of types based on a second plurality of types of the plurality of objects. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing the electronic device to generate a recognition map based on the first plurality of types.

[0216] In one embodiment of the present disclosure, the perception map may include at least one of a region of interest with a high priority for correction for perspective distortion, a maintenance region where no correction for perspective distortion is performed, or a low-correction region with a low priority for correction for perspective distortion.

[0217] In one embodiment of the present disclosure, at least one instruction may be individually or collectively executed by at least one processor so that the electronic device can determine a first area among a plurality of areas as a region of interest based on whether a first object included in the first area includes a face. At least one instruction may be individually or collectively executed by at least one processor so that the electronic device can determine a second area among the plurality of areas as a maintenance area based on whether a second object included in the second area includes a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse. At least one instruction may be individually or collectively executed by at least one processor so that the electronic device can determine a third area among the plurality of areas as a low-correction area based on whether a third object included in the third area includes a repetitive pattern.

[0218] In one embodiment of the present disclosure, the first mesh information may include first movement information and second movement information. At least one instruction may be individually or collectively executed by at least one processor, such that the electronic device may determine the first movement information such that the movement of the first pixel position included in the region of interest is greater than the movement of the second pixel position included in the low-compensation region. At least one instruction may be individually or collectively executed by at least one processor, such that the electronic device may determine the second movement information such that the movement of the third pixel position included in the low-compensation region is greater than the movement of the fourth pixel position included in the maintenance region.

[0219] In one embodiment of the present disclosure, at least one instruction is individually or collectively executed by at least one processor, thereby causing an electronic device to determine a vanishing point of a first image based on at least one of an angle of view or a focal length of a lens through which the first image was captured. At least one instruction is individually or collectively executed by at least one processor, thereby causing the electronic device to correct the first image based on the vanishing point, thereby generating a projection image.

[0220] In one embodiment of the present disclosure, the first mesh information may include a motion vector. At least one instruction may be individually or collectively executed by at least one processor, thereby causing the electronic device to divide each of the first image and the projection image into a plurality of blocks. At least one instruction may be individually or collectively executed by at least one processor, thereby causing the electronic device to determine a motion vector based on a first block of the first image and a second block of the projection image corresponding to the first block.

[0221] In one embodiment of the present disclosure, at least one instruction may be individually or collectively executed by at least one processor, thereby enabling an electronic device to generate a second image by correcting a first image based on first mesh information. At least one instruction may be individually or collectively executed by at least one processor, thereby enabling the electronic device to identify whether the second image includes a blank area. At least one instruction may be individually or collectively executed by at least one processor, thereby enabling the electronic device to determine pixels of the blank area.

[0222] In one embodiment of the present disclosure, at least one instruction may be individually or collectively executed by at least one processor, thereby enabling an electronic device to determine depth information of an object included in a first image. At least one instruction may be individually or collectively executed by at least one processor, thereby enabling the electronic device to generate a projection image based on the depth information.

[0223] In one embodiment of the present disclosure, at least one instruction may be individually or collectively executed by at least one processor, thereby allowing an electronic device to acquire a third image after a first image. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing the electronic device to determine a motion vector between the third image and the first image. At least one instruction may be individually or collectively executed by at least one processor, thereby allowing the electronic device to generate second mesh information for the third image based on modifying the first mesh information using the motion vector.

[0224] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one of the above methods on a computer is provided.

[0225] In a method for correcting image distortion of an electronic device according to the present disclosure, as a method for recognizing a face and / or geometric element included in an image, face image data and / or geometric element image data may be used as input data of an artificial intelligence model to obtain output data recognizing the face and / or geometric element included in the image. The artificial intelligence model may be created through learning. Here, being created through learning means that the artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through an operation between the operation result of the previous layer and the plurality of weight values.

[0226] Visual understanding is a technology that recognizes and processes objects like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, spatial understanding (3D reconstruction / localization), or image enhancement.

[0227] An image distortion correction method according to one embodiment of the present disclosure may also be implemented in the form of a recording medium including computer-executable instructions, such as a program module executed by a computer. A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. A computer-readable recording medium may include a computer storage medium and a communication medium. A computer storage medium includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium may include other data in a modulated data signal, such as a computer-readable instruction, data structure, or program module.

[0228] A computer-readable storage medium according to one embodiment of the present disclosure may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0229] A method according to one embodiment of the present disclosure may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). At least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0230] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0231] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for an electronic device to correct distortion of an image, Step of acquiring a first image (S210); A step (S220) of generating a projection image in which perspective distortion of the first image is corrected based on the first image; A step (S230) of generating a cognitive map that divides the first image into a plurality of first types of regions according to a plurality of objects included in the first image; A step (S240) of generating first mesh information for correcting the first image based on the projection image and the recognition map, wherein the first mesh information includes movement information for a plurality of pixel locations; and It includes a step (S250) of generating an output image by correcting the first image based on the first message information. A method wherein the above movement information is determined to different degrees depending on the type of the first plurality.

2. In the first paragraph, the plurality of areas correspond to the plurality of objects, The step of generating the above cognitive map (S230) is: A step of recognizing a plurality of objects included in the first image; A step of extracting the above multiple areas; A step of determining the first plurality of types based on the second plurality of types of the plurality of objects; and A method characterized by comprising a step of generating a cognitive map based on the first plurality of types.

3. In paragraph 2, A method according to claim 1, wherein the above-described cognitive map includes at least one of an area of interest having a high priority for correction for perspective distortion, a maintenance area in which no correction for perspective distortion is performed, or a low correction area having a low priority for correction for perspective distortion.

4. In paragraph 3, The step of determining the first plurality of types is: A step of determining a first region among the plurality of regions as the region of interest based on the first object of the first region including a face; determining the second area among the plurality of areas as the maintenance area based on the second object of the second area having a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse; and A method comprising the step of determining a third area among the plurality of areas as the low-compensation area based on a third object in the third area including a repetitive pattern.

5. In the fourth paragraph, the first message information includes first movement information and second movement information, The step of generating the above first message information is: A step of determining the first movement information such that the movement of the first pixel position included in the region of interest is greater than the movement of the second pixel position included in the low-compensation region; and A method comprising the step of determining the second movement information such that the movement of the third pixel position included in the low-compensation area is greater than the movement of the fourth pixel position included in the maintenance area.

6. In at least one of paragraphs 1 to 5, The step (S220) of generating the above projection image is: A step of determining a vanishing point of the first image based on at least one of a field of view or a focal length of a lens through which the first image was captured; and A method comprising the step of generating the projection image by correcting the first image based on the vanishing point.

7. In any one of paragraphs 1 to 6, the first mesh information includes a movement vector, The step of generating the above first message information is: A step of dividing each of the first image and the projection image into a plurality of blocks; and A step of determining the movement vector based on a first block of the first image and a second block of the projection image corresponding to the first block, , method.

8. In any one of paragraphs 1 to 7, The step (S250) of generating the above output image is: A step of generating a second image by correcting the first image based on the first message information; a step of identifying whether the second image includes a blank area; and A method comprising the step of determining pixels of the above blank area.

9. In any one of paragraphs 1 to 8, the step of generating the projection image comprises: A step of determining depth information of an object included in the first image; and A method comprising the step of generating the projection image based on the depth information.

10. In any one of paragraphs 1 to 9, A step of acquiring a third image after the first image; A step of determining a motion vector between the third image and the first image; and A method comprising the step of generating second mesh information for the third image based on the first mesh information modified using the motion vector.

11. In an electronic device for correcting image distortion, At least one processor (1410) comprising a processing circuit; and A memory (1420) comprising one or more storage media storing at least one instruction, The electronic device is configured such that the at least one instruction is individually or collectively executed by the at least one processor (1410). Obtain the first image, Based on the first image, a projection image is generated in which perspective distortion of the first image is corrected, According to the plurality of objects included in the first image, a cognitive map is generated that divides the first image into a plurality of regions of the first plurality of types, Based on the projection image and the recognition map, first mesh information for correcting the first image is generated, wherein the first mesh information includes movement information of a plurality of pixel locations, Generating an output image by correcting the first image based on the first message information, A device wherein the above movement information is determined to different degrees depending on the type of the first plurality.

12. In the 11th paragraph, the plurality of areas correspond to the plurality of objects, The electronic device is configured such that the at least one instruction is individually or collectively executed by the at least one processor (1410). Recognize the plurality of objects included in the first image, Extract the above multiple areas, Based on the second plurality of types of the above plurality of objects, the first plurality of types are determined, A device for generating the cognitive map based on the first plurality of types.

13. In paragraph 12, A device wherein the above-described cognitive map includes at least one of a region of interest having a high priority for correction for perspective distortion, a maintenance region in which no correction for perspective distortion is performed, or a low-correction region having a low priority for correction for perspective distortion.

14. In paragraph 13, The electronic device is configured such that the at least one instruction is individually or collectively executed by the at least one processor (1410). Based on the first object included in the first region including a face, the first region among the plurality of regions is determined as the region of interest, Based on the second object included in the second region having a geometric configuration including at least one of a straight line, a curve, a polygon, a circle, or an ellipse, determining the second region among the plurality of regions as the maintenance region, A device that determines a third area among the plurality of areas as the low-compensation area based on the third object included in the third area including a repetitive pattern.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 1 to 10 on a computer.

Images