Electronic apparatus and control method thereof

Abstract

This electronic apparatus includes a memory that stores instructions, and at least one processor that executes the instructions, wherein the at least one processor is configured to execute the instructions to: obtain an input image; obtain line map information indicating positions of a plurality of lines in the input image; obtain 2D grid map information including a plurality of grids corresponding to the input image; identify a target line crossing the plurality of grids included in the 2D grid map information from among the plurality of lines included in the line map information; identify the size of the target line; identify a degree of radial distortion of the target line on the basis of the size of the target line; perform distortion correction on the target line on the basis of the degree of radial distortion; and obtain a corrected image corresponding to the input image on the basis of the distortion correction.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for correcting image distortion and a control method thereof.

[0002] Depending on the characteristics of the lens used for photography, images may contain distortion. For example, images captured with a wide-angle lens may appear to have a wider field of view. Distortion correction can be applied to the image to compensate for distortion caused by lens characteristics.

[0003] The correction method first identifies the user's face area to correct image distortion, and then corrects the distortion within that area. However, when the user's face area is difficult to identify or there are many user face areas, the distortion processing time can be unnecessarily long.

[0004] When performing distortion correction without identifying the user's face area, distortion correction can be performed based on the center of the image. Since distortion increases with distance from the center, areas farther from the center may require more correction than areas closer to the center. However, performing image correction in bulk can result in reduced correction quality or speed.

[0005] Additional aspects will be presented in part in the following description, and in part will be apparent from the description or may be learned by practice of the embodiments presented.

[0006] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof that first detects a line in an image and corrects distortion using the detected line.

[0007] According to one embodiment, an electronic device includes a memory storing instructions and at least one processor executing the instructions, wherein the at least one processor obtains an input image by executing the instructions, obtains line map information indicating positions of a plurality of lines in the input image, obtains 2D grid map information including a plurality of grids corresponding to the input image, identifies a target line intersecting the plurality of grids included in the 2D grid map information among the plurality of lines included in the line map information, identifies a size of the target line, identifies a degree of radial distortion of the target line based on the size of the target line, performs distortion correction for the target line based on the degree of radial distortion, and obtains a corrected image corresponding to the input image based on the distortion correction.

[0008] The at least one processor may, when a curve is identified in the line map information, change the curve into a straight line and update the line map information based on the changed straight line.

[0009] The at least one processor can change the position of one of the plurality of grids based on the position of one of the plurality of lines.

[0010] The at least one processor can identify an area in the input image where one of the plurality of lines is not identified, and delete a grid corresponding to the area where the line is not identified from the 2D grid map information.

[0011] The at least one processor may obtain distortion map information indicating a degree of radial distortion for a plurality of target lines included in the input image, and perform distortion correction for the plurality of target lines based on the distortion map information.

[0012] The at least one processor can identify location information of a target area on which distortion correction is to be performed based on the distortion map information, obtain transformation mask map information based on the location information of the target area, and perform the distortion correction based on the transformation mask map information.

[0013] The at least one processor can identify a reference point corresponding to the target line and a distortion point corresponding to the target line, obtain a first distance from the reference point to the distortion point, obtain a second distance from the reference point to a undistorted point corresponding to the target line based on the first distance and a lens coefficient corresponding to the input image, and perform the distortion correction based on the first distance and the second distance.

[0014] The at least one processor may obtain a third distance from the reference point to the stereoscopic projection position based on the first distance, the second distance, and the position of the target line, obtain a fourth distance from the reference point to the perspective stereoscopic projection position based on the third distance and the lens coefficient, and perform the distortion correction based on the third distance and the fourth distance.

[0015] The at least one processor may perform the distortion correction by changing the distortion point corresponding to the target line to a perspective stereoscopic projection position corresponding to the fourth distance.

[0016] The electronic device further includes a camera including a wide-angle lens, and the at least one processor can obtain the input image through the camera including the wide-angle lens.

[0017] According to one embodiment, a control method of an electronic device includes the steps of: acquiring an input image; acquiring line map information indicating positions of a plurality of lines in the input image; acquiring 2D grid map information including a plurality of grids corresponding to the input image; identifying a target line intersecting the plurality of grids included in the 2D grid map information among the plurality of lines included in the line map information; identifying a size of the target line; identifying a degree of radial distortion of the target line based on the size of the target line; performing distortion correction for the target line based on the degree of radial distortion; and acquiring a corrected image corresponding to the input image based on the distortion correction.

[0018] The above control method may further include a step of changing the curve into a straight line when a curve is identified in the line map information, and a step of updating the line map information based on the changed straight line.

[0019] The control method may further include a step of changing a position of one of the plurality of grids based on a position of one of the plurality of lines.

[0020] The control method may further include a step of identifying an area in the input image where one of the plurality of lines is not identified, and a step of deleting a grid corresponding to the area in which the line is not identified from the 2D grid map information.

[0021] The above control method further includes a step of obtaining distortion map information indicating a degree of radial distortion for a plurality of target lines included in the input image, and the step of performing distortion correction can perform the distortion correction based on the distortion map information.

[0022] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0023] FIG. 1 is a drawing for explaining an image correction operation according to one embodiment.

[0024] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0025] FIG. 3 is a block diagram illustrating the configuration of the electronic device of FIG. 2, according to one embodiment.

[0026] FIG. 4 is a drawing for explaining distortion of an object according to one embodiment.

[0027] FIG. 5 is a diagram for explaining a distortion correction operation for a face area according to one embodiment.

[0028] FIG. 6 is a drawing for explaining an operation of correcting distortion after line detection according to one embodiment.

[0029] FIG. 7 is a drawing for explaining an operation of performing perspective stereoscopic correction to remove distortion, according to one embodiment.

[0030] FIG. 8 is a diagram for explaining a calculation process of image correction according to one embodiment.

[0031] FIG. 9 is a diagram for explaining an image correction operation according to one embodiment.

[0032] FIG. 10 is a drawing for explaining an image correction operation according to one embodiment.

[0033] FIG. 11 is a diagram for explaining an image correction operation according to one embodiment.

[0034] FIG. 12 is a diagram illustrating a 2D grid for an input image, according to one embodiment.

[0035] FIG. 13 is a drawing for explaining an operation of correcting an image using a mask, according to one embodiment.

[0036] FIG. 14 is a drawing for explaining a correction operation for a video of a moving object, according to one embodiment.

[0037] FIG. 15 is a drawing for explaining a line detection operation according to one embodiment.

[0038] FIG. 16 is a drawing for explaining an image correction operation according to one embodiment.

[0039] FIG. 17 is a diagram for explaining a mesh transformation process according to one embodiment.

[0040] FIG. 18 is a drawing for explaining an image correction result according to one embodiment.

[0041] FIG. 19 is a diagram for explaining an operation of correcting an image based on line detection and a mask, according to one embodiment.

[0042] FIG. 20 is a drawing for explaining an image correction operation according to one embodiment.

[0043] FIG. 21 is a diagram for explaining a mesh transformation process according to one embodiment.

[0044] FIG. 22 is a drawing for explaining an image correction result according to one embodiment.

[0045] FIG. 23 is a drawing for explaining a device capable of applying image correction according to one embodiment.

[0046] FIG. 24 is a drawing for explaining a device capable of applying image correction according to one embodiment.

[0047] FIG. 25 is a drawing for explaining a method for controlling an electronic device according to one embodiment.

[0048] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0049] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0050] In this specification, expressions such as “has”, “may have”, “includes”, “may include”, “consists of” or “may be composed of” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0051] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B". The term "or" includes any combination of one or more of the relevant listed items.

[0052] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0053] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0054] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0055] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0056] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0057] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0058] FIG. 1 is a drawing for explaining an image correction operation according to one embodiment.

[0059] Referring to FIG. 1, an electronic device (100) may receive an input image (10). The input image (10) may contain distortion. The distortion may occur depending on the characteristics of the camera lens. While the subject is in a three-dimensional space, the image acquired through the camera may be two-dimensional. Distortion may occur in the subject or a portion of the subject in the two-dimensional image.

[0060] The electronic device (100) can correct distortion included in an input image (10). The electronic device (100) can obtain a corrected image (20) as a correction result. The electronic device (100) can use various algorithms to remove distortion included in the input image.

[0061] The electronic device (100) can be implemented as a smartphone, tablet, wearable device including a camera, TV, laptop, desktop, AR (or VR) glasses, HMD device, video camera, etc.

[0062] FIG. 2 is a block diagram illustrating an electronic device (100) according to one embodiment.

[0063] Referring to FIG. 2, the electronic device (100) may include at least one of a memory (110) and at least one processor (120).

[0064] The memory (110) can store an input image. The memory (110) can store a distortion correction model for performing distortion correction. At least one processor (120) can perform distortion correction on the input image using the distortion correction model stored in the memory (110). The distortion correction model can be an artificial intelligence model. The distortion correction model can include at least one of an RNN module or a CNN module. The distortion correction model can receive an input image as input data and obtain a corrected image with the distortion corrected as output data.

[0065] At least one processor (120) can obtain an input image, obtain 2D grid map information including a plurality of grids corresponding to the input image, identify a target line intersecting a plurality of grids included in the 2D grid map information among a plurality of lines included in the line map information, identify a size of the target line, identify a degree of radial distortion of the target line based on the size of the target line, perform distortion correction for the target line based on the degree of distortion, and obtain a corrected image corresponding to the input image based on the distortion correction.

[0066] At least one processor (120) can obtain an input image.

[0067] For example, at least one processor (120) can obtain an input image through a camera (190) included in the electronic device (100). At least one processor (120) can obtain a photographed image through the camera (190). At least one processor (120) can obtain the photographed image as an input image. At least one processor (120) can obtain a corrected image by inputting the input image into a distortion correction model stored in the memory (110).

[0068] For example, at least one processor (120) may receive an input image through an external device (or external server) connected to the electronic device (100). At least one processor (120) may be connected to the external device (or external server) through a communication interface (130).

[0069] At least one processor (120) can obtain line map information indicating the positions of a plurality of lines in an input image.

[0070] At least one processor (120) can analyze an input image to identify a line (or a line object). At least one processor (120) can analyze the input image to identify a plurality of lines. At least one processor (120) can identify the location of each of the plurality of lines identified in the input image. At least one processor (120) can obtain (or generate) line map information indicating the location of each of the plurality of lines.

[0071] A line may include a straight line and / or a curved line. At least one processor (120) may identify whether the input image includes a straight line and a curved line. A straight line may be classified as a first type, and a curved line may be classified as a second type.

[0072] Line map information may be information indicating the location of a line. At least one processor (120) may generate line map information using the size (or coordinates) of the input image. For example, if the size of the input image is 100*100 (number of pixels), the size of the line map information may also be 100*100.

[0073] The line map information may include coordinates at which at least one line is located in all coordinates of the input image.

[0074] Line map information can be described as line information, line map, line information, line location information, line location, etc.

[0075] At least one processor (120) can obtain 2D grid map information including a plurality of grids corresponding to an input image.

[0076] A grid can be a line or pattern divided into preset sizes (or units). A grid can be a guide line or guide pattern for dividing an image into preset units. A 2D grid can represent a grid used in two dimensions. An example related to a 2D grid is described in Fig. 12.

[0077] At least one processor (120) can obtain 2D grid map information including a plurality of grids for distinguishing an input image.

[0078] Multiple grids can include vertical grids and horizontal grids. When vertical and horizontal lines intersect, multiple grids can form a grid pattern.

[0079] The 2D grid map information may be information indicating the location of a line. At least one processor (120) may generate the 2D grid map information using the size (or coordinates) of the input image. For example, if the size of the input image is 100*100 (number of pixels), the size of the 2D grid map information may also be 100*100.

[0080] For example, a grid may represent either vertical grid lines or horizontal grid lines. A grid may be described as a grid line.

[0081] For example, a grid may represent a grid pattern formed by the intersection of vertical and horizontal lines. The grid may be described as a grid pattern.

[0082] At least one processor (120) can identify a target line that intersects a plurality of grids included in the 2D grid map information among a plurality of lines included in the line map information.

[0083] At least one processor (120) can identify whether any one of the plurality of lines included in the line map information intersects the grid. At least one processor (120) can compare the positions of each of the plurality of lines included in the line map information with the positions of each of the plurality of grids included in the 2D grid map information.

[0084] At least one processor (120) can compare the positions of lines and grids. If the positions of a first line among a plurality of lines and a first grid among a plurality of grids are identical, the line can be identified as a target line. At least one processor (120) can identify a plurality of target lines in an input image.

[0085] At least one processor (120) can identify the size of the target line.

[0086] At least one processor (120) can identify the total size (or total length) of a target line that intersects the grid. A single target line may intersect the grid multiple times. At least one processor (120) can identify the total length of the target line regardless of the number of intersections.

[0087] At least one processor (120) can identify the degree of radial distortion of the target line based on the size of the target line.

[0088] At least one processor (120) can identify (or acquire or calculate) a radial distortion degree indicating the degree of distortion of the target line using the size of the target line. The larger the size of the target line, the greater the degree of radial distortion. The electronic device (100) can identify a general, non-radial distortion degree. The degree of radial distortion can be described as a distortion degree.

[0089] At least one processor (120) can perform distortion correction for the target line based on the degree of radial distortion.

[0090] At least one processor (120) may perform distortion correction to remove (or reduce) distortion of the target line. At least one processor (120) may change the length of the target line. At least one processor (120) may change the position of an end point (edge ​​point) of the target line to reduce the length of the target line. At least one processor (120) may change the edge point closer (or further) from the reference point.

[0091] For example, at least one processor (120) may change the first edge point of the target line to the second edge point. The distance from the reference point to the second edge point may be changed to be greater than the distance from the reference point to the first edge point. Distortion may be corrected according to the change in the edge point.

[0092] At least one processor (120) can obtain a corrected image corresponding to the input image based on distortion correction.

[0093] At least one processor (120) can perform distortion correction on an input image to obtain a corrected image. At least one processor (120) can input (or provide) the input image to a distortion correction model stored in the memory (110). At least one processor (120) can obtain a corrected image with the distortion corrected from the distortion correction model.

[0094] At least one processor (120) can, when a curve is identified in the line map information, change the curve into a straight line and update the line map information based on the changed straight line.

[0095] Once a curve is identified in the input image, at least one processor (120) can perform a correction to change the curve into a straight line. At least one processor (120) can determine a target line representing the curve and change a line model (or line function) forming the target line to represent a straight line.

[0096] For example, an action that changes a curve into a straight line may include an action that changes the curve into a straight line at preset intervals. For example, a 5 cm curve may be changed into five straight lines separated by 1 cm.

[0097] For example, an action of changing a curve into a straight line may include an action of connecting edge points of a target line with a straight line. For example, a curve extending from a first position to a second position may be changed into a straight line connecting the first and second positions.

[0098] The operation of changing a curve into a straight line can be performed based on equation (1300) of Fig. 13. The operation of changing a curve into a straight line can be performed based on step S1630 of Fig. 16.

[0099] After changing the curve into a straight line, at least one processor (120) may update line map information. The updated line map information may not include information indicating the shape of the curve.

[0100] At least one processor (120) can change the position of the grid included in the 2D grid map information based on the position of the line included in the line map information.

[0101] At least one processor (120) can align the positions of the grids included in the 2D grid map information. The initially generated grids can be generated in preset units. For example, 2D grid map information of a size of 100*100 can include 9 vertical line grids and 9 horizontal line grids. The spacing between each grid can be 10 (pixels).

[0102] At least one processor (120) can change the position of the grid so that the position of the line included in the line map information and the position of the grid included in the 2D grid map information are the same.

[0103] For example, at least one processor (120) can change the position of each grid.

[0104] For example, at least one processor (120) can change the position of the representative grid. The representative grid can be changed according to the user's settings.

[0105] At least one processor (120) can identify the line closest to the grid. At least one processor (120) can change the grid position to the position of the line closest to the grid. The position change operation can be applied to each grid.

[0106] The line alignment action may not be required. The line alignment action may be omitted.

[0107] At least one processor (120) can identify an area in which a line is not identified in the line map information and delete a grid corresponding to the area in which a line is not identified in the 2D grid map information.

[0108] At least one processor (120) can identify an area (or coordinates) where no lines are identified based on line map information. At least one processor (120) can delete a grid existing in the identified area (or coordinates). At least one processor (120) can change (or update) the 2D grid map information so that no grid is included in the area where no lines are identified.

[0109] For example, only a portion of the grid may be deleted in areas where lines are not identified, either in the vertical or horizontal grid. The entire horizontal or vertical grid may not be deleted.

[0110] For example, pattern grids for areas where lines are not identified among multiple pattern grids may be deleted.

[0111] The grid deletion operation is described in Fig. 12.

[0112] At least one processor (120) can obtain distortion map information indicating the degree of distortion for a plurality of target lines included in an input image, and perform distortion correction based on the distortion map information.

[0113] At least one processor (120) can obtain (or generate) distortion map information indicating the degree of distortion for the entire input image. At least one processor (120) can identify target lines included in the input image and obtain distortion map information indicating the degree of distortion for each target line.

[0114] The distortion map information may be information indicating the position of a line. At least one processor (120) may generate the distortion map information using the size (or coordinates) of the input image. For example, if the size of the input image is 100*100 (number of pixels), the size of the distortion map information may also be 100*100.

[0115] At least one processor (120) can identify location information of a target area for which distortion correction is to be performed based on distortion map information, obtain transpose mask map information based on the location information of the target area, and perform distortion correction based on the transpose mask map information.

[0116] At least one processor (120) can determine whether the degree of distortion contained in the distortion map information is greater than or equal to a threshold value. At least one processor (120) can perform distortion correction only for target lines whose degree of distortion is greater than or equal to the threshold value. At least one processor (120) can identify target lines whose degree of distortion is greater than or equal to the threshold value, and perform distortion correction only for the identified target lines.

[0117] At least one processor (120) can identify a target area including a target line greater than or equal to a threshold value. At least one processor (120) can obtain transformation mask map information indicating the location of the target area.

[0118] A transformation mask can represent a tool required for image correction. The transformation mask can be filtering information for specifying the area where image correction is to be performed. The transformation mask can represent an area for specifying the location of the target area for distortion correction.

[0119] The transformation mask map information may include information about a target area in the input image on which distortion correction is to be performed.

[0120] The transformation mask map information may be information indicating the position of a line. At least one processor (120) may generate the transformation mask map information using the size (or coordinates) of the input image. For example, if the size of the input image is 100*100 (number of pixels), the size of the transformation mask map information may also be 100*100.

[0121] Descriptions of the transformation mask map information are described in FIGS. 13, 19, and 20.

[0122] At least one processor (120) can identify a reference point (Po) corresponding to a target line and a distortion point (Pd) corresponding to the target line.

[0123] At least one processor (120) can obtain a first distance (rd) from a reference point (Po) to a distortion point (Pd).

[0124] At least one processor (120) can obtain a second distance (ru) from a reference point (Po) to an undistorted point corresponding to a target line based on a first distance (rd) and a lens coefficient corresponding to an input image.

[0125] At least one processor (120) can perform distortion correction based on the first distance (rd) and the second distance (ru).

[0126] At least one processor (120) can obtain a third distance (rs) from a reference point (Po) to a stereoscopic projection position (Ps) based on the first distance (rd), the second distance (ru) and the position of the target line.

[0127] At least one processor (120) can obtain a fourth distance (rm) from a reference point (Po) to a perspective stereoscopic projection position (Pm) based on a third distance (rs) and a lens coefficient.

[0128] At least one processor (120) can perform distortion correction based on the third distance (rs) and the fourth distance (rm).

[0129] At least one processor (120) can perform distortion correction by changing the distortion point (Pd) corresponding to the target line to a perspective stereoscopic projection position (Pm) corresponding to the fourth distance (rm).

[0130] At least one processor (120) can change the distortion point (Pd), which is the first edge point of the target line, to the perspective stereoscopic projection position (Pm), which is the second edge point.

[0131] The change operation of the edge point is described in FIGS. 8 to 11.

[0132] The electronic device (100) may further include a camera including a wide-angle lens. At least one processor (120) may acquire an input image through the camera including the wide-angle lens. The wide-angle lens may refer to a lens having a field of view greater than a critical angle. The input image may be a wide-angle image.

[0133] The electronic device (100) may include a camera including an ultra-wide lens. An ultra-wide lens may refer to a lens having a field of view of 90 degrees or more.

[0134] The electronic device (100) can perform distortion correction on a user's face area where distortion has occurred without separately identifying the user's face area. Since there is no need to separately identify the user's face area, processing time can be shortened.

[0135] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0136] FIG. 3 is a block diagram for explaining the specific configuration of the electronic device (100) of FIG. 2.

[0137] Referring to FIG. 3, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).

[0138] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). The memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0139] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0140] At least one processor (120) can perform overall control operations of the electronic device (100). At least one processor (120) has a function of controlling overall operations of the electronic device (100).

[0141] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (120) may perform various functions by executing computer executable instructions stored in a memory (110).

[0142] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0143] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0144] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID (Service Set Identifier) ​​and session key, is first transmitted and received. This information is then used to establish a connection before various other information can be transmitted and received.

[0145] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0146] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0147] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0148] In one example, the communication interface (130) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

[0149] In one example, the communication interface (130) may utilize different communication modules to communicate with external devices, such as remote control devices, and external servers. For example, the communication interface (130) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is merely an example, and the communication interface (130) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.

[0150] The display (140) may be implemented as a display of various forms, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form, such as an amorphous silicon thin film transistor (a-si TFT), a low temperature poly silicon (LTPS) TFT, an organic TFT (OTFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional dispaly), etc. According to an embodiment of the present disclosure, the display (140) may include a bezel that houses the display panel as well as a display panel that outputs an image. According to an embodiment of the present disclosure, the bezel may include a touch sensor configured to detect user interaction.

[0151] According to one embodiment, the electronic device (100) may include a display (140). The electronic device (100) may directly display an acquired image or content on the display (140).

[0152] According to one embodiment, the electronic device (100) may not include a display (140). The electronic device (100) may be connected to an external display device and may transmit images or content stored in the electronic device (100) to the external display device. The electronic device (100) may transmit the images or content together with a control signal for controlling the display of the images or content on the external display device. The external display device may be connected to the electronic device (100) through a communication interface (130) or an input / output interface (160). For example, the electronic device (100) may not include a display, such as a set-top box (STB). The electronic device (100) may include only a small display capable of displaying simple information such as text information. The electronic device (100) may transmit images or content to the external display device via the communication interface (130) wired or wirelessly, or via the input / output interface (160).

[0153] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0154] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). An output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0155] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0156] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0157] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0158] The camera (190) is a device configured to capture a subject and generate a captured image, and the captured image includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, lens, infrared sensor, or the like.

[0159] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0160] FIG. 4 is a drawing for explaining distortion of an object according to one embodiment.

[0161] Referring to the embodiment (410) of Fig. 4, a sphere in three-dimensional space can be represented. It is assumed that a sphere composed of three-dimensional coordinates exists.

[0162] Referring to embodiment (420) of Fig. 4, it is shown that the sphere of embodiment (410) is expressed in two dimensions. When a rectilinear projection is applied to a three-dimensional sphere in a two-dimensional space, the shape of the sphere may become an ellipse. This is because distortion occurs due to the rectilinear projection.

[0163] Referring to the embodiment (430) of FIG. 4, the electronic device (100) can correct an ellipse into a circle to remove distortion. The electronic device (100) can remove distortion based on stereoscopic projection. When the ellipse included in the embodiment (420) is corrected into a circle, a corrected empty space may occur. If an empty space occurs near the boundary of the ellipse, the electronic device (100) can expand the surrounding background of the empty space. The electronic device (100) can fill the empty space with the surrounding background. The electronic device (100) can perform a correction operation by optimizing a mesh grid loss function.

[0164] FIG. 5 is a diagram for explaining a distortion correction operation for a face area according to one embodiment.

[0165] Referring to FIG. 5, the electronic device (100) can first identify a face area when correcting an image. The electronic device (100) can obtain an input image (S510). The input image may contain distortion.

[0166] The electronic device (100) can perform radial distortion correction (S520). Radial distortion correction may include an operation of performing distortion caused by a lens based on the center of the image.

[0167] The electronic device (100) can identify a face area (S530). The face area can represent an area that includes a human face.

[0168] As an example, the electronic device (100) can identify a face area in an input image.

[0169] For example, the electronic device (100) can identify a face area in the first corrected image obtained based on the correction result performed in step S520. The electronic device (100) can obtain location information (or coordinate information) of the face area.

[0170] The electronic device (100) can correct distortion based on the identified facial region (S540). The electronic device (100) can correct distortion present in the facial region based on the location of the identified facial region.

[0171] FIG. 6 is a drawing for explaining an operation of correcting distortion after line detection according to one embodiment.

[0172] Referring to FIG. 6, the electronic device (100) can obtain an input image (610). The input image (610) may include distortion.

[0173] Referring to image (611), the electronic device (100) can detect a line in the input image (610). The line can be described as an edge. The electronic device (100) can detect the line included in the image and obtain position information of the line.

[0174] The electronic device (100) can perform perspective-stereographic correction based on the input image (610) and the positions of the detected lines. A description related to perspective-stereographic correction is provided in FIGS. 7 and 8 . The electronic device (100) can obtain a corrected image (620) with distortion removed through perspective-stereographic correction.

[0175] FIG. 7 is a drawing for explaining an operation of performing perspective stereoscopic correction to remove distortion, according to one embodiment.

[0176] Referring to FIG. 7, the electronic device (100) can acquire an input image (S710). The electronic device (100) can detect a line from the input image (S720). The electronic device (100) can acquire location information of the detected line. The electronic device (100) can acquire location information (or coordinate information) indicating where the line is located in the input image.

[0177] The electronic device (100) can perform perspective stereoscopic correction based on the position information of the input image and line (S730).

[0178] Perspective stereoscopic correction may include perspective correction and stereoscopic correction. Perspective correction may include operations that perform perspective-based corrections. Stereoscopic correction may refer to corrections that remove distortions that occur when representing objects in three-dimensional space in two-dimensional space. Perspective stereoscopic correction may be described as mixed perspective stereoscopic correction.

[0179] For example, the electronic device (100) can first perform perspective correction and then perform stereoscopic correction.

[0180] For example, the electronic device (100) can perform stereoscopic correction first and then perspective correction.

[0181] For example, the electronic device (100) can perform perspective correction and stereoscopic correction together.

[0182] The electronic device (100) can obtain a corrected image with distortion removed through perspective stereoscopic correction (S740).

[0183] FIG. 8 is a diagram for explaining a calculation process of image correction according to one embodiment.

[0184] Referring to the embodiment (810) of FIG. 8, the electronic device (100) can identify an optical center and an object position in object space.

[0185] Example (810) may represent a plane through which an input ray passes with respect to a distorted pixel and a primary optical axis. For example, the input ray may represent a target line among a plurality of lines. The plane may intersect with other projections and be orthogonal to the XY plane (image plane). The correction may be performed in a radial direction.

[0186] The electronic device (100) can identify an image center point (principal point, Po), a distortion point (distorted image point, Pd), and an undistorted point (undistorted image point, Pu) from the optical center. The electronic device (100) can identify a stereoscopic projection point (Ps). The electronic device (100) can identify a mixed projection point (Pm).

[0187] The mixed projection point (Pm) can be written as a perspective stereoscopic projection position.

[0188] The distortion point (Pd) is the actual input of the camera and can be a point relative to the input without any processing (or correction). The distortion point (Pd) can typically reflect distortion related to the radial direction of the lens.

[0189] The undistorted point (Pu) may be an image point obtained by applying camera calibration. For example, the camera calibration may include the Brown-Conrady distortion model. The electronic device (100) may obtain the undistorted point (Pu) by removing radial distortion included in the distorted image.

[0190] The mixed projection point (Pm) can be closer to the image center point (Po) than the undistorted point (Pu) and further than the stereoscopic projection point (Ps).

[0191] The distortion point (Pd) may be further away from the image center point (Po) than the stereoscopic projection point (Ps).

[0192] Embodiment (810) of Fig. 8 illustrates an embodiment in which the mixed projection point (Pm) is corrected to be further away from the reference point (Po) than the distortion point (Pd). Depending on the type of distortion, there may also be an embodiment in which the mixed projection point (Pm) is corrected to be closer to the reference point (Po) than the distortion point (Pd).

[0193] f can represent the focal length of the optical system.

[0194] Embodiment (820) of FIG. 8 may represent a mathematical formula used to calculate the points shown in embodiment (810).

[0195] The electronic device (100) can calculate the distance (ru) from the image center point (Po) to the undistorted point (Pu) on the image plane.

[0196] The electronic device (100) can calculate the distance (ru) using equation (1).

[0197] Distance (rd) can represent the distance from the image center point (Po) to the distortion point (Pd).

[0198] (xu, yu) can represent the coordinates of the undistorted point (Pu).

[0199] (xu, yu) may represent coordinates used in a pinhole camera. The pinhole camera may be described as a first type camera. The first type camera may represent a camera that does not include a lens. (xu, yu) may be described as a first type coordinate.

[0200] (xu, yu) can be an estimate of the ideal projection (xp, yp) in a pinhole camera.

[0201] (xs, ys) can represent the coordinates of the stereoscopic projection point (Ps).

[0202] (xs, ys) may represent coordinates used in a fisheye camera. The fisheye camera may be described as a second type of camera. The second type of camera may include a fisheye lens. (xs, ys) may be described as a second type of coordinate.

[0203] There are several types of fisheye lenses. Equation (3) applies only to stereoscopic projection. Other types may require additional transformations, as they require stereoscopic correction for facial enhancement. Fisheye cameras may require an anti-distortion step.

[0204] The electronic device (100) can obtain the (xu, yu) coordinates of the first type based on the coordinates (xs, ys) corresponding to the second type through equation (2).

[0205] The electronic device (100) can obtain coordinates (xs, ys) corresponding to the second type based on the (xu, yu) coordinates of the first type through equation (3).

[0206] Depending on the type of lens used in the camera, either equation (2) or (3) can be used. Both types of lenses can be directly mixed and used in hardware design without calculation.

[0207] The electronic device (100) can mutually calculate the distorted coordinates and the undistorted coordinates using equations (2) and (3).

[0208] The electronic device (100) can calculate the distance (rm) from the image center point (Po) to the mixed projection point (Pm) on the image plane using equation (4) based on the distance (rs) and lens correction coefficients (ks1, ks2, ks3).

[0209] An electronic device (100) can generate a regular grid of texture coordinates from a distorted image. The electronic device (100) can identify a blended projection point (Pm) corresponding to a grid node. The electronic device (100) can identify a distorted but regular grid and a corrected mesh node. The electronic device (100) can render the corrected mesh and use the original image as a texture target image.

[0210] The mixed projection point (Pm) can exist between the undistorted point (Pu) and the stereoscopic projection point (Ps). In areas where lines are not discernible (e.g., areas where faces are discernible), the mixed projection point (Pm) can be closer to the image center point (Po) than the distortion point (Pd).

[0211] The electronic device (100) can use various models to obtain a mixed projection point (Pm).

[0212] According to one embodiment, the electronic device (100) can obtain a mixed projection point (Pm) using a polynomial model. A mathematical approach related to this may be Equation (4) of embodiment (820). The electronic device (100) can apply the polynomial model to stereoscopic projection. The electronic device (100) can perform calculations in the order of rd, ru, rs, and rm.

[0213] According to an implementation example, the electronic device (100) may omit the operation of calculating ru and perform calculations in the order of rd, rs, and rm.

[0214] k may be a stereo projection correction coefficient. k may represent a vector of polynomial coefficients. If no correction is required, the value of k may be 0. The electronic device (100) may use a larger value of k as the distortion increases. For example, k1 may be a preset constant (e.g., -1 / 4). k (stereo projection correction coefficient) may be described as a lens correction coefficient or a visual appearance correction coefficient.

[0215] In various embodiments, k may represent a spatial function k(x, y). Knowing k(x, y) on an image allows one to compute a correction shift for a ray passing through P(x, y). The electronic device (100) may utilize k(x, y) together with a distortion map. The distortion map may have a lower resolution than the original image.

[0216] The electronic device (100) can obtain a mixed projection point (Pm) based on perspective projection in an area near the line. The electronic device (100) can obtain a mixed projection point (Pm) based on stereoscopic shooting in an area other than the area near the line.

[0217] The electronic device (100) can divide the input image into a regular grid. The electronic device (100) can calculate k based on all cells. If a line intersects a specific cell, the electronic device (100) can compensate for the specific cell. If multiple lines intersect a specific cell, the electronic device (100) can compensate for the specific cell based on the line with the longest and most curved degree.

[0218] The electronic device (100) can detect a line in an image using a line detection model (e.g., a Line Segment Detector (LSD). A line (or candidate line) detected by the line detection model may have radial distortion reflected therein. The electronic device (100) can identify a loss function that depends on k related to a distortion map for a distorted line (or candidate line). The electronic device (100) can identify a coefficient k by minimizing the loss function.

[0219] Assume that the coordinates of the distorted line pixel are (xi, yi). The electronic device (100) can identify the distortion correction model using equation (4) of the embodiment (820). The electronic device (100) can obtain the coefficient (a, b, c) values ​​using the linear equation ax+by+c=0. The electronic device (100) can perform linear regression analysis by substituting ax+by+c=0 into equation (4) of the embodiment (820). The electronic device (100) can minimize the residual sum based on the linear regression analysis. The electronic device (100) can identify the coefficient k for the distortion map through minimizing the residual sum. If divergence is observed, the curve cannot be considered a straight line. If the loss is within the acceptable range, the electronic device (100) can complete the minimization of the residual sum.

[0220] The electronic device (100) can perform a correction for perspective projection using k. The mixed projection point (Pm) can be close to the undistorted point (Pu) only along the line. In areas other than the line, the mixed projection point (Pm) can be closer to the stereoscopic projection position (Ps).

[0221] According to one embodiment, the electronic device (100) can obtain a mixed projection point (Pm) using an additive model. A mathematical approach related thereto is described in embodiment (1330) of FIG. 13. The distortion map can be associated with a coefficient λ.

[0222] FIG. 9 is a diagram for explaining an image correction operation according to one embodiment.

[0223] Referring to FIG. 9, the electronic device (100) can obtain an input image (S910). The input image may include distortion. The electronic device (100) can detect lines included in the input image (S920).

[0224] The electronic device (100) can determine whether a curved line (curve, bended line) is detected among the detected lines (S930).

[0225] When a curved line is detected (S930-Y), the electronic device (100) can add an area corresponding to the curved line to the distortion map (S940). The area corresponding to the curved line can include information indicating the location where the curved line is detected. The electronic device (100) can determine the area corresponding to the curved line as an area requiring correction.

[0226] A distortion map can be information indicating the location of a part where distortion is detected in the overall coordinates of the image.

[0227] The electronic device (100) can add an area for a curved line to the distortion map.

[0228] For example, a map before the area for a curved line is reflected may be described as a “distortion map,” and a map after the area for a curved line is reflected may be described as an “updated distortion map.”

[0229] For example, a map before the area for a curved line is reflected may be described as a “first distortion map,” and a map after the area for a curved line is reflected may be described as a “second distortion map.”

[0230] When a distortion map reflecting an area for a curved line is obtained, the electronic device (100) can correct a 2D grid (S950). The electronic device (100) can generate a 2D grid based on an input image. The electronic device (100) can perform correction on the 2D grid for mesh-based transform. The electronic device (100) can perform a warping function on the 2D grid. Warping can include an operation of deforming a specific portion (specific area) included in the input image. Warping can indicate an operation of deforming a 2D grid corresponding to a specific area of ​​the input image for image correction. The 2D grid can be described as a mesh 2D grid.

[0231] If a curved line is not detected (S930-N), the electronic device (100) may correct the 2D grid (S950). The electronic device (100) may delete the grid for areas where lines are not detected. If a curved line is not detected, the distortion map may not include information about areas where distortion exists.

[0232] The electronic device (100) can divide the original image into a regular rectangular grid including nodes (Pd(i,j)). i, j may be grid node indices in the x, y directions. i may be greater than 0 and less than N. j may be greater than 0 and less than M. N, M may represent the same or different constants. Each node may represent a point in the distorted space. The grid may be expressed as a matrix of 2D points. By using equations (3) and (4) of the embodiment (820) of FIG. 8, the electronic device (100) may obtain a corrected mesh including a mixed projection point (Pm). In performing step S950, the electronic device (100) may use a distortion map as input data. By performing step S950, the electronic device (100) may obtain matrix data including a corrected 2D grid as output data.

[0233] In various embodiments, a triangular grid may be used.

[0234] The electronic device (100) can optimize a 2D grid using distortion energy optimization (S960). The electronic device (100) can obtain an optimized 2D grid. Step S960 may be omitted depending on the implementation example.

[0235] The electronic device (100) may further modify (or adjust) the corrected mesh through step S960. For example, the electronic device (100) may adjust node positions in a video sequence. Analyzing multiple frames individually may result in mesh jitter. The corrected node positions may contain small errors. These errors may arise due to quantization, algorithmic fixed thresholds, and other reasons. To reduce these errors, the electronic device (100) may perform step S960.

[0236] The electronic device (100) can correct an input image based on a corrected 2D grid or an optimized 2D grid (S970). The electronic device (100) can perform distortion correction on the input image using the corrected (or optimized) 2D grid. The distortion correction may include performing a warping function. The electronic device (100) can obtain a corrected image.

[0237] The electronic device (100) can perform step S970 based on a texture warping technique.

[0238] Once the correction image is acquired, the electronic device (100) can determine whether an additional image is input (S980). If it is determined that an additional image is input (S980-Y), the electronic device (100) can repeat steps S910 to S980. For example, a video may include multiple images. The electronic device (100) can perform steps S910 to S980 for a video including consecutive image frames.

[0239] FIG. 10 is a drawing for explaining an image correction operation according to one embodiment.

[0240] Referring to FIG. 10, the electronic device (100) can obtain an input image (S1005). The electronic device (100) can obtain line map information indicating line positions in the input image (S1010). The electronic device (100) can determine whether a curve is identified in the line map information (S1015).

[0241] Once a curve is identified (S1015-Y), the electronic device (100) can change the curve into a straight line (S1020). The electronic device (100) can update line map information (S1021).

[0242] If the curve is not identified (S1015-N), the electronic device (100) can obtain 2D grid map information corresponding to the input image (S1025).

[0243] The electronic device (100) can align (or change) the grid (or grid line) of the 2D grid map information based on the line position of the line map information (S1030).

[0244] The electronic device (100) can delete grids in which lines included in the line map information are not identified in the 2D grid map information (S1035). Operations related to this are described in FIG. 12.

[0245] The electronic device (100) can identify a target line that intersects a grid (or a line of the grid) included in 2D grid map information among a plurality of lines included in line map information (S1040).

[0246] The electronic device (100) can identify the size (or length) of the target line (S1045). The electronic device (100) can identify the degree (or coefficient) of radial distortion of the target line based on the size of the target line (S1050).

[0247] The electronic device (100) can obtain distortion map information indicating the degree of distortion in the input image (S1055).

[0248] FIG. 11 is a drawing for explaining an image correction operation according to one embodiment.

[0249] Referring to FIG. 11, after performing step S1055 of FIG. 10, the electronic device (100) can obtain location information of a target area for performing distortion correction based on distortion map information (S1160).

[0250] The electronic device (100) can obtain transformation mask map information based on the location information of the target area (S1165).

[0251] The electronic device (100) can correct distortion based on transformation mask map information (S1170). The distortion correction may be perspective stereoscopic distortion correction.

[0252] The electronic device (100) can identify a reference point (Po) corresponding to a target line and a distortion point (Pd) corresponding to the target line (S1171).

[0253] The electronic device (100) can obtain a first distance (rd) from a reference point (Po) to a distortion point (Pd) (S1172).

[0254] The electronic device (100) can obtain a second distance (ru) from the reference point (Po) to the undistorted point (Pu) based on the first distance (rd) and the lens coefficient (k1) (S1173).

[0255] The electronic device (100) can obtain a third distance (rs) from a reference point (Po) to a stereoscopic projection position (Ps) based on the first distance (rd), the second distance (ru), and the position of the target line (S1174).

[0256] The electronic device (100) can obtain the fourth distance (rm) from the reference point (Po) to the perspective stereoscopic projection position (Pm) based on the third distance (rs) and the lens coefficient (Ks) (S1175).

[0257] The electronic device (100) can change the distortion point (Pd) of the target line to a perspective stereoscopic projection position (Pm) corresponding to the fourth distance (rm) (S1176).

[0258] The electronic device (100) can obtain a corrected image corresponding to the input image based on distortion correction (S1180).

[0259] FIG. 12 is a diagram illustrating a 2D grid for an input image, according to one embodiment.

[0260] Embodiment (1211) of FIG. 12 may represent a 2D grid corresponding to an input image. The electronic device (100) may generate a 2D grid based on the input image. The electronic device (100) may generate a 2D grid having a preset size (or unit) based on each pixel included in the input image. The electronic device (100) may acquire a first intermediate image including the 2D grid.

[0261] Embodiment (1212) of FIG. 12 may represent a deformed 2D grid. The electronic device (100) may deform the 2D grid based on detected lines. The electronic device (100) may delete the grid for areas where lines are not detected. The electronic device (100) may acquire a second intermediate image in which the grid for areas where lines are not detected is deleted. The operation of deforming the 2D grid may be described as an operation of warping the 2D grid. The calculation method related thereto may utilize Equations (3) and (4) of embodiment (820) of FIG. 8.

[0262] The electronic device (100) can perform distortion correction based on the second intermediate image.

[0263] FIG. 13 is a drawing for explaining an operation of correcting an image using a mask, according to one embodiment.

[0264] Referring to FIG. 13, the electronic device (100) can warp a 2D grid using stereographic projection to perform distortion correction.

[0265] The electronic device (100) can correct a curved line using equation (1300) of FIG. 13. The electronic device (100) can correct a curved line into a line (a straight line). The electronic device (100) can apply a curved line to a line model using equation (1300) of FIG. 13.

[0266] Equation (1300) of FIG. 13 may represent an example of correction using a polynomial model. According to one example, the electronic device (100) may obtain a mixed projection point (Pm) using a polynomial model.

[0267] n can be the index of the detected line.

[0268] u can represent the pixel coordinates of a line without distortion.

[0269] d can represent the pixel coordinates of the distorted line.

[0270] k3 and k5 can be preset coefficients. They can be adjusted according to user settings. k3 and k5 can reflect weights to maintain a balance between stereoscopic and perspective projection, indicating the degree to which curves are transformed into straight lines. Adjusting k3 and k5 may result in changes in certain curves while leaving other areas unchanged.

[0271] The electronic device (100) can detect lines and calculate line model coefficients. The electronic device (100) can fit lines (different from image lines, as lines representing the grid themselves) for each 2D grid in the input image to the line model. For example, the electronic device (100) can change the position of the 2D grid to match the lines detected in the input image with the grid lines of the 2D grid map information.

[0272] The electronic device (100) can identify a line model using the line equation coefficients (a, b, c) and the distortion model coefficients (k3, k5). Applying a curved line to the line model may include an operation of converting the curve into a straight line. The operation of converting the curve into a straight line may include an operation of calculating equations (3) and (4) of the embodiment (820) of FIG. 8.

[0273] Equation (1300) of Fig. 13 may correspond to Equation (4) of Example (820) of Fig. 8.

[0274] According to various embodiments, the electronic device (100) can be compensated using an additive model. The electronic device (100) can obtain a mixed projection point (Pm) using an additive model.

[0275] The electronic device (100) may acquire a transformation mask. The transformation mask may represent a grid or pattern (or area) for transforming an image. The transformation mask may be information for specifying an area for transforming a specific area (or part) of the image. The transformation mask may be an identifying tool for a user to indicate a specific area for distortion correction. The transformation mask may include a filtering function for indicating a specific area.

[0276] An electronic device (100) can obtain an input image (1310). The electronic device (100) can obtain a transformation mask (1315) for the input image (1310). The electronic device (100) can use the transformation mask (1315) to specify an area for which distortion is to be corrected. The electronic device (100) can use the transformation mask (1315) to obtain a corrected image (1320).

[0277] The electronic device (100) can additionally utilize equation (1330) in obtaining a corrected image (1320). Equation (1330) can represent a projective mixture model.

[0278] ρ can represent the radius of polar coordinates.

[0279] λ can represent a constant (or weight) related to the angle of distortion.

[0280] r can represent the distance from the center point to a specific location of an image pixel.

[0281] R can represent the size (or distance) of the distortion.

[0282] rm, ru, rs, and f may correspond to the embodiment of Fig. 8. fs may be an effective stereoscopic focal length. λ may be a linear mixing coefficient in the range [0, 1]. The linear mixing coefficient may be used to determine a model suitable for the image region.

[0283] The additive model may be a more theoretically proven equation than the polynomial model. Because the additive model exhibits linearity compared to the polynomial model, it may be intuitive. The additive model may depend only on the parameter λ.

[0284] If a line intersects a cell, λ can be 1. If a line does not intersect a cell, λ can be 0.

[0285] FIG. 14 is a drawing for explaining a correction operation for a video of a moving object, according to one embodiment.

[0286] Referring to FIG. 14, it is assumed that the video includes a plurality of consecutive images (1401, 1402, 1403). The electronic device (100) can perform distortion correction on each of the plurality of consecutive images. The video can show a person's face moving from the right area to the left area.

[0287] The electronic device (100) can perform distortion correction on a video using equation (1410).

[0288] The equation (Et) may represent node smoothing energy. The equation (Et) may be used to prevent node jitter. The equation (Et) may be a calculation equation used in step S960 of FIG. 9. A single image frame may not require a smoothing operation. Image content including multiple image frames may require a smoothing operation. The electronic device (100) may optimize the 2D grid by minimizing the equation (Et). The equation (Et) may represent a distortion energy function.

[0289] Vi^n can represent the i-node of the n-th frame. Multiple frames can be divided into grids for further warping.

[0290] FIG. 15 is a drawing for explaining a line detection operation according to one embodiment.

[0291] The electronic device (100) can detect a line using an input image (1510). The electronic device (100) can detect a line included in the input image (1510). The electronic device (100) can distinguish the type (or type) of the line. The type of the line can include a straight line (first type) and a curved line (second type).

[0292] The electronic device (100) can determine straight lines and curves based on an input image (1510). The electronic device (100) can determine the positions of straight lines and curves in the input image (1510). The electronic device (100) can obtain line map information (1511) including the positions of straight lines and curves based on the input image (1510).

[0293] The electronic device (100) can display a first UI at a straight line position and a second UI at a curved line position in an input image (1510). The first UI and the second UI included in the line map information (1511) can be provided in different colors (or types).

[0294] FIG. 16 is a drawing for explaining an image correction operation according to one embodiment.

[0295] Referring to FIG. 16, the electronic device (100) can acquire an input image (S1610). The electronic device (100) can search for a curve in the input image (S1620). The electronic device (100) can correct the searched curve into a straight line (S1630). Equation (1) can be used for the correction operation.

[0296] The electronic device (100) can divide the input image into a 2D grid (S1640). The electronic device (100) can evaluate radial distortion by a weighted average (k3, k5) based on the length of a line intersecting the grid. (k3, k5) may denote coefficients representing distortion.

[0297] The electronic device (100) can obtain first type 2D grid information for image warping.

[0298] The electronic device (100) can acquire second type 2D grid information for distortion evaluation. The 2D grid used in step S1640 may be a second type grid for distortion evaluation. The second type 2D grid may include a grid of cells related to distortion correction coefficients. The second type 2D grid may be used to prevent distortion changes that suddenly change depending on the image.

[0299] The electronic device (100) can perform an interpolation (k3, k5) operation on the input image (S1650).

[0300] The electronic device (100) can obtain distortion coefficients from a low-resolution distortion map to calculate the corrected positions of all mesh nodes using bilinear interpolation. Distortion coefficients may be required because the resolutions of the distortion map and the 2D mesh are different.

[0301] The electronic device (100) can evaluate the distortion grid to apply stereoscopic correction to the node coordinates and correct the distortion of the line based on the distortion coefficients (k3, k5) (S1660).

[0302] The electronic device (100) can perform distortion correction on the entire image (S1670). The electronic device (100) can obtain a final corrected image through an unwarping operation (S1670). The unwarping operation can be described as an image distortion removal operation. The electronic device (100) can obtain an image without distortion through step S1670.

[0303] FIG. 17 is a diagram for explaining a mesh transformation process according to one embodiment.

[0304] The code (1700) of Fig. 17 can be used for mesh transformation. The electronic device (100) can perform mesh transformation using a polynomial distortion model using the code (1700).

[0305] The electronic device (100) can obtain a distortion map including polynomial coefficient vectors (k3, k5). The electronic device (100) can input the distortion map as input data to a first model including a code (1700). The electronic device (100) can obtain matrix data (uniformMesh) representing regular grid points and matrix data (undistortedMesh) representing undistorted grid points as output data through the first model.

[0306] FIG. 18 is a drawing for explaining an image correction result according to one embodiment.

[0307] The image (1810) of FIG. 18 may represent an input image with distortion. For example, distortion may exist in areas (1811, 1812). The electronic device (100) may perform distortion correction on the entire image (1810).

[0308] The image (1820) of FIG. 18 may represent a corrected image in which distortion correction has been performed. Distortion may be removed (or corrected) in areas (1821, 1822).

[0309] FIG. 19 is a diagram for explaining an operation of correcting an image based on line detection and a mask, according to one embodiment.

[0310] Referring to FIG. 19, the electronic device (100) can obtain an input image (1910). The electronic device (100) can detect a line based on the input image (1910). The electronic device (100) can obtain line map information (1911) for the input image (1910).

[0311] The electronic device (100) can obtain a transformation mask (1915) based on an input image (1910) and line map information (1911). The electronic device (100) can specify (or filter) an area for which distortion is to be corrected based on the transformation mask (1915).

[0312] The electronic device (100) can perform distortion correction based on an input image (1910), line map information (1911), and a transformation mask (1915). The electronic device (100) can perform distortion correction to obtain a corrected image (1920).

[0313] FIG. 20 is a drawing for explaining an image correction operation according to one embodiment.

[0314] Referring to FIG. 20, the electronic device (100) can search for a line for an input image (S2010). The electronic device (100) can calculate a radial undistortion coefficient (S2020).

[0315] The electronic device (100) can calculate a transformation mask (or mixing mask) (S2030).

[0316] The electronic device (100) can evaluate the distortion grid (S2040). The electronic device (100) can use Equation (1) to apply radial distortion to the node coordinates. Equation (1) can correspond to Equation (1330) of FIG. 13.

[0317] The electronic device (100) can perform distortion correction (S2050). The electronic device (100) can perform a restoration operation for image distortion correction. The electronic device (100) can perform an unwarping operation for distortion correction.

[0318] FIG. 21 is a diagram for explaining a mesh transformation process according to one embodiment.

[0319] The code (2100) of Fig. 21 can be used for mesh transformation. The electronic device (100) can perform mesh transformation using an additive model using the code (2100).

[0320] The electronic device (100) can obtain a distortion map including polynomial coefficient vectors (k3, k5). The electronic device (100) can input the distortion map as input data to a second model including a code (2100). The electronic device (100) can obtain matrix data (uniformMesh) representing regular grid points and matrix data (undistortedMesh) representing undistorted grid points as output data through the second model.

[0321] The λ used in the additive model can represent information about the lines forming the mask. The code (2100) can use exponential weighting.

[0322] FIG. 22 is a drawing for explaining an image correction result according to one embodiment.

[0323] The image (2210) of FIG. 22 may represent an input image with distortion. For example, distortion may exist in areas (2211, 2212). The electronic device (100) may perform distortion correction on the entire image (2210).

[0324] The image (2220) of FIG. 22 may represent a corrected image in which distortion correction has been performed. Distortion may be removed (or corrected) in areas (2221, 2222).

[0325] FIG. 23 is a drawing for explaining a device capable of applying image correction according to one embodiment.

[0326] Referring to the embodiment (2300) of FIG. 23, the electronic device (100) can perform distortion correction when performing a face recognition function.

[0327] The electronic device (100) may be a device that performs a facial recognition function. To perform the facial recognition function, the electronic device (100) may acquire an input image containing a facial object. The input image acquired by the electronic device (100) may include a distorted facial object. If the facial object is distorted, the performance of the facial recognition function may deteriorate.

[0328] The electronic device (100) can perform distortion correction on an input image to obtain a corrected image with improved (or removed) distortion. The electronic device (100) can correct distortion in a face area by performing correction on the entire image without separately identifying the face area.

[0329] FIG. 24 is a drawing for explaining a device capable of applying image correction according to one embodiment.

[0330] Referring to the embodiment (2400) of FIG. 24, the electronic device (100) can perform distortion correction when providing an AR (Augmented Reality) service.

[0331] The electronic device (100) may be a device that performs AR functions. The electronic device (100) may acquire an input image containing various objects to provide the AR function. The electronic device (100) may perform distortion correction on the acquired input image and acquire a corrected image as a result of the correction.

[0332] The electronic device (100) can identify multiple objects based on the corrected image. The electronic device (100) can search for (or acquire) information corresponding to the multiple objects. Since the corrected image has distortion removed, the electronic device (100) can improve its performance in acquiring AR information.

[0333] For example, the electronic device (100) may identify the first object as an apple object. The electronic device (100) may obtain information (text: apple) about the apple object. The electronic device (100) may provide an AR image in which information (text: apple) about the apple object is displayed at a location corresponding to the first object.

[0334] For example, the electronic device (100) may be a refrigerator. The electronic device (100) may obtain an input image including various objects existing inside the refrigerator. Since the camera lens that captures the inside of the refrigerator is fixed, some of the objects included in the input image may be distorted. The electronic device (100) may correct the distortion of the input image to obtain a corrected image. The electronic device (100) may obtain information about each object based on the corrected image, and the electronic device (100) may provide an AR image that additionally includes information about each object in the input image.

[0335] FIG. 25 is a drawing for explaining a method for controlling an electronic device according to one embodiment.

[0336] Referring to FIG. 25, a control method of an electronic device includes a step of obtaining an input image (S2505), a step of obtaining line map information indicating positions of a plurality of lines in the input image (S2510), a step of obtaining 2D grid map information including a plurality of grids corresponding to the input image (S2515), a step of identifying a target line intersecting a plurality of grids included in the 2D grid map information among the plurality of lines included in the line map information (S2520), a step of identifying a size of the target line (S2525), a step of identifying a degree of radial distortion of the target line based on the size of the target line (S2530), a step of performing distortion correction for the target line based on the degree of radial distortion (S2535), and a step of obtaining a corrected image corresponding to the input image based on the distortion correction (S2540).

[0337] The control method may further include a step of changing the curve into a straight line when a curve is identified in the line map information, and a step of updating the line map information based on the changed straight line.

[0338] The control method may further include a step of changing a position of a grid included in the 2D grid map information based on a position of a line included in the line map information.

[0339] The control method may further include a step of identifying an area in which a line is not identified in the line map information and a step of deleting a grid corresponding to an area in which a line is not identified in the 2D grid map information.

[0340] The control method further includes a step of obtaining distortion map information indicating a degree of radial distortion for a plurality of target lines included in an input image, and the step of performing distortion correction can perform distortion correction based on the distortion map information.

[0341] The control method further includes a step of identifying position information of a target area where distortion correction is to be performed based on distortion map information and a step of obtaining transformation mask map information based on the position information of the target area, and the step of performing distortion correction can perform distortion correction based on the transformation mask map information.

[0342] The step of performing distortion correction may include identifying a reference point corresponding to a target line and a distortion point corresponding to the target line, obtaining a first distance from the reference point to the distortion point, obtaining a second distance from the reference point to a undistorted point corresponding to the target line based on the first distance and a lens coefficient corresponding to the input image, and performing distortion correction based on the first distance and the second distance.

[0343] The step of performing distortion correction may include obtaining a third distance from a reference point to a stereoscopic projection position based on a first distance, a second distance, and a position of a target line, obtaining a fourth distance from a reference point to a perspective stereoscopic projection position based on the third distance and a lens coefficient, and performing distortion correction based on the third distance and the fourth distance.

[0344] The step of performing distortion correction can perform distortion correction by changing the distortion point corresponding to the target line to a perspective stereoscopic projection position corresponding to the fourth distance.

[0345] The electronic device further includes a camera including a wide-angle lens, and the step of acquiring an input image may acquire the input image through the camera including the wide-angle lens.

[0346] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0347] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0348] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0349] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0350] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0351] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0352] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, Memory for storing instructions; and At least one processor executing the above instructions; At least one of the above processors executes the instructions, Obtain the input image, Obtain line map information indicating the locations of multiple lines in the above input image, Obtain 2D grid map information including multiple grids corresponding to the input image, Identifying a target line that intersects the plurality of grids included in the 2D grid map information among the plurality of lines included in the line map information, Identify the size of the above target line, Identify the degree of radial distortion of the target line based on the size of the target line, Distortion correction is performed for the target line based on the degree of radial distortion, An electronic device that obtains a corrected image corresponding to the input image based on the distortion correction.

2. In paragraph 1, At least one processor of the above, If a curve is identified in the above line map information, change the curve to a straight line, An electronic device that updates the line map information based on the changed straight line.

3. In paragraph 1, At least one processor of the above, An electronic device that changes the position of one of the plurality of grids based on the position of one of the plurality of lines.

4. In paragraph 1, At least one processor of the above, Identifying an area in the input image where one of the above multiple lines is not identified, An electronic device that deletes a grid corresponding to an area in which the line is not identified in the above 2D grid map information.

5. In paragraph 1, At least one processor of the above, Obtain distortion map information indicating the degree of radial distortion for multiple target lines included in the input image, An electronic device that performs distortion correction for the plurality of target lines based on the distortion map information.

6. In paragraph 5, At least one processor of the above, Identifying location information of a target area where distortion correction is to be performed based on the distortion map information, Obtain transformation mask map information based on the location information of the target area, An electronic device that performs distortion correction based on the above transformation mask map information.

7. In paragraph 1, At least one processor of the above, Identify a reference point corresponding to the target line and a distortion point corresponding to the target line, Obtain the first distance from the above reference point to the above distortion point, A first distance is obtained based on a lens coefficient corresponding to the input image, and a second distance is obtained from the reference point to the undistorted point corresponding to the target line. An electronic device that performs distortion correction based on the first distance and the second distance.

8. In paragraph 7, At least one processor of the above, Obtain a third distance from the reference point to the stereoscopic projection position based on the first distance, the second distance, and the position of the target line, The third distance is obtained based on the lens coefficient and the fourth distance from the reference point to the perspective stereoscopic projection position, An electronic device that performs distortion correction based on the third distance and the fourth distance.

9. In paragraph 8, At least one processor of the above, An electronic device that performs distortion correction by changing the distortion point corresponding to the target line to a perspective stereoscopic projection position corresponding to the fourth distance.

10. In paragraph 1, The above electronic device, A camera including a wide-angle lens; further comprising: At least one processor of the above, An electronic device that acquires the input image through the camera including the wide-angle lens.

11. In a method for controlling an electronic device, Step of obtaining an input image; A step of obtaining line map information indicating the positions of a plurality of lines in the input image; A step of obtaining 2D grid map information including a plurality of grids corresponding to the input image; A step of identifying a target line intersecting the plurality of grids included in the 2D grid map information among the plurality of lines included in the line map information; A step of identifying the size of the above target line; A step of identifying the degree of radial distortion of the target line based on the size of the target line; A step of performing distortion correction for the target line based on the degree of radial distortion; and A control method, comprising: a step of obtaining a corrected image corresponding to the input image based on the distortion correction; 12. In paragraph 11, The above control method is, When a curve is identified in the above line map information, a step of changing the curve into a straight line; and A control method further comprising: a step of updating the line map information based on the changed straight line.

13. In paragraph 11, The above control method is, A control method further comprising: a step of changing a position of one of the plurality of grids based on a position of one line of the plurality of lines.

14. In paragraph 11, The above control method is, A step of identifying an area in the input image in which one of the plurality of lines is not identified; and A control method further comprising: a step of deleting a grid corresponding to an area in which the line is not identified in the 2D grid map information.

15. In paragraph 11, The above control method is, A step of obtaining distortion map information indicating the degree of radial distortion for a plurality of target lines included in the input image is further included; The step of performing the above distortion correction is: A control method for performing distortion correction for the plurality of target lines based on the distortion map information.

Images