Electronic device performing image inpainting and method of operating the same

Abstract

A method for performing image inpainting to restore a restoration target region within an image is provided in an electronic device comprising: a step of determining a restoration target region within an original image; a step of generating a restoration target image to be restored from the original image based on the determined restoration target region; a step of obtaining a mask image representing the restoration target region; a step of determining whether to extract structural features of the restoration target region based on attributes of the restoration target region; a step of obtaining structural vectors composed of one or more lines and one or more junctions by applying the restoration target image and the mask image to a first model for extracting structural features of the restoration target image including structural features of the restoration target region; and a step of obtaining an inpainted image in which the restoration target region of the restoration target image is restored by applying a structural vector image transformed from the restoration target image, the mask image, and the structural vectors to a second model for restoring the restoration target image.

Description

Technology Field

[0001] An electronic device for performing image inpainting, which restores an image by filling pixels within an image, and a method of operating the same are provided. Background Technology

[0002] As an image processing method for image editing in electronic devices, image editing functions utilizing artificial intelligence are being provided. As an example of an AI-based image editing function, electronic devices provide image inpainting technology that restores images by naturally filling in damaged or empty areas within the image. By utilizing image inpainting technology to restore damaged areas or delete specific objects within an image, electronic devices can provide users with an improved image editing experience.

[0003] When an electronic device performs image inpainting, noise such as blurriness or distortion may exist within the inpainted image depending on the shape, location, size, etc., of damaged or empty areas.

[0004] In image inpainting technology, we propose a method to generate a clear and natural inpainted image without being affected by the shape, location, or size of the area to be restored. The problem to be solved

[0005] The disclosed embodiments are intended to solve the aforementioned problem and to provide an electronic device for performing image inpainting and a method of operating the same. means of solving the problem

[0006] As a technical means for achieving the technical problem described above, the first aspect of the present disclosure provides a method for performing image inpainting in which an electronic device restores a region to be restored within an image, comprising: determining a region to be restored within an original image; generating a target image to be restored from the original image based on the determined target image to be restored; obtaining a mask image representing the target image to be restored; determining whether to extract structural features of the target image to be restored based on attributes of the target image to be restored; obtaining structural vectors composed of one or more lines and one or more junctions by applying the target image to be restored and the mask image to a first model for extracting structural features of the target image to be restored, which includes structural features of the target image to be restored; and obtaining an inpainted image in which the target image to be restored has the restored region to be restored by applying a structural vector image transformed from the target image to be restored, the mask image, and the structural vectors to a second model for restoring the target image to be restored.

[0007] Additionally, a second aspect of the present disclosure relates to an electronic device for performing image inpainting, comprising: a communication interface; a memory for storing one or more instructions; The electronic device may include at least one processor that executes one or more instructions stored in the memory, wherein the at least one processor determines a region to be restored within an original image, generates a target image to be restored from the original image based on the determined target region to be restored, obtains a mask image representing the target region to be restored, determines whether to extract structural features of the target region to be restored based on the attributes of the target region to be restored, obtains structural vectors composed of one or more lines and one or more junctions by applying the target image to be restored and the mask image to a first model for extracting structural features of the target image to be restored including structural features of the target region to be restored, and obtains an inpainted image in which the target region to be restored of the target image to be restored is restored by applying a structural vector image transformed from the target image to be restored, the mask image, and the structural vectors to a second model for restoring the target image to be restored.

[0008] Additionally, a third aspect of the present disclosure may provide a recording medium storing a program that enables the method of the first aspect to be performed. Brief explanation of the drawing

[0009] FIG. 1 is a diagram schematically illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image in which a restoration target area within the image is restored. FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to restore a target area and obtain an inpainted image. FIG. 3 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to determine a restoration target area within an original image. FIG. 4 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to determine a restoration target area within an original image. FIG. 5 is a drawing for explaining a first model and a second model used by an electronic device according to one embodiment of the present disclosure to acquire an inpainted image. FIG. 6 is a diagram illustrating the architecture of a first model for extracting structural features of an image to be restored, according to one embodiment of the present disclosure. FIG. 7 is a diagram illustrating a structure vector obtained by an electronic device according to one embodiment of the present disclosure using a first model. FIG. 8 is a drawing for explaining the architecture of a second model that outputs an inpainted image according to one embodiment of the present disclosure. FIG. 9 is a drawing for illustrating another architecture of a second model that outputs an inpainted image according to one embodiment of the present disclosure. FIG. 10 is a drawing for illustrating an inpainted image obtained by an electronic device according to one embodiment of the present disclosure using a second model. FIG. 11 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image. FIG. 12 is a flowchart illustrating another method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image. FIG. 13 is a diagram illustrating the training of a structure vector element extraction unit and a Hough transform unit of a first model when an electronic device according to one embodiment of the present disclosure trains a first model. FIG. 14 is a diagram for explaining the content related to the generative adversarial network of the first model when an electronic device according to one embodiment of the present disclosure trains the first model. FIG. 15 is a diagram illustrating a method for determining the input order of training data for training a first model when an electronic device according to one embodiment of the present disclosure trains a first model. FIG. 16 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to generate training data for training a first model. FIG. 17 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an image in which structure vectors are pseudo-labeled. FIG. 18 is another diagram illustrating a method for generating verification data to verify the accuracy of an inpainted image acquired by an electronic device according to one embodiment using a trained second model. FIG. 19 is a drawing illustrating a method for an electronic device according to one embodiment of the present disclosure to train a second model to acquire an inpainted image. FIG. 20 is a drawing for illustrating embodiments in which an electronic device according to one embodiment of the present disclosure performs image inpainting. FIG. 21 is a drawing for illustrating other embodiments in which an electronic device according to one embodiment of the present disclosure performs image inpainting. FIG. 22 is a drawing for explaining an inpainted video generated by an electronic device according to one embodiment performing image inpainting. FIG. 23 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure. FIG. 24 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure. FIG. 25 is a flowchart of a method for an electronic device according to one embodiment of the present disclosure to obtain an inpainted image from an original image by interacting with a server. Specific details for implementing the invention

[0010] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0011] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0012] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification. Additionally, terms including ordinal numbers, such as "first" or "second," used in this specification may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0013] When a part of a specification is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part" or "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0014] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0015] In the present disclosure, the image to be restored refers to an image that is subject to restoration by applying an image inpainting algorithm according to the embodiments described below. The image to be restored may include a region to be restored.

[0016] In the present disclosure, the restoration target area refers to an area that is subject to restoration by applying an image inpainting algorithm according to the embodiments described below. The restoration target area refers to an area within the restoration target image where pixel values ​​have been changed, so that visual meaning, features, etc., are not identifiable. The inpainting algorithm according to the disclosed embodiments may generate pixels corresponding to the restoration target area and fill the restoration target area. The pixels filled in the restoration target area may be pixels generated so that the visual features are naturally connected with the surrounding areas of the restoration target area. By restoring the restoration target area of ​​the restoration target image according to the disclosed embodiments, an inpainted image can be obtained.

[0017] In the present disclosure, a structure vector refers to a vector representing the structural characteristics of structures existing within an image. A structure vector may be composed of points and lines. In the present disclosure, a structure vector of a restoration target area refers to a vector representing the structural characteristics of structures determined to exist within a restoration target area, obtained by inferring structures determined to exist within the restoration target area for a restoration target area where pixel values ​​have been changed and visual meanings, features, etc., are not identifiable.

[0018] FIG. 1 is a diagram schematically illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image in which a restoration target area within the image is restored.

[0019] Referring to FIG. 1, an electronic device (2000) according to one embodiment can acquire a restoration target image (110), extract geometric features of structures within the restoration target image to acquire structure vectors (120), and generate pixels corresponding to a restoration target area (115) of the restoration target image (110) using the acquired structure vectors (120) to acquire an inpainted image (130) in which the restoration target area (115) of the restoration target image (110) is filled with generated pixels. In this case, the structure vectors (120) may include structure vectors of the restoration target area (115).

[0020] In one embodiment, when an electronic device (2000) performs image inpainting to restore a restoration target area (115) of a restoration target image (110), the electronic device (2000) may restore the image based on surrounding pixels of the restoration target area (115) in order to restore the restoration target area (115) in accordance with the context of the restoration target image. In this case, if the size of the restoration target area (115) becomes larger than a predetermined standard, the information obtained from the surrounding pixels of the restoration target area (115) is insufficient, so the restoration target area (115) may be restored blurry or distorted. An electronic device (2000) according to one embodiment may restore the restoration target area (115) by inferring content (e.g., structures, etc.) that is determined to exist within the restoration target area using a generative adversarial neural network. Additionally, the electronic device (2000) can restore the restoration target area (115) by using a structure vector of the restoration target area obtained by inferring a structure that is determined to exist within the restoration target area (115).

[0021] An electronic device (2000) according to one embodiment may be a device capable of photographing a subject using a camera. For example, the electronic device (2000) may be a device such as a smartphone, tablet PC, laptop PC, desktop PC, TV, etc., but is not limited thereto.

[0022] In one embodiment, the electronic device (2000) may include a camera module composed of a plurality of cameras. The camera module may include a depth camera for acquiring a depth image and an RGB camera for acquiring an RGB image. The electronic device (2000) may acquire an image using the camera module.

[0023] In one embodiment, the electronic device (2000) can acquire a restoration target image (110) including a restoration target area (115) in which the pixel values ​​of a portion of the image have been changed.

[0024] The image to be restored (110) can be obtained in various ways.

[0025] In one embodiment, the electronic device (2000) can obtain a restoration target image (110) including a restoration target area (115) in which the pixel values ​​of a portion of the image have been changed from another electronic device (e.g., a server (3000)).

[0026] In one embodiment, the electronic device (2000) may acquire a restoration target image (110) generated based on user input of the electronic device (2000). For example, if the user of the electronic device (2000) wishes to delete a specific object within an undamaged original image, the electronic device (2000) may generate a restoration target image (110) based on user input. The electronic device (2000) may receive input from the user of the electronic device (2000) selecting a portion of the original image, and by changing the pixel values ​​of the pixels corresponding to the selected portion of the original image, generate a restoration target image (110) including a restoration target area (115). In this case, the original image may be one that the electronic device (2000) acquired from another electronic device (e.g., a server (3000)), or one that the electronic device (2000) acquired by capturing a scene using a camera module.

[0027] In one embodiment, the electronic device (2000) can extract structural features of structures within the image to be restored (110). The electronic device (2000) can extract structural features such as points, lines, and surfaces of structures within the image to be restored to obtain structural vectors (120) composed of one or more lines and one or more junctions. In this case, the extracted structural vectors may include structural vectors of the area to be restored.

[0028] An electronic device (2000) according to one embodiment can generate pixels corresponding to a restoration target area (115) within a restoration target image (110) based on structure vectors (120). The electronic device (2000) can generate pixels corresponding to a restoration target area (115) by guiding a frame structure of a structure corresponding to a restoration target area (115) based on structure vectors (120). The electronic device (2000) can obtain an inpainted image (130) in which the restoration target area (115) is restored to match the context of the restoration target image (110) by filling the generated pixels into the restoration target area (115).

[0029] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to restore a target area and obtain an inpainted image.

[0030] In step S210, an electronic device (2000) according to one embodiment can determine a restoration target area within an original image. The electronic device (2000) can acquire an original image and determine a restoration target area within the original image.

[0031] In one embodiment, the electronic device (2000) may acquire an original image to be inpainted. The original image may include, for example, an image in which some areas within the image are damaged or an image containing an object to be deleted. The electronic device (2000) may acquire the original image by photographing the subject using the camera of the electronic device (2000) or by receiving the original image from another electronic device (e.g., a server (3000)).

[0032] In the present disclosure, user input may be of various forms. For example, if the electronic device (2000) includes a touchscreen including a display and a touchpad, the user input may be touch data received through the touchscreen. In another example, if there is a separate display and input device connected to the electronic device (2000), the user input may be input data received through the input device.

[0033] In one embodiment, the electronic device (2000) can determine a restoration target area based on user input. The electronic device (2000) receives user input selecting a damaged area or an area to be deleted within the original image, and can determine a restoration target area based on the received user input.

[0034] For example, the electronic device (2000) may receive user input selecting a damaged area within the original image. Based on the received user input, the electronic device (2000) may determine the selected area as the area to be restored.

[0035] For example, the electronic device (2000) may receive user input (e.g., closed curve type input, brush type input, etc.) for selecting a specific area to delete within the original image. Based on the received user input, the electronic device (2000) may determine the selected specific area as the area to be restored.

[0036] For example, a user of the electronic device (2000) may receive user input selecting a specific object to delete a specific object within an original image. Based on the input in which the user selects an object within the original image, the electronic device (2000) may detect the boundary of an area corresponding to an object within the original image and determine the area corresponding to the detected object as a restoration target area.

[0037] In one embodiment, an electronic device (2000) can recognize objects within an original image and segment regions corresponding to the objects within the original image. The electronic device (2000) receives input from a user selecting a portion of the image and can determine the segmentation region related to the received user input as the region to be restored.

[0038] In step S220, an electronic device (2000) according to one embodiment can generate a restoration target image based on a restoration target area determined in step S210.

[0039] The electronic device (2000) can generate a restoration target image including a restoration target area by changing the pixel values ​​of pixels corresponding to a determined restoration target area. In this case, the electronic device (2000) can change the pixel values ​​of pixels corresponding to a portion of the area to a preset value. For example, the electronic device (2000) can change the pixel values ​​of pixels corresponding to a portion of the area to 0, or change the pixel values ​​of pixels corresponding to a portion of the area to the average value of the pixel values ​​of all images in the training data. However, the pixel values ​​to be changed are not limited to this.

[0040] In step S230, an electronic device (2000) according to one embodiment may acquire a mask image representing a region to be restored. In this case, the mask image may be composed of a region to be restored and a region not to be restored. The mask image may include masking information that masks the region to be restored. The masking information may include information indicating the location of the region to be restored in the original image (e.g., coordinate information), but is not limited thereto.

[0041] In step S240, the electronic device (2000) according to one embodiment may determine whether to extract structural features of the restoration target area based on the attributes of the restoration target area.

[0042] In one embodiment, the electronic device (2000) may determine whether to extract structural features of the area to be restored based on the size of the area to be restored. For example, the electronic device (2000) may decide to extract structural features of the area to be restored if the number of pixels corresponding to the area to be restored is greater than or equal to a preset number.

[0043] In one embodiment, the electronic device (2000) may determine whether to extract structural features of the area to be restored based on the ratio that the area to be restored occupies within the entire area of ​​the image to be restored. For example, the electronic device (2000) may decide to extract structural features of the area to be restored if the number of pixels of the area to be restored relative to the number of pixels of the image to be restored is greater than or equal to a preset value.

[0044] In one embodiment, the electronic device (2000) may determine whether to extract structural features of the restoration target area based on the width of the restoration target area. The electronic device (2000) may obtain skeletal lines of the restoration target area by applying a thinning algorithm to find pixels representing the skeletal line of the restoration target area, and may decide to extract structural features of the restoration target area if the width of the restoration target area, measured in a vertical direction relative to the skeletal line of the restoration target area, is greater than or equal to a preset value. For example, the electronic device (2000) may decide to extract structural features of the restoration target area if the maximum value of the width within the restoration target area is greater than or equal to a preset value.

[0045] In step S250, an electronic device (2000) according to one embodiment can obtain structure vectors composed of one or more lines and one or more junction points by applying the image to be restored and the mask image to the first model. The first model may be a neural network model for receiving the image to be restored and the mask image as inputs and detecting geometric features such as points, lines, and surfaces representing the frame structure of structures within the image to be restored. The structure vectors may include structural features of the area to be restored, which are inferred structural features determined to exist within the area to be restored for the area to be restored where pixel values ​​within the image to be restored have been changed. In this case, the structural features of the area to be restored may be inferred from structural features of an area other than the area to be restored within the entire area of ​​the image to be restored. Based on the input image to be restored and the mask image, the first model may output structure vectors related to the entire image to be restored, including the area to be restored.

[0046] The restoration target area of ​​the restoration target image is an area where pixel values ​​corresponding to a part of the restoration target image have been changed, making it impossible to visually understand the meaning of that part of the area, and where structures, etc. exist within the restoration target area, making it impossible to know what structural features exist. The electronic device (2000) can receive the restoration target image and infer the structural features of the restoration target area at once by using a first model capable of extracting structural features of the restoration target area, rather than a method such as detecting edge regions within the restoration target image and inferring the structure of the restoration target area where structural features are not detected based on the detected edge regions.

[0047] In one embodiment, training data for training a first model may consist of training images including a region to be restored and mask images corresponding to the training images and representing the region to be restored of the training images. Each of the training images may have structure vectors corresponding to the training images labeled on it. An electronic device (2000) may train a first model using the training data. Additionally, the electronic device (2000) may receive the trained first model from another electronic device (e.g., a server (3000)).

[0048] In step S260, an electronic device (2000) according to one embodiment can obtain an inpainted image in which the restoration target area of ​​the restoration target image is restored by applying a structure vector image converted from a restoration target image, a mask image, and structure vectors to a second model. The structure vectors may be output from a first model. The electronic device (2000) can convert the structure vectors output from the first model into a structure vector image. The electronic device (2000) can convert the structure vectors into a structure vector image based on coordinate information corresponding to the positions of one or more lines and one or more junction points and one or more lines and one or more junction points. For example, the electronic device (2000) can generate a structure vector image by converting the structure vectors into a binary map. However, the format of the structure vector image converted from the structure vectors is not limited thereto, and the electronic device (2000) can generate a structure vector image by converting the structure vectors into an image representation type of vector.

[0049] The inpainted image output from the second model may be an image in which pixels corresponding to the area to be restored are generated, and the area to be restored is filled with the generated pixels. Additionally, the inpainted image may be an image restored to conform to the context of the image to be restored, in which the area to be restored naturally connects with other areas outside the area to be restored.

[0050] FIG. 3 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to determine a restoration target area within an original image.

[0051] In describing FIG. 3, steps S310 to S320 correspond to step S210 of FIG. 2, and step S330 may correspond to step S220 of FIG. 2.

[0052] In step S310, an electronic device (2000) according to one embodiment may receive input from a user selecting a portion of an original image (302). In one embodiment, the user of the electronic device (2000) may want to delete a specific area or a specific object within the original image. The electronic device (2000) may receive user input from a user selecting a specific area or a specific object.

[0053] In step S320, an electronic device (2000) according to one embodiment may determine a restoration target area (307) based on input received from a user. The electronic device (2000) may receive user input from a user specifying an area to be deleted within the original image (302), determine the restoration target area (307), and obtain a mask image (308) representing the restoration target area (307).

[0054] In one embodiment, the electronic device (2000) may determine an area corresponding to user input as the area to be restored (307). User input may be input in various ways. For example, the electronic device (2000) may receive input in which the user draws a closed curve within the image so that the closed curve area within the image becomes the area to be restored (307). Based on the received input, the electronic device (2000) may determine the closed curve area within the original image (302) as the area to be restored (307). In another example, the electronic device (2000) may receive brush input in which the user masks a specific area within the image. Based on the received input, the electronic device (2000) may determine the area within the original image (302) where the brush input was received as the area to be restored (307).

[0055] In one embodiment, the electronic device (2000) can determine the restoration target area (307) by inferring the restoration target area (307) based on user input. In this case, the electronic device (2000) may use a neural network model that detects object regions based on sparse data input.

[0056] For example, the electronic device (2000) may receive input (e.g., dot type input, scribble type input, etc.) in which a user selects a portion of an object to be deleted within the original image (302). Based on the received input, the electronic device (2000) may detect the boundary line of the object area selected by the user within the original image (302) and determine the detected area as the area to be restored (307).

[0057] In one embodiment, a user of the electronic device (2000) may want to restore an original image (304) in which some areas within the image are damaged (e.g., an original image obtained by scanning an old photograph). In this case, the electronic device (2000) may receive user input to mask the area to be restored within the original image (304) in which some areas within the image are damaged. Based on the received input, the electronic device (2000) may determine the area masked by the user within the original image (304) in which some areas within the image are damaged as the area to be restored.

[0058] In the embodiments described above, embodiments in which the electronic device (2000) determines a restoration target area based on user input have been described, but the user input of the embodiments described above is merely an example, and the electronic device (2000) can receive various types of user input to determine a restoration target area.

[0059] In step S330, an electronic device (2000) according to one embodiment can generate a restoration target image by changing pixels corresponding to the restoration target area (307) based on a determined restoration target area (307).

[0060] In one embodiment, the electronic device (2000) can change the pixel values ​​of pixels corresponding to the restoration target area (307), which is determined based on user input selecting a portion of the original image (302). For example, the electronic device (2000) can calculate the average value of the pixel values ​​of all images in the training data. The electronic device (2000) can obtain the restoration target image (306) by changing the pixel values ​​of the pixels corresponding to the restoration target area (307) to the average value of the pixel values ​​of all images in the training data. In another example, the electronic device (2000) can obtain the restoration target image (306) by changing the pixel values ​​of the pixels corresponding to the restoration target area (307) to 0.

[0061] Additionally, the electronic device (2000) may acquire a mask image (308) representing a restoration target area (307). In this case, the mask image (308) may consist of a restoration target area (307) and a non-restoration target area. The mask image may include masking information that masks the restoration target area (307).

[0062] Meanwhile, there may be multiple restoration target areas (307) determined from the original image (302). When there are multiple restoration target areas, the electronic device (2000) can restore each of the multiple restoration target areas according to the embodiments described below. In this case, the electronic device (2000) can restore each determined restoration target area whenever one restoration target area is determined, or restore the determined multiple restoration target areas collectively after multiple restoration target areas are determined.

[0063] In one embodiment, the electronic device (2000) may determine whether to extract structural features of the image to be restored (306) based on the ratio that the size of the determined area to be restored (307) occupies within the entire area of ​​the image to be restored (306). If there are multiple areas to be restored, the electronic device (2000) may identify each area to be restored and determine whether to extract structural features of the image to be restored (306). This will be described further in the description of FIGS. 11 and FIGS. 12.

[0064] FIG. 4 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to determine a restoration target area within an original image.

[0065] In describing FIG. 4, steps S410 to S430 correspond to step S210 of FIG. 2, and step S440 may correspond to step S220 of FIG. 2.

[0066] In step S410, an electronic device (2000) according to one embodiment can recognize objects within an original image and segment regions corresponding to the objects within the original image. In this case, the electronic device (2000) may use a neural network model that detects objects within an image based on image input.

[0067] For example, an electronic device (2000) can receive an original image (402) as input and use a neural network model to detect recognizable objects present in the image. The electronic device (2000) can segment regions corresponding to the recognized objects.

[0068] In step S420, an electronic device (2000) according to one embodiment may receive input from a user selecting a portion of the original image (402).

[0069] In one embodiment, the electronic device (2000) can display segmented regions and receive user input selecting at least some of the segmented regions.

[0070] In one embodiment, the electronic device (2000) receives user input selecting a portion of the original image (402) and can match segmented regions corresponding to the selected portion.

[0071] In step S430, an electronic device (2000) according to one embodiment can determine a recovery target region based on segmented regions related to the received input.

[0072] In one embodiment, the electronic device (2000) displays segmented regions within the original image (402) through object recognition and can determine a restoration target region based on user input selecting at least some of the segmented regions. For example, the electronic device (2000) can display all objects recognized within the original image (402), such as people in the foreground region of the original image (402) and people in the background region of the original image (402), in a guided form so that the user can select them. Among the objects within the original image (402), if the user selects people in the background region of the original image (402), the electronic device (2000) can determine the segmented regions corresponding to the people in the background region as restoration target regions.

[0073] In one embodiment, the electronic device (2000) receives user input selecting a portion of an original image (402) and can match segmented regions corresponding to the selected portion. The electronic device (2000) can determine the matched segmented regions as regions to be restored.

[0074] In step S440, an electronic device (2000) according to one embodiment can generate a restoration target image by changing pixels corresponding to the restoration target area based on a determined restoration target area.

[0075] In one embodiment, the electronic device (2000) can change the pixel values ​​of pixels corresponding to the area to be restored. For example, the electronic device (2000) can calculate the average value of the pixel values ​​of all images in the training data. The electronic device (2000) can obtain the image to be restored (404) by changing the pixel values ​​of pixels corresponding to the area to be restored to the average value of the pixel values ​​of all images in the training data. In another example, the electronic device (2000) can obtain the image to be restored (404) by changing the pixel values ​​of pixels corresponding to the area to be restored to 0.

[0076] Additionally, the electronic device (2000) may acquire a mask image (406) representing a region to be restored. In this case, the mask image (406) may consist of a region to be restored and a region not to be restored. The mask image may include masking information that masks the region to be restored.

[0077] Meanwhile, there may be multiple restoration target areas determined from the original image (402). If there are multiple restoration target areas, the electronic device (2000) can restore each of the multiple restoration target areas according to the embodiments described below. In this case, the electronic device (2000) may restore each determined restoration target area whenever one restoration target area is determined, or restore the determined multiple restoration target areas collectively after multiple restoration target areas are determined.

[0078] FIG. 5 is a drawing for explaining a first model and a second model used by an electronic device according to one embodiment of the present disclosure to acquire an inpainted image.

[0079] Referring to FIG. 5, an electronic device (2000) according to one embodiment can obtain a structure vector (540) by extracting structural features of structures within an image using a restoration target image (510) and a mask image (520) obtained according to the above embodiments. The structure vector (540) may be data composed of one or more lines and one or more junction points that extract structural features of structures existing within the image.

[0080] The electronic device (2000) inputs the image to be restored (510) and the mask image (520) into a first model (530), which is a neural network model, and can obtain a structure vector (540) output from the first model (530). The first model (530) may be a neural network model trained to extract structural features of the image to be restored (510). Additionally, the first model (530) can extract structural features of the area to be restored within the image to be restored (510). Structural features of the area to be restored refer to a structure vector (540) generated by predicting structural features within the area to be restored for the area to be restored, where pixel values ​​within the image to be restored (510) have been changed and structural features have disappeared.

[0081] The electronic device (2000) inputs a structure vector image (545), obtained by converting the restoration target image (510), the mask image (520), and the structure vector (540) output from the first model, into a second model (550), which is a neural network model, and can obtain an inpainted image (560) using the second model (550). The second model (550) can obtain an inpainted image by generating pixels to be filled in the restoration target area within the restoration target image (510) and filling the restoration target area with the generated pixels. In this case, when the second model (550) generates pixels to be filled in the restoration target area, the structure vector (540) of the restoration target area, which is a structural feature of the restoration target area, can be used.

[0082] FIG. 6 is a diagram illustrating the architecture of a first model for extracting structural features of an image to be restored, according to one embodiment of the present disclosure.

[0083] Referring to FIG. 6, the first model (600) can receive a restoration target image (610) and a mask image (620) as inputs and output a structure vector (640).

[0084] The first model (600) may include a structure vector element extraction unit (602), a Hough transform unit (604), a structure vector verification unit (606), and a Generative Adversarial Network (GAN) (608). Additionally, the first model (600) may be a model trained based on training data generated through a series of data processing processes. The method by which the electronic device (2000) trains the first model will be described in the description of FIGS. 13 through 18.

[0085] The structure vector element extraction unit (602) may include a plurality of neural network layers. The structure vector element extraction unit (602) may receive a restoration target image (610) and a mask image (620) as inputs and extract structure vector elements within the restoration target image. Here, the restoration target image (610) includes a restoration target area, and the mask image (620) refers to an image that masks the restoration target area. The structure vector element extraction unit (602) may extract features from the input images and obtain a plurality of image maps representing junction points where lines that can form the structure of structures within the image meet, and line segments that are candidates for lines that can form the structure of structures within the image. The obtained image maps may be converted one-to-one into junction point coordinate elements and line segment coordinate elements and transmitted to the structure vector verification unit (606).

[0086] The Hough transform unit (604) includes a plurality of Hough transform modules, and each Hough transform module may include a plurality of neural network layers. Each Hough transform module may receive features of an image as input and extract linear features. Each Hough transform module may extract linear features from the features of the input image, combine the extracted linear features with the features of the input image, and transmit them to a neural network layer included in the structure vector element extraction unit (602). Additionally, the Hough transform unit (604) may transmit the linear features extracted from the input image to the structure vector verification unit (606).

[0087] A detailed description of the training of the Hough transform unit (604) during the training of the first model (600) will be further described in the description of FIG. 13.

[0088] The structure vector verification unit (606) can match the junction points and line segments output from the structure vector element extraction unit (602) and verify the matched lines using linear features output from the Hough transform unit (604). The structure vector verification unit (606) can output the verified structure vectors as a structure vector (640) representing structural features within the image to be restored (610). In this case, the structure vector (640) represents structural features within the image to be restored (610) and may include a structure vector of the area to be restored, which predicts structural features for the area to be restored where some regions within the image to be restored (610) have changed pixel values.

[0089] A Generative Adversarial Network (GAN) (608) may include a generator model that outputs an RGB image generated from a target image (610) to be restored, and a discriminator model that receives an RGB image generated from the target image (610) or a ground truth RGB image and determines whether the input image is genuine. The discriminator model of the Generative Adversarial Network (608) may be trained to determine whether the RGB image input to the discriminator model is genuine during the training process of the first model (600). Specifically, the discriminator model may be trained to classify the input image into the 'Real' class if the input RGB image is determined to be a ground truth RGB image that has not undergone any processing, and to classify the input image into the 'Fake' class if the input RGB image is determined to be an RGB image generated by processing the target image to be restored, which was generated by the generator model. Additionally, the generator model of the generative adversarial network (608) can be trained to generate images close to the ground truth during the training process of the first model (600).

[0090] A detailed explanation of how the generative adversarial network (608) is trained during the training of the first model (600) will be further described in the explanation of FIG. 14.

[0091] FIG. 7 is a diagram illustrating a structure vector obtained by an electronic device according to one embodiment of the present disclosure using a first model.

[0092] Referring to FIG. 7, the structure vectors obtained using the first model may include junction points (710) and lines (720).

[0093] In one embodiment, the structural vectors composed of junction points (710) and lines (720), obtained by the electronic device (2000) using the first model, may include the structural vector (740) of the restoration target area within the restoration target image. Here, the restoration target area refers to an area within the restoration target image where pixel values ​​corresponding to a part of the area have been changed, making it impossible to visually understand the meaning of that part of the area.

[0094] The electronic device (2000) can obtain a structure vector (740) of a restoration target area by using a first model, regardless of whether a restoration target area (730) of the restoration target image exists or the size of the restoration target area. The first model can obtain a structure vector (740) of a restoration target area by receiving a restoration target image and a mask image as inputs and inferring structural features of a structure predicted to have existed in the restoration target area. The structure vector (740) of the restoration target area can be used to guide the frame structure of a structure corresponding to the restoration target area when the electronic device (2000) inpaints the restoration target image and inserts pixels into the restoration target area. The electronic device (2000) can obtain a structure vector image by converting the structure vectors obtained using the first model into an image form.

[0095] The electronic device (2000) can obtain an inpainted image in which the restoration target area of ​​the restoration target image is restored by applying the restoration target image, the mask image, and the structure vector image to a second model that performs image inpainting.

[0096] FIG. 8 is a drawing for explaining the architecture of a second model that outputs an inpainted image according to one embodiment of the present disclosure.

[0097] Referring to FIG. 8, the second model may be a Generative Adversarial Network (GAN), which is a neural network model composed of a generator model (810) and a discriminator model (820). The second model may use the generator model (810) to input a restoration target image (830), a mask image (840), and a structure vector image (850) obtained by transforming a structure vector obtained from the first model, and output an inpainted image (860) in which the restoration target region (835) of the restoration target image (830) is restored.

[0098] The generator model (810) can restore the restoration target area (835) based on the area corresponding to the restoration target area (835) of the restoration target image (830) among the structure vector images (850). Specifically, the generator model (810) can generate an inpainted image (860) in which the restoration target area (835) is filled with generated pixels by generating pixels corresponding to the restoration target area (835) of the restoration target image (830) using an RGB decoder.

[0099] The electronic device (2000) can train a second model. Additionally, the electronic device (2000) can obtain a second model trained from another electronic device (e.g., a server (3000)).

[0100] During the training of the second model, the generator model (810) of the second model can be trained to generate images having the same features as the ground truth image (870). Additionally, during the training of the second model, the discriminator model (820) of the second model can be trained to determine the authenticity of an image input to the discriminator model (820). Specifically, if the discriminator model (820) determines that the input image is a ground truth image (870) that has not undergone any processing, it can classify the input image as a 'Real' class, and if the input image is determined to be an inpainted image (870) generated by restoring the target image generated by the generator model (810), it can classify the input image as a 'Fake' class.

[0101] A detailed explanation of how the second model is trained will be described further in the explanation of Fig. 19.

[0102] FIG. 9 is a drawing for illustrating another architecture of a second model that outputs an inpainted image according to one embodiment of the present disclosure.

[0103] The second model may be a Generative Adversarial Network (GAN), which is a neural network model composed of a generator model (910) and a discriminator model (920). The second model can use the generator model (910) to receive a restoration target image (930), a mask image (940), and a structure vector image (950) as inputs, and output an inpainted image (960) in which the restoration target region (935) of the restoration target image (930) is restored. Additionally, in describing FIG. 9, the description of the configuration corresponding to FIG. 8 will be omitted.

[0104] Referring to FIG. 9, a second model according to one embodiment may further include a vector image decoder (980). The vector image decoder (980) may share the neural network layers included in the generator model (910). The vector image decoder (980) may output a reconstructed structure vector image (985) that reproduces the structure vector image (950) input to the generator model (910) based on feature values ​​output from the neural network layers included in the generator model (910). By generating the reconstructed structure vector image (985) based on feature values ​​output from the neural network layers included in the generator model (910), the structure vector image features extracted from the structural features of the image to be restored within the generator model (910) may be maintained.

[0105] The electronic device (2000) can train a second model. Additionally, the electronic device (2000) can obtain a second model trained from another electronic device (e.g., a server (3000)). During the training of the second model, a vector image decoder (980) included in the second model can be trained to generate a reconstructed structure vector image (985) having the same features as the input structure vector image (950).

[0106] FIG. 10 is a drawing for illustrating an inpainted image obtained by an electronic device according to one embodiment of the present disclosure using a second model.

[0107] In one embodiment, the electronic device (2000) may generate a restoration target image (1010) according to the embodiments described above. The restoration target area of ​​the restoration target image (1010) is determined according to the embodiments described above, and the restoration target area within the restoration target image (1010) may differ from one another for each restoration target image (1010). For example, as shown in FIG. 10, the restoration target area of ​​the first restoration target image (1012) may be smaller than the restoration target area of ​​the second restoration target image (1014).

[0108] The electronic device (2000) can input a restoration target image (1010) and a mask image representing a restoration target region into a first model to obtain a structure vector (1020) that extracts structural features of the restoration target image (1010).

[0109] The electronic device (2000) can obtain a structure vector (1020) by extracting structural features of the image to be restored, regardless of whether the size of the area to be restored of the image to be restored (1010) is large or small. In this case, the structure vector (1020) may include a structure vector of the area to be restored that represents the structural features of the area to be restored.

[0110] For example, although there are no visually meaningful pixels within the restoration target area of ​​the first restoration target image (1012), the first model can infer structural features within the restoration target area of ​​the first restoration target image (1012) to obtain a first structural vector (1022) including a structural vector of the restoration target area of ​​the first restoration target image (1012).

[0111] Additionally, although there are no visually meaningful pixels within the restoration target area of ​​the second restoration target image (1014), the first model can infer structural features within the restoration target area of ​​the second restoration target image (1014) and obtain a second structural vector (1024) including the structural vector of the restoration target area of ​​the second restoration target image (1014).

[0112] The electronic device (2000) inputs the restoration target image (1010), the mask image representing the restoration target area, and the structure vector image (1030) converted from the structure vector (1020) into the second model, thereby obtaining an inpainted image (1040) in which the restoration target area of ​​the restoration target image (1010) has been restored.

[0113] For example, the electronic device (2000) can obtain a first inpainted image (1042) restored by using a second model, wherein the restoration target region within the first restoration target image (1012) is guided by the structure vector image (1032) of the first structure vector.

[0114] Additionally, the electronic device (2000) can obtain a second inpainted image (1044) that is restored by using a second model, wherein the restoration target area within the second restoration target image (1014) is guided by the structure vector image (1034) of the second structure vector.

[0115] An electronic device (2000) according to one embodiment can obtain an inpainted image (1040) in which the area to be restored is naturally restored to match the context of the image to be restored (1010), regardless of the size of the area to be restored of the image to be restored (1010), by using a first model for extracting structural features of the image to be restored (1010) and a second model for restoring the image to be restored (1010) using a structural vector image of the structural vector (1020). That is, by obtaining a structural vector (1020) for the area to be restored of the image to be restored (1010), the electronic device (2000) can perform robust inpainted image processing without blurring or distortion of the restored area, without being affected by the size of the area to be restored when restoring the area to be restored.

[0116] FIG. 11 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image.

[0117] In one embodiment, the electronic device (2000) may determine whether to perform step S1110 based on the attributes of the area to be restored before performing step S1110. For example, if the determined area to be restored is an area to be restored determined from an original image (304) in which a portion of the image shown in FIG. 3 is damaged, extraction of structural features of the area to be restored may not be necessary. The electronic device (2000) may determine whether to perform step S1110 based on the width of the area to be restored. For example, the electronic device (2000) may perform step S1110 if the maximum width of the area to be restored is greater than or equal to a preset value. The electronic device (2000) may perform step S1150 if the maximum width of the area to be restored is less than a preset value. Meanwhile, if there are multiple restoration target areas, the electronic device (2000) performs S1110 if the width (e.g., maximum width) of one or more of the multiple restoration target areas is greater than or equal to a preset value, and if the width (e.g., maximum width) of each of the multiple restoration target areas is less than a preset value, it may perform step S1150 without extracting structural features of the restoration target areas. Since the method by which the electronic device (2000) determines the width of the restoration target areas is described in step S240 of FIG. 2, the same description is omitted.

[0118] In step S1110, an electronic device (2000) according to one embodiment may determine whether the relative ratio of the area to be restored is greater than or equal to a preset value. Here, the relative ratio of the area to be restored refers to the relative ratio occupied by the area to be restored within the image to be restored (1102). If the number of pixels of the area to be restored relative to the number of pixels of the image to be restored is greater than or equal to a preset value, the electronic device (2000) may decide to extract structural features of the area to be restored. If the relative ratio of the area to be restored is greater than or equal to a preset value, the electronic device (2000) may perform step S1120 to extract structural features of the area to be restored. If the relative ratio of the area to be restored is less than a preset value, the electronic device (2000) may perform step S1150 without extracting structural features of the area to be restored.

[0119] When the electronic device performs step S1110, there may be multiple restoration target regions. In this case, the electronic device (2000) may restore each determined restoration target region whenever one restoration target region is determined, or restore multiple determined restoration target regions collectively after multiple restoration target regions are determined.

[0120] The electronic device (2000) can identify each restoration target area forming an isolated simple closed curve based on whether the pixels of the restoration target area are connected.

[0121] When the electronic device (2000) restores each determined restoration target area whenever a single restoration target area is determined, the electronic device (2000) can operate in the same way as when there is only one restoration target area. Since this has been previously described, the same explanation is omitted.

[0122] When the electronic device (2000) restores a plurality of restoration target regions in batches, the electronic device (2000) may determine, for each of the plurality of restoration target regions, whether the relative ratio of each restoration target region is greater than or equal to a preset value. If the relative ratio of one or more of the plurality of restoration target regions is greater than or equal to a preset value, the electronic device (2000) may perform step S1120 to extract structural features of the plurality of restoration target regions in batches. If the relative ratio of each of the plurality of restoration target regions is less than or equal to a preset value, the electronic device (2000) may perform step S1150 without extracting structural features of the restoration target regions.

[0123] In step S1120, an electronic device (2000) according to one embodiment may apply the image to be restored (1102) and the mask image (1104) to a first model to obtain a structure vector representing the structural features of the structures within the image to be restored (1102). The structure vector may include one or more lines and one or more junction points. Since the method by which the electronic device (2000) obtains the structure vector using the first model has been described in the previously described embodiments, the same description will be omitted.

[0124] In step S1130, an electronic device (2000) according to one embodiment can identify whether a structure vector exists within the area to be restored. If a structure vector exists within the area to be restored, the structure vector of the area to be restored can be used to restore the area to be restored. If the structure vector of the area to be restored exists, the electronic device (2000) can perform step S1140. Alternatively, if the structure vector of the area to be restored does not exist, the electronic device (2000) can perform step S1150.

[0125] In step S1140, an electronic device (2000) according to one embodiment may obtain an inpainted image by applying a restoration target image (1102), a mask image (1104), and a structure vector image to a second model. Since the method by which the electronic device (2000) obtains an inpainted image using the second model has been described in the previously described embodiments, the same description will be omitted.

[0126] In step S1150, an electronic device (2000) according to one embodiment may obtain an inpainted image by applying a restoration target image (1102) and a mask image (1104) to a second model. In this case, the second model may output an inpainted image by receiving only the restoration target image (1102) and the mask image (1104) as input. Alternatively, the second model may output an inpainted image by receiving the restoration target image (1102), the mask image (1104), and any data to be used when not using a structure vector as input.

[0127] FIG. 12 is a flowchart illustrating another method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image.

[0128] In one embodiment, the electronic device (2000) may determine whether to perform step S1210 based on the attributes of the area to be restored before performing step S1210. For example, if the area to be restored is an area to be restored determined from an original image (304) in which some area within the image shown in FIG. 3 is damaged, extraction of structural features of the area to be restored may not be necessary. The electronic device (2000) may determine whether to perform step S1210 based on the width of the area to be restored. For example, the electronic device (2000) may perform step S1210 if the maximum width of the area to be restored is greater than or equal to a preset value. The electronic device (2000) may perform step S1280 if the maximum width of the area to be restored is less than a preset value. Meanwhile, if there are multiple restoration target areas, the electronic device (2000) performs S1210 if the width (e.g., maximum width) of one or more of the multiple restoration target areas is greater than or equal to a preset value, and if the width (e.g., maximum width) of each of the multiple restoration target areas is less than a preset value, it may perform step S1280 without extracting structural features of the restoration target areas. Since the method by which the electronic device (2000) determines the width of the restoration target areas is described in step S240 of FIG. 2, the same description is omitted.

[0129] In step S1210, the electronic device (2000) according to one embodiment may determine whether the relative ratio of the restoration target area is greater than or equal to a preset value. Since this corresponds to step S1110 of FIG. 11, the same description is omitted. Meanwhile, if there are multiple restoration target areas, the electronic device (2000) may determine whether the relative ratio of the restoration target area is greater than or equal to a preset value each time one restoration target area is determined. Additionally, if the electronic device (2000) restores multiple restoration target areas in batches, the electronic device (2000) may determine whether the relative ratio of each restoration target area is greater than or equal to a preset value for each of the multiple restoration target areas. If the relative ratio of one or more of the restoration target areas among the multiple restoration target areas is greater than or equal to a preset value, the electronic device (2000) may perform step S1220 to identify whether the structural characteristics of the multiple restoration target areas satisfy other conditions. The electronic device (2000) may perform step S1280 without extracting structural features of the restoration target area when the relative ratio of each of the plurality of restoration target areas is all less than a preset value.

[0130] In step S1220, an electronic device (2000) according to one embodiment can identify whether the electronic device (2000) has pre-set conditions for performing steps S1230 to S1240.

[0131] In one embodiment, the preset condition may be whether the relative ratio of the area to be restored is greater than or equal to a threshold value. The electronic device (2000) may perform step S1230 if the relative ratio of the area to be restored is greater than or equal to the threshold value. For example, the preset value that is the criterion for step S1210 may be 20%, and the threshold value included in the preset condition may be 40%. If the relative ratio of the area to be restored is 30%, step S1220 is performed because the relative ratio of the area to be restored is greater than or equal to the preset value of 20%, but since it is less than the threshold value of 40%, the electronic device (2000) may not perform step S1230 and may perform step S1250. Alternatively, if the relative ratio of the area to be restored is 50%, the electronic device (2000) may perform step S1230 because the relative ratio of the area to be restored is greater than or equal to the threshold value of 40%. If there are multiple restoration target areas, the electronic device (2000) can perform step S1230 if the relative ratio of one or more restoration target areas is greater than or equal to a threshold value.

[0132] An electronic device (2000) according to one embodiment may perform a step S1230 of classifying the category of a restoration target image (1202) for a restoration target image (1202) in which the relative ratio of the restoration target area is greater than or equal to a threshold value, in order to precisely and efficiently acquire structural features of a restoration target in which the relative ratio of the restoration target area is greater than the threshold value.

[0133] In one embodiment, a pre-set condition may be whether the computing budget available in the electronic device (2000) is sufficient for the electronic device (2000) to perform step S1230. The electronic device (2000) may identify the available computing budget of the electronic device (2000). Based on the identified computing budget, the electronic device (2000) may determine whether to perform step S1230. For example, when performing step S1230, the size of the image category classification model used may be S0, the memory usage may be M0, and the runtime execution time may be T0; when performing step S1250, the size of the first model used may be S1, the memory usage may be M1, and the runtime execution time may be T1; and when performing step S1270, the size of the second model used may be S2, the memory usage may be M2, and the runtime execution time may be T2. The electronic device (2000) may identify the ROM free limit, the memory availability limit, and the runtime execution time requirement limit, which are the computing budgets of the electronic device (2000). An electronic device (2000) according to one embodiment may determine that the available computing budget of the electronic device (2000) is sufficient to perform step S1230 if the ROM free limit is greater than or equal to the sum of S0, S1 and S2, the memory available limit is greater than or equal to the sum of M0, M1 and M2, and the runtime execution time requirement limit is greater than or equal to the sum of T0, T1 and T2.

[0134] In step S1230, an electronic device (2000) according to one embodiment can classify the category of the image to be restored. For example, the electronic device (2000) can classify the scene category of the image to be restored. The electronic device (2000) can classify the scene category of the image to be restored (1202) using an image category classification model.

[0135] In step S1240, an electronic device (2000) according to one embodiment can identify whether the scene category of the classified image to be restored (1202) is included in a pre-set category list. If the scene category of the classified image to be restored (1202) is included in a pre-set category list, the electronic device (2000) can perform step S1250 to extract structural features of the area to be restored. If the scene category of the classified image to be restored (1202) is not included in a pre-set category list, the electronic device (2000) can perform step S1280.

[0136] The pre-configured list of categories may consist of categories of scenes having structural characteristics. For example, categories of scenes having structural characteristics may include, but are not limited to, airport terminals, apartment buildings, hallways, bedrooms, hotel rooms, movie theaters, museums, etc.

[0137] In step S1250, an electronic device (2000) according to one embodiment may apply the image to be restored (1202) and the mask image (1204) to a first model to obtain a structure vector representing the structural features of the structures within the image to be restored (1202). Since the method by which the electronic device (2000) obtains the structure vector using the first model has been described in the previously described embodiments, the same description will be omitted.

[0138] In step S1260, an electronic device (2000) according to one embodiment can identify whether a structure vector exists within the area to be restored. If a structure vector exists within the area to be restored, the structure vector of the area to be restored can be used to restore the area to be restored. If the structure vector of the area to be restored exists, the electronic device (2000) can perform step S1270. Alternatively, if the structure vector of the area to be restored does not exist, the electronic device (2000) can perform step S1280.

[0139] In step S1270, an electronic device (2000) according to one embodiment may obtain an inpainted image by applying the restoration target image (1202), the mask image (1204), and the structure vector to a second model. Since the method by which the electronic device (2000) obtains the inpainted image using the second model has been described in the previously described embodiments, the same description will be omitted.

[0140] In step S1280, an electronic device (2000) according to one embodiment may obtain an inpainted image by applying a restoration target image (1202) and a mask image (1204) to a second model. In this case, the second model may output an inpainted image by receiving only the restoration target image (1202) and the mask image (1204) as input. Alternatively, the second model may output an inpainted image by receiving the restoration target image (1202), the mask image (1204), and any data to be used when not using a structure vector as input.

[0141] FIG. 13 is a diagram illustrating the training of a structure vector element extraction unit and a Hough transform unit of a first model when an electronic device according to one embodiment of the present disclosure trains a first model.

[0142] Referring to FIG. 13, the structure vector element extraction unit (1300) of the first model receives a restoration target image and a mask image according to the embodiments described above, and can extract structure vector elements within the restoration target image.

[0143] The structure vector element extraction unit (1300) may include a plurality of neural network layers. In this case, for each neural network layer within the structure vector element extraction unit (1300), a feature map output from the previous layer of the structure vector element extraction unit (1300) may be input to a Hough transform module, and a linear feature map output from the Hough transform model may be input to the next layer of the structure vector element extraction unit (1300).

[0144] For example, the Hough transform module (1310) may receive an input feature map output from a previous layer, perform multiple operations including convolution operations, and extract linear features (1330) from the input feature map (1320). In this case, the weights of the nodes included in the layers corresponding to the trainable portion (1325) may be updated as the first model is trained. The Hough transform module (1310) may combine the extracted linear features (1330) with the input feature map (1320) and transmit the combined feature map to the next neural network layer of the structure vector element extraction unit (1300).

[0145] FIG. 14 is a diagram for explaining the content related to the generative adversarial network of the first model when an electronic device according to one embodiment of the present disclosure trains the first model.

[0146] In one embodiment, the first model outputting the structure vector (1404) may include a generative adversarial network. The generative adversarial network may include a generator model (1410) and a discriminator model (1420). The generator model (1410) may receive a training image (1402) as input and output a generated RGB image. For example, the generator model (1410) may receive a first training image (1430) as input and output a first generated RGB image (1440). Additionally, the generator model (1410) may receive a second training image (1450) as input and output a second generated RGB image (1460). The discriminator model (1420) may receive a generated RGB image or a ground truth RGB image as input and determine whether the input image is true or false.

[0147] The discriminator model (1420) of the Generative Adversarial Network can be trained to determine the authenticity of an RGB image input to the discriminator model (1420) during the training process of the first model. Specifically, the discriminator model (1420) can update its weights based on a ground truth RGB image that has not undergone any processing. In this case, the labeled label value of the ground truth RGB image may be a label value corresponding to the 'Real' class. Additionally, the discriminator model (1420) can update its weights based on an RGB image generated by processing a training image (1402) generated by the generator model (1410). In this case, the labeled label value of the generated RGB image may be a label value corresponding to the 'Fake' class.

[0148] The generator model (1410) of the generative adversarial network can be trained to generate a realistic image that can be classified as a 'Real' class by the discriminator model (1420) during the training process of the first model. The electronic device (2000) can label the generated RGB image output from the generator model (1410) with a label value corresponding to the 'Real' class in order to update the weights of the generator model (1410). That is, the true label value of the generated RGB image output from the generator model (1410) is a label value corresponding to the 'Fake' class, but the electronic device (2000) can label the label value of the generated RGB image with a label value corresponding to the 'Real' class in order to update the weights of the generator model (1410). The electronic device (2000) can input a generated RGB image labeled with a label value corresponding to the 'Real' class into the discriminator model (1420). In this case, the discriminator model (1420) can classify the input image into the 'Real' class if the generated RGB image labeled as the 'Real' class is determined to be the correct RGB image, and classify the input image into the 'Fake' class if the generated RGB image labeled as the 'Real' class is determined to be the RGB image generated by the generator model (1410). The electronic device (2000) can calculate the loss value output from the discriminator model (1420) and update the weights of the generator model (1410) based on the loss value.

[0149] FIG. 15 is a diagram illustrating a method for determining the input order of training data for training a first model when an electronic device according to one embodiment of the present disclosure trains a first model.

[0150] In describing FIG. 15, the training data (1520) for training the first model (1510) by the electronic device (2000) may be generated through a process in which the electronic device (2000) pseudo-labels the structural vector on data that does not have the correct label of the structural vector, and masks the area to be restored on the image in which the structural vector is pseudo-labeled. The method of the electronic device (2000) masking the area to be restored on the image in which the structural vector is labeled is described in FIG. 16, and the method of the electronic device (2000) pseudo-labeling the structural vector on data that does not have the correct label of the structural vector is described later in FIG. 17.

[0151] In one embodiment, the electronic device (2000) can train a first model (1510) based on training data (1520) comprising a training image including a region to be restored and a mask image corresponding to the training image and representing the region to be restored.

[0152] In one embodiment, the size of the area to be restored in the training image and mask image included in the training data (1520) may differ. Additionally, the training data (1520) may be classified based on the size of the area to be restored. The training data (1520) may be classified based on the size of the area to be restored, for example, such as a first training image (1522) and a first mask image (1524) in which the size of the area to be restored is 0.1% to 1%, and a second training image (1526) and a second mask image (1528) in which the size of the area to be restored is 3% to 4%. In the same way, the training data (1520) may include training images and mask images classified based on the size of the area to be restored. For example, the training data (1520) may include training images and mask images in which the size of the area to be restored is 6% to 7%, and training images and mask images in which the size of the area to be restored is 9% to 10%. However, although training images with a restoration target area size of 0.1% to 1%, 3% to 4%, 6% to 7%, and 9% to 10% have been described above, this is merely an example. The size of the restoration target area of ​​the training images and mask images included in the training data (1520) may have a value between, for example, 0.1% and 10%. Additionally, the training images and mask images included in the training data (1520) may be classified among training data having a restoration target area size within a predetermined range based on the size of the restoration target area in order to perform steps S1502 to S1506 described later.

[0153] When training the first model, the electronic device (2000) can determine the training order of the training images used for training the first model (1510) according to the size of the restoration target area of ​​the training images.

[0154] Steps S1502 through S1506 represent a flowchart for schematically explaining how the electronic device (2000) trains the first model (1510).

[0155] In step S1502, an electronic device (2000) according to one embodiment may input training data (1520) into the first model (1510) for training the first model (1510).

[0156] In step S1504, an electronic device (2000) according to one embodiment can update the weights of the first model (1510) based on the loss output from the first model (1510).

[0157] In step S1506, an electronic device (2000) according to one embodiment may change the training data (1520) to training data in which the size of the area to be restored within the training image and mask image is larger. For example, the electronic device (2000) may change the training data (1520) after performing at least one epoch, which is a training unit in which operations are performed on all training data within the training data (1520) and weights are updated. In another example, the electronic device (2000) may train the first model such that for each batch, which is a data input unit for loss output and weight update, after performing loss output and weight update, the next batch is composed of training data in which the size of the area to be restored is larger. In this case, the electronic device (2000) performs an operation for each batch to output a loss and iterates through the operation to update the weights. When the operation is performed on all training data within the training data and the weights are updated, it can be said that one epoch of training of the first model has been performed.

[0158] An electronic device (2000) according to one embodiment may perform fine-tuning of the first model (1510) after performing a plurality of epochs of updating the weights of the first model (1510) using training data (1520). The electronic device (2000) may perform fine-tuning of training the first model using another dataset. In this case, the other dataset may be a dataset consisting of images in which the correct labels of the structure vectors exist.

[0159] FIG. 16 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to generate training data for training a first model.

[0160] Referring to FIG. 16, an electronic device (2000) according to one embodiment can generate training data (1610) by masking a restoration target area on an image in which structure vectors are labeled. In this case, the image in which structure vectors are labeled refers to a pseudo-labeled image in which structure features are extracted within an image of an image without a correct label, and structure vectors corresponding to the image are detected.

[0161] The training images and mask images included in the training data (1610) may include a restoration target area, and the size, shape, and location of the restoration target area may be determined in various ways. For example, the training data (1610) may include a first training image (1620) and a first mask image (1625) in which the size of the restoration target area is 0.1% to 1%, a second training image (1630) and a second mask image (1635) in which the size of the restoration target area is 3% to 4%, a third training image (1640) and a third mask image (1645) in which the size of the restoration target area is 6% to 7%, and a fourth training image (1650) and a fourth mask image (1655) in which the size of the restoration target area is 9% to 10%. However, the above description uses training images with restoration target area sizes of 0.1% to 1%, 3% to 4%, 6% to 7%, and 9% to 10% as examples, but this is for convenience of explanation. The size of the restoration target area of ​​the training image and mask image included in the training data (1520) may have a value between, for example, 0.1% and 10%.

[0162] In one embodiment, an electronic device (2000) may select images for detecting structure vectors according to a predetermined process and label the structure vectors to obtain an image in which the structure vectors are pseudo-labeled. The method by which the electronic device obtains an image in which the structure vectors are pseudo-labeled will be described in the description of FIG. 17.

[0163] In the following, a method for an electronic device (2000) to generate training data (1610) is described. Additionally, for convenience of explanation, an image in which structure vectors are pseudo-labeled will be referred to as a 'pseudo-labeled image'. Furthermore, since the image generated by the electronic device (2000) by masking the area to be restored on the pseudo-labeled image is the training image, the pseudo-labeled image and the training image have the same structural characteristics with respect to the scene elements within the image.

[0164] In one embodiment, the electronic device (2000) can generate a training image containing a region to be restored and a mask image corresponding to the training image and representing the region to be restored by masking a region to be restored on a pseudo-labeled image. In this case, the electronic device (2000) can determine the size, shape, and location of the region to be restored that is masked on the pseudo-labeled image according to the structural features of the pseudo-labeled image (i.e., the structural features of the training image).

[0165] In one embodiment, the electronic device (2000) may determine the size of the area to be restored within a pseudo-labeled image based on the structural features of the pseudo-labeled image. For example, the electronic device (2000) may determine the size of the area to be restored based on the structural vectors of the pseudo-labeled image such that the area to be restored does not completely obscure scene elements (e.g., structures, objects, etc.) within the pseudo-labeled image, but only partially obscures them. Additionally, the electronic device (2000) may determine the size of the area to be restored such that the size of the area to be restored is between 0.1% and 10% of the size of the pseudo-labeled image.

[0166] In one embodiment, the electronic device (2000) can determine the shape of the area to be restored within a pseudo-labeled image based on the structural features of the pseudo-labeled image. The electronic device (2000) can determine the shape of the area to be restored based on the structural vectors of the pseudo-labeled image such that the area to be restored does not completely obscure scene elements (e.g., structures, objects, etc.) within the pseudo-labeled image, but obscures at least a portion thereof. Additionally, the electronic device (2000) can determine the shape of the area to be restored such that the shape of the area to be restored becomes the shape of a specific object.

[0167] In one embodiment, the electronic device (2000) may determine the location of the area to be restored within the pseudo-labeled image based on the structural features of the pseudo-labeled image. Based on the structural vectors of the pseudo-labeled image, the electronic device (2000) may determine the shape of the area to be restored such that the area to be restored does not completely obscure scene elements (e.g., structures, objects, etc.) within the pseudo-labeled image, but obscures at least a portion thereof. Additionally, the electronic device (2000) may determine the location of the area to be restored to any location within the pseudo-labeled image.

[0168] The electronic device (2000) can determine the area to be restored based on the structural features of the pseudo-labeled image when determining the size, shape, and location of the area to be restored within the pseudo-labeled image. Training data (1610) can be obtained such that structural vectors can be well extracted even when the area to be restored is masked in the pseudo-labeled image.

[0169] That is, the electronic device (2000) can obtain training data (1610) that is masked such that, based on the generation conditions of the training data (1610) set as in the embodiments described above, at least some of the scene elements that have meaning within the image are retained even if the structure vectors mask the area to be restored in the pseudo-labeled image.

[0170] The method of generating a training image and a mask image by masking the area to be restored on the above-mentioned pseudo-labeled image can be applied in the same way to an image in which a correct label of a structure vector exists. The electronic device (2000) can apply the method according to the embodiments described above to another dataset, which is a dataset consisting of images in which a correct label of a structure vector exists. The electronic device (2000) can obtain training data for fine-tuning by masking the area to be restored on each of the images in which a correct label of a structure vector exists, which are included in the other dataset. The electronic device (2000) can perform fine-tuning of the first model using the training data for fine-tuning. Since this has been described above in FIG. 15, the same description will be omitted.

[0171] FIG. 17 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to acquire an image in which structure vectors are pseudo-labeled.

[0172] In step S1710, an electronic device (2000) according to one embodiment can acquire an image set consisting of a plurality of images.

[0173] In step S1720, an electronic device (2000) according to one embodiment can identify first images from an image set for labeling a structure vector for training a first model onto the images. In this case, the images included in the image set consist of images that do not have a correct label for the structure vector.

[0174] The electronic device (2000) can identify the scene categories of images in an image set using an image category classification model, and can identify images in which the scene category of the image is a scene category from which structural features can be extracted as first images.

[0175] In step S1730, an electronic device (2000) according to one embodiment may obtain second images in which each of the first images is pseudo-labeled with each of the structure vectors for training the first model by applying the first images to a structure vector detection model. Here, the structure vector detection model may be a neural network model that receives an original image (e.g., an RGB image) as input and detects a structure vector. Additionally, the structure vector detection model may be a neural network model trained based on a ground truth structure vector dataset, which consists of images in which ground truth structure vectors are labeled. Additionally, pseudo-labeling refers to labeling a structure vector detected using the structure vector detection model onto an image that does not have a label for the structure vector.

[0176] The electronic device (2000) can obtain second images in which the structure vector is pseudo-labeled by labeling the first images with the structure vector detected from the first images.

[0177] In step S1740, an electronic device (2000) according to one embodiment may select at least some of the second images to generate a training image.

[0178] In one embodiment, the electronic device (2000) can generate a training image by masking a restoration target area on each of the second images according to the method described in FIG. 16 for all of the acquired second images.

[0179] In one embodiment, the electronic device (2000) can obtain a third image by filtering some of the acquired second images based on a predetermined criterion, and generate a training image by masking a restoration target area on each of the third images according to the method described in FIG. 16 for the third images. The electronic device (2000) can obtain the third image among the second images based on the distribution of another reference dataset composed of images labeled with correct structure vectors. The electronic device (2000) can obtain the third image among the second images based on the average across the entire dataset of the number of lines within each image labeled with correct structure vectors, the average across the entire dataset of the total sum of all line lengths within each image labeled with correct structure vectors, and the average across the entire dataset of the number of junction points relative to the number of lines within each image labeled with correct structure vectors, for all data included in the other reference dataset. For example, the electronic device (2000) can obtain a third image by selecting images from among the second images in which the number of lines in the image in which the detected structure vectors are pseudo-labeled is greater than the average value of 74.98 (numbers) obtained from the distribution of another reference dataset composed of images in which the correct structure vectors are labeled, the total sum of all line lengths in the image in which the detected structure vectors are pseudo-labeled is greater than the average value of 6356.57 (pixels) obtained from the distribution of another reference dataset composed of images in which the correct structure vectors are labeled, and the number of junction points relative to the number of lines in the image in which the detected structure vectors are pseudo-labeled is less than the average value of 1.34 (ratio) obtained from the distribution of another reference dataset composed of images in which the correct structure vectors are labeled.

[0180] The electronic device (2000) can perform steps S1710 to S1740 to obtain a set of images (1700) in which structure vectors are pseudo-labeled, which are images for generating training data to train a first model.

[0181] FIG. 18 is another diagram illustrating a method for generating verification data to verify the accuracy of an inpainted image acquired by an electronic device according to one embodiment using a trained second model.

[0182] In one embodiment, the electronic device (2000) can generate a verification image (1700) for verifying a trained first model by damaging a portion of the image. The electronic device (2000) can generate a verification image (1800) by using various methods to mask the restoration target area that has damaged a portion of the image.

[0183] For example, the electronic device (2000) can generate a verification image (1810) by masking the image with the randomly determined restoration target area (1810), which is determined as a randomly determined restoration target area within the image.

[0184] In another example, the electronic device (2000) can generate a verification image (1800) by detecting an object within an image and masking the image with a randomly determined restoration target area (1810) in the remaining area that does not overlap with the existing object area (1820) determined by object detection.

[0185] In another example, the electronic device (2000) can generate a verification image (1800) by masking the existing object area (1820) determined by object detection, which is determined by object detection, into the image as the area to be restored, and the area corresponding to the detected object is determined as the area to be restored.

[0186] In another example, the electronic device (2000) can generate a verification image (1800) by masking an image with a randomly determined restoration target area (1810) and an existing object area (1820) determined through object detection as a restoration target area.

[0187] The electronic device (2000) can numerically or qualitatively evaluate the accuracy of the inpainted image obtained using the trained second model using the generated verification image (1800). Based on the evaluation results, the electronic device (2000) can further train the second model.

[0188] FIG. 19 is a drawing illustrating a method for an electronic device according to one embodiment of the present disclosure to train a second model to acquire an inpainted image.

[0189] In one embodiment, the second model may be composed of a generative adversarial neural network.

[0190] In describing FIG. 19, the first dataset (1910) may be a dataset composed of training images, mask images, and structure vector images that include a region to be restored. Additionally, the second dataset (1920) may be a dataset composed of unprocessed ground truth images. An electronic device (2000) according to one embodiment may obtain a training image (1902), a mask image (1904), and structure vector images (1906) that include a region to be restored from the first dataset (1910).

[0191] In one embodiment, the electronic device (2000) may repeat the operation of updating the weights of the discriminator model multiple times and apply the inpainted image (1912) output from the generator model (1930) to the discriminator model (1940) to determine whether the image is authentic. By determining whether the inpainted image (1912) is authentic, the electronic device (2000) may calculate a loss value from the loss function (1950) and update the weights of the generator model (1930) based on the loss value.

[0192] In one embodiment, the electronic device (2000) can update the weights of the discriminator model (1940).

[0193] In one embodiment, the electronic device (2000) may update the weights of a discriminator model (1940) that determines the authenticity of an image based on a ground truth image (1922) obtained from a second dataset (1920). In this case, the label value of the ground truth image (1922) obtained from the second dataset (1920) may be 1. A label value of 1 refers to a label value corresponding to the 'Real' class, which indicates that the image is a ground truth image that has not undergone any processing.

[0194] In one embodiment, the electronic device (2000) can update the weights of a discriminator model (1940) that determines the authenticity of an image based on an inpainted image (1912) obtained from a generator model (1930). The electronic device (2000) can obtain a training image (1902), a mask image (1904), and structure vectors (1906) from among the data included in a first dataset (1910), apply the training image (1902), the mask image (1904), and the structure vectors (1906) to a generator model (1930), and obtain an inpainted image (1912) output from the generator model. The electronic device (2000) can label the inpainted image (1912) output from the generator model (1930) with a label value of '0'. In this case, the labeling value '0' refers to the label value corresponding to the 'Fake' class, which represents an inpainted image generated by processing the image to be restored.

[0195] An electronic device (2000) according to one embodiment may receive a correct image (1922) or an inpainted image (1912) as input and perform an operation to update the weights of a discriminator model (1940) multiple times (iteration).

[0196] In one embodiment, the electronic device (2000) can update the weights of the generator model (1930). The electronic device (2000) can apply the training image (1902), the mask image (1904), and the structure vectors (1906) to the generator model (1930) and label the inpainted image (1912) output from the generator model with a label value of '1' corresponding to the 'Real' class. The electronic device (2000) can apply the inpainted image (1912) labeled with the label value '1' to the discriminator model (1940) and determine whether the image is genuine. The electronic device (2000) obtains a loss value based on a loss function (1950) and can update the weights of the generator model (1930) based on the loss value calculated by the loss function (1950) of the discriminator model (1940).

[0197] FIG. 20 is a drawing for illustrating embodiments in which an electronic device according to one embodiment of the present disclosure performs image inpainting.

[0198] In step S2010, an electronic device (2000) according to one embodiment can acquire an original image and determine a restoration target area.

[0199] In one embodiment, a user of the electronic device (2000) may want to remove unwanted objects within an original image. The electronic device (2000) receives input from the user selecting objects within the original image, and by changing the pixel values ​​of the pixels corresponding to the selected objects based on the received input, it can determine a restoration target area (2002) where the user has deleted the objects.

[0200] In one embodiment, a user of the electronic device (2000) may use a damaged photograph (e.g., an old photo scan) as the original image and wish to restore a damaged area within the original image. The electronic device (2000) receives input from the user selecting damaged areas within the original image, and based on the received input, may determine a restoration target area (2006) corresponding to the damaged area.

[0201] Specific methods for determining a restoration target area based on user input by an electronic device (2000) have been described above in the description of FIGS. 3 and FIGS. 4, so the same description will be omitted.

[0202] In step S2020, an electronic device (2000) according to one embodiment may perform an image inpainting algorithm. Based on a determined restoration target area, the electronic device (2000) may obtain an inpainted image in which the restoration target area of ​​the restoration target image is restored. In this case, the electronic device (2000) may perform steps S1110 to S1150 of FIG. 11 or steps S1210 to S1280 of FIG. 12.

[0203] In step S2030, an electronic device (2000) according to one embodiment can acquire an inpainted image.

[0204] In one embodiment, if the restoration target area determined by the electronic device (2000) is the restoration target area (2002) where the user has deleted objects, the electronic device (2000) can perform an image inpainting algorithm to obtain an inpainted image (2004) in which objects within the image have been deleted.

[0205] In one embodiment, if the restoration target area determined by the electronic device (2000) is a restoration target area (2006) corresponding to a damaged area, the electronic device (2000) can perform an image inpainting algorithm to obtain an inpainted image (2008) in which the restoration target areas within the image are restored.

[0206] FIG. 21 is a drawing for illustrating other embodiments in which an electronic device according to one embodiment of the present disclosure performs image inpainting.

[0207] An electronic device (2000) according to one embodiment can acquire an image containing depth information and, based on the depth information, acquire an inpainted video that can be viewed at any point in time different from the time when the image was taken.

[0208] In one embodiment, the electronic device (2000) may obtain depth information from an image captured using at least two cameras, or obtain depth information from an image containing depth information captured using an RGB-D (RGB-Depth) camera.

[0209] In another embodiment, the electronic device (2000) can acquire an RGB image and perform step S2105 to acquire depth information.

[0210] In step S2105, the electronic device (2000) can predict the depth of the image based on at least one object identified within the acquired image. For example, the electronic device (2000) can identify different depths, such as a first depth, a second depth, ..., a Nth depth, from an RGB image. The electronic device (2000) can acquire the segmentation levels of the different depths as depth information.

[0211] In step S2110, an electronic device (2000) according to one embodiment can extract a layered depth image (LDI). The electronic device (2000) can acquire a layered depth image based on acquired depth information. For example, based on the depth information, the electronic device (2000) can divide the image into a first layer image, a second layer image, a third layer image, etc., having different depths.

[0212] In step S2120, an electronic device (2000) according to one embodiment can determine a restoration target area for each layered depth image.

[0213] In one embodiment, the electronic device (2000) may determine as a restoration target area an area that is not captured at a specific point in time, which represents the parallax that occurs as the camera moves its viewpoint and views the subject from different points in time. For example, when the camera views the subject from the left, the right area of ​​the subject is not captured by the camera, so the electronic device (2000) may determine as a restoration target area the right rear area of ​​the subject that was obscured by the subject. In this case, the electronic device (2000) may select one or more layer images corresponding to the depth of the area obscured by the subject from among the layered depth images extracted in step S2110, and determine the restoration target area within the selected layer image.

[0214] In one embodiment, the electronic device (2000) may determine an area outside the boundaries of an image captured by a camera as a restoration target area. In this case, the electronic device (2000) may select one or more layer images corresponding to the depth of the background area among the layered depth images extracted in step S2110, and determine the areas outside the boundaries of the selected layer images as restoration target areas.

[0215] In step S2130, an electronic device (2000) according to one embodiment may perform an image inpainting algorithm. Based on a determined restoration target area, the electronic device (2000) may obtain one or more inpainted images in which the restoration target area of ​​one or more layer images is restored. In this case, the electronic device (2000) may perform steps S1110 to S1150 of FIG. 11 or steps S1210 to S1280 of FIG. 12.

[0216] In step S2140, an electronic device (2000) according to one embodiment can generate a video that can be viewed at any point in time different from the time when the image was captured by 3D rendering. Based on one or more layer images on which image inpainting was performed, the electronic device (2000) can generate visual effects such as being captured at a different point in time while the camera moves over a scene that has already been captured.

[0217] FIG. 22 is a drawing for explaining an inpainted video generated by an electronic device according to one embodiment performing image inpainting.

[0218] Referring to FIG. 22, an electronic device (2000) according to one embodiment can acquire an original image in which a camera captures a subject at a specific point in time. The electronic device (2000) can acquire depth information for the original image and, based on the acquired depth information, acquire a layered depth image.

[0219] The electronic device (2000) determines the area to be restored within the layered depth image and, when the camera viewpoint changes, performs an inpainting algorithm on the areas requiring restoration, thereby obtaining an inpainted image in which image frames of the subject viewed from different viewpoints over time are played back.

[0220] FIG. 23 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0221] Referring to FIG. 23, an electronic device (2000) according to one embodiment may include a communication interface (2100), a memory (2200), and a processor (2300).

[0222] The communication interface (2100) can perform data communication with the server (3000) under the control of the processor (2300). In addition, the communication interface (2100) can perform data communication not only with the server (3000) but also with other peripheral electronic devices (not shown).

[0223] The communication interface (2100) can perform data communication with a server (3000) or other peripheral electronic devices using at least one of the data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0224] A communication interface (2100) according to one embodiment can transmit and receive data for performing image inpainting with an external device (not shown). For example, the communication interface (2100) can receive a first model (2230) pre-trained to detect structure vectors and a second model (2240) for performing image inpainting from an external server (3000).

[0225] The memory (2200) may store instructions, data structures, and program code that can be read by the processor (2300). In the disclosed embodiments, operations performed by the processor (2300) may be implemented by executing the instructions or code of the program stored in the memory (2200).

[0226] The memory (2200) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a non-volatile memory and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0227] A memory (2200) according to one embodiment may store various types of data that can be used to perform image inpainting. For example, the memory (2200) may store an image restoration module (2210) for performing image inpainting, a training data generation module (2220) for training a first model (2230) and a second model (2240), a first model (2230) for extracting structural features of an image to be restored, and a second model (2240) for restoring an image to be restored.

[0228] In the disclosed embodiment, the image restoration module (2210) may include a structure vector detection module (2212) and an inpainting module (2214). Additionally, the training data generation module (2220) may include a labeling module (2222) and a masking module (2224).

[0229] The processor (2300) can control the overall operations of the electronic device (2000). For example, the processor (2300) can control the overall operations of the electronic device (2000) to perform image inpainting to restore an image by executing one or more instructions of a program stored in memory (2200).

[0230] The processor (2300) may be composed of at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), APs (Application Processor), a Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for processing AI models, but is not limited thereto.

[0231] In one embodiment, the processor (2300) can restore the image to be restored by executing an image restoration module (2210).

[0232] The processor (2300) can obtain a structure vector representing the structural features of the image to be restored by applying the image to be restored and the mask image to the first model (2230) using the structure vector detection module (2212) and the first model (2230). In this case, the structure vector of the image to be restored may include a structure vector of the area to be restored, which represents the structural features of the area to be restored. Since specific methods for the processor (2300) to obtain the structure vector have already been described in the aforementioned embodiments, the same description will be omitted.

[0233] The processor (2300) can obtain an inpainted image in which the restoration target region of the restoration target image is restored by using the inpainting module (2214) and the second model (2240) and applying the structure vector image converted from the restoration target image, the mask image, and the structure vector to the second model (2240). Since specific methods for the processor (2300) to obtain the inpainted image have already been described in the previously described embodiments, the same description will be omitted.

[0234] In one embodiment, the processor (2300) can generate training data for training the first model (2230) and the second model (2240) using a training data generation module (2220).

[0235] The processor (2300) can obtain images in which structure vectors are pseudo-labeled using a labeling module (2222). The processor (2300) can obtain an image set consisting of multiple images in which there are no correct labels for the structure vectors, and can identify first images from the image set for pseudo-labeling the structure vectors for training the first model. The processor (2300) can use an image category classification model to identify the scene categories of the images within the image set, and can identify images in which the scene category of the image is a scene category in which structural features can be extracted as the first images. The processor (2300) can apply the first images to a structure vector detection model. The processor (2300) can obtain second images in which the structure vectors are pseudo-labeled by labeling the structure vectors detected from the first images onto the first images. The processor (2300) can obtain third images by filtering some of the obtained second images based on a predetermined criterion.

[0236] The processor (2300) can generate training data by using a masking module (2224) to mask a restoration target area on second images, which are images in which a structure vector is pseudo-labeled, or on a third image that is a portion of the second images filtered. When the processor (2300) masks a restoration target area on the second images or the third image, it can determine the size, shape, and location of the restoration target area to be masked based on the structural features of the second images or the third images.

[0237] The processor (2300) can train the first model (2230) and the second model (2240) based on training data generated using the training data generation module (2220). Since this has already been described in the previously mentioned embodiments, the same description will be omitted.

[0238] FIG. 24 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0239] A server (3000) according to one embodiment is interconnected with an electronic device (2000) via wired or wireless communication and can perform data communication.

[0240] A server (3000) according to one embodiment may include at least a communication interface (3100), a DB (3200), a memory (3300), and a processor (3400).

[0241] A communication interface (3100) according to one embodiment may include one or more components that enable communication through a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, a satellite communication network, and a combination thereof.

[0242] A communication interface (3100) according to one embodiment can transmit a first model and a second model to an electronic device (2000). Additionally, the communication interface (3100) can receive training data from the electronic device (2000) and transmit the updated first model and second model to the electronic device (2000).

[0243] DB (3200) can store data received from an electronic device (2000) or data received from an external device.

[0244] DB (3200) can store training data to be used to train the first model and the second model generated through training on the server (3000).

[0245] The memory (3300) can store various data, programs, or applications for operating and controlling the server (3000). A program stored in the memory (3300) may include one or more instructions. A program (one or more instructions) or application stored in the memory (3300) may be executed by a processor (3400). The memory (3300) may store a module that performs the same function as a module stored in the electronic device (2000).

[0246] For example, data and program instruction codes corresponding to an image restoration module (not shown) and a training data generation module (not shown) can be stored in the memory (3300).

[0247] The processor (3400) can control the server (3000) overall. According to one embodiment, the processor (3400) can execute one or more programs stored in memory (3300).

[0248] A processor (3400) according to one embodiment may include an AP (Application Processor), CPU (Central Processing Unit), GPU (Graphic Processing Unit), Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for processing AI models.

[0249] The processor (3400) can perform operations that can be performed in the electronic device (2000) according to the embodiments described above.

[0250] The processor (3400) can generate an inpainted image by restoring the restoration target area of ​​the restoration target image. The processor (3400) receives data regarding the original image and user input, or the restoration target image and mask image from the electronic device (2000), and uses the first model and the second model stored in the DB to generate an inpainted image in which the restoration target area of ​​the restoration target image is restored. Since the method by which the server (3000) generates the inpainted image corresponds to the method by which the electronic device (2000) generates the inpainted image, the same description will be omitted.

[0251] The processor (3400) can generate a first model and a second model by performing training of a neural network model using data stored in the DB (3200). The generated first model and second model can be transmitted to an electronic device (2000).

[0252] FIG. 25 is a flowchart of a method for an electronic device according to one embodiment of the present disclosure to acquire an inpainted image from an original image by interacting with a server.

[0253] In step S2510, an electronic device (2000) according to one embodiment can determine a region to be restored within an original image. The electronic device (2000) can acquire an original image and determine a region to be restored within the original image.

[0254] The operation of the electronic device (2000) in step S2510 determining the restoration target area corresponds to step S210 of FIG. 2, so the same description will be omitted.

[0255] In step S2520, an electronic device (2000) according to one embodiment may request a server (3000) to generate an inpainting image. The electronic device (2000) may request the server (3000) to generate an inpainting image while providing the server (2000), for example, area information of the original image and the area to be restored.

[0256] In step S2530, a server (3000) according to one embodiment can generate a restoration target image based on a determined restoration target area.

[0257] The server (3000) can generate a restoration target image including a restoration target area by changing the pixel values ​​of pixels corresponding to a determined restoration target area. In this case, the server (3000) can change the pixel values ​​of pixels corresponding to a portion of the area to a preset value. For example, the server (3000) can change the pixel values ​​of pixels corresponding to a portion of the area to 0, or change the pixel values ​​of pixels corresponding to a portion of the area to the average value of the pixel values ​​of all images in the training data. Additionally, the server (3000) can obtain a mask image representing the restoration target area. In this case, the mask image may consist of a restoration target area and a non-restoration target area.

[0258] However, although it has been described that in step S2530, the electronic device (2000) requests the server (3000) to generate an inpainting image while providing area information of the original image and the area to be restored to the server (3000), it is not limited thereto. The operation of step S2530 may not be performed at the server (3000) but may be performed at the electronic device (2000) as described in step S210 of FIG. 2. In this case, the server (3000) may receive the image to be restored and the mask image from the electronic device (2000).

[0259] In step S2540, a server (3000) according to one embodiment may obtain structure vectors composed of one or more lines and one or more junction points by applying the image to be restored and the mask image to a first model stored in the server (3000). The first model may be a neural network model for receiving the image to be restored and the mask image as inputs and detecting geometric features such as points, lines, and surfaces representing the frame structure of structures within the image to be restored. The structure vectors may include structural features of the area to be restored, which are inferred structural features determined to exist within the area to be restored for the area to be restored where pixel values ​​within the image to be restored have been changed. In this case, the structural features of the area to be restored may be inferred from structural features of an area other than the area to be restored within the entire area of ​​the image to be restored. Based on the input image to be restored and the mask image, the first model may output structure vectors related to the entire image to be restored, including the area to be restored.

[0260] In step S2550, a server (3000) according to one embodiment can obtain an inpainted image in which the restoration target area of ​​the restoration target image is restored by applying a structure vector image transformed from a restoration target image, a mask image, and a structure vector to a second model stored in the server (3000). The structure vectors may be output from a first model, and the server (3000) can obtain a structure vector image by transforming the structure vectors. The inpainted image output from the second model may be an image in which pixels corresponding to the restoration target area are generated and the restoration target area is filled with the generated pixels. Additionally, the inpainted image may be an image restored to conform to the context of the restoration target image, in which the restoration target area of ​​the restoration target image is naturally connected to other areas outside the restoration target area.

[0261] In step S2560, an electronic device (2000) according to one embodiment can receive an inpainted image generated from a server (3000).

[0262] Meanwhile, the block diagram of the electronic device (2000) shown in FIG. 23 and the block diagram of the server (3000) shown in FIG. 24 are block diagrams for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of each device actually implemented. That is, as needed, two or more components may be combined into one component, or one component may be subdivided into two or more components. Furthermore, the functions performed in each block are intended to explain the embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0263] A method of operation of an electronic device according to one embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The computer-readable medium may be provided in the form of a non-transitory recording medium. Here, a non-transient recording medium simply means a tangible device that does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transient storage medium' may include a buffer where data is stored temporarily.

[0264] In addition, the method of operation of the electronic device according to the disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0265] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0266] A computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, if a third device (e.g., a smartphone) is communicationally connected to the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0267] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0268] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a client device connected to the server in communication to perform a method according to the disclosed embodiments.

[0269] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 A method for performing image inpainting in which an electronic device restores a restoration target region within an image, comprising: determining a restoration target region within an original image; generating a restoration target image from the original image to be restored based on the determined restoration target region; obtaining a mask image representing the restoration target region; determining whether to extract structural features of the restoration target region based on attributes of the restoration target region; obtaining structural vectors composed of one or more lines and one or more junctions by applying the restoration target image and the mask image to a first model for extracting structural features of the restoration target image including structural features of the restoration target region based on the determination to extract structural features of the restoration target region; and obtaining an inpainted image in which the restoration target region of the restoration target image is restored by applying a structural vector image transformed from the restoration target image, the mask image, and the structural vectors to a second model for restoring the restoration target image. Claim 2 A method according to claim 1, wherein the step of determining the area to be restored comprises: receiving input from a user selecting a portion of the area within the original image; and determining the area to be restored based on the received input, and the step of generating the image to be restored comprises generating the image to be restored by changing pixels corresponding to the area to be restored within the original image. Claim 3 A method according to claim 1, wherein the step of determining whether to extract structural features of the restoration target area based on the attributes of the restoration target area includes the step of determining whether to extract structural features of the restoration target area based on the width of the restoration target area. Claim 4 A method according to claim 1, wherein the step of determining whether to extract structural features of the restoration target area based on the attributes of the restoration target area includes the step of determining to extract structural features of the restoration target area when the number of pixels of the restoration target area relative to the number of pixels of the restoration target image is greater than or equal to a preset value. Claim 5 In claim 4, the step of determining whether to extract structural features of the restoration target area based on the attributes of the restoration target area comprises: identifying an available computing budget of the electronic device; identifying a category of the restoration target image based on the identified computing budget; and determining to extract structural features of the restoration target area if the category of the restoration target image is a pre-set scene category, wherein the computing budget includes a ROM free limit, a memory availability limit, and a runtime execution time requirement limit. Claim 6 A method according to claim 1, wherein training data for training the first model comprises a training image including a region to be restored, a mask image vector corresponding to the training image and representing the region to be restored of the training image, and a structure vector corresponding to the training image, and wherein the first model is trained based on the training data. Claim 7 In claim 6, the method wherein the training sequence of the training images used for training the first model is determined according to the size of the restoration target area. Claim 8 A method according to claim 6, wherein the first model is trained using a generator model that receives the training image as input and outputs a generated RGB image, and a discriminator model that receives the generated RGB image or ground truth RGB image as input and determines the authenticity of the input image. Claim 9 A method according to claim 6, wherein the size, shape, and location of the restoration target area within the training image are determined according to the structural features of the training image. Claim 10 A method according to claim 6, wherein the training images are generated by identifying first images for labeling a structure vector for training a first model in an image set composed of multiple images, applying the first images to a structure vector detection model to obtain second images in which each of the first images is labeled with the structure vectors for training, and selecting at least some of the second images. Claim 11 An electronic device for performing image inpainting comprises: a communication interface; a memory for storing one or more instructions; An electronic device comprising at least one processor that executes one or more instructions stored in the memory, wherein the at least one processor determines a region to be restored within an original image, generates a target image to be restored from the original image based on the determined target region to be restored, obtains a mask image representing the target region to be restored, determines whether to extract structural features of the target region to be restored based on the attributes of the target region to be restored, and, based on the determination to extract structural features of the target region to be restored, applies the target image to be restored and the mask image to a first model for extracting structural features of the target image to be restored including structural features of the target region to obtain structural vectors composed of one or more lines and one or more junctions, and applies a structural vector image transformed from the target image to be restored, the mask image, and the structural vectors to a second model for restoring the target image to obtain an inpainted image in which the target region to be restored of the target image to be restored is restored. Claim 12 An electronic device according to claim 11, wherein at least one processor receives input from a user selecting a portion of the original image, determines the area to be restored based on the received input, and generates the image to be restored by changing pixels corresponding to the area to be restored within the original image. Claim 13 In claim 12, the electronic device wherein at least one processor determines whether to extract structural features of the restoration target area based on the width of the restoration target area. Claim 14 An electronic device according to claim 11, wherein at least one processor determines to extract structural features of the restoration target area when the number of pixels of the restoration target area relative to the number of pixels of the restoration target image is greater than or equal to a preset value. Claim 15 An electronic device according to claim 14, wherein at least one processor identifies an available computing budget of the electronic device, identifies a category of the image to be restored based on the identified computing budget, and if the category of the image to be restored is a pre-set category of a scene, decides to extract structural features of the area to be restored, and the computing budget includes a ROM free limit, a memory availability limit, and a runtime execution time requirement limit. Claim 16 An electronic device according to claim 11, wherein training data for training the first model comprises a training image including a region to be restored, a mask image vector corresponding to the training image and representing the region to be restored of the training image, and a structure vector corresponding to the training image, and wherein the first model is trained based on the training data. Claim 17 In paragraph 16, the first model is an electronic device in which the training sequence of the training images used for training the first model is determined according to the size of the area to be restored. Claim 18 An electronic device according to claim 16, wherein the first model is trained using a generator model that receives the training image as input and outputs a generated RGB image, and a discriminator model that receives the generated RGB image or ground truth RGB image as input and determines the authenticity of the input image. Claim 19 An electronic device according to claim 16, wherein the size, shape, and location of the restoration target area within the training image are determined according to the structural features of the training image. Claim 20 A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 1 through 10 on a computer. Claim 21 delete

Images