Image processing method using generative model and computing device for performing same

Abstract

This method for editing an image by using a generative model may comprise the steps of: receiving a user input requesting movement of at least one object included in an input image; expanding the input image in a direction determined on the basis of the movement of the object; determining a generation required area, which is an area requiring generation of a partial image of the at least one object, on the basis of the expanded input image; generating an image for the generation required area by using at least one generative model, and outputting a reconstructed image on the basis of the input image and the partial image for the at least one object.

Description

Image processing method using a generative model and computing device for performing the same

[0001] The present disclosure relates to a method, device, and system for editing an image using a generative model, and more particularly, to a method, device, and system for editing an image after an object has been transformed when the position or size of the object in the image is transformed.

[0002] Generative AI technology learns the patterns and structures of massive training data and, based on that data, generates new data similar to the input data. Generative AI can be used to obtain images that correspond to text or to expand images to areas not included in the original image.

[0003] Generative AI technology can be applied to image processing to support outpainting or inpainting. Outpainting expands an image while preserving its style and content, while inpainting generates an image to fill a specific area within the image.

[0004] Meanwhile, recent advancements in image processing technology, which prioritize user experience, have led to some devices and programs supporting features that allow users to change the position and size of objects within an image. However, if parts of an object are cropped or appear out of place in the resulting image after transformation, user satisfaction may be reduced.

[0005] A method executed by at least one processor for processing an image using one or more generative models according to one embodiment of the present disclosure may include the steps of: receiving a user input requesting movement of at least one object included in an input image; expanding the input image in a direction determined based on the movement of the object; determining a generation required area, which is an area requiring generation of a partial image for the at least one object, based on the expanded input image; generating an image for the generation required area using at least one generative model; and outputting a reconstructed image based on the input image and the partial image for the at least one object.

[0006] According to one embodiment of the present disclosure, a computing device includes an input / output interface for receiving a user input requesting processing of an input image and outputting a reconstructed image processed according to the user input, a memory storing instructions for processing the input image, and at least one processor for executing the instructions, wherein, when the at least one processor receives a user input requesting movement of at least one object included in the input image, the at least one processor expands the input image in a direction determined based on the movement of the object, determines a generation required area, which is an area requiring generation of a partial image for the at least one object, based on the expanded input image, and generates an image for the generation required area using at least one generative model, and then outputs a reconstructed image based on the input image and the partial image for the at least one object.

[0007] According to one embodiment of the present disclosure, a computer-readable, non-transitory recording medium is stored with instructions executed by at least one processor, such that the at least one processor receives a user input requesting processing of an input image, and includes an input / output interface for outputting a reconstructed image processed according to the user input, a memory storing instructions for processing the input image, and at least one processor for executing the instructions, wherein the at least one processor, when receiving a user input requesting movement of at least one object included in the input image, expands the input image in a direction determined based on the movement of the object, determines a generation required area, which is an area requiring generation of a partial image for the at least one object, based on the expanded input image, and generates an image for the generation required area using at least one generative model, and then outputs a reconstructed image based on the input image and the partial image for the at least one object.

[0008] According to one embodiment of the present disclosure, a computer program may be stored in a medium for performing at least one of the embodiments of the disclosed method on a computer.

[0009] FIG. 1 is a diagram illustrating a process of editing an image using a creation model when converting an object included in an image according to one embodiment of the present disclosure.

[0010] FIG. 2 is a diagram illustrating a process of expanding an image by considering inverse transformation when an object included in the image is reduced, according to one embodiment of the present disclosure.

[0011] FIG. 3 is a diagram illustrating a process of expanding an image by considering inverse transformation when an object included in the image is rotated, according to one embodiment of the present disclosure.

[0012] FIG. 4A is a diagram illustrating a method for expanding an image based on the size of a selected object, according to one embodiment of the present disclosure.

[0013] FIG. 4b is a diagram illustrating a process of expanding an image based on the size of the image, according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram illustrating a process of expanding an image based on a transformable range of a selected object, according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating a process of expanding an image by moving all borders located in the reverse transformation direction according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating a process of inferring an object proposal region from an image according to one embodiment of the present disclosure.

[0017] Figures 8 and 9 are diagrams illustrating a process of inferring an object proposal region without expanding an input image and expanding the object proposal region by considering the inverse transformation of the object.

[0018] Figure 10 is a diagram illustrating a process of editing an image by using the object image generated in the process of expanding the input image by considering the inverse transformation of the object's transformation.

[0019] FIG. 11 is a diagram illustrating an example of a configuration of a computing device for performing image editing using a generative model according to one embodiment of the present disclosure.

[0020] FIG. 12 is a diagram illustrating the types and roles of generation models according to one embodiment of the present disclosure.

[0021] FIGS. 13 to 19 are flowcharts illustrating a process of editing an image using a generation model according to embodiments of the present disclosure.

[0022] FIGS. 20 to 23 are flowcharts illustrating a process of editing an image using a generative model according to embodiments of the present disclosure.

[0023] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.

[0024] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0025] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the specification. In addition, when describing an embodiment of the present disclosure, if a detailed description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

[0026] In one embodiment, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and the instructions, when executed by the processor of the computer or other programmable data processing apparatus, can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0027] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0028] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided through a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In addition, in one embodiment, the '~ unit' may include one or more processors.

[0029] Embodiments of the present disclosure relate to a method for editing an image using a generative model. Before describing specific embodiments, the meanings of terms frequently used in this specification are defined.

[0030] "Editing" an image may refer to processing an image so that at least a portion of the image is changed. In embodiments of the present disclosure, when an object included in an image is transformed (moved, enlarged, reduced, rotated, etc.), a device performing image processing may generate an image of the object after the transformation, which may be referred to as editing the image. Terms such as "recomposition" of an image may also be used instead of "editing" an image.

[0031] The 'transformation' of an object may include movement of the object, resizing (e.g., enlarging or reducing), rotation of the object, etc. In the present disclosure, processes involving movement of an object are described. However, those skilled in the art will understand that the present disclosure is not limited to such embodiments. Additionally, even though the embodiments relate to cases where an object is moved, other types of transformations, such as resizing or rotation, may be applied to the object. Instead of 'transformation' of an object, terms such as 'change' of an object may be used. Also, instead of 'movement' of an object, terms such as 'translation' of an object may be used.

[0032] A "partial object" may refer to an object whose appearance is not at least partially displayed in an image. For example, if an object is located at the edge of an image so that only a portion of the object is displayed in the image and the rest is not displayed in the image, the object may be referred to as a partial object. Alternatively, if a portion of the object is obscured by another object or the background and is not displayed, the object may be referred to as a partial object. According to one embodiment of the present disclosure, a computing device may determine whether an object is a partial object based on whether the object touches at least one of the edges of an image. Furthermore, according to one embodiment of the present disclosure, the computing device may perform object recognition on an image and determine whether the object is a partial object based on the result of the object recognition. Terms such as "incomplete object" may be used instead of "partial object."

[0033] A "full object" can refer to an object whose entire form is displayed in the image. Terms such as "complete object" or "whole object" can also be used instead of "full object."

[0034] "Generative AI" can refer to AI technology capable of generating new text, images, etc. in response to input data (e.g., text, images, etc.). Representative examples of generative AI are described in the "Generative Models" section below.

[0035] A "generative model" can refer to a neural network model that implements generative AI technology. By learning the patterns and structure of training data, a generative model can generate new data with similar characteristics to the input data or new data corresponding to the input data. For example, if the input data is an image and the generative model is asked to expand the image, the generative model can create a new image outside the original image while maintaining the content or style of the original image. Or, for example, if the input data is text containing a question, the generative model can generate and output an answer to the question.

[0036] The term "object proposal area" may refer to an area within an image where an object is judged or predicted to exist. According to one embodiment, a computing device performing image processing may infer, determine, or extract a certain area (e.g., a bounding box) containing an object as an object proposal area. Alternatively, the computing device may separate an object from an image through segmentation and infer (determine) only the separated object as an object proposal area. Alternatively, the computing device may infer (determine) the object proposal area to include not only the separated object as a result of segmentation, but also a portion of the margin around the object.

[0037] The term "generation required area" may refer to an area where an image needs to be generated for image editing. For example, when moving an object within an image, an image needs to be generated for a part of the object that was not visible in the original image, and the area where a new image needs to be generated may correspond to the generation required area. In embodiments of the present disclosure, one or more generation models can generate an image for the generation required area. In embodiments of the present disclosure, by inferring (determining) the minimum area where an image needs to be generated as the generation required area, it is possible to prevent the output image from becoming unnatural due to the instability of the generation model or to prevent a lot of time from being consumed in image generation.

[0038] "Outpainting" may refer to a technique for expanding the outer border of an image while maintaining the image's style and content. In embodiments of the present disclosure, outpainting may be used when expanding an input image considering inverse transformation.

[0039] "Inpainting" may refer to a technique for creating a new image and filling in a specific area within an image. In the examples of the present disclosure, inpainting may be used when creating an image for an area requiring creation.

[0040] Hereinafter, a method of editing an image using a generation model according to embodiments of the present disclosure and a computing device for performing the same will be described with reference to drawings.

[0041] The processes described with reference to FIGS. 1 through 10 are assumed to be performed by a computing device supporting image processing functions. Therefore, in describing FIGS. 1 through 10 , the computing device is described as performing the processes. Detailed components included in a computing device according to one embodiment are illustrated in FIG. 11 , and these components will be described in detail below.

[0042] In embodiments of the present disclosure, a computing device may perform outpainting to expand an image, or inpainting when a transformation of an object requires the creation of a new image (object or background) within the image. This image creation may be performed using a single generative model, but efficiency and performance can also be improved by utilizing generative models with different characteristics for each operation.

[0043] In embodiments of the present disclosure, unless otherwise specified, it is assumed that an image is edited using three generative models as illustrated in FIG. 12. Referring to FIG. 12, the first generative model (1210) is a model for performing outpainting for image expansion. The second generative model (1220) and the third generative model (1230) are both models for performing inpainting for image generation after object transformation. The second generative model (1220) is a model for generating an image of an object, and the third generative model (1230) is a model for generating an image of a background.

[0044] In one embodiment, the first generative model (1210) may be a model that focuses on the generation speed rather than the quality of the generated image. This is because, when expanding an image for object proposal region inference (determination), fast generation is more important than the quality of the generated image. In contrast, the second generative model (1220) may be a model that focuses on the quality of the generated image rather than the generation speed. This is because the image generated by the second generative model (1220) corresponds to the output image. Therefore, the second generative model (1220) may be a model with higher performance than the first generative model (1210). In addition, the first generative model (1210) may be a model with lower quality output data in exchange for faster generation speed compared to the second generative model (1220). In addition, since the image of the object may have a greater impact on the quality of the overall image than the image of the background, the second generative model (1220) may be a model with higher performance than the third generative model (1230). Of course, it is self-evident that processes described in the following embodiments using different generation models may also be performed using one identical generation model or multiple identical generation models.

[0045] FIG. 1 is a diagram illustrating a process of editing an image using a generation model when transforming an object included in the image, according to one embodiment of the present disclosure. FIG. 1 illustrates images corresponding to the first operation (1a) to the sixth operation (1f), respectively, in the order in which the images are processed.

[0046] Referring to FIG. 1, in a first operation (1a), a computing device can obtain an input image including an object (10). In a second operation (1b), the computing device can receive a user input requesting a transformation of the object. For example, a user can request a movement of the object (10) by touching the object (10) on the screen of the computing device on which the input image is displayed and moving a finger while maintaining the touch. Of course, the user input can also be configured to select (specify) an object (10) in various other forms and request a transformation of the object (10). (e.g., clicking a mouse button while the pointer is positioned on the object and moving the pointer while maintaining the click)

[0047] In the embodiment disclosed in Fig. 1, the transformation of the object (10) requested by the user is a 'translation' of the object, but is not limited thereto, and the user may also request a resize (enlargement, reduction) or rotation of the object (10).

[0048] In the third operation (1c), the computing device can expand the input image by considering the transformation (movement) requested by the user, or by considering the inverse transformation for the transformation requested by the user. In other words, the computing device can expand the input image in a direction determined based on the movement of the object requested by the user. To expand the input image, the computing device can perform outpainting using the first generative model (1210) described above. To cause the first generative model (1210) to perform outpainting, the computing device can input a prompt instructing the first generative model (1210) to perform outpainting. For example, the computing device can input a prompt to the first generative model (1210) to expand the input image by moving a certain border of the input image in a certain direction and a certain distance, and to generate an image for the expanded area while maintaining the features of the input image (e.g., the overall atmosphere and the contents of the foreground and background, etc.). The computing device can generate a prompt instructing outpainting based on user input (a transformation request) and the input image. That is, the computing device can generate a prompt instructing the first generative model (1210) to perform outpainting according to the method described in the embodiments below. A detailed description of the specific method by which the computing device expands the input image by considering inverse transformation in the third operation (1c) is as follows.

[0049] A computing device can determine whether an object (10) for which a user has requested a transformation (movement), i.e., an object (10) selected by a user input, is in contact with at least one border among the borders of an input image. Referring to FIG. 1, it can be seen that a first border (B1) among the borders of the input image is in contact with the object (10). Accordingly, the computing device can select, among the borders of the input image, the first border (B1) that is in contact with the object (10).

[0050] The computing device can move the first border (B1) in a distance and direction according to the inverse transformation (movement) according to the user input. In other words, the computing device can move the first border (B1) in the opposite direction to the movement (distance and direction) of the object (10) requested by the user input in the second operation (1b). The user input received by the computing device in the second operation (1b) includes a transformation request to move the object (10) in the upper left diagonal direction. Accordingly, the computing device can move the first border (B1) in the lower right diagonal direction. At this time, the distance by which the first border (B1) is moved can be determined based on the distance by which the object (10) is moved by the user input in the second operation (1b). For example, the distance by which the first border (B1) is moved can be the same as the distance by which the object (10) is moved by the user input in the second operation (1b).

[0051] When the first border (B1) is moved according to the inverse transformation, the computing device can expand the input image by generating (outpainting) an image up to the area where the first border (B1) is moved using the first generation model (1210). In other words, the computing device can generate a prompt that instructs the first generation model (1210) to generate an image up to the area where the first border (B1) is moved while maintaining the characteristics of the input image, and input the prompt into the first generation model (1210).

[0052] A computing device can also expand an input image using various other methods. Furthermore, the method by which the computing device expands an input image may vary even when a transformation other than translation (e.g., resizing, rotation, etc.) is applied to the object (10). This will be described in detail with reference to FIGS. 2 through 6.

[0053] FIG. 2 is a diagram illustrating a method for expanding an image by considering inverse transformation when an object included in the image is reduced, according to one embodiment of the present disclosure. FIG. 2 illustrates images corresponding to the first operation (2a) to the third operation (2c), respectively, in the order in which the images are processed.

[0054] Referring to FIG. 2, in a first operation (2a), a computing device can obtain an input image including an object (10). In a second operation (2b), the computing device can receive a user input requesting a reduction of the object (10). For example, a user can request a reduction of the object (10) by touching the object (10) with two fingers on a screen of the computing device on which the input image is displayed and performing a pinch-in operation.

[0055] In the third operation (2c), the computing device can expand the input image by considering the inverse transformation for the transformation (reduction) requested by the user. Specifically, the computing device can determine whether the object (10) for which the user requested reduction, i.e., the object (10) selected by the user input, touches at least one border among the borders of the input image. Referring to FIG. 2, it can be seen that the first border (B1) among the borders of the input image touches the object (10). Therefore, the computing device can select the first border (B1) that touches the object (10) among the borders of the input image.

[0056] The computing device can move the first border (B1) according to the inverse transformation (reduction) according to the user input. In other words, the computing device can move the first border (B1) so that the input image expands in the direction of the first border (B1) by the ratio in which the object (10) is reduced by the user input. (For example, if the size of the object (10) is reduced by 15% by the user input, the computing device moves the first border (B1) so that the input image expands in the direction of the first border (B1) by 15%.) Therefore, as illustrated in the third operation (2c), the computing device can extend the length of the first border (B1) by moving the first border (B1) in a direction away from the center of the object (10).

[0057] When the first border (B1) is moved according to the inverse transformation, the computing device can expand the input image by generating an image up to the area where the first border (B1) is moved using the first generation model (1210).

[0058] FIG. 3 is a diagram illustrating a method for expanding an image by considering inverse transformation when an object included in the image is rotated, according to one embodiment of the present disclosure. FIG. 3 illustrates images corresponding to the first operation (3a) to the third operation (3c), respectively, in the order in which the images are processed.

[0059] Referring to FIG. 3, in a first operation (3a), a computing device can obtain an input image including an object (10). In a second operation (3b), the computing device can receive a user input requesting rotation of the object (10). For example, a user can touch the object (10) with two fingers on a screen of the computing device on which the input image is displayed, and rotate the object (10) while maintaining the touch.

[0060] In the third operation (3c), the computing device can expand the input image by considering the inverse transformation for the transformation (rotation) requested by the user. Specifically, the computing device can determine whether the object (10) for which the user requested rotation, i.e., the object (10) selected by the user input, touches at least one border among the borders of the input image. Referring to FIG. 3, it can be seen that the first border (B1) among the borders of the input image touches the object (10). Therefore, the computing device can select the first border (B1) that touches the object (10) among the borders of the input image.

[0061] The computing device can move the first border (B1) according to the inverse transformation (rotation) according to the user input. In other words, the computing device can move the first border (B1) opposite to the rotation (direction and angle) of the object (10) requested by the user input in the second operation (3b). The user input received in the second operation (3b) requests that the object (10) be rotated counterclockwise. Therefore, as illustrated in the third operation (3c), the computing device can move the first border (B1) by rotating the first border (B1) clockwise about the left end of the first border (B1) as an axis. At this time, the size of the angle by which the first border (B1) is rotated can be determined based on the size of the angle by which the object (10) is rotated by the user input in the second operation (3b). For example, the size of the angle at which the first frame (B1) rotates may be the same as the size of the angle at which the object (10) rotates by user input in the second operation (3b).

[0062] When the first border (B1) is moved according to the inverse transformation, the computing device can expand the input image by generating an image up to the area where the first border (B1) is moved using the first generation model (1210).

[0063] According to one embodiment, the computing device can expand the input image even before a request for transformation of the object (10) is received when the object (10) is selected by user input. For example, the user can request the movement of the object (10) by touching the object (10) included in the input image with a finger for a certain period of time and moving the finger while maintaining the touch. The computing device can also expand the input image in advance before the user moves the finger. Since it may take some time for the user to decide how to transform the object (10) (e.g., where to move the object (10)) after selecting the object (10) (touching it for a certain period of time), the computing device can improve the speed at which the image is edited by expanding the image in advance before the user makes a decision on the transformation. From the user's perspective, the time required for the edited image to be output is shortened, so the user experience can be improved.

[0064] According to one embodiment, the computing device may expand the input image in advance before any user input is received, such as selecting an object (10) or requesting a transformation of the object (10), and then, when the user input is received, infer (determine) the object suggestion region based on the user input. For example, the computing device may expand the original image by a predetermined ratio (e.g., 1.5 times) through outpainting, temporarily store the expanded original image, and then, when the user input is received, perform a process for inferring (determining) the object suggestion region.

[0065] If the input image is expanded in a state where only the object (10) to be transformed is selected in this way and a specific transformation request is not received, the computing device cannot consider the reverse transformation. Therefore, according to one embodiment, the computing device can expand the image based on the size or position of the selected object (10), the size of the input image, etc. For example, the computing device can move the border of the input image by a distance determined based on the length of the selected object (10) (the length measured based on an axis parallel to the direction of movement of the border), or can move the border of the input image by a distance determined based on the range in which the selected object (10) can be transformed within the input image (e.g., based on how far and in which direction the object (10) can be moved within the input image). Alternatively, the computing device can move the border of the input image by a distance determined based on the length (width or height) of the input image. This will be described in detail below with reference to FIGS. 4A, 4B, and 5.

[0066] FIG. 4a is a diagram illustrating a method for expanding an image based on the size of a selected object, according to one embodiment of the present disclosure. FIG. 4a illustrates images corresponding to the first operation (4aa) to the third operation (4ac), respectively, in the order in which the images are processed.

[0067] Referring to FIG. 4a, in a first operation (4aa), a computing device can obtain an input image including an object (10). In a second operation (4ab), the computing device can receive a user input for selecting the object (10). For example, in the second operation (4ab), the user may select the object (10) as a target of transformation by touching the object (10) with a finger for a certain period of time, but may not yet have input a request for transformation (movement, resizing, rotation, etc.) of the object (10). Furthermore, according to one embodiment, in the second operation (4ab), the computing device may also receive a user input for requesting transformation of the object (10).

[0068] In the third operation (4ac), the computing device can expand the input image based on the size of the object (10). To expand the input image, the computing device can determine whether the selected object (10) is in contact with at least one border among the borders of the input image. Referring to FIG. 4a, it can be seen that the first border (B1) among the borders of the input image is in contact with the object (10). Accordingly, the computing device can select the first border (B1) that is in contact with the object (10) among the borders of the input image.

[0069] The computing device can move the first border (B1) away from the center of the object (10) and determine the distance by which the first border (B1) is to be moved based on the size of the object (10). According to one embodiment, the computing device can measure the length (d1) of the object (10) based on an axis parallel to the direction of movement of the first border (B1). Furthermore, according to one embodiment, the computing device can also measure the length (d1) of the object (10) based on an axis perpendicular to the first border (B1).

[0070] The computing device can move the first border (B1) by a distance (d2) obtained by multiplying the length (d1) of the measured object (10) by a preset ratio (e.g., 0.2). At this time, the preset ratio used in calculating the distance (d2) can be set to an appropriate value depending on conditions such as the required image quality or processing speed.

[0071] When the first border (B1) is moved, the computing device can expand the input image by generating an image up to the area where the first border (B1) is moved using the first generation model (1210).

[0072] FIG. 4b is a diagram illustrating a method for expanding an image based on its size, according to one embodiment of the present disclosure. FIG. 4b illustrates images corresponding to the first operation (4ba) to the third operation (4bc), respectively, in the order in which the images are processed.

[0073] Referring to FIG. 4b, in a first operation (4ba), a computing device can obtain an input image including an object (10). In a second operation (4bb), the computing device can receive a user input for selecting the object (10). For example, in the second operation (4bb), the user may select the object (10) as a target of transformation by touching the object (10) with a finger for a certain period of time, but may not yet have input requesting transformation (movement, resizing, rotation, etc.) of the object (10). Furthermore, according to one embodiment, in the second operation (4bb), the computing device may also receive a user input requesting transformation of the object (10).

[0074] In the third operation (4bc), the computing device can expand the input image based on the size of the input image. For example, the computing device can expand the input image based on the length (width or height) of the input image. To expand the input image, the computing device can determine whether the selected object (10) touches at least one border among the borders of the input image. Referring to FIG. 4b, it can be seen that the first border (B1) among the borders of the input image touches the object (10). Accordingly, the computing device can select the first border (B1) among the borders of the input image that touches the object (10).

[0075] The computing device can move the first border (B1) away from the center of the object (10) and determine the distance by which the first border (B1) is to be moved based on the length (width or height) of the input image. In one embodiment, the computing device can measure the length (d1) of the input image based on an axis perpendicular to the first border (B1). In addition, in one embodiment, the computing device can measure the length (d1) of the input image based on an axis parallel to the movement direction of the first border (B1).

[0076] The computing device can move the first border (B1) by a distance (d2) obtained by multiplying the length (d1) of the measured image by a preset ratio (e.g., 0.1). At this time, the preset ratio used in calculating the distance (d2) can be set to an appropriate value depending on conditions such as the required image quality or processing speed.

[0077] When the first border (B1) is moved, the computing device can expand the input image by generating an image up to the area where the first border (B1) is moved using the first generation model (1210).

[0078] FIG. 5 is a diagram illustrating a method for expanding an image based on the transformable range of a selected object, according to one embodiment of the present disclosure. FIG. 5 illustrates images corresponding to the first operation (5a) to the third operation (5c), respectively, in the order in which the images are processed.

[0079] Referring to FIG. 5, in a first operation (5a), a computing device can obtain an input image including an object (10). In a second operation (5b), the computing device can receive a user input for selecting the object (10). For example, in the second operation (5b), the user may select the object (10) as a target of transformation by touching the object (10) with a finger for a certain period of time, but may not yet have made an input requesting transformation (movement, resizing, rotation, etc.) of the object (10). Furthermore, according to one embodiment, in the second operation (5b), the computing device may also receive a user input requesting transformation of the object (10).

[0080] In the third operation (5c), the computing device can expand the input image based on the transformable range of the object (10). Specifically, the computing device can determine the transformable (e.g., moveable) range of the object (10) within the input image based on the size and position of the object (10), and expand the input image based on the determined range.

[0081] To expand the input image, the computing device can determine whether the selected object (10) is in contact with at least one border among the borders of the input image. Referring to FIG. 5, it can be seen that the first border (B1) among the borders of the input image is in contact with the object (10). Accordingly, the computing device can select the first border (B1) that is in contact with the object (10) among the borders of the input image.

[0082] The computing device can move the first border (B1) away from the center of the object (10) and determine the distance by which the first border (B1) is to be moved based on the transformable range of the object (10). According to one embodiment, the computing device can measure the distance (d1) by which the object (10) can move based on an axis parallel to the moving direction of the first border (B1) in the third operation (5c). The computing device can move the first border (B1) by a distance (d2) obtained by multiplying the measured distance (d1) by a preset ratio (e.g., 1). At this time, the preset ratio used in calculating the distance (d2) can be set to an appropriate value depending on conditions such as the quality of the required image or the processing speed.

[0083] FIG. 6 is a diagram illustrating a method for expanding an image by moving all borders located in the inverse transformation direction according to one embodiment of the present disclosure. FIG. 6 illustrates images corresponding to the first operation (6a) to the third operation (6c), respectively, in the order in which the images are processed.

[0084] Referring to FIG. 6, in a first operation (6a), a computing device can obtain an input image including an object (10). In a second operation (6b), the computing device can receive a user input requesting movement of the object (10). For example, a user can request movement of the object (10) by touching the object (10) on a screen of the computing device on which the input image is displayed and moving a finger while maintaining the touch.

[0085] In the third operation (6c), the computing device can expand the input image by considering the inverse transformation for the transformation (movement) requested by the user. Specifically, the computing device can select all borders located in the inverse transformation direction among the borders of the input image. In the second operation (6b), the user input received by the computing device includes a transformation request to move the object (10) in the upper left diagonal direction. Since the inverse transformation direction is the lower right direction, the computing device can select the borders (B1, B2) located on the lower and right sides.

[0086] The computing device can move the first border (B1) and the second border (B2) by a distance and direction according to the inverse transformation (movement) according to the user input. For example, the computing device can move the first border (B1) downward by the distance by which the object (10) moves upward by the user input received in the second operation (6b). Similarly, the computing device can move the second border (B2) to the right by the distance by which the object (10) moves leftward by the user input received in the second operation (6b).

[0087] When the first border (B1) and the second border (B2) are moved according to the inverse transformation, the computing device can expand the input image by generating an image up to the area where the borders (B1, B2) are moved using the first generation model (1210).

[0088] So far, we have looked at various methods for expanding an input image. Returning to Fig. 1, when the input image is expanded in the third operation (1c), the computing device can infer (determine) the object proposal area (100a) in the fourth operation (1d). Meanwhile, the image included in the expanded area in the third operation (1c) can also be used as a guide when performing inpainting after object transformation (e.g., generating an image for the area requiring generation). In other words, when performing inpainting after object transformation, the computing device can refer to the image generated when expanding the input image in the third operation (1c), thereby increasing image generation efficiency.

[0089] As described above, the 'object suggestion region' may mean a region in which an object (10) is judged or predicted to exist within an image. The computing device may infer (determine) a region including an object (10) in an extended input image as an object suggestion region (100a). According to one embodiment, the computing device may infer (determine) an object suggestion region (100b) corresponding to the object (10) transformed (moved) in a fifth operation (1e) based on the object suggestion region (100a) inferred (determined) in a fourth operation (1d). For convenience of explanation, the object suggestion region (100a) inferred (determined) in the fourth operation (1d) is referred to as a first object suggestion region, and the object suggestion region (100b) inferred (determined) in the fifth operation (1e) is referred to as a second object suggestion region.

[0090] In the fourth operation (1d), the computing device can infer (determine) the first object suggestion region (100a) in the form of a bounding box. According to one embodiment, the computing device can infer (determine) the bounding box including the object (10) as the first object suggestion region (100a) through a method used in general pedestrian detection technology, and can also infer (determine) the first object suggestion region (100a) through various other methods. Furthermore, according to one embodiment, the computing device can infer (determine) the first object suggestion region (100a) in a form other than a bounding box, which will be described in detail with reference to FIG. 7.

[0091] FIG. 7 is a diagram illustrating a method for inferring (determining) an object proposal region from an image, according to one embodiment of the present disclosure. FIG. 7 illustrates three examples of different methods for inferring (determining) an object proposal region, represented by the first to third screens (7a, 7b, and 7c).

[0092] The first screen (7a) of Fig. 7 illustrates an example in which a computing device infers (determines) an object suggestion area (700a) in the form of a bounding box containing an object (10). This is the same method as described in the fourth operation (1d) of Fig. 1 above.

[0093] The second screen (7b) of Fig. 7 illustrates an example in which a computing device separates an object (10) through segmentation and infers (determines) only the separated object (10) as an object suggestion area (700b).

[0094] The third screen (7c) of FIG. 7 illustrates an example in which a computing device separates an object (10) through segmentation and infers (determines) an object proposal area (700c) to include not only the separated object (10) but also a portion of the margin around the object (10).

[0095] In the embodiments illustrated in FIGS. 1 and 8 to 10, the computing device infers (determines) the object suggestion region in the form of a bounding box, but the computing device may also infer (determine) the object suggestion region in the other two ways illustrated in FIG. 7, and may also infer (determine) the object suggestion region in various other ways.

[0096] Returning to FIG. 1 again, if the computing device has inferred (determined) the first object proposal region (100a) from the expanded input image in the fourth operation (1d), it can transform the object (10) in the fifth operation (1e) and infer (determine) the second object proposal region (100b) corresponding to the transformed object (10).

[0097] The operation performed by the computing device in the fifth operation (1e) is described in detail as follows. The computing device can transform the object (10) according to a request included in the user input received in the second operation (1b). That is, the computing device can move the object (10) in the upper left direction on the input image. The computing device can infer (determine) a second object proposal area (100b) corresponding to the object (10) at the position after movement based on the first object proposal area (100a) inferred (determined) in advance. For example, the computing device can infer (determine) a second object proposal area (100b) corresponding to the moved object (10) by reflecting the size and shape of the first object proposal area (100a), the positional relationship between the first object proposal area (100a) and the object (10), etc. According to one embodiment, the object (10) included in the second object proposal area (100b) may be smaller than the object (10) included in the first object proposal area (100a) because it is extracted from an unexpanded input image. However, as illustrated in FIG. 1, the second object proposal area (100b) may have the same size as the first object proposal area (100a), and thus, an area (200) in which generation of an additional image of the object (10) is required may exist in the second object proposal area (100b). The computing device may infer (determine) this area (200) as a generation required area. Alternatively, for example, the computing device may infer (determine) the second object proposal area (100b) by moving the first object proposal area (100a) in the distance and direction in which the object (10) moves according to a user's request.

[0098] In summary, the computing device can infer (determine) the region requiring generation (200) based on the second object proposal region (100b). Specifically, the computing device can infer (determine) the region requiring generation (200) as the region requiring generation, by comparing the converted object (10) with the second object proposal region (100b).

[0099] As previously explained, the 'region requiring generation' may mean a region requiring generation of an image for image editing. According to one embodiment, the region requiring generation may be divided into an object portion and a background portion. In the fifth operation (1e) of FIG. 1, the portion (200) requiring additional generation of the object (10) image may be referred to as a first region requiring generation, and the empty spaces (regions requiring additional generation of a background image) created due to transformation (movement) of the object (10) may be referred to as a second region requiring generation. That is, the computing device may infer (determine) the region requiring generation of a background image due to transformation of the object (10) as the second region requiring generation.

[0100] To summarize the fourth operation (1d) and the fifth operation (1e), the computing device can infer (determine) an area including an object (10) in the expanded input image as a first object proposal area (100a), and move (transform) the object (10) on the input image according to a user input received in the second operation (1b). Next, the computing device can infer (determine) a second object proposal area (100b) corresponding to the moved object (10) based on the first object proposal area (100a). Finally, the computing device can infer (determine) a first generation-necessary area (200) and a second generation-necessary area (empty spaces created due to the movement of the object (10)) based on the second object proposal area (100b).

[0101] According to one embodiment of the present disclosure, the computing device may infer (determine) a region requiring generation (200) by considering the object (10). For example, the computing device may determine a region requiring generation of an image of the object (10) based on the size of the object (10) and the direction of the object (10) (e.g., the direction in which the front of the object (10) is facing), and infer (determine) the determined region as the region requiring generation (200). Alternatively, for example, the computing device may identify the object (10) by performing object recognition on an input image (e.g., identify the type of the object), and determine a region requiring generation of an image of the object (10) based on the identification result of the object (10) (e.g., the type of the object), and infer (determine) the determined region as the region requiring generation (200).

[0102] In the sixth operation (1f), the computing device can generate images for the first generation-required area (200) and the second generation-required area (empty spaces created due to movement of the object (10)), edit the images, and output the edited images. According to one embodiment, as described above with reference to FIG. 12, the computing device can generate an image for the first generation-required area (200) using the second generation model (1220), and can generate an image for the second generation-required area using the third generation model (1230).

[0103] Meanwhile, according to one embodiment of the present disclosure, in the fifth operation (1e), the computing device may infer (determine) the entire second object proposal area (100b) as a region requiring generation. Accordingly, the computing device may generate a new image for the entire second object proposal area (100b) using the second generation model (1220) for object image generation. In this case, since the image for the entire second object proposal area (100b) is generated by reflecting the surrounding background at the position after the transformation of the object (10), the effect of generating the image so as to better match the surrounding background can be expected.

[0104] According to one embodiment of the present disclosure, in the sixth operation (1f), the computing device may input a prompt to the generative model to inpaint the region requiring generation. For example, the computing device may generate a prompt to instruct the second generative model (1220) to generate an image while maintaining the characteristics of the input image (e.g., the overall atmosphere and the contents of the foreground and background, etc.) for the first region requiring generation (200) inferred (determined) in the fifth operation (1e). To this end, the computing device may generate a prompt to inpaint based on a user input (a conversion request) and the input image.

[0105] Meanwhile, as described above, the image generated when the input image is expanded in the third operation (1c) can be used as a guide when generating an image for the first generation-required area (200). According to one embodiment, the computing device can generate an image of an object (10) in the first generation-required area (200) similar to the image of the object (10) included in the expanded input image in the third operation (1c), but can generate the image so as to match the background in the first generation-required area (200). To this end, the computing device can generate a prompt that instructs to generate an image for the first generation-required area (200) while maintaining the characteristics of the image of the expanded area in the third operation (1c), and input the prompt into the second generation model (1220).

[0106] So far, embodiments of a method of inferring (determining) an object proposal region after expanding an input image by considering the transformation of the object (10) or the inverse transformation of the transformation of the object (10) have been described. Hereinafter, with reference to FIGS. 8 and 9, embodiments of a method of inferring (determining) an object proposal region without expanding the input image and expanding the object proposal region by considering the inverse transformation of the object (10) will be described. According to these embodiments, since the image (the image within the object proposal region) is expanded by reflecting the surrounding background of the transformed object (10), it is expected that the image is generated so that the object (10) better matches the surrounding background compared to the embodiment described above with reference to FIG. 1.

[0107] FIGS. 8 and 9 are diagrams for explaining embodiments of a method for inferring (determining) an object proposal region without expanding an input image and expanding the object proposal region by considering the inverse transformation of the object's transformation. FIG. 8 shows images corresponding to the first operation (8a) to the eighth operation (8h) in the order of image processing. FIG. 9 shows images corresponding to the first operation (9a) to the seventh operation (9g) in the order of image processing.

[0108] Referring to FIG. 8, in a first operation (8a), a computing device can obtain an input image including an object (10). In a second operation (8b), the computing device can receive a user input requesting a transformation of the object. For example, a user can request a movement of the object (10) by touching the object (10) on a screen of the computing device on which the input image is displayed and moving a finger while maintaining the touch. In the embodiment disclosed in FIG. 8, the transformation of the object (10) requested by the user is a 'movement' of the object, but is not limited thereto, and the user can also request a resize (enlarge, reduce) or rotation of the object (10).

[0109] In the third operation (8c), the computing device can infer (determine) an area including an object (10) in the input image as an object proposal area (100a). According to one embodiment, the computing device can infer (determine) an object proposal area (100b) corresponding to the transformed (moved) object (10) in the fourth operation (8d) based on the object proposal area (100a) inferred (determined) in the third operation (8c). For convenience of explanation, the object proposal area (100a) inferred (determined) in the third operation (8c) is referred to as a first object proposal area, and the object proposal area (100b) inferred (determined) in the fourth operation (8d) is referred to as a second object proposal area.

[0110] In the third operation (8c), the computing device can infer (determine) the first object suggestion region (100a) in the form of a bounding box. According to one embodiment, the computing device can infer (determine) the bounding box including the object (10) as the first object suggestion region (100a) through a method used in general pedestrian detection technology, and can also infer (determine) the first object suggestion region (100a) through various other methods. Furthermore, according to one embodiment, the computing device can infer (determine) the first object suggestion region (100a) in a form other than a bounding box, as described above with reference to FIG. 7.

[0111] If the computing device infers (determines) the first object proposal area (100a) from the input image in the third operation (8c), it can transform the object (10) in the fourth operation (8d) and infer (determine) the second object proposal area (100b) corresponding to the transformed object (10).

[0112] The operation performed by the computing device in the fourth operation (8d) is described in detail as follows. The computing device can transform the object (10) according to a request included in the user input received in the second operation (8b). That is, the computing device can move the object (10) in the upper left direction on the input image. The computing device can infer (determine) the second object proposal area (100b) corresponding to the object (10) at the position after movement based on the first object proposal area (100a) extracted in advance. For example, the computing device can infer (determine) the second object proposal area (100b) corresponding to the moved object (10) by reflecting the size and shape of the first object proposal area (100a), the positional relationship between the first object proposal area (100a) and the object (10), etc.

[0113] In the fifth operation (8e), the computing device can expand the second object proposal area (100b) by considering the inverse transformation of the transformation of the object (10). The method of expanding the second object proposal area (100b) by considering the inverse transformation may be similar to the third operation (1c) of FIG. 1 and the method described with reference to FIGS. 2 to 6. In the previous embodiments, at least one border that touches the object (10) among the borders of the input image is moved by a distance and direction according to the inverse transformation. In the fifth operation (8e) of FIG. 8, at least one border that touches the object (10) among the borders of the second object proposal area (100b) is moved by a distance and direction according to the inverse transformation, and an image can be generated (outpainted) using the first generation model up to the area where the border is moved. The method of moving the border according to the transformation of the object (10) may be the same as described in the previous embodiments. The extended second object proposal area (100c) considering the inverse transformation is shown in the sixth movement (8f).

[0114] In the seventh operation (8g), the computing device can infer (determine) the generation-required region (200) based on the extended second object proposal region (100c). A detailed description of how the computing device infers (determines) the generation-required region (200) is as follows. The computing device can transform (move) the object (10) according to a request included in the user input and apply the extended second object proposal region (100c) to the transformed object (10). By comparing the transformed object (10) and the extended second object proposal region (100c), the computing device can infer (determine) the region in which an image for the transformed object (10) must be additionally generated as the generation-required region (200). In the seventh operation (8g), the object (10) included in the expanded second object proposal area (100c) may be smaller than the object (10) included in the expanded second object proposal area (100c) in the sixth operation (8f), since it is extracted from the non-expanded input image. Accordingly, in the expanded second object proposal area (100c), there may be an area (200) where generation of an additional image of the object (10) is required. The computing device may infer (determine) this area (200) as a region requiring generation.

[0115] In summary, the computing device can infer (determine) the region requiring generation (200) based on the extended second object proposal region (100c). Specifically, the computing device can infer (determine) the region requiring generation (200) as the region requiring generation, by comparing the converted object (10) and the extended second object proposal region (100c).

[0116] According to one embodiment, the region requiring generation can be divided into an object portion and a background portion. In the seventh operation (8g) of FIG. 8, the portion (200) requiring additional generation of the object (10) image can be referred to as a first region requiring generation, and the empty spaces (part requiring additional generation of a background image) created due to transformation (movement) of the object (10) can be referred to as a second region requiring generation. That is, the computing device can infer (determine) the region requiring generation of a background image due to transformation of the object (10) as the second region requiring generation.

[0117] In the eighth operation (8h), the computing device can generate images for the first generation-required area (200) and the second generation-required area (empty spaces created due to movement of the object (10)), edit the images, and output the edited images. According to one embodiment, as described above with reference to FIG. 12, the computing device can generate an image for the first generation-required area (200) using the second generation model (1220), and can generate an image for the second generation-required area using the third generation model (1230).

[0118] According to the embodiment described above, as the second object suggestion region (100b) is expanded in the fifth operation (8e), the expanded second object suggestion region (100c) in the sixth operation (8f) already includes an image of the expanded object (10). However, in the seventh operation (8g), the computing device compares the object (10) before expansion (an object whose position has only been moved due to transformation) with the expanded second object suggestion region (100c) to infer (determine) the region requiring generation (200), and expands the object (10) by generating an image for the region requiring generation (200) using the second generation model (1220). The reason for this is as follows.

[0119] In the sixth operation (8f), the image included in the expanded second object proposal area (100c) is an image generated using the first generation model (1210). As described above, the first generation model (1210) is a model used to 'temporarily' expand an image in order to infer (determine) the generation-required area (200). In contrast, the second generation model (1220) is a model used to generate an image to be output, and can generate a more precise image than the first generation model (1210). Therefore, the computing device expands the second object proposal area (100b) using the first generation model (1210), and then infers (determines) the generation-required area (200) based on the expanded second object proposal area (100c), and generates an image for the generation-required area (200) again using the second generation model (1220), thereby improving the quality of the output image.

[0120] According to one embodiment, the process of inferring (determining) the generation-required area (200) in the seventh operation (8g) and generating an image for the generation-required area (200) may be omitted, and the output image may be generated by directly using the image within the expanded second object proposal area (100c) in the sixth operation (8f). Such an embodiment is illustrated in Fig. 9.

[0121] Comparing Fig. 9 with Fig. 8, all remaining operations are identical except that the seventh operation (8g) of Fig. 8 has been deleted. Therefore, descriptions of identical operations are omitted, and only the sixth operation (9f) and seventh operation (9g) of Fig. 9 are described.

[0122] In the sixth operation (9f), the computing device can infer (determine) only the empty spaces (areas requiring additional generation of a background image) resulting from the transformation (movement) of the object (10) as the generation-required area (second generation-required area). In the seventh operation (9g), the computing device can output an edited image by using the image of the object (10) included in the extended second object proposal area (100c) as it is and generating an image for the second generation-required area using the third generation model.

[0123] Below, the differences between the method according to the embodiment illustrated in Fig. 1 (hereinafter, “first method”) and the method according to the embodiment illustrated in Fig. 8 (hereinafter, “second method”) will be described.

[0124] In the case of the first method, the image is expanded at the position before the object (10) is transformed, so the image is generated by reflecting the background at the position before the object (10) is transformed, and the object proposal area can be inferred (determined) accordingly.

[0125] In the second method, the object suggestion area is inferred (determined) from the position after the transformation of the object (10), and the object suggestion area can be inferred (determined) by reflecting the background of the position after the transformation of the object (10).

[0126] Therefore, when comparing the object proposal area of ​​the first method (the first object proposal area (100a) of the fourth operation (1d) of FIG. 1) with the object proposal area of ​​the second method (the extended second object proposal area (100c) of the sixth operation (8f) of FIG. 8), the object proposal area (100c) of the second method is inferred (determined) to better match the surroundings of the position after transformation of the object (10), and therefore, it is expected that a more natural output image will be obtained when using the second method.

[0127] Figure 10 is a diagram illustrating an embodiment of image editing using an object image generated in the process of expanding an input image by considering the inverse transformation of the object. Figure 10 illustrates images corresponding to the first operation (10a) to the sixth operation (10f) in the order of image processing.

[0128] Comparing Fig. 10 with Fig. 1, only the fifth motion (1e, 10e) is different, and the remaining motions are identical. Therefore, descriptions of identical motions are omitted, and only the fifth motion (10e) of Fig. 10 is described.

[0129] In the fifth operation (10e), the computing device can transform the object (10) according to a request included in the user input received in the second operation (10b). That is, the computing device can move the object (10) in the upper left direction on the input image. The computing device can infer (determine) a second object suggestion area (100b) corresponding to the object (10) at the position after movement, based on the first object suggestion area (100a) inferred (determined) in advance. For example, the computing device can infer (determine) a second object suggestion area (100b) corresponding to the moved object (10) by reflecting the size and shape of the first object suggestion area (100a), and the positional relationship between the first object suggestion area (100a) and the object (10).

[0130] Instead of generating a new image of an object (10) to be included in the second object suggestion area (100b), the computing device can insert the image of the object (10) included in the first object suggestion area (100a) into the second object suggestion area (100b). Subsequently, the computing device can infer (determine) only the empty spaces (areas requiring additional generation of a background image) resulting from the transformation (movement) of the object (10) as the area requiring generation (the second area requiring generation).

[0131] In the sixth operation (10f), the computing device can output an edited image by generating an image for the second generation required area using the third generation model.

[0132] As described above, in the embodiment illustrated in FIG. 10, the computing device can increase the processing speed by using the image of the object (10) generated by the first generation model (1210) as it is at the position after the transformation of the object (10) when expanding the input image for inference (determination) of the object proposal area. According to one embodiment, the computing device can adjust the color or brightness, etc. of the image of the object (10) according to the surrounding background by using a harmonizer or the like so that the image of the object (10) generated by the first generation model (1210) can blend well with the surrounding background at the position after the transformation of the object (10).

[0133] Hereinafter, with reference to FIG. 11, a configuration of a computing device for performing the image processing operations described so far will be described. FIG. 11 is a diagram illustrating the configuration of a computing device for performing image editing using a generative model according to one embodiment of the present disclosure.

[0134] Referring to FIG. 11, a computing device (1000) according to one embodiment may include an input / output interface (1100), a memory (1200), and a processor (1300). However, the components of the computing device (1000) are not limited to the examples described above, and the computing device (1000) may include more or fewer components than the components described above. In one embodiment, some or all of the input / output interface (1100), the memory (1200), and the processor (1300) may be implemented in the form of a single chip, and the processor (1300) may include one or more processors.

[0135] The input / output interface (1100) may include an input interface (e.g., touch screen, hard button, microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., display panel, speaker, etc.) for displaying the results of execution of an operation according to the user's control or the status of the computing device (1000).

[0136] The memory (1200) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (1200) may not exist separately and may be configured to be included in the processor (1300). The memory (1200) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. Programs or instructions for performing operations according to the embodiments described with reference to FIGS. 1 to 10 may be stored in the memory (1200). The memory (1200) may also provide stored data to the processor (1300) upon request of the processor (1300).

[0137] The processor (1300) controls a series of processes so that the computing device (1000) operates according to the embodiments described with reference to FIGS. 1 to 10, and may be configured with one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. For example, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0138] The processor (1300) can record data in the memory (1200) or read data stored in the memory (1200), and in particular, process data according to predefined operation rules or artificial intelligence models by executing programs or commands stored in the memory (1200). Accordingly, the processor (1300) can perform the operations described in the embodiments described above, and the operations described as being performed by the computing device (1000) in the embodiments described above can be regarded as being performed by the processor (1300) unless otherwise specified.

[0139] FIGS. 13 to 19 are flowcharts illustrating a method for editing an image using a generation model according to embodiments of the present disclosure. Hereinafter, a method for editing an image using a generation model according to embodiments of the present disclosure will be described with reference to FIGS. 13 to 19 . Since the operations described below are performed by the computing device (1000) described so far, the contents included in the previously described embodiments may be equally applied even if omitted below.

[0140] Referring to FIG. 13, in operation 1301, the computing device may receive user input requesting transformation of at least one object included in an input image.

[0141] In operation 1302, the computing device can transform at least one object according to the user's request, and infer (determine) a generation-required region, which is an area requiring image generation, by considering the inverse transformation of the transformation. Operation 1302 is identical to operation 1403 of FIG. 14, and therefore, the descriptions given below for operation 1403 can also be applied to operation 1302.

[0142] In operation 1303, the computing device can generate an image of a region requiring generation using one or more generative models.

[0143] Referring to FIG. 14, at operation 1401, the computing device may receive user input requesting transformation of at least one object included in an input image.

[0144] In operation 1402, the computing device can determine whether at least one border among the borders of the input image touches at least one object (an object that is the target of the transformation request). If the determination result indicates that at least one border does not touch at least one object, the computing device proceeds to operation 1405, where the computing device can transform at least one object according to a request included in the user input. Conversely, if the determination result indicates that at least one border touches at least one object, the computing device proceeds to operation 1403.

[0145] In operation 1403, the computing device can transform at least one object according to the user's request, and infer (determine) a generation-required area, which is an area where image generation is required, by considering the inverse transformation of the transformation. Detailed operations included in operation 1403 are illustrated in FIGS. 15 and 19, respectively. The operations in FIG. 15 may correspond to the first method described above with reference to FIGS. 1 to 6, and the operations in FIG. 19 may correspond to the second method described above with reference to FIGS. 8 and 9.

[0146] Referring to FIG. 15, in operation 1501, the computing device can expand the input image by considering the inverse transformation of the object's transformation. The detailed operations included in operation 1501 are described with reference to FIGS. 16 to 18.

[0147] Referring to FIG. 16, in operation 1601, the computing device may select at least one border that touches at least one object among the borders of the input image. In operation 1602, the computing device may move the selected at least one border by a distance and direction according to the inverse transformation. In operation 1603, the computing device may generate an image up to the area where the border has been moved using one or more generative models.

[0148] Referring to FIG. 17, in operation 1701, the computing device may select at least one border that touches at least one object selected from among borders of the input image or at least one border that cuts at least one object. In operation 1702, the computing device may move the at least one selected border by a distance determined based on a size of the at least one object. For example, the computing device may move the at least one selected border by a distance that is calculated by multiplying the length of the at least one object by a preset ratio. Alternatively, for example, the computing device may determine a movement distance based on a transformable range of the at least one object within the input image, and move the at least one selected border by the determined movement distance.

[0149] In operation 1703, the computing device can generate an image using one or more generative models up to a region where at least one border has been moved.

[0150] Referring to FIG. 18, in operation 1801, the computing device may select at least one border that touches at least one object selected from among the borders of the input image or at least one border that cuts off at least one object. In operation 1802, the computing device may move the at least one selected border by a distance determined based on a length of the input image measured relative to an axis perpendicular to the at least one selected border. For example, the computing device may move the at least one selected border by a distance that is calculated by multiplying the length of the input image by a preset ratio. In operation 1803, the computing device may generate an image up to an area where the at least one border has been moved using one or more generative models.

[0151] Returning to FIG. 15 again, in operation 1502, the computing device can infer (determine) a region including at least one object in the expanded input image as a first object proposal region. In operation 1503, the computing device can transform at least one object according to a request included in the user input received in operation 1401. In operation 1504, the computing device can infer (determine) a second object proposal region corresponding to the transformed at least one object based on the first object proposal region. In operation 1505, the computing device can infer (determine) a generation-required region based on the second object proposal region.

[0152] Referring to FIG. 19, in operation 1901, the computing device can infer (determine) an area including at least one object in an input image as a first object proposal area. In operation 1902, the computing device can transform at least one object according to a request included in a user input received in operation 1401. In operation 1903, the computing device can infer (determine) a second object proposal area corresponding to the transformed at least one object based on the first object proposal area. In operation 1904, the computing device can expand the second object proposal area by considering an inverse transformation of the object. In operation 1905, the computing device can infer (determine) a generation-required area based on the expanded second object proposal area.

[0153] Returning to FIG. 14 again, at operation 1404, the computing device may generate an image of a region requiring generation using one or more generative models.

[0154] Hereinafter, a method of editing an image using a generation model according to one embodiment of the present disclosure will be described with reference to FIGS. 20 to 23.

[0155] Referring to FIG. 20, in operation 2001, the computing device may receive user input requesting movement of at least one object included in an input image. As described above, the user may request movement of an object included in the input image in various ways.

[0156] In operation 2002, the computing device can expand the input image in a direction determined based on the movement of the requested object. The computing device can determine the direction in which to expand the input image by considering the movement of the requested object. For example, the computing device can determine the direction opposite the direction of movement of the object as the expansion direction of the input image. Figure 21 illustrates detailed operations included in operation 2002.

[0157] Referring to FIG. 21, in operation 2101, the computing device may determine whether at least one object is a partial object in which only a portion of the object is displayed. According to one embodiment of the present disclosure, the computing device may perform object recognition on an input image and then determine whether the object is a partial object based on the recognition result for at least one object. Alternatively, according to one embodiment of the present disclosure, the computing device may determine whether the object is a partial object based on whether the object touches the border of the input image or whether the object is cut off by the border of the input image. Detailed operations included in operation 2101 are illustrated in FIG. 22.

[0158] Referring to FIG. 22, in operation 2201, the computing device can determine whether at least one object is in contact with at least one border among the borders of the input image or whether at least one object is cut off by at least one border among the borders of the input image. Then, in operation 2202, the computing device can determine at least one object as a partial object if at least one object is in contact with at least one border or at least one object is cut off by at least one border.

[0159] Returning to FIG. 21 again, in operation 2102, the computing device may generate an image to extend the input image in a direction opposite to the movement direction of the requested object, if at least one object is a partial object. According to one embodiment of the present disclosure, the computing device may select at least one border that touches at least one object or at least one border that cuts at least one object from among the borders of the input image, move the selected at least one border in a direction opposite to the movement direction of the requested object, and then generate an image up to the area to which the at least one border has moved using one or more generation models.

[0160] Returning to FIG. 20 again, in operation 2003, the computing device may infer (determine) a region requiring generation of an image of at least one object based on the expanded input image. According to one embodiment of the present disclosure, the computing device may infer (determine) a region requiring generation of an image of at least one object as a region requiring generation based on the size of at least one object and the orientation of the at least one object. Alternatively, according to one embodiment of the present disclosure, the computing device may identify at least one object included in the input image (e.g., determine the type of object), and then infer (determine) a region requiring generation of an image of at least one object as a region requiring generation based on the identification result (e.g., type of object). Alternatively, according to one embodiment of the present disclosure, the computing device may infer (determine) a region including an object in the expanded input image as an object suggestion region, and then infer (determine) a region requiring generation based on the object suggestion region. Detailed operations included in operation 2003 are illustrated in FIG. 23.

[0161] Referring to FIG. 23, in operation 2301, the computing device can infer (determine) an area including at least one object in the extended input image as a first object proposal area.

[0162] In operation 2302, the computing device can move at least one object upon request.

[0163] In operation 2303, the computing device can infer (determine) a second object proposal area corresponding to at least one moved object based on the first object proposal area.

[0164] In operation 2304, the computing device can infer (determine) a region requiring generation based on the second object suggestion region. According to one embodiment of the present disclosure, the computing device can infer (determine) a region requiring additional generation of an image for the at least one moved object as the first region requiring generation by comparing the at least one moved object with the second object suggestion region, and can infer (determine) a region requiring generation of a background image due to the movement of the object as the second region requiring generation.

[0165] The specific method by which the computing device infers (determines) the first object proposal area, the second object proposal area, and the generation-required area is as described above with reference to FIG. 1.

[0166] Returning to FIG. 20 again, in operation 2004, the computing device can generate an image of a region requiring generation using at least one generation model. According to one embodiment of the present disclosure, the computing device can generate an image of a region requiring generation by generating a prompt based on at least one of information regarding a location of the region requiring generation, information regarding a type of at least one object, and information regarding a background including the region requiring generation, and inputting the generated prompt into at least one generation model.

[0167] According to the embodiments described above, when a user requests a transformation of an object included in an image, an area requiring image generation (a region requiring generation) is inferred (determined) by considering the inverse transformation for the transformation, and an image is generated using a generation model for the inferred (determined) region requiring generation, thereby editing the image so that the transformed object blends well with the surrounding background. In addition, by inferring (determining) the minimum area requiring image generation as the region requiring generation, the speed and efficiency of image processing can be improved.

[0168] According to one embodiment, a method for editing an image using a generative model may include the steps of: receiving a user input requesting movement of at least one object included in an input image; expanding the input image in a direction determined based on the requested movement of the object; determining a generation required area, which is an area requiring generation of an image for the at least one object, based on the expanded input image; and generating an image for the generation required area using at least one generative model.

[0169] According to one embodiment, the step of expanding the input image may include the step of determining whether the at least one object is a partial object in which only a portion of the object is displayed, and if the at least one object is a partial object, the step of generating an image to expand the input image in a direction opposite to a movement direction of the requested object.

[0170] According to one embodiment, the step of determining whether the object is a partial object may include the step of checking whether the at least one object is in contact with at least one border among the borders of the input image, and the step of determining the at least one object as a partial object if the at least one object is in contact with the at least one border.

[0171] According to one embodiment, the step of generating an image to extend the input image may include the steps of selecting at least one border among borders of the input image that touches the at least one object, moving the selected at least one border in a direction opposite to a movement direction of the requested object, and generating an image up to an area to which the at least one border has moved using the at least one generation model.

[0172] According to one embodiment, the step of determining whether the object is a partial object may include the step of performing object recognition on the input image and the step of determining whether the object is a partial object based on the recognition result for the at least one object.

[0173] According to one embodiment, the step of inferring the region requiring generation may infer the region requiring generation of an image of the at least one object as the region requiring generation based on the size of the at least one object and the direction of the at least one object.

[0174] According to one embodiment, the step of inferring the region requiring generation may include the step of identifying the at least one object and the step of inferring the region requiring generation of an image of the at least one object as the region requiring generation based on the identification result.

[0175] According to one embodiment, the step of inferring the region requiring generation may include the steps of inferring an area including the at least one object in the extended input image as a first object proposal area, moving the at least one object according to the request, inferring a second object proposal area corresponding to the moved at least one object based on the first object proposal area, and inferring the region requiring generation based on the second object proposal area.

[0176] According to one embodiment, the step of inferring the region requiring generation based on the second object suggestion region may include a step of inferring a region in which an image for the at least one object moved must be additionally generated as a first region requiring generation by comparing the at least one object moved with the second object suggestion region, and a step of inferring a region in which a background image must be generated due to movement of the object as a second region requiring generation.

[0177] According to one embodiment, the input image is expanded using a first generation model, the image for the first generation-required region is generated using a second generation model, and the image for the second generation-required region is generated using a third generation model, wherein the second generation model may have higher performance than the first generation model or the third generation model.

[0178] According to one embodiment, the step of generating an image for the region requiring generation may include the step of generating a prompt based on at least one of information about a location of the region requiring generation, information about a type of the at least one object, and information about a background including the region requiring generation, and the step of inputting the generated prompt into the at least one generation model.

[0179] According to one embodiment, a computing device includes an input / output interface for receiving a user input requesting processing of an image, and outputting an image processed according to the user input, a memory storing commands for processing the image, and at least one processor, wherein the at least one processor executes the commands so that, when a user input requesting movement of at least one object included in an input image is received, the input image is expanded in a direction determined based on the requested movement of the object, and a generation required area, which is an area requiring generation of an image for the at least one object, is inferred based on the expanded input image, and then an image for the generation required area is generated using at least one generative model.

[0180] According to one embodiment, the at least one processor may generate an image by determining whether the at least one object is a partial object in which only a portion of the object is displayed when expanding the input image, and if the at least one object is a partial object, expanding the input image in a direction opposite to the direction of movement of the requested object.

[0181] According to one embodiment, when determining whether the at least one processor is a partial object, the processor may determine whether the at least one object is in contact with at least one border among the borders of the input image, and if the at least one object is in contact with the at least one border, the processor may determine the at least one object to be a partial object.

[0182] According to one embodiment, the at least one processor may generate an image to expand the input image by selecting at least one border among the borders of the input image that is in contact with the at least one object, moving the selected at least one border in a direction opposite to a movement direction of the requested object, and then generating an image up to an area to which the at least one border has been moved using the at least one generation model.

[0183] According to one embodiment, the at least one processor may perform object recognition on the input image to determine whether the object is a partial object, and then determine whether the object is a partial object based on the recognition result for the at least one object.

[0184] According to one embodiment, in inferring the region requiring generation, the at least one processor may infer a region requiring generation of an image of the at least one object as the region requiring generation based on a size of the at least one object and a direction of the at least one object.

[0185] According to one embodiment, in inferring the region requiring generation, the at least one processor may identify the at least one object and then infer the region requiring generation of an image of the at least one object as the region requiring generation based on the identification result.

[0186] According to one embodiment, when inferring the generation-required area, the at least one processor may infer an area including the at least one object in the extended input image as a first object proposal area, move the at least one object according to the request, infer a second object proposal area corresponding to the moved at least one object based on the first object proposal area, and then infer the generation-required area based on the second object proposal area.

[0187] According to one embodiment, when inferring the generation need area based on the second object suggestion area, the at least one processor may compare the at least one moved object with the second object suggestion area, thereby inferring an area in which an image for the at least one moved object must be additionally generated as a first generation need area, and then inferring an area in which a background image must be generated due to movement of the object as a second generation need area.

[0188] According to one embodiment, the at least one processor expands the input image using a first generation model, generates an image for the first generation-required region using a second generation model, and generates an image for the second generation-required region using a third generation model, wherein the second generation model may have higher performance than the first generation model or the third generation model.

[0189] According to one embodiment, the at least one processor may generate a prompt based on at least one of information about a location of the region requiring generation, information about a type of the at least one object, and information about a background including the region requiring generation, when generating an image for the region requiring generation, and then input the generated prompt into the at least one generation model.

[0190] According to one embodiment, a method for editing an image using a generative model may include the steps of: receiving a user input requesting transformation of at least one object included in an input image; determining whether at least one border among borders of the input image touches the at least one object; transforming the at least one object according to the request if the at least one border touches the at least one object, and inferring a generation required area, which is an area requiring generation of an image, by considering an inverse transformation for the transformation; and generating an image for the generation required area using one or more generative models.

[0191] According to one embodiment, the step of inferring the region requiring generation may include the steps of expanding the input image by considering the inverse transformation, inferring an area including the at least one object in the expanded input image as a first object proposal area, transforming the at least one object according to the request, inferring a second object proposal area corresponding to the transformed at least one object based on the first object proposal area, and inferring the region requiring generation based on the second object proposal area.

[0192] According to one embodiment, the step of expanding the input image may include the steps of selecting at least one border that is in contact with the at least one object among the borders of the input image, moving the at least one selected border by a distance and direction according to the inverse transformation, and generating an image up to an area where the at least one border has been moved using the at least one generation model.

[0193] According to one embodiment, the step of expanding the input image may include the step of selecting at least one border that is in contact with the at least one selected object from among the borders of the input image, the step of moving the at least one selected border by a distance determined according to a size of the at least one object, and the step of generating an image up to an area where the at least one border has been moved using the at least one generation model.

[0194] According to one embodiment, the step of moving the at least one object by a distance determined based on the size of the at least one object may move the at least one selected border by a distance obtained by multiplying the length of the at least one object by a preset ratio.

[0195] According to one embodiment, the step of moving the at least one object by a distance determined based on a size of the at least one object may determine a movement distance based on a range in which the at least one object can be transformed within the input image, and move the at least one selected border by the determined movement length.

[0196] According to one embodiment, the step of expanding the input image may include the steps of selecting at least one border among the borders of the input image that touches the at least one selected object, moving the at least one selected border by a distance determined based on a length of the input image measured relative to an axis perpendicular to the at least one selected border, and generating an image up to an area to which the at least one border has moved using the at least one generation model.

[0197] According to one embodiment, the step of moving the at least one selected border by a distance determined based on the length of the input image may move the at least one selected border by a distance obtained by multiplying the length of the input image by a preset ratio.

[0198] According to one embodiment, the step of inferring the region requiring generation based on the second object suggestion region may include a step of inferring a region in which an image for the at least one object that has been converted must be additionally generated as a first region requiring generation by comparing the at least one object that has been converted with the second object suggestion region, and a step of inferring a region in which a background image must be generated due to conversion of the object must be generated as a second region requiring generation.

[0199] According to one embodiment, the input image is expanded using a first generation model, the image for the first generation-required region is generated using a second generation model, and the image for the second generation-required region is generated using a third generation model, wherein the second generation model may have higher performance than the first generation model or the third generation model.

[0200] According to one embodiment, the step of inferring the region requiring generation may include the steps of inferring a region including the at least one object in the input image as a first object proposal region, transforming the at least one object according to the request, inferring a second object proposal region corresponding to the transformed at least one object based on the first object proposal region, expanding the second object proposal region by considering the inverse transformation, and inferring the region requiring generation based on the expanded second object proposal region.

[0201] According to one embodiment, the step of inferring the region requiring generation based on the extended second object proposal region may include the step of inferring a region in which an image for the at least one object converted must be additionally generated as a first region requiring generation by comparing the at least one object converted and the extended second object proposal region, and the step of inferring a region in which a background image must be generated due to conversion of the object must be generated as a second region requiring generation.

[0202] According to one embodiment, a computing device includes an input / output interface for receiving a user input requesting processing of an image, outputting an image processed according to the user input, a memory storing commands for processing the image, and at least one processor, wherein the at least one processor executes the commands so that, when receiving a user input requesting transformation of at least one object included in an input image, the computing device can determine whether at least one border among borders of the input image touches the at least one object, and if the at least one border touches the at least one object, transform the at least one object according to the request, infer a generation required area, which is an area where generation of an image is required, by considering an inverse transformation for the transformation, and generate an image for the generation required area using one or more generative models.

[0203] According to one embodiment, the at least one processor may, in inferring the generation-required area, expand the input image by considering the inverse transformation, infer an area including the at least one object in the expanded input image as a first object proposal area, transform the at least one object according to the request, infer a second object proposal area corresponding to the transformed at least one object based on the first object proposal area, and infer the generation-required area based on the second object proposal area.

[0204] According to one embodiment, the at least one processor may, when expanding the input image, select at least one border that is in contact with the at least one object among the borders of the input image, move the selected at least one border by a distance and direction according to the inverse transformation, and generate an image up to an area where the at least one border has been moved using the at least one generation model.

[0205] According to one embodiment, the at least one processor may, when expanding the input image, select at least one border that is in contact with the at least one selected object from among the borders of the input image, move the at least one selected border by a distance determined according to a size of the at least one object, and generate an image up to an area where the at least one border has been moved using the at least one generation model.

[0206] According to one embodiment, the at least one processor may move the at least one selected border by a distance determined based on a size of the at least one object, by multiplying the length of the at least one object by a preset ratio.

[0207] According to one embodiment, the at least one processor may determine a movement distance based on a range in which the at least one object is transformable within the input image, and move the at least one selected border by the determined movement length, wherein the movement distance is determined based on a size of the at least one object.

[0208] According to one embodiment, when inferring the generation need area based on the second object suggestion area, the at least one processor may compare the at least one transformed object with the second object suggestion area, thereby inferring an area in which an image for the at least one transformed object must be additionally generated as a first generation need area, and inferring an area in which a background image must be generated due to the transformation of the object as a second generation need area.

[0209] According to one embodiment, the at least one processor expands the input image using a first generation model, generates an image for the first generation-required region using a second generation model, and generates an image for the second generation-required region using a third generation model, wherein the second generation model may have higher performance than the first generation model or the third generation model.

[0210] According to one embodiment, the at least one processor may, in inferring the generation-required region, infer a region including the at least one object in the input image as a first object proposal region, transform the at least one object according to the request, infer a second object proposal region corresponding to the transformed at least one object based on the first object proposal region, expand the second object proposal region by considering the inverse transformation, and infer the generation-required region based on the expanded second object proposal region.

[0211] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0212] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0213] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0214] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0215] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method executed by at least one processor for processing an image using one or more generative models, A step of receiving a user input requesting movement of at least one object included in an input image; A step of expanding the input image in a direction determined based on the movement of the object; A step of determining a generation required area, which is an area requiring generation of a partial image for at least one object, based on the extended input image; A step of generating an image for the region requiring generation using at least one generative model; and A method comprising the step of outputting a reconstructed image based on the input image and a partial image of at least one object.

2. In paragraph 1, The step of expanding the above input image is: a step of determining whether at least one object is a partial object in which only a part of the object is displayed; and A method characterized by comprising the step of generating an image so as to expand the input image in a direction opposite to the movement direction of the object, if at least one object is a partial object.

3. In either of paragraphs 1 and 2, The step for determining whether the above is a partial object is: a step of determining whether at least one object is cut off by at least one border among the borders of the input image; and A method characterized by comprising a step of determining the at least one object as a partial object if the at least one object is cut by the at least one border.

4. In any one of paragraphs 1 to 3, The step of generating an image to extend the above input image is: A step of selecting a first border among at least one border that cuts off at least one object from among the borders of the input image; a step of moving the first border in the opposite direction to the movement direction of the requested object; and A method characterized by comprising the step of generating an image up to an area where at least one border has been moved using the one or more generative models.

5. In any one of paragraphs 1 to 4, The step of determining the area that needs to be created above is: A method characterized in that an area requiring generation of a partial image of the at least one object is determined as the generation-required area based on the size of the at least one object and the movement direction of the at least one object.

6. In any one of paragraphs 1 to 5, The step of determining the area that needs to be created above is: A step of determining an area including at least one object in the extended input image as a first object proposal area; A step of moving at least one object according to the user input; A step of determining a second object proposal area corresponding to at least one object that has been moved based on the first object proposal area; and A method characterized by comprising a step of determining the generation requirement area based on the second object proposal area.

7. In any one of paragraphs 1 to 6, The step of determining the generation requirement area based on the second object proposal area is as follows: A step of determining a first region in which a partial image for the at least one object moved must be additionally generated as a first generation required region by comparing the at least one object moved with the second object suggestion region; and A method characterized by comprising a step of determining a second area in which a background image must be generated due to movement of the object as a second generation-required area.

8. In any one of paragraphs 1 to 7, The above input image is extended using the first generative model, A partial image of at least one object moved within the first generation required area is generated using a second generation model, The background image within the above second generation required area is generated using the third generation model, A method characterized in that the second generation model has higher performance than the first generation model or the third generation model.

9. In any one of paragraphs 1 to 8, The step of generating an image for the above generation-required area is: A step of generating a prompt based on at least one of information about the location of the area requiring creation, information about the type of the at least one object, and information about the background including the area requiring creation; and A method characterized by comprising a step of inputting the generated prompt into at least one generation model.

10. An input / output interface (1100) for receiving a user input requesting processing of an input image and outputting a reconstructed image processed according to the user input; A memory (1200) in which instructions for processing the above input image are stored; and comprising at least one processor (1300) for executing the above instructions, By the above instructions, the at least one processor, When receiving user input requesting movement of at least one object contained in the input image, Expanding the input image in a direction determined based on the movement of the object, Based on the extended input image, a generation required area is determined, which is an area where a partial image for at least one object needs to be generated, After generating an image for the above generation-required area using at least one generative model, A computing device that outputs a reconstructed image based on the input image and a partial image of at least one object.

11. In paragraph 10, In expanding the input image, the at least one processor (1300) by the instructions, After determining whether at least one of the above objects is a partial object that displays only a part of the object, A computing device characterized in that the image is generated so as to extend the input image in the opposite direction to the movement direction of the object, if at least one object is a partial object.

12. In either of paragraphs 10 and 11, In determining whether the at least one object is a partial object, the at least one processor (1300) is, by the instructions, After checking whether at least one object is cut off by at least one border among the borders of the input image, A computing device characterized in that if the at least one object is cut by the at least one border, the at least one object is determined to be a partial object.

13. In any one of paragraphs 10 to 12, In generating an image to extend the input image, the at least one processor (1300) is, by the instructions, Selecting a first border among at least one border that cuts off at least one object from among the borders of the input image, After moving the first border in the opposite direction to the movement direction of the requested object, A computing device characterized in that it generates an image up to an area where at least one border has moved using the one or more generative models.

14. In any one of paragraphs 10 to 13, In determining the above generation requirement area, at least one processor (1300) by the instructions, In the extended input image, an area including at least one object is determined as a first object proposal area, Moving at least one object according to the above request, Based on the first object proposal area, a second object proposal area corresponding to at least one object that has been moved is determined, A computing device characterized in that the generation-required area is determined based on the second object proposal area.

15. In any one of paragraphs 10 to 14, In determining the generation need area based on the second object proposal area, the at least one processor (1300) by the instructions, By comparing the at least one object that has been moved with the second object suggestion area, a first area in which a partial image for the at least one object that has been moved must be additionally generated is determined as the first generation required area. A computing device characterized in that a second area in which a background image must be generated due to movement of the object is determined as a second generation-required area.

Images