Device and method for acquiring images from which unwanted objects have been removed

Abstract

An image acquisition device and method are disclosed. An image acquisition method using an artificial intelligence neural network may include: acquiring a first image through a camera; detecting objects included in the first image; classifying the detected objects into a main object and a sub-object; selecting an object to be removed among the detected objects; removing an object region in the first image where the object to be removed is located; and generating a second image by performing an inpainting restoration process on the removed object region.

Description

Technology Field

[0001] The disclosure relates to an image acquisition device and method, and specifically to an device and method for acquiring an image from which unwanted object image portions have been removed. Background Technology

[0003] Image inpainting is a technique that originated from the restoration of damaged artwork and is used to restore damaged areas in images, such as paintings, photographs, and videos, or to remove parts of an image.

[0004] When an area to be restored exists within an image, image inpainting technology is used by correcting pixels in the image region at the boundary of the restoration area and filling the restoration area with the corrected pixels. Conventional image inpainting technology requires a high level of creativity and concentration, so it is mostly performed by professional image technicians; consequently, ordinary people without expertise have often been unable to easily utilize image inpainting technology.

[0005] Meanwhile, advancements in computer programs are replacing human processing capabilities, and recently, deep learning-based artificial intelligence systems are being adopted, with processing methods evolving according to user preferences through machine learning. These AI systems can be used to acquire images such as photos or videos, and AI systems also need to use inpainting technology.

[0006] The aforementioned background technology is one that the inventor possessed or acquired in the process of deriving the content of the disclosure of the present application, and it cannot be considered as prior art disclosed to the general public prior to the filing of this application. The problem to be solved

[0008] The objective according to one embodiment is to provide an image acquisition device and method capable of distinguishing an object region from an image.

[0009] The objective according to one embodiment is to provide an image acquisition device and method capable of removing unwanted object regions from an image and providing them to the user.

[0010] The objective according to one embodiment is to provide an image acquisition device and method capable of automatically distinguishing a main object and a sub-object from an image and providing the image with the sub-object removed to a user. means of solving the problem

[0012] An image acquisition method according to one embodiment is performed through an artificial intelligence neural network and may include the operation of acquiring a first image through a camera, the operation of detecting objects included in the first image, the operation of classifying the detected objects into a main object and a sub-object, the operation of selecting an object to be removed among the detected objects, the operation of removing an object region in the first image where the object to be removed is located, and the operation of generating a second image by performing an inpainting restoration process on the removed object region.

[0013] In one embodiment, the operation of selecting the object to be removed may include selecting the object to be removed among the sub-objects based on the relative position information of the sub-object with respect to the main object within the first image.

[0014] In one embodiment, the operation of selecting the object to be removed may include the operation of detecting the gaze direction of the objects within the first image. The operation of selecting the object to be removed may include the operation of selecting the object to be removed among the sub-objects based on the detected gaze direction.

[0015] In one embodiment, the operation of selecting the object to be removed among the sub-objects can select the object to be removed depending on whether the detected line of sight direction is facing the camera.

[0016] In one embodiment, the operation of selecting the object to be removed among the sub-objects can select the object to be removed based on whether the detected gaze direction of the sub-object has similarity to the detected gaze direction of the main object.

[0017] In one embodiment, the operation of selecting the object to be removed may include selecting the object to be removed among the sub-objects based on the relative size of the sub-object to the main object within the first image.

[0018] In one embodiment, the operation of selecting the object to be removed may include an operation of tracking the movement of the objects within the first image. The operation of selecting the object to be removed may include an operation of determining the state of each of the objects to be either a dynamic state or a static state based on the tracked movement of the objects. The operation of selecting the object to be removed may include an operation of selecting the object to be removed among the sub-objects based on the determined state of the objects.

[0019] In one embodiment, the operation of selecting an object to be removed among the sub-objects may select a sub-object determined to be in a static state among the sub-objects as the object to be removed when the main object is in a dynamic state.

[0020] In one embodiment, the operation of classifying the detected objects into a main object and a sub-object may include receiving input from a user selecting the main object within the first image. The operation of classifying the detected objects into a main object and a sub-object may include classifying the sub-object based on the user input.

[0021] In one embodiment, the image acquisition method may further include an operation of pre-learning images of specific objects from a plurality of images. The operation of classifying the detected objects into main objects and sub-objects may include an operation of determining identity with the pre-learned specific object among the objects detected on the first image, and an operation of classifying the object determined to be identical to the specific object as the main object.

[0022] In one embodiment, the artificial intelligence neural network may be a model trained to extract external images with high similarity to the first image from an external server and to generate the second image using the extracted external images as training data.

[0023] In one embodiment, the artificial intelligence neural network may be a model trained to generate the second image using the second image as training data, using photos taken in real-time at the site with the background as the focus.

[0024] In one embodiment, the artificial intelligence neural network may be configured to extract external images from the external server that have the same location information as the shooting location information stored in the first image.

[0025] An image acquisition device according to one embodiment may include a camera for acquiring a first image, a processor for generating a second image by processing the first image through an artificial intelligence neural network, and a display for displaying at least one of the first image and the second image. In one embodiment, the processor may operate to input the first image into the artificial intelligence neural network to detect objects included in the first image, classify the detected objects into a main object and a sub-object, select an object to be removed among the classified objects, remove an object region in the first image where the object to be removed is located, and perform inpainting restoration processing on the removed object region to generate a second image.

[0026] In one embodiment, the processor may operate to detect the gaze direction of the objects within the first image and select the object to be removed among the sub-objects based on the similarity of the detected gaze direction of the main object and the sub-object.

[0027] In one embodiment, the image acquisition device may further include a communication unit that communicates with an external server. The processor may extract external images from the external server that have the same location information as the shooting location information stored in the first image, and operate to utilize external images similar to the first image among the extracted external images as training data for the artificial intelligence neural network. Effects of the invention

[0029] An image acquisition device and method according to one embodiment can process an image by distinguishing an object region from the image.

[0030] An image acquisition device and method according to one embodiment can remove an object region that the user does not want from an initial image and provide it to the user.

[0031] An image acquisition device and method according to one embodiment can automatically distinguish between a main object and a sub-object from an image and provide the image with the sub-object removed to the user. Brief explanation of the drawing

[0033] FIG. 1 is a block diagram of an image acquisition device according to one embodiment. FIG. 2 is a flowchart of an image acquisition method according to one embodiment. FIG. 3a is a flowchart of an operation for distinguishing objects based on user input according to one embodiment. FIG. 3b is a drawing showing an example of distinguishing objects according to user input, according to one embodiment. FIG. 4a is a flowchart of an operation in which a target object to be removed is selected, according to one embodiment. FIG. 4b is a drawing showing an example in which a target object for removal is selected based on relative position information with respect to a main object, according to one embodiment. FIG. 5a is a flowchart of an operation in which a target object to be removed is selected, according to one embodiment. FIG. 5b is a drawing showing an example in which a target object for removal is selected according to the viewing direction of an object on an image, according to one embodiment. FIG. 6a is a flowchart of an operation in which a target object to be removed is selected, according to one embodiment. FIG. 6b is a drawing showing an example in which objects to be removed are selected based on the relative sizes between objects in an image according to one embodiment. FIG. 7a is a flowchart of an operation in which a target object for removal is selected according to one embodiment. FIG. 7b is a diagram showing an example in which a target object for removal is selected according to the movement state of an object detected on an image, according to one embodiment. Specific details for implementing the invention

[0034] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.

[0035] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, actions, components, parts, or combinations thereof.

[0036] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0037] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.

[0038] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments. These terms are intended only to distinguish the components from other components, and the nature, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "combined," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "combined," or "connected" between each component.

[0039] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0040] Components included in any one embodiment and components having common functions shall be described using the same names in other embodiments. Unless otherwise stated, the description in any one embodiment may also apply to other embodiments, and specific descriptions shall be omitted to the extent of overlap.

[0041] FIG. 1 is a block diagram of an image acquisition device according to one embodiment.

[0042] Referring to FIG. 1, an image acquisition device (1) according to one embodiment can process a first image containing an object to generate a second image from the first image in which an object not desired by the user is removed.

[0043] In one embodiment, the first image refers to an image in a state before image processing is performed through the image acquisition device (1) (e.g., original image), and the second image may refer to an image in which image processing has been performed on the first image (e.g., final image). In one embodiment, an object may refer to a single target (e.g., person, animal, etc.) detected within the image. The main object refers to an object among the objects detected in the image that is to be included in the second image, and the secondary object refers to an object among the objects detected in the image that excludes the main object. The object to be removed may refer to an object among the objects detected in the image that is to be removed from the second image through image processing. In one embodiment, the main object may be automatically determined by the processor (11), but may be changed according to user input and settings. In one embodiment, the image processing device may be an electronic device.

[0044] In one embodiment, the image acquisition device (1) detects objects from a first image acquired and can classify the detected objects into a main object and a sub-object. The image acquisition device (1) can select an object to be removed among the objects on the classified first image and remove the object area where the object to be removed is located from the first image. In one embodiment, the image acquisition device (1) can operate to generate a second image by performing an inpainting restoration process on the object area removed from the first image.

[0045] The image processing device may include a camera (12), a display (10), a processor (11), a communication unit (13), and a memory (14).

[0046] In one embodiment, the camera (12) can acquire a first image. The first image acquired through the camera (12) may include one or more objects. In one embodiment, the camera (12) may include a sensor for detecting the phase difference of each object included in the first image during the process of acquiring the first image. The camera (12) can acquire information about the movement state of each object by detecting a change in phase that occurs according to the movement of each object included in the first image through the sensor.

[0047] In one embodiment, the display (10) can display an image to the user. For example, the display (10) can display at least one of a first image, a second image, an image with an object display, and an image currently being processed on the screen.

[0048] In one embodiment, the processor (11) can control the operation of the image acquisition device (1). In one embodiment, the processor (11) can control the operation of the image acquisition device (1) by executing a program stored in memory (14). In one embodiment, the processor (11) can perform the image processing operation described below by executing a program stored in memory (14).

[0049] In one embodiment, the processor (11) can perform image processing operations through an artificial intelligence neural network (110). The processor (11) can display a first image to a user through a display (10). The processor (11) can input the first image acquired through a camera (12) into the artificial intelligence neural network (110) to detect objects included in the first image. For example, the processor (11) can detect each object within the first image through the artificial intelligence neural network (110) and partition an object region corresponding to the shape of each detected object on the first image.

[0050] In one embodiment, the artificial intelligence neural network (110) used by the processor (11) may be implemented as one or more learning models. In one embodiment, the artificial intelligence neural network may be implemented as a learning model such as a deep neural network (DNN), a recurring neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), and a generative adversarial network (GAN), but is not limited thereto. In one embodiment, the artificial intelligence neural network (110) may be implemented as a learning model that learns learning data about objects according to pre-set criteria to detect objects from an image. For example, the learning model may learn to distinguish objects from an image by performing supervised learning, unsupervised learning, and reinforcement learning on the learning data. The learning model may learn the learning data according to deep neural network technology. In one embodiment, the artificial intelligence neural network (110) may be implemented as a learning model that learns training data according to pre-set criteria to perform inpainting restoration processing on an image. For example, the learning model may be trained to restore areas removed from an image by performing supervised learning, unsupervised learning, and reinforcement learning on the training data. In one embodiment, the artificial intelligence neural network (110) may be a model trained to generate a second image by extracting an external image with high similarity to a first image from an external server (1000) and using the extracted external image as training data to perform inpainting restoration processing on the first image.In one embodiment, the artificial intelligence neural network (110) may be configured to extract external images having the same location information as the shooting location information stored in the first image from an external server (1000) (e.g., cloud).

[0051] In one embodiment, the processor (11) can use an artificial intelligence neural network (110) to classify objects detected in the first image into main objects or sub-objects.

[0052] In one embodiment, the processor (11) can distinguish objects detected in the first image based on user input. For example, the processor (11) may partition an object area for each object detected in the first image and display it on the first image, and select an object corresponding to the object area where user input is received as the main object. The processor (11) may select an object corresponding to the object area where user input is not received as the sub-object. The distinction between the main object and the sub-object based on the reception of user input may be performed in the opposite manner to the example described above.

[0053] In one embodiment, the processor (11) can pre-learn an image of a main object and automatically distinguish between a main object and a sub-object from a first image. For example, the processor (11) can pre-learn images of specific objects from a plurality of images stored in memory (14) through an artificial intelligence neural network (110). The processor (11) can determine identity by comparing objects detected from the first image with a specific object that has been pre-learned through the artificial intelligence neural network (110), and automatically classify an object that is identical to or substantially identical to a specific object as a main object, thereby distinguishing the objects included in the first image into a main object and a sub-object.

[0054] In one embodiment, the processor (11) can select objects to be removed among the objects distinguished in the first image using an artificial intelligence neural network (110).

[0055] In one embodiment, the processor (11) can select an object to be removed based on location information between objects included in the first image. In one embodiment, the processor (11) can obtain relative location information between a main object and a sub-object distinguished in the first image. In one embodiment, relative location information between objects can be detected through a sensor included in the camera (12). The processor (11) can select an object to be removed among the sub-objects based on the relative location information of the sub-objects with respect to the main object within the first image. For example, the processor (11) can select a sub-object having location information that is more than a set distance away from the main object as an object to be removed. However, this is merely an example, and the method of selecting an object to be removed based on location information between objects is not limited thereto.

[0056] In one embodiment, the processor (11) can select an object to be removed based on the gaze direction of an object detected in the first image. In one embodiment, the processor (11) can detect the gaze direction of each object detected within the first image. For example, the processor (11) can detect information about the head and eyes of each object included in the first image through an artificial intelligence neural network (110), and predict the direction of the object's head, the position of the eyes, and the direction the detected eyes are looking to detect the gaze direction of each object. In one embodiment, the processor (11) can select an object to be removed based on whether the detected gaze direction of each object included in the first image is facing the camera (12). For example, the processor (11) can select an object among the sub-objects that is detected as not having a gaze direction facing the camera (12) (e.g., the direction of the camera (12) lens) as an object to be removed. In one embodiment, the processor (11) may select an object to be removed based on the similarity between the gaze direction of a sub-object included in the first image and the gaze direction of a main object. For example, the processor (11) may select a sub-object having a gaze direction different from that of a main object as an object to be removed.

[0057] In one embodiment, the processor (11) may select an object to be removed based on the relative size between objects detected in the first image. In one embodiment, the processor (11) may detect the relative size of a sub-object to a main object within the first image. The processor (11) may select an object to be removed among the sub-objects based on the relative size between the main object and the sub-object. For example, the processor (11) may select a sub-object that has a size greater than or equal to a set size or a size less than or equal to a set size as an object to be removed. However, this is merely an example, and the operation of selecting an object to be removed based on the relative size between objects is not limited thereto.

[0058] In one embodiment, the processor (11) can select an object to be removed based on the movement of objects detected in the first image. In one embodiment, the processor (11) can track the movement of each object detected within the first image. For example, the processor (11) can track the movement of the object based on the appearance of each object within the first image. In one embodiment, the processor (11) can determine the state of each object detected in the first image as either a dynamic state or a static state based on the movement of the tracked objects. In one embodiment, the processor (11) can select an object to be removed among the sub-objects based on whether the sub-object has a dynamic state or a static state. For example, the processor (11) can select a sub-object having a movement state different from that of the main object as an object to be removed. For example, if the main object is determined to be in a dynamic state, the processor (11) can select a sub-object determined to be in a static state among the sub-objects as an object to be removed.

[0059] In one embodiment, the processor (11) can use an artificial intelligence neural network (110) to remove an object region where an object to be removed is located, and can perform inpainting restoration on the removed object region. The processor (11) can generate a second image from a first image in which an object that the user does not want has been removed. The processor (11) can display the second image to the user through a display (10).

[0060] In one embodiment, the processor (11) obtains relative position information between the camera (12) and the main object, and based on the obtained relative position information, can restore at least a portion of the image of the main object that is obscured by the object to be removed.

[0061] In one embodiment, the processor (11) can extract external images having the same shooting location information as the first image from an external server (1000). The processor (11) can restore a part of the image (e.g., background image) on the first image that is obscured by the object to be removed through the extracted external images.

[0063] Hereinafter, an image acquisition method using an image acquisition device (1) according to one embodiment will be described. In describing the image acquisition method, unless otherwise specified, terms identical to those previously described may be understood to mean identical or similar configurations.

[0064] FIG. 2 is a flowchart of an image acquisition method according to one embodiment. FIG. 3a is a flowchart of an operation for distinguishing objects based on user input according to one embodiment. FIG. 3b is a diagram showing an example of distinguishing objects based on user input according to one embodiment. FIG. 4a is a flowchart of an operation for selecting an object to be removed according to one embodiment. FIG. 4b is a diagram showing an example of selecting an object to be removed based on relative position information with respect to a main object according to one embodiment. FIG. 5a is a flowchart of an operation for selecting an object to be removed according to one embodiment. FIG. 5b is a diagram showing an example of selecting an object to be removed based on the line of sight of an object on an image according to one embodiment. FIG. 6a is a flowchart of an operation for selecting an object to be removed according to one embodiment. FIG. 6b is a diagram showing an example of selecting an object to be removed based on the relative size between objects on an image according to one embodiment. FIG. 7a is a flowchart of an operation in which a target object for removal is selected according to one embodiment. FIG. 7b is a diagram showing an example in which a target object for removal is selected according to the movement state of an object detected on an image according to one embodiment.

[0065] Referring to FIGS. 2 to 3b, the image acquisition method can be performed through an image acquisition device (1) (e.g., the image acquisition device (1) of FIG. 1). The image acquisition method can be performed by a processor (11) using an artificial intelligence neural network (110). In one embodiment, the image acquisition method may include an operation S210 of acquiring a first image through a camera (12), an operation S220 of detecting objects included in the first image, an operation S230 of classifying the detected objects into main objects and sub-objects, an operation S240 of selecting an object to be removed among the detected objects, an operation S250 of removing an object region where the object to be removed is located in the first image, and an operation S260 of generating a second image by performing inpainting restoration processing on the removed object region.

[0066] According to one embodiment, in operation S210, the image acquisition device (1) can acquire a first image through a camera (12). In another example, the image acquisition device (1) can acquire a first image from an external camera (12). The first image can be displayed on a display (10). In operation S210, the image acquisition device (1) can acquire location information where the first image is captured during the process of acquiring the first image. In operation S210, the image acquisition device (1) can acquire relative location information of each object included in the first image with respect to the camera (12).

[0067] In operation S220, according to one embodiment, the image acquisition device (1) can detect objects included in the first image. In one embodiment, the objects may include people. In one embodiment, the image acquisition device (1) can detect buildings, structures, and natural landscapes as background objects from the first image. The operation of detecting objects from the first image can be performed through a pre-trained artificial intelligence neural network (110). The image acquisition device (1) can recognize the shapes of the objects detected in the first image and divide the first image area corresponding to the shapes of the recognized objects into object areas. The image acquisition device (1) can display each object area divided in the first image on a display (10) (e.g., see FIG. 3b) and provide it to the user.

[0068] In operation S230, according to one embodiment, the image acquisition device (1) may classify the detected objects into main objects and sub-objects. In one embodiment, the main object may refer to an object intended to be included in the second image. For example, the main object may include the user of the image acquisition device (1). The main object may include objects related to the user of the image acquisition device (1) (e.g., the user's family, friends, pets, etc.). In one embodiment, the sub-object may refer to an object that can be removed from the second image. For example, the sub-object may refer to an object unrelated to the main object. In one embodiment, the main object may be pre-set by the user or automatically classified according to prior training of an artificial intelligence learning network.

[0069] In operation S230, the distinction of detected objects can be performed by inputting the first image into an artificial intelligence neural network (110). The artificial intelligence neural network (110) can distinguish the objects detected in the first image into main objects and sub-objects based on the results of learning the training data.

[0070] In operation S230, the operation of distinguishing detected objects into main objects and sub-objects can be performed according to user input. In one embodiment, operation S230 may include operation S310 of receiving user input for selecting a main object within a first image, and operation S320 of distinguishing sub-objects among the detected objects based on user input.

[0071] In operation S310, the image acquisition device (1) can display each object area detected on the first image by partitioning and displaying it on the display (10). For example, as shown in FIG. 3b, each detected object (10a, 30a, 30b) can be partitioned and displayed on the first image displayed on the display (10). In one embodiment, the image acquisition device (1) can receive a user input signal through an input unit (e.g., a touch screen). For example, if the user inputs a selection signal for object 10a in the main object area, the image acquisition device (1) can recognize object 10a as the main object.

[0072] In operation S320, the image acquisition device (1) can classify objects detected in the first image into main objects and sub-objects based on the received user input. For example, the image acquisition device (1) can classify the remaining objects that the user did not select as main objects as sub-objects.

[0073] In one embodiment, in operation S230, the operation of classifying detected objects into main objects and sub-objects can be automatically performed by prior learning of the artificial intelligence neural network (110).

[0074] For example, the image acquisition method may include an operation of pre-learning images of specific objects from a plurality of images. The operation of pre-learning images of specific objects may be performed, for example, through pre-learning of a plurality of images stored in memory (14). The processor (11) may pre-learn images of specific objects that are repeatedly detected in a plurality of images through an artificial intelligence neural network (110). In this case, operation S230 may include an operation of determining whether there is an object among the objects detected in the first image that has identity with a specific object that has been pre-learned, and an operation of classifying an object determined to be identical to a specific object as a main object. Accordingly, in operation S230, the image acquisition device (1) may, before receiving user input, first classify some of the detected objects as main objects and display the classified result on the first image to display (10).

[0075] In operation S240, the image acquisition device (1) according to one embodiment may select an object to be removed among the detected objects. The object to be removed may refer to an object that needs to be removed from the first image. The image acquisition device (1) may perform the operation of selecting an object to be removed among the distinguished objects through various methods using an artificial intelligence neural network (110). In one embodiment, the object to be removed may be selected according to the input of a received user. A detailed explanation thereof will be provided later.

[0076] In operation S250, an image acquisition device (1) according to one embodiment can remove an object region where an object to be removed is located from a first image. For example, the image acquisition device (1) can remove an object region from the first image that corresponds to the shape of an object selected as an object to be removed.

[0077] According to one embodiment, in operation S260, the image acquisition device (1) can generate a second image in which only the image of the object desired by the user is displayed by performing an inpainting restoration process on the object area from which the object to be removed has been removed. The inpainting restoration process can be performed through a pre-trained artificial intelligence neural network (110). In one embodiment, the artificial intelligence neural network (110) can extract external images that have high similarity to the first image from an external server (1000) and perform an inpainting restoration process on the first image based on the extracted external images. For example, the artificial intelligence neural network (110) can use the extracted external images as training data. In one embodiment, the artificial intelligence neural network (110) can extract external images from the external server (1000) that have the same location information as the shooting location information stored in the first image. In this case, the artificial intelligence neural network (110) can perform an inpainting restoration process on the object area from which the object to be removed has been removed through the background images appearing in the extracted external images.

[0078] In one embodiment, the artificial intelligence neural network may be a model trained to generate the second image using the second image as training data, utilizing photos taken in real-time at the site with a focus on the background. For example, the artificial intelligence neural network (110) may generate the second image using photos taken in real-time by an individual at the same location as training data. At this time, GPS information included in the photos may be utilized, and in the case of tourist attractions, information such as names or hashtags may be used to distinguish photos of the same location and use them as training data.

[0079] In one embodiment, referring to FIG. 3b, the image acquisition device (1) can partition and display an object area for an object detected in the first image. The image acquisition device (1) receives input from a user selecting a main object (10a) and can select a sub-object (30a, 30b) that the user did not select as an object to be removed. The processor (11) can remove the object area for the object (30a, 30b) selected as the object to be removed from the first image and generate a second image by restoring the image of the main object (10a) or background object (30a) obscured by the object to be removed through an artificial intelligence neural network (110) by inpainting restoration processing of the removed object area.

[0080] Referring to FIGS. 4a and 4b, operation S240 may include operation S410, which selects an object to be removed among the sub-objects based on the relative position information of the sub-objects to the main object within the first image. For example, an image acquisition device (1) may acquire relative position information of each object to the camera (12) through a sensor included in the camera (12), and detect relative position information of sub-objects (40a, 40b) to the main object (10a) based on the detected information. In one embodiment, the image acquisition device (1) may select an object to be removed among the sub-objects based on the relative position information of the sub-objects (40a, 40b) to the main object (10a). For example, the processor (11) may select a sub-object (40a, 40b) located at a position greater than a set distance from the main object (10a) as an object to be removed through an artificial intelligence neural network (110). For example, a sub-object (40a) that is closer than a set distance from the main object (10a) or a sub-object (40b) that is further away than a set distance from the camera (12) may be selected as the object to be removed.

[0081] In one embodiment, the processor (11) can remove an object region for a sub-object (40a, 40b) selected as a target object for removal as in FIG. 4b from the first image, and restore the removed object region (40c) through an artificial intelligence neural network (110) to restore an image of a main object (10a) or background object (20a) obscured by the target object for removal, thereby generating a second image.

[0082] Referring to FIGS. 5a and 5b, operation S240 can select an object to be removed according to the gaze direction of objects detected in the first image. In one embodiment, operation S240 may include operation S510 of detecting the gaze direction of each object in the first image, and operation S520 of selecting an object to be removed among the sub-objects based on the detected gaze direction.

[0083] In operation S510, the image acquisition device (1) can detect the gaze direction (fa, fb) of each object (10a, 50b) detected in the first image. For example, the processor (11) can detect information about the head and eye of each object (10a, 50b) included in the first image through an artificial intelligence neural network (110), and can detect the gaze direction of each object by predicting the direction of the object's head, the position of the eye, and the direction in which the detected eye is looking.

[0084] In operation S520, the processor (11) can select an object to be removed based on whether the detected gaze direction (fa, fb) of each object (10a, 50b) included in the first image is facing the camera (12). For example, the processor (11) can select an object (50b) among the sub-objects that is detected as not having a gaze direction facing the camera (12) (e.g., the direction of the camera (12) lens) as an object to be removed. In one embodiment, the processor (11) can select an object to be removed based on the similarity between the gaze direction of the sub-object (50b) included in the first image and the gaze direction of the main object (10a). For example, the processor (11) can select a sub-object (50b) that has a gaze direction (fb) different from the gaze direction (fa) of the main object (10a) as an object to be removed.

[0085] In one embodiment, the processor (11) can remove an object region for a sub-object (50b) selected as a target object for removal as in FIG. 5b from the first image, and restore the removed object region (50c) through an artificial intelligence neural network (110) to restore an image of a main object (10a) or background object obscured by the target object for removal, thereby generating a second image.

[0086] Referring to FIGS. 6a and 6b, operation S240 can select an object to be removed based on the relative sizes of objects detected within the first image. In one embodiment, operation S240 may include operation S610, which selects an object to be removed based on the relative sizes of a sub-object to a main object within the first image.

[0087] In operation S610, the processor (11) can detect the relative size of the sub-objects (60a, 60b) to the main object (10a) within the first image. The processor (11) can select an object to be removed among the sub-objects (60a, 60b) based on the relative size of the main object (10a) and the sub-objects (60a, 60b). For example, the processor (11) can select an object (60b) that has a size greater than or equal to a set size relative to the main object, or a sub-object (60a) that has a size less than or equal to a set size, as an object to be removed. However, this is merely an example, and the operation of selecting an object to be removed based on the relative size between objects is not limited thereto.

[0088] In one embodiment, the processor (11) can remove an object region for a sub-object (60a, 60b) selected as a target object for removal as in FIG. 6b from the first image, and restore the removed object region (60c) through an artificial intelligence neural network (110) to restore an image of a main object (10a) or background object obscured by the target object for removal, thereby generating a second image.

[0089] Referring to FIGS. 7a and 7b, operation S240 can select an object to be removed based on the movement state of objects detected in the first image. In one embodiment, operation S240 may include operation S710 of tracking the movement of each object in the first image, operation S720 of determining the state of each object as either a dynamic state or a static state based on the movement of the tracked objects, and operation S730 of selecting an object to be removed among the sub-objects based on the determined movement state of the objects.

[0090] In operation S710, the processor (11) can track the movement of each of the objects (10a, 70a, 70b) detected in the first image. For example, the processor (11) can track the movement of the objects based on the appearance of each object in the first image.

[0091] In operation S720, the processor (11) can determine the state of each object (10a, 70a, 70b) detected in the first image as either a dynamic state or a static state, depending on the movement of the tracked objects (10a, 70a, 70b). In one embodiment, the processor (11) can select an object to be removed among the sub-objects based on whether the sub-objects (70a, 70b) have a dynamic state or a static state. For example, the processor (11) can select a sub-object (70a) having a movement state different from that of the main object (10a) as an object to be removed. In one embodiment, if the main object (10a) is determined to be in a dynamic state, the processor (11) can select a sub-object (70b) determined to be in a static state among the sub-objects (70a, 70b) as an object to be removed.

[0092] In one embodiment, the processor (11) can remove an object region for a sub-object (70a) selected as a target object for removal as in FIG. 7b from the first image, and restore the removed object region (70c) through an artificial intelligence neural network (110) to restore the image of the main object (10a) or background object obscured by the target object for removal, thereby generating a second image.

[0093] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0094] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0096] 1: Image acquisition device 10: Display 11: Processor 110: Artificial Intelligence Neural Network 12: Camera 13: Communications Department 14: Memory 1000: External Server*

Claims

Claim 1 A method for acquiring an image through an artificial intelligence neural network comprises: an operation of acquiring a first image through a camera; an operation of detecting objects included in the first image; an operation of classifying the detected objects into a main object and a sub-object; an operation of selecting an object to be removed among the detected objects; an operation of removing an object region in the first image where the object to be removed is located; and an operation of generating a second image by performing an in-painting restoration process on the removed object region. The operation of selecting an object to be removed includes an operation of detecting the gaze direction of the objects within the first image by predicting the head direction, eye position, and the direction in which the detected eyes are looking of each object through an artificial intelligence neural network. Based on the detected gaze direction, the method includes an operation of selecting an object to be removed according to whether the detected gaze direction of the sub-object has similarity to the detected gaze direction of the main object and whether the detected gaze direction is directed toward the camera. The artificial intelligence neural network extracts external images having the same location information as the shooting location information of the first image from an external server and uses the extracted external images as training data or the background at the site An image acquisition method, wherein the model is trained to generate the second image in which the background where the object to be removed was located is restored, using photos taken in real-time with the center as training data. Claim 2 An image acquisition method according to claim 1, wherein the operation of selecting the object to be removed includes selecting the object to be removed among the sub-objects based on the relative position information of the sub-object with respect to the main object within the first image. Claim 3 delete Claim 4 delete Claim 5 An image acquisition method according to claim 1, wherein the operation of selecting the object to be removed includes selecting the object to be removed among the sub-objects based on the relative size of the sub-object to the main object in the first image. Claim 6 In claim 1, the operation of selecting the object to be removed comprises: an operation of tracking the movement of the objects within the first image; an operation of determining the state of each of the objects to be either a dynamic state or a static state based on the tracked movement of the objects; and an operation of selecting the object to be removed among the sub-objects based on the determined state of the objects, wherein the operation of selecting the object to be removed among the sub-objects selects the sub-object determined to be in a static state among the sub-objects as the object to be removed. Claim 7 An image acquisition method according to claim 1, wherein the operation of classifying the detected objects into a main object and a sub-object comprises: receiving input from a user selecting the main object within the first image; and classifying the sub-object based on the user input. Claim 8 An image acquisition method according to claim 1, further comprising an operation of pre-learning images of specific objects from a plurality of images, wherein the operation of classifying the detected objects into a main object and a sub-object comprises: an operation of determining identity with the pre-learned specific object among the objects detected on the first image; and an operation of classifying an object determined to be identical to the specific object as a main object. Claim 9 delete Claim 10 An image acquisition device comprising: a camera for acquiring a first image; a processor for generating a second image by image processing the first image through an artificial intelligence neural network; and a display for displaying at least one of the first image and the second image, wherein the processor detects information regarding the head and eye of each object included in the first image through an artificial intelligence neural network of objects within the first image, predicts the direction in which the eye looks to detect the direction of gaze, and, based on the detected direction of gaze, selects a target object to be removed according to whether the detected direction of gaze of a sub-object has similarity to the detected direction of gaze of a main object and whether the detected direction of gaze is directed toward the camera, and wherein the artificial intelligence neural network is a model trained to generate the second image in which the background where the target object to be removed was located is restored by extracting external images having the same location information as the shooting location information of the first image from an external server and using the extracted external images as training data or by using photos taken in real-time at the site centered on the background as training data. Claim 11 In claim 10, the image acquisition device further includes a communication unit that communicates with an external server, and the processor extracts external images having the same location information as the shooting location information stored in the first image from the external server, and operates to utilize external images similar to the first image among the extracted external images as training data for the artificial intelligence neural network.

Images