Image acquisition apparatus and method of controlling image acquisition apparatus

Abstract

The invention discloses an image acquisition apparatus and a method of controlling the image acquisition apparatus. An artificial intelligence (AI) system is provided in which the system mimics functions, such as recognition and determination by a human brain, by utilizing a machine learning algorithm and application of the AI system. Disclosed is an image acquisition apparatus, in which the image acquisition apparatus includes: a camera configured to acquire a first image; at least one processor configured to input the first image to a first AI neural network, detect, by the first AI neural network, first data corresponding to a master object and second data corresponding to a sub-object from data corresponding to a plurality of objects included in the first image, and using a second AI neural network to generate a second image by recovering third data corresponding to at least a portion of the master object hidden by the quilt object, where the third data replaces the second data; and a display configured to display at least one of the first image and the second image.

Description

[0001] This application is a divisional application of an invention patent application with an application date of December 26, 2018, application number "201880080204.7", and invention name "Image acquisition device and method for controlling image acquisition device". Technical Field

[0002] One or more embodiments relate to an image acquisition device and a control method thereof. Background Art

[0003] Artificial intelligence (AI) systems are computer systems that are configured to achieve human-level intelligence, train themselves, and make decisions autonomously to become smarter, compared to existing rule-based intelligent systems. As the recognition rate of AI systems has improved and the more AI systems are used, the more accurately they understand user preferences, the existing rule-based intelligent systems are gradually being replaced by deep learning AI systems.

[0004] AI technology includes machine learning (deep learning) and element technology that uses machine learning.

[0005] Machine learning is an algorithmic technology that self-classifies / learns features of input data, and each of the element technologies is a technology that uses a machine learning algorithm (such as deep learning), and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operation control.

[0006] The various fields to which AI technology is applied are as follows. Language understanding is a technology that recognizes human language / characters and applies / processes human language / characters, and includes natural language processing, machine translation, conversational systems, question-answering, speech recognition / synthesis, etc. Visual understanding is a technology that recognizes and processes objects as in human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, etc. Reasoning / prediction is a technology that logically performs reasoning and prediction by determining information, and includes knowledge / probability-based reasoning, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation is a technology that automatically processes human experience information into knowledge data, and includes knowledge establishment (data generation / classification), knowledge management (data utilization), etc. Operation control is a technology that controls the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision avoidance, and driving), manipulation control (behavior control), etc.

[0007] AI technology can also be used to capture images, such as pictures or motion pictures. Summary of the invention

[0008] Technical issues According to one aspect of the present disclosure, there is provided a method for acquiring an image by using an artificial intelligence (AI) neural network, the method comprising: acquiring a first image by using a camera, wherein a sub-object causes a portion of a main object to be hidden from the camera; inputting the first image to a first AI neural network; detecting, by the first AI neural network, first data corresponding to the main object and second data corresponding to the sub-object from data corresponding to a plurality of objects included in the first image; removing the second data corresponding to the sub-object from the first image; and generating a second image by restoring third data corresponding to at least a portion of the main object hidden by the sub-object using a second AI neural network, wherein the third data replaces the second data.

[0009] In some embodiments of the method, the step of detecting the first data corresponding to the main object and the second data corresponding to the sub-object includes displaying an indicator indicating the second data corresponding to the sub-object together with the first image.

[0010] In some embodiments of the method, the step of restoring the third data corresponding to the at least part of the main object includes: acquiring information of a first relative position between the camera and the main object; and restoring the third data corresponding to the at least part of the main object based on the information of the first relative position.

[0011] In some embodiments of the method, the method includes: determining a first sharpness of third data corresponding to the at least part of the main object; and performing image processing so that a second sharpness of the third data corresponding to the at least part of the main object corresponds to the first sharpness.

[0012] In some embodiments of the method, the step of detecting first data corresponding to the main object and second data corresponding to the sub-object includes: receiving a user selection of the sub-object from the first image; and detecting the second data corresponding to the sub-object from the first image based on the user selection.

[0013] In some embodiments of the method, the step of displaying the indicator includes: tracking the movement of the sub-object; and displaying the indicator based on the movement of the sub-object.

[0014] In some embodiments of the method, the step of tracking the movement of the sub-object includes: acquiring information of a second relative position between the camera and the sub-object; inputting the information of the second relative position into the first AI neural network; and tracking the movement of the sub-object by the first AI neural network by detecting changes in the information of the second relative position.

[0015] In some embodiments of the method, the step of detecting first data corresponding to the main object and second data corresponding to the sub-object includes: in response to user input driving the camera, forming a communication link with a server, wherein the server includes a first AI neural network and a second AI neural network; sending the first image to the server through the communication link; and receiving information about the results of detecting the first data corresponding to the main object and the second data corresponding to the sub-object from the server through the communication link.

[0016] In some embodiments of the method, the method includes receiving data from a server for updating at least one of the first AI neural network and the second AI neural network in response to a user input actuating the camera.

[0017] In some embodiments of the method, the first AI neural network is a model trained to detect first data corresponding to the main object by using at least one pre-stored image as learning data.

[0018] According to one aspect of the present disclosure, an image acquisition device is provided.

[0019] In some embodiments, the image acquisition device includes: a camera configured to acquire a first image, wherein a sub-object causes a portion of a main object to be hidden from the camera; at least one processor configured to: input the first image into a first AI neural network; detect, by the first AI neural network, first data corresponding to the main object and second data corresponding to the sub-object from data corresponding to multiple objects included in the first image, and generate a second image using a second AI neural network by restoring third data corresponding to at least a portion of the main object hidden by the sub-object, wherein the third data replaces the second data; and a display configured to display at least one of the first image and the second image.

[0020] In some embodiments of the image acquisition device, the display is further configured to: display an indicator indicating second data corresponding to the sub-object together with the first image.

[0021] In some embodiments of the image acquisition device, the at least one processor is further configured to: acquire information of a first relative position between the camera and the main object; and based on the information of the first relative position, restore third data corresponding to the at least a portion of the main object.

[0022] In some embodiments of the image acquisition device, the at least one processor is also configured to: determine a first sharpness of third data corresponding to the at least part of the main object; and perform image processing so that a second sharpness of the third data corresponding to the at least part of the main object corresponds to the first sharpness.

[0023] In some embodiments, the image acquisition device includes: a user input interface configured to receive a user selection of a sub-object from the first image, and the at least one processor is further configured to detect second data corresponding to the sub-object based on the user selection.

[0024] In some embodiments of the image acquisition apparatus, the at least one processor is further configured to track the motion of the sub-object, and the display is further configured to display the indicator based on the motion of the sub-object.

[0025] In some embodiments of the image acquisition device, the at least one processor is further configured to: acquire information of a second relative position between the camera and the sub-object; input the information of the second relative position into the first AI neural network; and track the movement of the sub-object by detecting changes in the information of the second relative position.

[0026] In some embodiments, the image acquisition device includes: a user input interface configured to receive user input for driving a camera; and a communication interface configured to: in response to the user input, form a communication link with a server including a first AI neural network and a second AI neural network, send a first image to the server through the communication link, and receive information from the server about the results of detecting first data corresponding to a main object and second data corresponding to a sub-object from the first image through the communication link.

[0027] In some embodiments, the image acquisition device includes: a user input interface configured to receive user input for driving a camera; and a communication interface configured to: in response to the user input, form a communication link with a server that updates at least one of the first AI neural network and the second AI neural network, and receive data for updating at least one of the first AI neural network and the second AI neural network from the server through the communication link.

[0028] In some embodiments of the image acquisition device, the first AI neural network is a model trained to detect first data corresponding to the main object by using at least one pre-stored image as learning data.

[0029] According to one aspect of the present disclosure, a computer program product includes a non-transitory computer-readable storage medium having recorded thereon a method of acquiring an image by using an artificial intelligence (AI) neural network, wherein the computer-readable storage medium includes instructions for performing the following operations: acquiring a first image by using a camera, wherein a sub-object causes a portion of a main object to be hidden from the camera; inputting the first image to a first AI neural network; detecting, by the first AI neural network, first data corresponding to the main object and second data corresponding to the sub-object from data corresponding to a plurality of objects included in the first image; removing the second data corresponding to the sub-object from the first image; and generating a second image by restoring third data corresponding to at least a portion of the main object hidden by the sub-object using a second AI neural network, wherein the third data replaces the second data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 An example of an image acquisition device acquiring an image according to some embodiments is shown; Figure 2 is a flowchart of a method of acquiring an image performed by an image acquisition device according to some embodiments; Figure 3 , Figure 4 and Figure 5 is a diagram illustrating an example of displaying an indicator indicating a child object according to some embodiments; Figure 6 is a flow chart of a method of restoring at least a portion of a main object that was hidden by a child object according to some embodiments; Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 An example showing restoration of at least a portion of a main object hidden by a child object according to some embodiments; Fig.12 is a flow chart of a method of detecting a sub-object in response to user input according to some embodiments; Fig.13 and Fig.14 illustrates an example of detecting a sub-object in response to user input according to some embodiments; Fig.15 is a flow chart of a method of restoring at least a portion of a main object that was hidden by a child object in response to user input according to some embodiments; Fig.16 , Fig.17 and Fig.18An example showing restoring at least a portion of a main object hidden by a child object in response to user input according to some embodiments; Fig.19 is a flow chart of a method of tracking motion of a sub-object within an image according to some embodiments; Fig. 20 , Fig.21 , Fig. 22 and Fig.23 is a diagram illustrating an example of tracking the motion of a sub-object within an image according to some embodiments; Fig.24 is a flow chart of a method of restoring at least a portion of a main object hidden by a sub-object based on a relative position between a camera and at least one of the main object and the sub-object according to some embodiments; Fig.25 and Fig.26 shows an example of detecting a main object and a sub-object when a camera moves according to some embodiments; Fig. 27 is a diagram illustrating a method of displaying an indicator indicating a child object by using a server according to some embodiments; Fig.28 and Fig.29 is a diagram illustrating an example of displaying an indicator indicating a child object by using a server according to some embodiments; Fig.30 and Fig.31 is a block diagram of an image acquisition apparatus according to some embodiments; Fig.32 is a block diagram of a server according to some embodiments; Fig.33 is a block diagram of a processor included in an image acquisition device according to some embodiments; Fig.34 is a block diagram of a data learner included in a processor according to some embodiments; Fig.35 is a block diagram of a data identifier included in a processor according to some embodiments; and Fig.36 is a block diagram illustrating an example of an image acquisition device and a server interoperating to learn and recognize data according to some embodiments. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that a person skilled in the art can easily perform the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments described herein. In the accompanying drawings, for simplicity of description, parts not related to the description are omitted, and the same reference numerals always refer to the same elements.

[0032] The aforementioned embodiments can be described according to function block components and various processing steps. Some or all of these function blocks can be implemented by any number of hardware and / or software components configured to perform specified functions. For example, function blocks according to the present disclosure can be implemented by one or more microprocessors or by circuit components for predetermined functions. In addition, for example, function blocks according to the present disclosure can be implemented with any programming language or scripting language. Function blocks can be implemented with algorithms executed on one or more processors. In addition, the present disclosure described herein can adopt any number of conventional techniques for electronic configuration, signal processing and / or control, data processing, etc. The words "mechanism", "element", "means" and "configuration" are widely used and are not limited to mechanical embodiments or physical embodiments.

[0033] Throughout the specification, when an element is referred to as being "connected" or "coupled" to another element, the element may be directly connected or coupled to the other element, or may be electrically connected or coupled to the other element with an intermediate element interposed therebetween. In addition, when used in this specification, the terms "include" and / or "comprising..." or "includes" and / or "comprising..." indicate the presence of the elements, but do not exclude the presence or addition of one or more other elements.

[0034] In addition, the connecting lines or connectors between the components shown in the various figures presented are intended to represent exemplary functional relationships and / or physical or logical couplings between the components. The connections between the components may be represented by many alternative or additional functional relationships, physical connections or logical connections in actual devices.

[0035] Although such terms as "first", "second", etc. may be used to describe various components, these components are not necessarily limited to the above terms. The above terms are only used to distinguish one component from another.

[0036] A main object used herein means an object that a user of the image acquisition apparatus 1000 desires to include in an image, and a sub-object used herein means an object that the user does not desire to include in an image.

[0037] The present disclosure will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown.

[0038] Figure 1 An example of the image acquisition device 1000 acquiring an image according to some embodiments is shown.

[0039] Reference Figure 1 , the image acquisition device 1000 may acquire an image by using a camera included in the image acquisition device 1000. Figure 1In the example of , objects 1a, 1b, 2a, 2b, and 2c are in the field of view of the camera. Figure 1 In the example of , object 1b is a building, and objects 2a, 2b, 2c, and 1b are people. Parts of building 1b are not visible to the camera because they are hidden behind objects 2a, 2b, 2c, and 1a. The image acquired by the image acquisition device 1000 may include main objects 1a and 1b and sub-objects 2a, 2b, and 2c.

[0040] When necessary for clarity, a three-dimensional object in the field of view of a camera will be distinguished from an image of the three-dimensional object captured by the camera. A three-dimensional thing in the field of view may be directly referred to as an object, while an image of an object may be referred to as data corresponding to the object. For example, building 1b is an object in the field of view of a camera. The portion of the image representing building 1b may be referred to as data corresponding to building 1b. When there is no risk of confusion when discussing the content of an image, data representing an object may be referred to as an object in an image, etc.

[0041] The image acquisition device 1000 may detect the main objects 1a and 1b and the sub-objects 2a, 2b and 2c from the acquired image. The image acquisition device 1000 may remove the sub-objects 2a, 2b and 2c from the image. In some embodiments, the removal of the sub-objects may be achieved by replacing the data associated with the sub-objects with default data representing a blank screen. In some embodiments, the removal of the sub-objects 2a, 2b and 2c from the image is performed based on the result of detecting the main objects 1a and 1b and the sub-objects 2a, 2b and 2c. The image acquisition device 1000 may restore at least a portion of the main object 1b that is hidden by the sub-objects 2a, 2b and 2c, thereby acquiring an image. For example, in Figure 1 In the lower part of the diagram, the image acquisition device 1000 is shown with a second image, after the sub-objects 2a, 2b and 2c have been removed, the building 1b now appears complete and continuous behind the object 1a. For example, the complete vertical line marking the corner of the building 1b (previously partially hidden by the object 2c) now appears complete to the viewer of the image.

[0042] The image acquired by the image acquiring device 1000 may include a final image to be stored in the image acquiring device 1000 , and a preview image displayed on the image acquiring device 1000 in order to obtain the final image.

[0043] The image acquisition device 1000 can detect at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the acquired image by using the training model 3000. The image acquisition device 1000 can restore at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c by using the training model 3000. After restoring the hidden portion of the object 1b, the cumulative effect on the viewer of the image is that the object 1b including the portion previously hidden by one or more sub-objects is completely and continuously presented.

[0044] The training model 3000 may include a plurality of training models. In other words, the training models corresponding to each of the various uses may be collectively referred to as the training model 3000. For example, the training model 3000 includes a first training model for detecting at least one of the main objects 1a and 1b and detecting at least one of the sub-objects 2a, 2b, and 2c among a plurality of objects included in an image, and a second training model for restoring at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c. Hereinafter, various training models for implementing the disclosed embodiments are collectively described as the training model 3000.

[0045] The training model 3000 may be established in consideration of, for example, the application field of the recognition model, the purpose of learning, or the computer performance of the device. The training model 3000 may include, for example, a model based on an AI neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent DNN (BRDNN), and a generative adversarial network (GAN) may be used as the training model 3000, but the embodiment is not limited thereto.

[0046] According to an embodiment, the training model 3000 may learn the learning data according to a preset standard so as to detect at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the acquired image. For example, the training model 3000 may detect at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c by performing supervised learning, unsupervised learning, and reinforcement learning on the learning data. The training model 3000 may detect at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the acquired image by learning the learning data according to the DNN technology.

[0047] According to an embodiment, the training model 3000 may learn the learning data according to a preset standard so as to restore at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c. For example, the training model 3000 may restore at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c by performing supervised learning, unsupervised learning, and reinforcement learning on the learning data. The training model 3000 may restore at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c by learning the learning data according to the DNN technology.

[0048] According to an embodiment, the user may acquire at least one of the preview image and the final image as an image including only the main object.

[0049] Figure 2 is a flowchart of a method of acquiring an image performed by the image acquisition device 1000 according to some embodiments. Figures 3 to 5 is a diagram illustrating an example of displaying an indicator indicating a child object according to some embodiments.

[0050] Reference Figure 2 In operation S210, the image acquisition device 1000 may acquire a first image by using a camera. Generally, a camera captures light from a three-dimensional object in a field of view of the camera and records a two-dimensional image composed of data corresponding to the three-dimensional object. In operation S230, Figure 2 The flowchart of FIG. 1 shows that the image acquisition device 1000 can detect at least one of a main object and a sub-object from a first image. Specifically, this can be referred to as detecting first data corresponding to the main object and detecting second data corresponding to the sub-object. However, generally, when there is no risk of confusion, the data in the image can be directly referred to as an object. In operation S250, Figure 2 The flowchart of FIG. 1 shows that the image acquisition device 1000 may display an indicator indicating a detected sub-object.

[0051] According to an embodiment, the image acquiring device 1000 may acquire a first image by using a camera included in the image acquiring device 1000. Alternatively, the image acquiring device 1000 may acquire an image from an external camera connected to the image acquiring device 1000 according to at least one of a wired manner and a wireless manner. The first image acquired by the camera may include a preview image for acquiring a final image to be stored in the image acquiring device 1000. The preview image may be displayed on a display of the image acquiring device 1000, or may be displayed on an external display connected to the image acquiring device 1000 according to at least one of a wired manner and a wireless manner.

[0052] In operation S230, the image acquisition device 1000 may detect a main object and a sub-object from the first image. The main object may include an object that the user of the image acquisition device 1000 desires to include in the final image. The sub-object may also include another object that the user does not desire to include in the final image.

[0053] Reference Figures 3 to 5 , the image acquisition device 1000 can detect the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the first image.

[0054] According to an embodiment, the main object 1a may include a person. For example, the main object 1a may include a user of the image acquisition device 1000. Alternatively, the main object 1a may include a person associated with the user of the image acquisition device 1000. In detail, the main object 1a may include a family member, a lover, or a relative of the user of the image acquisition device 1000.

[0055] According to an embodiment, the main object 1b may include a building, a sculpture, and / or a natural landscape. For example, the main object 1b may include a landmark in the area where the image acquisition device 1000 is located. As another example, the main object 1b may include a sculpture located around or near the image acquisition device 1000. As another example, the main object 1b may include a natural landscape such as the sky, a lawn, a lake, and / or an ocean.

[0056] According to an embodiment, the sub-objects 2a, 2b, and 2c may include people. For example, the sub-objects 2a, 2b, and 2c may include people located near the user. In detail, the sub-objects 2a, 2b, and 2c may include passers-by passing near the user, people taking photos with the main object 1b as a background, and people selling things near the user. Generally, the people included in the sub-objects 2a, 2b, and 2c may have nothing to do with the user, but the embodiment is not limited thereto.

[0057] According to an embodiment, the sub-object may include a specific object. For example, the sub-object may include at least one object within the first image that hides the main object 1b or a portion of the main object 1b. For example, when a sub-object in the field of view blocks some light reaching the camera from the main object 1b, this blocking of light means that the sub-object hides a portion of the main object 1b from the camera. Due to the blocking of light, the data present in the image captured by the camera is incomplete in the sense that a portion of the main object 1b is not visible in the image captured by the camera. As an example, the sub-object may include an object that hides the main object 1b (such as a tree, a trash can, a newsstand, a wall, and a wire mesh fence), but the embodiment is not limited thereto.

[0058] According to an embodiment, the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c may be detected from the first image by inputting the first image into an AI neural network. For example, the AI ​​neural network may detect data of respective parts within the first image, wherein the data of the respective parts respectively correspond to different sub-objects. For example, the image acquisition device 1000 may input the first image into the AI ​​neural network, and the AI ​​neural network may detect at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the first image based on a result of learning the learning data.

[0059] According to an embodiment, the AI ​​neural network may learn by using at least one image stored in the image acquisition device 1000 as learning data. For example, the AI ​​neural network may use an image including a face of a user and an image including a face of a person associated with the user as learning data, wherein the images are stored in the image acquisition device 1000.

[0060] In many cases, the user of the image acquisition device 1000 takes a photo including his or her face and the face of a person associated with the user. Therefore, a plurality of images each including the face of the user and a plurality of images each including people associated with the user (family members, lovers, and relatives) may be stored in the image acquisition device 1000. Therefore, when the AI ​​neural network learns at least one image stored in the image acquisition device 1000 as learning data, the AI ​​neural network may detect at least one of the user and the person associated with the user as the main object 1a.

[0061] The image acquisition device 1000 may store a plurality of images each associated with a specific area where the user is located. When the user wants to go to a specific area, the user generally searches for materials associated with the area on the Internet and stores the images associated with the area in the image acquisition device 1000. In particular, the user generally stores images associated with the main buildings, sculptures, and natural landscapes of the specific area in the image acquisition device 1000. Therefore, when the AI ​​neural network learns at least one image stored in the image acquisition device 1000 as learning data, the AI ​​neural network may detect at least one of the main buildings, sculptures, and natural landscapes of the specific area where the user is located as the main object 1b.

[0062] According to an embodiment, the AI ​​neural network may learn by using at least one image disclosed on the Internet as learning data. For example, the AI ​​neural network may use an image disclosed on the Internet and associated with a specific area as learning data. When a user wants to go to a specific area, the user may search for images associated with the specific area on the Internet. The AI ​​neural network may detect buildings, sculptures, and / or natural landscapes in the specific area where the user is located as main objects 1b. The AI ​​neural network may perform detection by learning using images associated with the specific area found by the user on the Internet as learning data.

[0063] The AI ​​neural network may use images that are publicly available on the Internet and associated with general objects as learning data. The AI ​​neural network may use multiple images publicly available on the Internet as learning data in order to recognize general objects (e.g., trees, trash cans, street lights, traffic lights, parking spaces, people, and animals) within an image. The AI ​​neural network may detect sub-objects 2a, 2b, and 2c from the first image by learning using images publicly available on the Internet and associated with general objects as learning data.

[0064] In operation S250, the image acquiring device 1000 may display an indicator indicating that the detected object is a sub-object. For example, the image acquiring device 1000 may display both the first image and the indicator indicating the sub-object on a display included in the image acquiring device 1000. Alternatively, the image acquiring device 1000 may display both the first image and the indicator indicating the sub-object on an external display connected to the image acquiring device 1000, wherein the connection may be a wired connection or a wireless connection.

[0065] Reference Figures 3 to 5 , the image acquisition device 1000 may display an indicator indicating a sub-object according to various methods.

[0066] According to the embodiment, Figure 3 As shown in , the image acquisition device 1000 may display indicators 3a, 3b, and 3c near the sub-objects 2a, 2b, and 2c. In detail, the image acquisition device 1000 may display indicators 3a, 3b, and 3c each formed in a star shape near the sub-objects 2a, 2b, and 2c. Those of ordinary skill in the art will recognize that the indicators may be displayed in various shapes.

[0067] According to the embodiment, Figure 4As shown in , the image acquisition device 1000 may display an indicator indicating the sub-object by indicating the sub-objects 2a, 2b, and 2c with dotted lines. For example, the image acquisition device 1000 may display the sub-objects 2a, 2b, and 2c by drawing the outlines of the sub-objects 2a, 2b, and 2c with dotted lines. As another example, the image acquisition device 1000 may display the sub-objects 2a, 2b, and 2c with semi-transparent dotted lines so that at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c is displayed within the sub-objects 2a, 2b, and 2c.

[0068] According to the embodiment, Figure 5 As shown in , the image acquisition device 1000 may display an indicator indicating a sub-object by covering at least corresponding portions of the sub-objects 2a, 2b, and 2c with at least one of a preset color and a preset pattern.

[0069] Figure 6 is a flow chart of a method of restoring at least a portion of a main object that was hidden by a child object according to some embodiments. Figures 7 to 11 An example of restoring at least a portion of a main object that was hidden by a child object is shown in accordance with some embodiments.

[0070] Reference Figure 6 In operation S610, the image acquiring device 1000 may acquire a first image by using a camera, in operation S630, the image acquiring device 1000 may detect at least one of a main object and a sub-object from the first image, in operation S650, the image acquiring device 1000 may remove the detected sub-object from the first image, in operation S670, the image acquiring device 1000 may generate a second image by restoring an area where the removed sub-object is located, and in operation S690, the image acquiring device 1000 may store the generated second image.

[0071] Operation S610 is similar to operation S210 , and thus a redundant description thereof will be omitted.

[0072] Operation S630 is similar to operation S230 , and thus a redundant description thereof will be omitted.

[0073] In operation S650, the image acquisition device 1000 may remove the detected sub-object from the first image. In some embodiments, the operation of removing the detected sub-object corresponds to an operation of identifying data within the image to be overwritten by the second image.

[0074] According to the embodiment, Figure 7 As shown in , the image acquisition device 1000 may remove data associated with at least some areas of the first image where the detected sub-objects 2a, 2b, and 2c are located from the first image. In detail, the image acquisition device 1000 may remove only data associated with the sub-objects 2a, 2b, and 2c from the first image. Figure 8 As shown in regions 5a, 5b, and 5c of the first image, the image acquisition device 1000 may remove data associated with regions 4a, 4b, and 4c including sub-objects 2a, 2b, and 2c from the first image. The image acquisition device 1000 may display or not display an image from which data associated with the sub-objects has been removed.

[0075] According to an embodiment, the image acquisition device 1000 can remove data associated with at least some areas of the first image where the sub-objects 2a, 2b, and 2c are located from the first image by using an AI neural network. The AI ​​neural network can detect areas associated with the sub-objects so that restoration is effectively performed, and can remove the detected areas by learning using an image to be partially removed and an image where the portion has been restored as learning data.

[0076] In operation S670, the image acquisition device 1000 may restore at least some areas of the first image from which data has been removed. In some embodiments, the restoration of the removed data corresponds to replacing the removed data with other data, wherein the other data corresponds to a portion of the object that was hidden from the camera when the first image was captured (e.g., a portion of a building).

[0077] According to an embodiment, the image acquisition device 1000 may restore the areas 5a, 5b, and 5c from which data has been removed so that at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c is included. Fig. 9 , the image acquisition device 1000 may restore the areas 5a, 5b, and 5c from which data has been removed so that the at least a portion of the main object 1b is included, thereby obtaining restored areas 6a, 6b, and 6c.

[0078] According to an embodiment, the image acquisition device 1000 may restore the area from which data has been removed by using an AI neural network so that the at least a portion of the main object 1b is included. For example, the AI ​​neural network may restore the area by learning at least one image associated with the main object 1b as learning data so that the at least a portion of the main object 1b is included. In detail, the AI ​​neural network may learn at least one image stored in the image acquisition device 1000 and associated with the main object 1b as learning data. The AI ​​neural network may learn at least one image that is disclosed on the Internet and associated with the main object 1b as learning data. The AI ​​neural network may learn images associated with general objects as learning data. The AI ​​neural network may generate a second image by restoring the area from which data has been removed using a model such as a generative adversarial network (GAN) so that the at least a portion of the main object 1b is included.

[0079] According to an embodiment, the image acquisition device 1000 may restore the area from which data has been removed so that the sharpness of the restored area corresponds to the sharpness near the restored area. Sharpness corresponds to a subjective perception related to the edge contrast of an image. The image acquisition device 1000 may acquire a first image in which the main object 1a is "in focus" and the main object 1b is "out of focus" (e.g., a first sharpness). In this case, the focused main object 1a will be clear, and the out-of-focus main object 1b will be unsharp, such as blurred. The image acquisition device 1000 may restore the area from which data has been removed so that the sharpness of the restored area (e.g., a second sharpness) corresponds to the sharpness (e.g., the first sharpness) that is blurred due to the out-of-focus of the main object 1b included in the first image in this example.

[0080] According to an embodiment, in order to restore the area from which data is removed so that the sharpness of the restored area corresponds to the sharpness near the restored area, the image acquisition device 1000 may determine the sharpness of the main object 1b included in the restored area to correspond to the sharpness of the main object 1b included in the first image. The image acquisition device 1000 may additionally perform image processing so that the sharpness of the main object 1b included in the restored area corresponds to the sharpness of the main object 1b included in the first image.

[0081] Reference Fig.10 , the area of ​​the main object 1b included in the restored areas 6d, 6e, and 6f can be clearer than the vicinity of the restored areas 6d, 6e, and 6f. Fig.11 , the image acquisition device 1000 can perform image processing so that Fig.10 The sharpness of the clearly restored area in the image processing area corresponds to the sharpness near the restored area. For example, the image acquisition device 1000 may perform image processing for reducing the sharpness of the main object 1b included in the restored areas 6d, 6e, and 6f. The sharpness of the main object 1b included in the image processing-completed areas 6g, 6h, and 6i may correspond to the sharpness of the main object 1b included in the first image.

[0082] Although it has been described above that the hidden area of ​​the main object 1b is clearly restored and then image processing for reducing the sharpness of the clearly restored area of ​​the main object 1b is additionally performed, the embodiment is not limited thereto. Without performing image processing for reducing the sharpness of the clearly restored area of ​​the main object 1b, the hidden area of ​​the main object 1b may be restored so that the sharpness of the main object 1b included in the restored area corresponds to the sharpness near the restored area.

[0083] Fig.12 is a flow chart of a method of detecting sub-objects in response to user input according to some embodiments. Fig.13 and Fig.14 An example of detecting sub-objects in response to user input according to some embodiments is shown.

[0084] Reference Fig.12 , in operation S1210, the image acquiring device 1000 may receive an input selecting an object included in the first image, in operation S1230, the image acquiring device 1000 may detect the selected object as a sub-object, and in operation S1250, the image acquiring device 1000 may display an indicator indicating that the detected object is a sub-object.

[0085] In operation S1210, the image obtaining apparatus 1000 may receive an input of selecting an object included in a first image from a user via a user input interface.

[0086] Reference Fig.13 , the user 10 may select the object 2c included in the first image displayed on the touch screen of the image acquisition device 1000. For example, the user 10 may select the area where the object 2c is located by using the touch screen.

[0087] In operation S1230, the image acquiring device 1000 may detect the selected object 2c as a sub-object.

[0088] According to an embodiment, the image acquisition device 1000 may detect the selected object 2c as a sub-object by using an AI neural network. The AI ​​neural network may detect the appearance of an object (e.g., a tree, a trash can, a street lamp, a traffic light, a parking space, a person, and an animal) included in the image by learning an image of the appearance of the object as learning data. The image acquisition device 1000 may detect an object including an area where a user input has been received or located near the area by using an AI neural network.

[0089] Reference Fig.13 , the image acquisition device 1000 may detect an object 2c located near an area where an input from the user 10 has been received. The image acquisition device 1000 may detect the outer shape of an object 2c located near an area where an input from the user 10 has been received by using an AI neural network, wherein the object 2c is also referred to as a person 2c. The image acquisition device 1000 may detect the person 2c as a sub-object based on the detected outer shape of the person 2c.

[0090] In operation S1250, the image acquiring device 1000 may display an indicator indicating that the selected object 2c is a child object.

[0091] Reference Fig.14 , the image acquisition device 1000 may display an indicator indicating that the selected object 2c is a sub-object in a star shape near the object 2c. However, the embodiment is not limited thereto. Figure 4 As shown in , object 2c may be marked by a dotted line to indicate that object 2c is a child object. Figure 5As shown in , at least a portion of the object 2c may be covered with at least one of a preset color and a preset pattern and then displayed. The image acquisition device 1000 may display an indicator indicating that the selected object 2c is a sub-object based on the outer shape of the object 2c detected using the AI ​​neural network.

[0092] Fig.15 is a flow chart of a method of restoring at least a portion of a main object that was hidden by a child object in response to user input according to some embodiments. Figures 16 to 18 An example of restoring at least a portion of a main object hidden by a sub-object in response to a user input according to some embodiments is shown.

[0093] Reference Fig.15 In operation S1510, the image acquisition device 1000 may receive an input selecting an object included in the first image (a “selection” of a representation, data, or appearance of an object in the first image), in operation S1530, the image acquisition device 1000 may detect the selected object as a sub-object, in operation S1550, the image acquisition device 1000 may remove the selected object from the first image, and in operation S1570, the image acquisition device 1000 may generate a second image by restoring the area where the removed object was located.

[0094] In operation S1510, the image obtaining apparatus 1000 may receive an input of selecting an object included in a first image from a user via a user input interface.

[0095] Reference Fig.16 , the user 10 may select the object 2c included in the first image displayed on the touch screen of the image acquisition device 1000. For example, the user 10 may select the area where the object 2c is located by using the touch screen.

[0096] In operation S1530, the image acquiring device 1000 may detect the selected object 2c as a sub-object. Operation S1530 is similar to operation S1230, and thus a redundant description thereof will be omitted.

[0097] In operation S1550, the image acquiring device 1000 may remove the selected object 2c from the first image.

[0098] According to an embodiment, the image acquisition device 1000 may remove data associated with at least some areas of the first image where the object 2c is located from the first image. For example, the image acquisition device 1000 may remove only data associated with the sub-object 2c from the first image. Fig.17 As shown in the region 7a of the first image, the image acquisition device 1000 may remove data associated with the region 7a including the sub-object 2c from the first image. Operation S1550 is similar to operation S650, and thus a redundant description thereof will be omitted.

[0099] In operation S1570, the image obtaining apparatus 1000 may restore at least a portion of the first image from which data has been removed.

[0100] According to an embodiment, the image acquisition device 1000 may restore the area 7a from which data is removed so that at least a portion of the main object 1b hidden by the sub-object 2c is included. Fig.18 , the image acquisition device 1000 may restore the data-removed region 7a so that at least a portion of the main object 1b is included, thereby generating the restored region 7b. Operation S1570 is similar to operation S670, and thus a redundant description thereof will be omitted.

[0101] Fig.19 is a flow chart of a method of tracking motion of a sub-object within an image, according to some embodiments. Figure 20 to Figure 23 is a diagram illustrating an example of tracking the motion of a sub-object within an image according to some embodiments.

[0102] Reference Fig.19 , in operation S1910, the image acquiring device 1000 may acquire a first image by using a camera, in operation S1930, the image acquiring device 1000 may detect at least one of a main object and a sub-object from the first image, in operation S1950, the image acquiring device 1000 may track movement of the detected sub-object, and in operation S1970, the image acquiring device 1000 may display an indicator indicating the detected sub-object based on the movement of the sub-object.

[0103] Operation S1910 is similar to operation S210, and thus a redundant description thereof will be omitted.

[0104] Operation S1930 is similar to operation S230, and thus a redundant description thereof will be omitted.

[0105] In operation S1950, the image acquiring device 1000 may track the motion of the detected sub-object.

[0106] According to an embodiment, the image acquisition device 1000 may track the motion of the sub-objects 2a and 2b by inputting the first image to the AI ​​neural network. Fig. 20 , the AI ​​neural network may detect the outer shapes of sub-objects 2a and 2b from the first image, and may track the movement of sub-objects 2a and 2b based on the detected outer shapes of sub-objects 2a and 2b. For example, the AI ​​neural network may track the movement of sub-object 2a from left to right by recognizing the outer shape of sub-object 2a as the shape of a person walking from left to right. As another example, the AI ​​neural network may track the movement of sub-object 2b from the lower left end to the upper right end by recognizing the outer shape of sub-object 2b as the shape of a person walking from the lower left end to the upper right end.

[0107] According to an embodiment, the image acquisition device 1000 may track the motion of the sub-objects 2a and 2b by using a sensor included in the camera. The sensor included in the camera may include a sensor capable of detecting a phase difference of an object. The phase difference detection sensor may detect a phase change caused by the motion of the sub-objects 2a and 2b. The image acquisition device 1000 may track the motion of the sub-objects 2a and 2b based on the detected phase change of the sub-objects 2a and 2b.

[0108] According to an embodiment, the image acquisition device 1000 may acquire a plurality of images by using a camera, and track the motion of the detected sub-objects based on the acquired plurality of images. The image acquisition device 1000 may track the motion of the sub-objects 2a and 2b by inputting the acquired plurality of images into the AI ​​neural network. For example, the image acquisition device 1000 may track the motion of the sub-objects 2a and 2b by detecting the motion vectors of the sub-objects 2a and 2b from the first image. The image acquisition device 1000 may detect the motion vectors of the sub-objects 2a and 2b by acquiring a plurality of images before and after the first image using a camera and inputting the plurality of images acquired before and after the first image into the AI ​​neural network. For example, the AI ​​neural network may detect the motion vectors of the sub-objects 2a and 2b by comparing the sub-objects 2a and 2b included in the first image with the sub-objects 2a and 2b included in the plurality of images acquired before and after the first image. As another example, the AI ​​neural network can detect the motion vectors of sub-objects 2a and 2b by segmenting the first image into multiple blocks and segmenting each of multiple images acquired before and after the first image into multiple blocks and comparing the multiple blocks of the first image with the multiple blocks of each of the multiple images acquired before and after the first image.

[0109] In operation S1970, the image acquiring device 1000 may display an indicator indicating the detected sub-object based on the motion of the detected sub-object.

[0110] Reference Fig. 20 and Fig.21 , the image acquisition device 1000 may display the indicators 3a and 3b indicating the sub-objects 2a and 2b to correspond to the movements of the sub-objects 2a and 2b. When the sub-object 2a moves to the right, the image acquisition device 1000 may display the indicator 3a so that the indicator 3a located near the sub-object 2a is located near the sub-object 2a moving to the right. When the sub-object 2b moves to the upper right end, the image acquisition device 1000 may display the indicator 3b so that the indicator 3b located near the sub-object 2b is located near the sub-object 2b moving to the upper right end.

[0111] Reference Fig. 22 and Fig.23, the image acquisition device 1000 may display the outer shapes of the sub-objects 2a and 2b in dotted lines. The image acquisition device 1000 may display the outer shapes of the sub-objects 2a and 2b in dotted lines to correspond to the movements of the sub-objects 2a and 2b.

[0112] When the image acquisition device 1000 covers at least corresponding portions of the sub-objects 2a and 2b with at least one of a preset color and a preset pattern and displays the covering result, the image acquisition device 1000 may cover at least corresponding portions of the moving sub-objects 2a and 2b with at least one of a preset color and a preset pattern to correspond to the movement of the sub-objects 2a and 2b, and may display the covering result.

[0113] Fig.24 is a flow chart of a method of restoring at least a portion of a main object hidden by a sub-object based on a relative position between a camera and at least one of the main object and the sub-object in accordance with some embodiments. Fig.25 and Fig.26 An example of detecting a main object and sub-objects when a camera moves is shown in accordance with some embodiments.

[0114] Reference Fig.24 , in operation S2410, the image acquiring device 1000 may acquire a first image by using a camera, in operation S2430, the image acquiring device 1000 may detect a main object and a sub-object from the first image, in operation S2450, the image acquiring device 1000 may acquire information about a relative position between the camera and at least one of the main object and the sub-object, and in operation S2470, the image acquiring device 1000 may display a second image based on the acquired information about the relative position.

[0115] Operation S2410 is similar to operation S210 , and thus a redundant description thereof will be omitted.

[0116] Operation S2430 is similar to operation S230, and thus a redundant description thereof will be omitted.

[0117] In operation S2450, the image acquiring device 1000 may acquire information about a relative position between a camera and at least one of a main object and a sub-object.

[0118] According to an embodiment, the image acquisition device 1000 may acquire the location information of the camera. For example, the image acquisition device 1000 may acquire the location information of the camera by using a location sensor such as a global positioning system (GPS) module. As another example, the image acquisition device 1000 may acquire the location information of the camera by using short-range communication technology such as Wifi, Bluetooth, Zigbee, and beacon.

[0119] According to an embodiment, the image acquiring device 1000 may acquire position information of the camera from the first image by using an AI neural network.

[0120] For example, the AI ​​neural network may identify the main object 1b included in the first image, and obtain the location information of the area where the identified object 1b is located. In detail, when the main object 1b is a main building in a specific area, the AI ​​neural network may obtain the location information of the area where the main object 1b is located. Because the camera that captures the first image including the main object 1b will be located in an area similar to the area where the main object 1b is located, the AI ​​neural network may obtain the location information of the camera.

[0121] As another example, the AI ​​neural network may acquire the position information of the main object 1b by using the position information of the area where the main object 1b is located included in the first image, the shooting information of the first image (eg, focal length), and the size of the main object 1b within the first image.

[0122] According to an embodiment, the image acquisition device 1000 may acquire the position information of the main object. For example, the image acquisition device 1000 may acquire the position information of the camera by using a position sensor such as a GPS module, and may acquire the position information of the main object by using the shooting information (e.g., focal length) of the first image and the size of the main object in the first image.

[0123] According to an embodiment, the image acquisition device 1000 may acquire the position information of the main object from the first image by using an AI neural network. For example, the AI ​​neural network may identify the main object 1b included in the first image, and acquire the position information of the area where the identified main object 1b is located. In detail, when the main object 1b is a main building in a specific area, the AI ​​neural network may acquire the position information of the area where the main object 1b is located.

[0124] According to an embodiment, the image acquisition device 1000 may acquire multiple pieces of position information of the sub-objects 2a, 2b, and 2c. For example, the image acquisition device 1000 may acquire the position information of the camera by using a position sensor such as a GPS module, and may acquire the position information of the main object by using the shooting information (e.g., focal length) of the first image and the corresponding sizes of the sub-objects 2a, 2b, and 2c within the first image.

[0125] According to an embodiment, the image acquisition device 1000 may acquire information on the relative position between the camera and the main object. For example, the image acquisition device 1000 may acquire information on the relative position between the camera and the main object based on the acquired position information of the camera and the acquired position information of the main object. The information on the relative position depends on, for example, the depth or distance from the camera to the object in the field of view of the camera. As another example, the image acquisition device 1000 may acquire information on the relative position between the camera and the main object by using an AI neural network by using shooting information (e.g., focal length) of the first image and the size of the main object within the first image.

[0126] According to an embodiment, the image acquisition device 1000 may acquire information about the relative position between the camera and the sub-object. For example, the image acquisition device 1000 may acquire information about the relative position between the camera and the sub-object based on the acquired position information of the camera and the position information of the sub-object. As another example, the image acquisition device 1000 may acquire information about the relative position between the camera and the sub-object by using an AI neural network by using shooting information (e.g., focal length) of the first image and the size of the sub-object within the first image.

[0127] In operation S2470, the image acquiring device 1000 may generate a second image based on information of a relative position between the camera and at least one of the main object and the sub-object.

[0128] According to an embodiment, the image acquisition device 1000 may remove data associated with at least some regions of the first image where each of the sub-objects 2a, 2b, and 2c is located from the first image by using an AI neural network.

[0129] For example, the AI ​​neural network may track the movement of sub-objects 2a, 2b, and 2c based on the information of the relative position between the camera and the sub-objects 2a, 2b, and 2c. For example, the AI ​​neural network may detect the change of the information of the relative position between the camera and the sub-objects 2a, 2b, and 2c by using a phase difference detection sensor included in the camera. For example, the relative position may be determined at both the first moment and the second moment. The change of information is the change of the relative position information from the first moment to the second moment. As another example, the AI ​​neural network may detect the change of the information of the relative position between the camera and the sub-objects 2a, 2b, and 2c by comparing a plurality of images with each other. As another example, the AI ​​neural network may detect the change of the information of the relative position between the camera and the sub-objects 2a, 2b, and 2c by detecting motion vectors from a plurality of images acquired using the camera. The AI ​​neural network may predict the direction and speed of the movement of the sub-objects 2a, 2b, and 2c based on the change of the information of the relative position between the camera and the sub-objects 2a, 2b, and 2c. The AI ​​neural network may track the movement of the sub-objects 2a, 2b, and 2c based on the predicted direction and speed of the movement of the sub-objects 2a, 2b, and 2c. The operation of tracking the motion of the sub-object by using the AI ​​neural network is similar to operation S1950, and thus a redundant description thereof will be omitted.

[0130] Reference Fig.25 and Fig.26 , when the camera becomes farther away from the main objects 1a and 1b and the sub-objects 2a, 2b and 2c, the information of the relative position between the camera and the main objects 1a and 1b and the sub-objects 2a, 2b and 2c may change. When the user takes a photo, the user may change the position of the camera to change the composition of the photo. For example, the user may take a photo by moving one step back from the main object 1b (which is also referred to as the building 1b) of a specific area so as to capture a wider view of the building 1b in the shot. Therefore, because the user with the camera moves a farther distance from the building 1b, the information of the relative position between the camera and each of the main objects 1a and 1b and the sub-objects 2a, 2b and 2c may change.

[0131] The image acquisition device 1000 may track the movement of the sub-objects 2a, 2b, and 2c based on the changed position information. The image acquisition device 1000 may display indicators indicating the sub-objects 2a and 2b corresponding to the movement of the sub-objects 2a and 2b. Fig.25 and Fig.26, the image acquisition device 1000 may track the movement of the sub-objects 2a and 2b in the direction toward the upper right end. The image acquisition device 1000 may display an indicator indicating the sub-objects 2a and 2b by marking or rendering the outlines of the sub-objects 2a and 2b with dotted lines corresponding to the movement of the sub-objects 2a and 2b. In some embodiments, data in an image corresponding to an object in the field of view of a camera may be referred to as an outline corresponding to the object, or simply an outline.

[0132] The AI ​​neural network can effectively remove data associated with at least some areas of the first image where the sub-objects 2a, 2b, and 2c are located based on the movements of the sub-objects 2a, 2b, and 2c. The operation of removing data associated with at least some areas of the first image where the sub-objects 2a, 2b, and 2c are located by the image acquisition device 1000 is similar to operation S650, and thus a redundant description thereof will be omitted.

[0133] According to an embodiment, the image acquisition device 1000 may track the motion of the sub-objects 2a, 2b, and 2c, and may remove data associated with at least some regions of the first image where the sub-objects 2a, 2b, and 2c are respectively located from the first image. The operation of tracking the motion of the sub-objects 2a, 2b, and 2c by the image acquisition device 1000 is similar to operation S1950, and the operation of removing data associated with at least some regions of the first image where the sub-objects 2a, 2b, and 2c are respectively located from the first image by the image acquisition device 1000 is similar to operation S650, and thus a redundant description thereof will be omitted.

[0134] According to an embodiment, the image acquisition device 1000 may restore the area from which data is removed based on information of the relative position between the camera and the main object 1b by using an AI neural network, so that at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c is included. For example, the AI ​​neural network may acquire the position information of the main object 1b, wherein the main object 1b is a building located in a specific area. As in operation S2450, the AI ​​neural network may acquire information of the relative position between the camera and the main object 1b. The AI ​​neural network may search for an image of the main object 1b similar to the first image based on the position information of the main object 1b and the information of the relative position between the camera and the main object 1b. The AI ​​neural network may restore the area from which data is removed of the first image by learning at least one found image as learning data. The AI ​​neural network may generate a second image by restoring the area from which data is removed within the first image.

[0135] According to an embodiment, the image acquisition device 1000 may display the second image on the display by restoring the area in the first image from which data is removed. The image acquisition device 1000 may store the second image in the memory. In this case, the image acquisition device 1000 may store the second image in the memory when generating the second image by restoring the area in the first image from which data is removed, or may store the second image in the memory in response to a user input.

[0136] Fig. 27 2000 is a diagram illustrating a method of displaying an indicator indicating a child object by using the server 2000 according to some embodiments. Fig.28 and Fig.29 2000 is a diagram illustrating an example of displaying an indicator indicating a child object by using the server 2000 according to some embodiments.

[0137] Reference Fig. 27 , in operation S2710, the image acquiring device 1000 may receive an input for driving a camera from a user, in operation S2720, the image acquiring device 1000 may access the server 2000, in operation S2730, the image acquiring device 1000 may acquire a first image by using the camera, and in operation S2740, the image acquiring device 1000 may transmit the first image to the server 2000. In operation S2750, the server 2000 may detect a main object and a sub-object from the received first image, and in operation S2760, the server 2000 may transmit information on a result of detecting the main object and the sub-object to the image acquiring device 1000. In operation S2770, the image acquiring device 1000 may display an indicator indicating the sub-object based on the information received from the server 2000.

[0138] In operation S2710, the image acquisition device 1000 may receive an input for driving a camera from a user. Fig.28 , the image acquisition device 1000 may receive an input for a button for driving the camera from the user 10. Alternatively, the image acquisition device 1000 may receive an input for driving an application related to the camera from the user 10.

[0139] In operation S2720 , the image acquiring device 1000 may access the server 2000 .

[0140] According to an embodiment, when the image acquisition device 1000 receives an input to drive the camera from a user, the image acquisition device 1000 may automatically access the server 2000 by forming a communication link with the server. The communication link may be, for example, an Internet-based session. The server 2000 may include an AI neural network that detects at least one of the main objects 1a and 1b and the sub-objects 2a, 2b, and 2c from the first image. The server 2000 may include the following AI neural network: wherein the AI ​​neural network removes data associated with at least some areas of the first image where the sub-objects 2a, 2b, and 2c are located, and restores the area where the data is removed, so that at least a portion of the main object 1b hidden by the sub-objects 2a, 2b, and 2c is included.

[0141] In operation S2730, the image acquiring apparatus 1000 may acquire a first image by using a camera. Operation S2730 is similar to operation S210, and thus a redundant description thereof will be omitted.

[0142] In operation S2740, the image acquiring device 1000 may transmit the acquired first image to the server 2000 through a communication link.

[0143] In operation S2750, the server 2000 may detect a main object and a sub-object from the received first image by using an AI neural network. Fig.28 and Fig.29 , the server 2000 may detect main objects 1a and 1b and sub-objects 2a, 2b, and 2c from the first image. The AI ​​neural network may be dedicated to the user of the image acquisition device 1000. For example, the AI ​​neural network may have learned data associated with the user (e.g., at least one image stored in the user's image acquisition device 1000) as learning data. The operation of the AI ​​neural network detecting the main object and the sub-object from the first image is similar to operation S230, and thus a redundant description thereof will be omitted.

[0144] In operation S2760, the server 2000 may transmit the result of detecting the main object and the sub-object from the first image by using the AI ​​neural network to the image acquisition device 1000 through the communication link. For example, the server 2000 may transmit the first image including indicators indicating the main object and the sub-object to the image acquisition device 1000. Alternatively, the server 2000 may transmit information indicating the objects included in the first image and information indicating whether each of the objects is the main object or the sub-object to the image acquisition device 1000.

[0145] In operation S2770, the image acquiring device 1000 may display indicators 3a, 3b, and 3c indicating the sub-objects 2a, 2b, and 2c, respectively. Operation S2770 is similar to operation S250, and thus a redundant description thereof will be omitted.

[0146] When the image acquisition device 1000 accesses the server 2000 in response to receiving an input for driving the camera from the user, the image acquisition device 1000 may receive data about the AI ​​neural network from the server 2000. For example, the image acquisition device 1000 may receive a software module implementing the AI ​​neural network from the server 2000. As another example, the image acquisition device 1000 may receive data for updating the AI ​​neural network from the server 2000. The image acquisition device 1000 may detect a main object and a sub-object from the first image based on the received data about the AI ​​neural network.

[0147] Fig.30 and Fig.31 is a block diagram of an image acquisition device 1000 according to some embodiments.

[0148] Reference Fig.30 , the image acquisition device 1000 may include a user input interface 1100 , a display 1210 , a processor 1300 , and a communication interface 1500 . Fig.30 All components shown in the figure are not essential components of the image acquisition device 1000. Fig.30 More or fewer components than those shown in the figure may constitute the image acquisition device 1000.

[0149] For example, refer to Fig.31 , the image acquiring device 1000 may further include a sensing unit 1400 , an audio / video (A / V) input interface 1600 , an output interface 1200 , and a memory 1700 in addition to the user input interface 1100 , the processor 1300 , and the communication interface 1500 .

[0150] The user input interface 1100 refers to a unit through which a user inputs data for controlling the image acquisition device 1000. For example, the user input interface 1100 may be, but is not limited to, a keypad, a dome switch, a touch pad (e.g., capacitive cover type, resistive cover type, infrared beam type, integral strain gauge type, surface acoustic wave type, piezoelectric type, etc.), a roller, or a roller switch. The user input interface 1100 may include a touch screen that receives a user's touch input by combining a touch layer with the display 1210.

[0151] The user input interface 1100 may receive a user input of selecting at least one object displayed on the display 1210 (“selection”).

[0152] The output interface 1200 may output an audio signal, a video signal, or a vibration signal, and may include a display 1210 , an audio output interface 1220 , and a vibration motor 1230 .

[0153] The display 1210 may display information processed by the image acquisition device 1000. For example, the display 1210 may display a first image and an indicator indicating a sub-object. The display 1210 may display a second image obtained by restoring a region from which data is removed so that at least a portion of the main object is included.

[0154] The audio output interface 1220 outputs audio data received from the communication interface 1500 or stored in the memory 1700. The audio output interface 1220 also outputs an audio signal related to the function of the image acquiring device 1000 (eg, a call signal reception sound, a message reception sound, or a notification sound).

[0155] The processor 1300 generally controls the overall operation of the image acquisition device 1000. For example, the processor 1300 may control the user input interface 1100, the output interface 1200, the sensing unit 1400, the communication interface 1500, the A / V input interface 1600, etc. by executing the program stored in the memory 1700. The processor 1300 may execute the program stored in the memory 1700. Figures 1 to 29 The functions of the image acquisition device 1000 are as follows.

[0156] In detail, the processor 1300 may control the user input interface 1100 to receive a user input for selecting at least one of the displayed objects. The processor 1300 may control the microphone 1620 to receive a user's voice input. The processor 1300 may run an application that performs the operation of the image acquisition device 1000 based on the user input, and may control the user input interface to receive the user input through the executed application. For example, the processor 1300 may execute a voice assistant application, and may control the A / V input interface 1600 to receive a user's voice input through the microphone 1620 by controlling the executed voice assistant application.

[0157] The processor 1300 may control the output interface 1200 and the memory 1700 of the image acquisition device 1000 to display the first image, the indicator indicating the sub-object, and the second image. The processor 1300 may include at least one microchip.

[0158] The processor 1300 may detect a main object and a sub-object from the first image. The processor 1300 may detect at least one of the main object and the sub-object by using an AI neural network.

[0159] The processor 1300 may remove the sub-object from the first image and generate the second image by restoring the area of ​​the first image from which the sub-object has been removed. The processor 1300 may remove the sub-object from the first image by using an AI neural network and restore the area of ​​the first image from which the sub-object has been removed. In this case, the AI ​​neural network may restore at least a portion of the main object hidden by the removed sub-object.

[0160] The processor 1300 may acquire information of a relative position between the camera and the main object, and may restore the at least a portion of the main object based on the information of the relative position.

[0161] The processor 1300 may perform image processing for restoring the at least a portion of the main object so that the sharpness of the restored portion of the main object corresponds to the sharpness of the main object in the first image. For example, the processor 1300 may determine the sharpness of the restored at least a portion of the main object to correspond to the sharpness of the main object in the first image, and may perform image processing so that the sharpness of the restored portion of the main object corresponds to the determined sharpness.

[0162] The processor 1300 may detect a sub-object selected from the first image based on the user input. For example, the processor 1300 may detect the shape of each of the plurality of objects included in the first image by using an AI neural network, and may detect an object located near the area where the user input has been received as a sub-object.

[0163] The processor 1300 may track the movement of the sub-object within the first image. For example, the processor 1300 may acquire information of the relative position between the camera and the sub-object, and may track the movement of the sub-object by detecting a change in the information of the relative position.

[0164] The sensing unit 1400 may sense a state of the image acquiring device 1000 or a state of a surrounding environment of the image acquiring device 1000 and may transmit information corresponding to the sensed state to the processor 1300 .

[0165] The sensing unit 1400 may include at least one selected from a magnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyro sensor 1450, a position sensor (e.g., GPS) 1460, an atmospheric pressure sensor 1470, a proximity sensor 1480, and an RGB sensor 1490 (i.e., an illumination sensor), but is not limited thereto. Given their names, those of ordinary skill in the art will instinctively understand the functions of most sensors, and thus their detailed description will be omitted herein.

[0166] The communication interface 1500 may include at least one component that enables the image acquisition device 1000 to communicate with other devices (not shown) and the server 2000. The other device may be a computing device (such as the image acquisition device 1000) or a sensing device, and the embodiment is not limited thereto. For example, the communication interface 1500 may include a short-range wireless communication interface 1510, a mobile communication interface 1520, and a broadcast receiver 1530.

[0167] Examples of the short-range wireless communication interface 1510 may include, but are not limited to, a Bluetooth communication interface, a Bluetooth low energy (BLE) communication interface, a near field communication (NFC) interface, a wireless local area network (WLAN) (e.g., Wi-Fi) communication interface, a ZigBee communication interface, an infrared data association (IrDA) communication interface, a Wi-Fi Direct (WFD) communication interface, an ultra-wideband (UWB) communication interface, and an Ant+ communication interface.

[0168] The mobile communication interface 1520 may exchange wireless signals with at least one selected from a base station, an external terminal, and a server on a mobile communication network. Here, examples of wireless signals may include voice call signals, video call signals, and various types of data according to text / multimedia message transmission.

[0169] The broadcast receiver 1530 receives a broadcast signal and / or broadcast-related information from an external source via a broadcast channel. The broadcast channel may be a satellite channel, a ground wave channel, etc. According to an embodiment, the image acquisition device 1000 may not include the broadcast receiver 1530.

[0170] According to an embodiment, the communication interface 1500 may transmit the first image to the server 2000 .

[0171] According to an embodiment, the communication interface 1500 may receive information on a result of detecting the main object and the sub-object from the server 2000 .

[0172] According to an embodiment, the communication interface 1500 may receive a software module implementing an AI neural network from the server 2000 .

[0173] According to an embodiment, the communication interface 1500 may receive data for updating the AI ​​neural network from the server 2000 .

[0174] The A / V input interface 1600 inputs an audio signal or a video signal and may include a camera 1610 and a microphone 1620. The camera 1610 may acquire an image frame such as a still image or a moving picture via an image sensor in a video call mode or a photographing mode. The image captured via the image sensor may be processed by the processor 1300 or a separate image processor (not shown).

[0175] Microphone 1620 receives an external audio signal and converts the external audio signal into electrical audio data. For example, microphone 1620 may receive an audio signal from an external device or a user. Microphone 1620 may receive a user's voice input. Microphone 1620 may use various noise removal algorithms to remove noise generated when receiving an external audio signal.

[0176] The memory 1700 may store a program used by the processor 1300 to perform processing and control, and may also store data input to or output from the image acquiring device 1000 .

[0177] The memory 1700 may include at least one type of storage medium selected from a flash memory type, a hard disk type, a micro multimedia card type, a card type memory (e.g., a secure digital (SD) or an extreme digital (XD) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0178] Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, a user interface (UI) module 1710 , a touch screen module 1720 , and a notification module 1730 .

[0179] The UI module 1710 may provide a UI, a graphical user interface (GUI), etc. that is dedicated to each application and interoperates with the image acquisition device 1000. The touch screen module 1720 may detect a touch gesture of a user on the touch screen and transmit information about the touch gesture to the processor 1300. The touch screen module 1720 according to an embodiment may recognize and analyze a touch code. The touch screen module 1720 may be configured by separate hardware including a controller.

[0180] The notification module 1730 may generate a signal for notifying that an event has been generated in the image acquiring device 1000. Examples of the event generated in the image acquiring device 1000 may include call signal reception, message reception, key signal input, schedule notification, etc. The notification module 1730 may output the notification signal in the form of a video signal via the display 1210, in the form of an audio signal via the audio output interface 1220, or in the form of a vibration signal via the vibration motor 1230.

[0181] Fig.32 is a block diagram of a server 2000 according to some embodiments.

[0182] Reference Fig.32 , the server 2000 may include a communication interface 2500 , a database (DB) 2700 , and a processor 2300 .

[0183] The communication interface 2500 may include at least one component that enables the server 2000 to communicate with the image acquiring device 1000 .

[0184] The communication interface 2500 may receive an image from the image acquiring device 1000 or transmit an image to the image acquiring device 1000 .

[0185] DB 2700 may store training models and learning data applied in the training models.

[0186] The processor 2300 generally controls the overall operation of the server 2000. For example, the processor 2300 may control the DB 2700 and the communication interface 2500 by executing a program stored in the DB 2700 of the server 2000. The processor 2300 may execute a program stored in the DB 2700 to perform Figures 1 to 29 Some operations of the image acquisition device 1000.

[0187] Processor 2300 can perform at least one of the following functions: a function of detecting at least one of a main object and a sub-object from a first image, a function of removing data associated with at least some areas of the first image where the sub-object is located, and a function of restoring the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0188] Processor 2300 may manage at least one of the following data: data required for detecting at least one of a main object and a sub-object from a first image, data required for removing data associated with at least some areas of the first image where the sub-object is located, and data required for restoring the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0189] Fig.33 is a block diagram of a processor 1300 according to some embodiments.

[0190] Reference Fig.33 , the processor 1300 may include a data learner 1310 and a data identifier 1320.

[0191] The data learner 1310 may learn a criterion for detecting a main object and a sub-object from a first image. The data learner 1310 may learn a criterion as to which data will be used in order to detect a main object and a sub-object from a first image. The data learner 1310 may learn a criterion for detecting a main object and a sub-object from a first image by obtaining data for learning and applying the obtained data to a data recognition model to be described later.

[0192] The data learner 1310 may learn a criterion for removing, from the first image, data associated with at least some regions of the first image where sub-objects are located. The data learner 1310 may learn a criterion regarding which data will be used in order to remove, from the first image, data associated with at least some regions of the first image where sub-objects are located. The data learner 1310 may learn a criterion for removing, from the first image, data associated with at least some regions of the first image where sub-objects are located, data by obtaining data for learning and applying the obtained data to a data recognition model.

[0193] The data learner 1310 may learn a criterion for restoring a region from which data has been removed so that at least a portion of a main object hidden by a sub-object is included. The data learner 1310 may learn a criterion as to which data will be used in order to restore a region from which data has been removed so that at least a portion of a main object hidden by a sub-object is included. The data learner 1310 may learn a criterion for restoring a region from which data has been removed so that at least a portion of a main object hidden by a sub-object is included by obtaining data for learning and applying the obtained data to a data recognition model.

[0194] The data identifier 1320 may detect a main object and a sub-object from the first image based on the data. The data identifier 1320 may detect a main object and a sub-object from the first image based on specific data by using a trained data recognition model. The data identifier 1320 may detect a main object and a sub-object from the first image by obtaining specific data according to a standard preset due to learning and using a data recognition model by using the obtained data as an input value. A result value output by the data recognition model by using the obtained data as an input value may be used to update the data recognition model.

[0195] The data identifier 1320 may remove data associated with at least some areas of the first image where the sub-object is located from the first image based on the data. The data identifier 1320 may remove data associated with at least some areas of the first image where the sub-object is located from the first image by using a trained data recognition model. The data identifier 1320 may remove data associated with at least some areas of the first image where the sub-object is located from the first image by obtaining specific data according to a standard preset due to learning and using the data recognition model by using the obtained data as an input value. A result value output by the data recognition model by using the obtained data as an input value may be used to update the data recognition model.

[0196] The data identifier 1320 may restore the area from which the data was removed based on the data so that at least a portion of the main object hidden by the sub-object is included. The data identifier 1320 may restore the area from which the data was removed by using a trained data recognition model so that at least a portion of the main object hidden by the sub-object is included. The data identifier 1320 may restore the area from which the data was removed by obtaining specific data according to a standard preset due to learning and using the data recognition model by using the obtained data as an input value so that at least a portion of the main object hidden by the sub-object is included. The result value output by the data recognition model by using the obtained data as an input value may be used to update the data recognition model.

[0197] At least one of the data learner 1310 and the data identifier 1320 may be manufactured in the form of at least one hardware chip and may be installed on a device. For example, at least one of the data learner 1310 and the data identifier 1320 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) or a processor dedicated to graphics (e.g., a graphics processing unit (GPU)), and may be installed on any of the aforementioned various devices.

[0198] In this case, the data learner 1310 and the data identifier 1320 may both be installed on a single device, or may be installed on separate devices, respectively. For example, one of the data learner 1310 and the data identifier 1320 may be included in the image acquisition device 1000, and the other may be included in the server 2000. The data learner 1310 and the data identifier 1320 may be connected to each other by wire or wirelessly, so that the model information established by the data learner 1310 may be provided to the data identifier 1320 and the data input to the data identifier 1320 may be provided to the data learner 1310 as additional learning data.

[0199] At least one of the data learner 1310 and the data identifier 1320 may be implemented as a software module. When at least one of the data learner 1310 and the data identifier 1320 is implemented using a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In this case, at least one software module may be provided by an operating system (OS) or by a specific application. Alternatively, some of the at least one software module may be provided by the OS, and other software modules may be provided by a specific application.

[0200] Fig.34 is a block diagram of a data learner 1310 according to some embodiments.

[0201] Reference Fig.34 , the data learner 1310 may include a data obtainer 1310-1, a preprocessor 1310-2, a learning data selector 1310-3, a model learner 1310-4 and a model evaluator 1310-5.

[0202] The data obtainer 1310-1 may obtain data required to detect the main object and the sub-object from the first image. The data obtainer 1310-1 may obtain data required to remove data associated with at least some areas of the first image where the sub-object is located from the first image. The data obtainer 1310-1 may obtain data required to restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0203] The data obtainer 1310-1 may obtain at least one image stored in the image obtaining device 1000. For example, the data obtainer 1310-1 may obtain an image including a face of the user and an image including faces of people associated with the user (e.g., family members, lovers, and relatives of the user), wherein the images are stored in the image obtaining device 1000. As another example, the data obtainer 1310-1 may obtain an image associated with an area where the user is located (e.g., main buildings, sculptures, and natural landscapes of the area), wherein the images are stored in the image obtaining device 1000. As another example, the data obtainer 1310-1 may obtain at least one image disclosed on the Internet (e.g., main buildings, sculptures, and natural landscapes of the area). As another example, the data obtainer 1310-1 may obtain at least one image associated with general objects (e.g., trees, trash cans, street lights, traffic lights, parking spaces, people, and animals), wherein the at least one image is disclosed on the Internet.

[0204] The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to detect at least one of the main object and the sub-object from the first image. The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to remove data associated with at least some areas of the first image where the sub-object is located from the first image. The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to restore the area where the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0205] The preprocessor 1310 - 2 may process the obtained data into a preset format so that a model learner 1310 - 4 to be described later may perform learning to detect at least one of a main object and a sub-object from a first image using the obtained data.

[0206] The learning data selector 1310-3 may select data required for learning from a plurality of pre-processed data. The selected data may be provided to the model learner 1310-4.

[0207] The learning data selector 1310-3 may select data required for learning from a plurality of preprocessed data according to a preset standard for detecting at least one of a main object and a sub-object from a first image. The learning data selector 1310-3 may select data required for learning from a plurality of preprocessed data according to a preset standard for removing data associated with at least some areas of the first image where the sub-object is located from the first image. The learning data selector 1310-3 may select data required for learning from a plurality of preprocessed data according to a preset standard for restoring an area from which data has been removed so that at least a portion of the main object hidden by the sub-object is included.

[0208] The learning data selector 1310 - 3 may select data according to a criterion preset due to learning by the model learner 1310 - 4 , which will be described later.

[0209] The model learner 1310-4 may learn a standard on how to detect at least one of the main object and the sub-object from the first image based on the learning data. The model learner 1310-4 may learn a standard on how to remove data associated with at least some areas of the first image where the sub-object is located from the first image based on the learning data. The model learner 1310-4 may learn a standard on how to restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included based on the learning data.

[0210] The model learner 1310-4 may learn a criterion regarding which learning data will be used in order to detect at least one of the main object and the sub-object from the first image. The model learner 1310-4 may learn a criterion regarding which learning data will be used in order to remove data associated with at least some areas of the first image where the sub-object is located from the first image. The model learner 1310-4 may learn a criterion regarding which learning data will be used in order to restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0211] The model learner 1310-4 may train a training model for detecting at least one of a main object and a sub-object from a first image by using the learning data. The model learner 1310-4 may train a training model for removing data associated with at least some areas of the first image where the sub-object is located from the first image by using the learning data. The model learner 1310-4 may train a training model for restoring the area from which the data is removed so that at least a portion of the main object hidden by the sub-object is included by using the learning data.

[0212] In this case, the training model may be a pre-established model. For example, the training model may be a model pre-established by receiving basic learning data (eg, sample data).

[0213] The training model may be established by considering, for example, the application field of the recognition model, the purpose of learning, or the computer performance of the device. The training model may include, for example, a model based on a neural network. For example, a model such as a DNN, RNN, or BRDNN may be used as a training model, but the embodiment is not limited thereto.

[0214] According to various embodiments, when there are multiple pre-established training models, the model learner 1310-4 may determine a training model having a high correlation between the input learning data and the basic learning data as a training model to be trained. In this case, the basic learning data may be pre-classified according to the type of data, and the training model may be pre-established according to the type of data. For example, the basic learning data may be pre-classified according to various criteria such as the region where the learning data is generated, the time when the learning data is generated, the size of the learning data, the type of the learning data, the generator of the learning data, and the type of the object in the learning data.

[0215] The model learner 1310-4 may train the training model by using a learning algorithm including, for example, error back propagation or gradient descent.

[0216] The model learner 1310-4 may train the training model through supervised learning using, for example, learning data as an input value. The model learner 1310-4 may train the training model through unsupervised learning to find a standard for detecting at least one of the main object and the sub-object from the first image by detecting at least one of the main object and the sub-object without supervision and self-learning the type of data required to provide a response operation corresponding to the result of the detection. The model learner 1310-4 may train the training model through reinforcement learning using feedback on whether the result of detecting at least one of the main object and the sub-object according to the learning is correct.

[0217] When the training model is trained, the model learner 1310-4 may store the trained training model. In this case, the model learner 1310-4 may store the trained training model in a memory of a device including the data identifier 1320. Alternatively, the model learner 1310-4 may store the trained training model in a memory of a server connected to the device via a wired network or a wireless network.

[0218] In this case, the memory storing the trained training model may also store, for example, commands or data related to at least one other component of the device. The memory may also store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API), and / or an application program (or application).

[0219] When the model evaluator 1310-5 inputs the evaluation data to the training model and the recognition result output from the training model does not meet the predetermined standard, the model evaluator 1310-5 may enable the model learner 1310-4 to learn again. In this case, the evaluation data may be preset data for evaluating the training model.

[0220] For example, when the number or percentage of evaluation data providing inaccurate recognition results in the recognition results of the trained training model for the evaluation data exceeds a preset threshold, the model evaluator 1310-5 may evaluate that the predetermined standard is not met. For example, when the predetermined standard is defined as 2% and the trained training model outputs erroneous recognition results for more than 20 evaluation data among a total of 1000 evaluation data, the model evaluator 1310-5 may evaluate that the trained training model is not suitable.

[0221] When there are multiple trained training models, the model evaluator 1310-5 may evaluate whether each of the multiple trained training models satisfies a predetermined criterion, and may determine the training model that satisfies the predetermined criterion as the final training model. In this case, when multiple training models meet the predetermined criterion, the model evaluator 1310-5 may determine one training model or a predetermined number of training models preset in descending order of evaluation scores as the final training model.

[0222] At least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 in the data learner 1310 may be manufactured in the form of at least one hardware chip and may be installed on a device. For example, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or AP) or a processor dedicated to graphics (e.g., a GPU), and may be installed on any of the aforementioned various devices.

[0223] The data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may all be installed on a single device, or may be installed on separate devices, respectively. For example, some of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be included in the device, and the others may be included in the server.

[0224] For example, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be implemented as a software module. When at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In this case, at least one software module may be provided by the OS or by a specific application. Alternatively, some of the at least one software module may be provided by the OS, and other software modules may be provided by a specific application.

[0225] The processor 1300 may use various training models, and may effectively learn a criterion for detecting at least one of a main object and a sub-object from a first image according to various methods via the various training models.

[0226] Fig.35 is a block diagram of a data identifier 1320 according to some embodiments.

[0227] Reference Fig.35 , the data identifier 1320 may include a data obtainer 1320-1, a preprocessor 1320-2, a recognition data selector 1320-3, a recognition result provider 1320-4 and a model improver 1320-5.

[0228] The data obtainer 1320-1 may obtain data required to detect the main object and the sub-object from the first image. The data obtainer 1320-1 may obtain data required to remove data associated with at least some areas of the first image where the sub-object is located from the first image. The data obtainer 1320-1 may obtain data required to restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0229] The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to detect at least one of the main object and the sub-object from the first image. The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to remove data associated with at least some areas of the first image where the sub-object is located from the first image. The preprocessor 1310-2 may preprocess the obtained data so that the obtained data may be used to restore the area where the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0230] The preprocessor 1320-2 may process the obtained data into a preset format so that the recognition result provider 1320-4, which will be described later, can use the obtained data to detect at least one of the main object and the sub-object from the first image. The preprocessor 1320-2 may preprocess the obtained data into a preset format so that the recognition result provider 1320-4 can use the obtained data to remove data associated with at least some areas of the first image where the sub-object is located from the first image.

[0231] The recognition data selector 1320-3 selects data required to detect at least one of the main object and the sub-object from the first image from the plurality of preprocessed data. The recognition data selector 1320-3 may select data required to remove data associated with at least some areas of the first image where the sub-object is located from the plurality of preprocessed data. The recognition data selector 1320-3 may select data required to restore the area from which the data has been removed so that at least a portion of the main object hidden by the sub-object is included from the plurality of preprocessed data. The selected data may be provided to the recognition result provider 1320-4.

[0232] The identification data selector 1320-3 may select some or all of the plurality of preprocessed data according to a preset standard for detecting at least one of the main object and the sub-object from the first image. The identification data selector 1320-3 may select some or all of the plurality of preprocessed data according to a preset standard for removing data associated with at least some areas of the first image where the sub-object is located from the first image. The identification data selector 1320-3 may select some or all of the plurality of preprocessed data according to a preset standard for restoring the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0233] The recognition data selector 1320 - 3 may select data according to a criterion preset due to learning by the model learner 1310 - 4 , which will be described later.

[0234] The recognition result provider 1320-4 can detect at least one of the main object and the sub-object from the first image by applying the selected data to the data recognition model. The recognition result provider 1320-4 can remove data associated with at least some areas of the first image where the sub-object is located from the first image by applying the selected data to the data recognition model. The recognition result provider 1320-4 can restore the area where the data was removed by applying the selected data to the data recognition model so that at least a portion of the main object hidden by the sub-object is included.

[0235] The recognition result provider 1320-4 may provide a recognition result that meets the purpose of data recognition. The recognition result provider 1320-4 may apply the selected data to the data recognition model by using the data selected by the recognition data selector 1320-3 as an input value. The recognition result may be determined by the data recognition model. For example, the recognition result of detecting at least one of the main object and the sub-object from the first image, the recognition result of removing the data associated with at least some areas of the first image where the sub-object is located from the first image, and the recognition result of restoring the area where the data is removed so that at least a portion of the main object hidden by the sub-object is included may be provided as text, an image, or an instruction (e.g., an application execution instruction or a module function execution instruction).

[0236] The model improver 1320-5 may enable the data recognition model to be updated based on the evaluation of the recognition result provided by the recognition result provider 1320-4. For example, the model improver 1320-5 may enable the model learner 1310-4 to update the data recognition model by providing the recognition result provided by the recognition result provider 1320-4 to the model learner 1310-4.

[0237] At least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 within the data identifier 1320 may be manufactured in the form of at least one hardware chip and may be installed on a device. For example, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or AP) or a processor dedicated to graphics (e.g., a GPU), and may be installed on any of the aforementioned various devices.

[0238] The data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 may all be installed on a single electronic device, or may be installed on separate electronic devices, respectively. For example, some of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 may be included in the electronic device, and the others may be included in the server.

[0239] At least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 may be implemented as a software module. When at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model improver 1320-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In this case, at least one software module may be provided by the OS or by a specific application. Alternatively, some of the at least one software module may be provided by the OS, and other software modules may be provided by a specific application.

[0240] The image acquisition device 1000 can provide the user with an image that meets the user's intention by using the training model to which the learning result has been applied.

[0241] Fig.36 is a block diagram illustrating an example in which the image acquisition device 1000 and the server 2000 interoperate to learn and recognize data according to some embodiments.

[0242] Reference Fig.36 The server 2000 may learn a standard for detecting at least one of a main object and a sub-object from a first image, a standard for removing data associated with at least some areas of the first image where the sub-object is located from the first image, and a standard for restoring the area from which the data has been removed so that at least a portion of the main object hidden by the sub-object is included, and the image acquisition device 1000 may detect at least one of the main object and the sub-object from the first image based on the learning result of the server 2000, remove data associated with at least some areas of the first image where the sub-object is located from the first image, and restore the area from which the data has been removed so that at least a portion of the main object hidden by the sub-object is included.

[0243] In this case, the model learner 2340 of the server 2000 may execute Fig.33 The function of the data learner 1310.

[0244] The model learner 2340 of the server 2000 may learn a criterion for detecting a main object and a sub-object from the first image. The model learner 2340 may learn a criterion about which data will be used in order to detect the main object and the sub-object from the first image. The model learner 2340 may learn a criterion for detecting a main object and a sub-object from the first image by obtaining data for learning and applying the obtained data to a data recognition model to be described later.

[0245] The model learner 2340 may learn a criterion for removing, from the first image, data associated with at least some areas of the first image where sub-objects are located. The model learner 2340 may learn a criterion as to which data will be used in order to remove, from the first image, data associated with at least some areas of the first image where sub-objects are located. The model learner 2340 may learn a criterion for removing, from the first image, data associated with at least some areas of the first image where sub-objects are located, data by obtaining data for learning and applying the obtained data to a data recognition model to be described later.

[0246] The model learner 2340 may learn a criterion for restoring a region from which data has been removed so that at least a portion of the main object hidden by the sub-object is included. The model learner 2340 may learn a criterion as to which data will be used in order to restore a region from which data has been removed so that at least a portion of the main object hidden by the sub-object is included. The model learner 2340 may learn a criterion for restoring a region from which data has been removed so that at least a portion of the main object hidden by the sub-object is included by obtaining data for learning and applying the obtained data to a data recognition model to be described later.

[0247] The model learner 2340 can learn at least one of a criterion for detecting at least one of a main object and a sub-object from a first image, a criterion for removing data associated with at least some areas of the first image where the sub-object is located, and a criterion for restoring the area from which the data has been removed so that at least a portion of the main object hidden by the sub-object is included, by obtaining data for learning and applying the obtained data to a training model to be described later.

[0248] The recognition result provider 1320-4 of the image acquisition device 1000 may detect at least one of the main object and the sub-object from the first image, remove data associated with at least some areas of the first image where the sub-object is located, and restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included by applying the data selected by the recognition data selector 1320-3 to the data recognition model generated by the server 2000. For example, the recognition result provider 1320-4 may send the data selected by the recognition data selector 1320-3 to the server 2000, and the server 2000 may request to detect at least one of the main object and the sub-object from the first image, remove data associated with at least some areas of the first image where the sub-object is located, and restore the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included by applying the data selected by the recognition data selector 1320-3 to the recognition model.

[0249] The recognition result provider 1320-4 may receive information about the following method from the server 2000: detecting at least one of a main object and a sub-object from a first image, removing data associated with at least some areas of the first image where the sub-object is located, and restoring the area from which the data was removed so that at least a portion of the main object hidden by the sub-object is included.

[0250] Alternatively, the recognition result provider 1320-4 of the image acquisition device 1000 may receive a recognition model generated by the server 2000 from the server 2000, and may detect at least one of the main object and the sub-object from the first image by using the received recognition model. The recognition result provider 1320-4 may remove data associated with at least some areas of the first image where the sub-object is located from the first image by using the received recognition model, and restore the area from which the data is removed so that at least a portion of the main object hidden by the sub-object is included. In this case, the recognition result provider 1320-4 of the image acquisition device 1000 may detect at least one of the main object and the sub-object from the first image by applying the data selected by the recognition data selector 1320-3 to the recognition model received from the server 2000, remove data associated with at least some areas of the first image where the sub-object is located from the first image, and restore the area from which the data is removed so that at least a portion of the main object hidden by the sub-object is included.

[0251] Each unit of the image acquisition device 1000 may be implemented by a combination of a processor, a memory, and a program code located in the memory and executed by the processor to perform various functions of the above-mentioned method and device.

[0252] Each unit of the server 2000 may be implemented by a combination of a processor, a memory, and a program code located in the memory and executed by the processor to perform various functions of the above-mentioned methods and devices.

[0253] The image acquisition device 1000 and the server 2000 can effectively issue and execute operations for learning and data recognition of the training model, and accordingly effectively perform data processing to provide the user with the desired image. Figure 1 consistent service and effectively protect user privacy.

[0254] Some embodiments may also be implemented as storage media including instruction codes executable by a computer, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes all volatile / nonvolatile and removable / non-removable media. In addition, computer-readable media may include all computer storage and communication media. Computer storage media includes all volatile / nonvolatile and removable / non-removable media implemented by a specific method or technology for storing information, such as computer-readable instruction codes, data structures, program modules, or other data.

[0255] The term "~ unit" used herein may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

[0256] Although the embodiments of the present disclosure have been disclosed for illustrative purposes, it will be appreciated by those skilled in the art that various changes and modifications are possible without departing from the spirit and scope of the present disclosure. Therefore, the above embodiments should be understood to be non-restrictive but illustrative in all respects. For example, the various elements described in an integrated form can be used separately, and the separated elements can be used in a combined state.

[0257] Although one or more example embodiments have been described with reference to the drawings, workers skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. A method for providing an image, the method comprising: acquiring, via a camera of an image acquisition device, a first image including a plurality of objects; receiving input selecting a region of a first image; detecting an object from the plurality of objects using a first trained model based on the selected region; generating, using the second trained model, a second image including image data to replace data associated with a region corresponding to the detected object with other data; as well as displaying a second image using a display of the image acquisition device, wherein a first size of the selected area is different from a second size of the area corresponding to the detected object, and The first training model is different from the second training model.

2. The method according to claim 1, wherein: The detected object is a person.

3. The method according to claim 1, wherein: The first training model is trained to recognize at least one object for removal from a given image.

4. The method according to claim 3, wherein: The first trained model is configured to detect a main object and sub-objects in a first image.

5. The method according to claim 1, wherein: The second training model is trained to recover data corresponding to areas occluded by the detected object.

6. The method according to claim 1, wherein: Generating a second image includes removing data associated with the detected object.

7. The method according to claim 6, wherein: Data associated with the detected object is removed using a third trained model that is trained to remove data associated with areas where sub-objects are located.

8. The method according to claim 1, wherein: Generating a second image includes replacing data associated with a region corresponding to the detected object included in the first image with data corresponding to a region of the background hidden by the detected object.

9. The method according to claim 1, further comprising: Based on the detected object, an indicator indicating the detected object is displayed.

10. The method according to claim 9, wherein: Displaying the indicator includes displaying the detected object covered with at least one of a preset color and a preset pattern.

11. The method according to claim 1, wherein: Generating a second image includes generating a portion of another object hidden by the detected object.

12. The method according to claim 1, wherein: The second training model includes a generative artificial intelligence (AI) model.

13. An image acquisition device, comprising: monitor; camera; at least one processor; as well as The memory is configured to store instructions, which, when executed by the at least one processor, cause the image acquisition device to perform the following operations: controlling the camera to acquire a first image including a plurality of objects; receiving input selecting a region of a first image; detecting an object from the plurality of objects using a first trained model based on the selected region; generating, using the second trained model, a second image including image data to replace data associated with a region corresponding to the detected object with other data; as well as controlling the display to display a second image, wherein a first size of the selected area is different from a second size of the area corresponding to the detected object, and The first training model is different from the second training model.

14. The image acquisition device according to claim 13, wherein: The detected object is a person.

15. The image acquisition device according to claim 13, in, A first training model is trained to recognize at least one object for removal from a given image, and The first training model detects the main object and the sub-object in the first image.

16. The image acquisition device according to claim 13, wherein: The second training model is trained to recover data corresponding to areas occluded by the detected object.

17. The image acquisition device according to claim 13, wherein: The instructions further cause the image acquisition device to perform the following operations: Data associated with the detected object is removed using a third trained model, the third trained model being trained to remove data associated with the area where the sub-object is located.

Images