Electronic device, method, and non-transitory computer-readable recording medium for determining whether to fill image

Abstract

An electronic device is disclosed. The electronic device may comprise a display. The electronic device may comprise a processor. The electronic device may comprise a memory for storing instructions. The electronic device may obtain a resulting image generated by filling an input image, on the basis of a first ratio at which an object exists in the input image being greater than or equal to a first reference ratio. The electronic device may cause the resulting image to refrain from being displayed through the display on the basis of a second ratio of a newly filled portion of the object exceeding a second reference ratio. The electronic device may display the resulting image through the display on the basis of the second ratio being equal to or less than the second reference ratio.

Description

Electronic device, method, and non-transitory computer-readable recording medium for determining whether an image is filled

[0001] The following descriptions relate to electronic devices, methods, and non-transitory computer-readable recording media for determining whether an image is filled.

[0002] An electronic device may provide multiple functions. For example, the electronic device may generate an image based on a prompt. For example, the electronic device may generate an image based on an object inputted as a prompt. For example, the electronic device may fill in an outer area of ​​an image inputted as a prompt (e.g., out-painting) or fill in an inner area of ​​an image (e.g., in-painting).

[0003] An electronic device is disclosed. The electronic device may include a display. The electronic device may include a processor. The electronic device may include a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to obtain a result image generated by a filling process on an input image based on a first ratio of an object present in the input image being greater than or equal to a first reference ratio. The instructions, when executed by the processor, may cause the electronic device to refrain from displaying the result image through the display based on a second ratio of a newly filled portion of the object exceeding a second reference ratio. The instructions, when executed by the processor, may cause the electronic device to display the result image through the display based on a second ratio being less than or equal to the second reference ratio.

[0004] An electronic device is disclosed. The electronic device may include a display. The electronic device may include a processor. The electronic device may include a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to obtain a first anomaly for an input image through a first anomaly detection module. The first anomaly detection module may be trained by images in which a first ratio of an object is greater than or equal to a first reference ratio. The instructions, when executed by the processor, may cause the electronic device to obtain a result image generated by a filling process for the input image based on the first anomaly being less than or equal to a first reference anomaly. The instructions, when executed by the processor, may cause the electronic device to obtain a second anomaly for the result image through a second anomaly detection module. The second anomaly detection module may be trained by images in which a second ratio of an object is greater than or equal to a specified second reference ratio. The instructions, when executed by the processor, may cause the electronic device to refrain from displaying the result image through the display based on whether the second abnormality is greater than or equal to a second reference abnormality. The instructions, when executed by the processor, may cause the electronic device to display the result image through the display based on whether the second abnormality is less than a second reference abnormality.

[0005] A method is disclosed. The method can be performed in an electronic device including a display. The method can include an operation of obtaining a result image generated by a filling process on an input image based on a first ratio of an object present in the input image being greater than or equal to a first reference ratio. The method can include an operation of refraining from displaying the result image through the display based on a second ratio of a newly filled portion of the object exceeding a second reference ratio. The method can include an operation of displaying the result image through the display based on a second ratio being less than or equal to the second reference ratio.

[0006] A method is disclosed. The method can be performed in an electronic device including a display. The method can include an operation of acquiring a first outlier for an input image through a first outlier detection module. The first outlier detection module can be trained by images in which a first ratio of an object is greater than or equal to a first reference ratio. The method can include an operation of acquiring a result image generated by a filling process for the input image based on whether the first outlier is less than or equal to the first reference outlier. The method can include an operation of acquiring a second outlier for the result image through a second outlier detection module. The second outlier detection module can be trained by images in which a second ratio of an object is greater than or equal to a designated second reference ratio. The method can include an operation of refraining from displaying the result image through the display based on whether the second outlier is greater than or equal to the second reference outlier. The method can include an operation of displaying the result image through the display based on whether the second outlier is less than or equal to the second reference outlier.

[0007] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when executed by a processor of an electronic device including a display, can cause the electronic device to obtain a result image generated by a filling process on an input image based on a first ratio of an object present in the input image being greater than or equal to a first reference ratio. The instructions, when executed by the processor, can cause the electronic device to refrain from displaying the result image through the display based on a second ratio of a newly filled portion of the object exceeding a second reference ratio. The instructions, when executed by the processor, can cause the electronic device to display the result image through the display based on a second ratio being less than or equal to the second reference ratio.

[0008] A non-transitory computer-readable recording medium is disclosed. The non-transitory computer-readable recording medium may store a program including instructions. The instructions, when executed by a processor of an electronic device including a display, may cause the electronic device to obtain a first anomaly for an input image through a first anomaly detection module. The first anomaly detection module may be trained by images in which a first ratio of an object is greater than or equal to a first reference ratio. The instructions, when executed by the processor, may cause the electronic device to obtain a result image generated by a filling process for the input image based on the first anomaly being less than or equal to a first reference anomaly. The instructions, when executed by the processor, may cause the electronic device to obtain a second anomaly for the result image through a second anomaly detection module. The second anomaly detection module may be trained by images in which a second ratio of an object is greater than or equal to a specified second reference ratio. The instructions, when executed by the processor, may cause the electronic device to refrain from displaying the result image through the display based on whether the second abnormality is greater than or equal to a second reference abnormality. The instructions, when executed by the processor, may cause the electronic device to display the result image through the display based on whether the second abnormality is less than a second reference abnormality.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2A is a block diagram of components within an electronic device for determining whether an image is filled.

[0011] FIG. 2b is a block diagram of components within an electronic device for determining whether to fill an image based on a conditional judgment.

[0012] FIG. 2c is a block diagram of components within an electronic device for determining whether to fill an image based on whether an anomaly is detected.

[0013] Figure 3a is a diagram illustrating an example of an input image.

[0014] Figure 3b is a diagram illustrating an example of an input image.

[0015] Figure 3c is a diagram illustrating an example of an input image.

[0016] Figure 3d is a drawing showing an example of a landmark.

[0017] Figure 3e is a drawing showing an example of a landmark.

[0018] Figure 3f is a diagram illustrating an example of an input image.

[0019] Figure 4a is a drawing showing an example of a result image.

[0020] Figure 4b is a drawing showing an example of a result image.

[0021] Figure 4c is a drawing showing an example of a result image.

[0022] Figure 4d is a drawing showing an example of the result image.

[0023] Figure 5 is a block diagram of components of an electronic device and server for generating a result image.

[0024] Figure 6a is a diagram illustrating an example of an input image and a masking area.

[0025] FIG. 6b is a diagram illustrating an example of an image for a masking area transmitted from a server to an electronic device.

[0026] Figure 6c is a drawing showing an example of a result image synthesized in an electronic device.

[0027] Figure 7a is a diagram illustrating an example of an input for an object included in an input image.

[0028] Figure 7b is a diagram illustrating an example of a masking area and a result image determined according to input.

[0029] Figure 7c is a diagram illustrating an example of a masking area and a result image determined according to input.

[0030] Figure 8a is a diagram illustrating an example of an input for an object included in an input image.

[0031] Figure 8b is a diagram illustrating an example of a masking area and a resulting image determined according to input.

[0032] Figure 8c is a diagram illustrating an example of a masking area and a result image determined according to input.

[0033] Figure 8d is a diagram illustrating an example of a masking area and a result image determined according to input.

[0034] FIG. 9 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0035] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0036] FIG. 11 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0037] FIG. 12 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0038] FIG. 13 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0039] FIG. 14 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0040] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0041] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0042] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0043] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0044] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0045] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0046] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0047] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0048] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0049] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0050] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0051] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0052] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0053] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0054] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0055] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0056] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0057] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0058] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0059] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0060] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0061] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0062] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0063] FIG. 2a is a block diagram of components within an electronic device (101) for determining whether to fill an image.

[0064] Referring to FIG. 2A, the electronic device (101) may include a preprocessor (210), an image filler (220), and a postprocessor (230). In one embodiment, the preprocessor (210), the image filler (220), and the postprocessor (230) may be included in a program (140) of the electronic device (101).

[0065] In one embodiment, the preprocessor (210) may determine whether to perform filling processing on the input image (201). In one embodiment, the filling processing may be a process of creating a new image in a masking area associated with the input image (201). For example, the filling processing may include an in-painting process of drawing a new image in a masking area selected within the input image (201). For example, the filling processing may include an out-painting process of drawing a new image in a masking area selected outside the input image (201).

[0066] In one embodiment, the preprocessor (210) may determine whether to perform a filling process on the input image (201) based on whether the input image (201) satisfies a specified condition. In one embodiment, the specified condition may be related to a state of one or more objects included in the input image (201). For example, the state of the object may include the size and / or ratio of the object within the input image (201). In one embodiment, the operation of the preprocessor (210) to determine whether the input image (201) satisfies the specified condition may be described with reference to FIGS. 2B, 3A, 3B, 3C, and 3F.

[0067] In one embodiment, the preprocessor (210) may determine whether to perform a filling process on the input image (201) based on whether the anomaly rate detected for the input image (201) exceeds a specified first reference anomaly rate. In one embodiment, the anomaly rate for the input image (201) may be obtained by an anomaly detector. In one embodiment, the anomaly detector may be described with reference to FIG. 2C.

[0068] In one embodiment, the preprocessor (210) may set a masking area associated with the input image (201). In one embodiment, the preprocessor (210) may set a masking area outside the input image (201) and / or a masking area inside the input image (201) based on an input. In one embodiment, the preprocessor (210) may set a masking area associated with an object selected based on a user input for the input image (201). For example, the masking area may be an area of ​​an object selected in the input image (201). For example, the masking area may be an area to which the object selected in the input image (201) has moved. For example, the masking area may be an area adjacent to an area to which the object selected in the input image (201) has moved. For example, if an object selected from the input image (201) is located on the periphery (e.g., a side) of the input image (201), an area extending in the direction of the side from the area where the selected object has moved may be a masking area. However, the present invention is not limited thereto. In one embodiment, the preprocessor (210) may set a masking area to enlarge the size of the input image (201). For example, the masking area may be an area extending in the direction of a selected side among one or more sides of the input image (201). The masking area set by the preprocessor (210) may be described with reference to FIGS. 7a, 7b, 7c, 8a, 8b, 8c, and 8d.

[0069] In one embodiment, the image filler (220) may include a generative AI (artificial intelligence) model. In one embodiment, the generative AI model may be a model that can draw a new image in a masking area based on an input image (201). For example, the generative AI model may draw a new image in a masking area related to the input image (201) based on at least one prompt. Here, the prompt may include at least one keyword related to the input image (201). The prompt may include at least one keyword that can be extracted from the input image (201). Here, the keyword may be at least one object included in the input image (201) and / or a word (or sentence) to describe the input image (201). In one embodiment, the keyword may be extracted from the image filler (220), but is not limited thereto. For example, the keyword may be extracted from the preprocessor (210). In one embodiment, the preprocessor (210) may transmit data about the input image (201), the masking area, and the keywords to the image filler (220).

[0070] In one embodiment, the image filler (220) may generate a result image (205) for an input image (201) based on a generative AI model. In one embodiment, the result image (205) may include a partial image (or filled image) generated by a masking area. In one embodiment, the result image (205) may include the partial image (or filled image) and the input image (201).

[0071] In one embodiment, the image filler (220) may pass the resulting image (205) to a post-processor (230).

[0072] In one embodiment, the post-processor (230) can determine whether to output the result image (205). In one embodiment, the post-processor (230) can determine whether to output the result image (205) through the display module (160).

[0073] In one embodiment, the post-processor (230) may determine whether to output the result image (205) based on whether the result image (205) satisfies a specified condition. In one embodiment, the specified condition may be related to the state of one or more objects included in the result image (205). For example, the state of the object may include the size and / or ratio of the object within the result image (205). In one embodiment, the operation of the post-processor (230) to determine whether the result image (205) satisfies the specified condition may be described with reference to FIGS. 2B, 4A, 4B, 4C, and 4D.

[0074] In one embodiment, the post-processor (230) may determine whether to output the result image (205) based on whether the outlier detected for the result image (205) exceeds a specified second reference outlier. In one embodiment, the outlier for the result image (205) may be obtained by an anomaly detection unit. In one embodiment, the anomaly detection unit may be described with reference to FIG. 2C.

[0075] In one embodiment, the post-processor (230) may determine whether to save the result image (205). For example, the post-processor (230) may determine whether to save the result image (205) based on whether the result image (205) satisfies a specified condition. For example, the post-processor (230) may determine whether to save the result image (205) based on whether an outlier detected for the result image (205) exceeds a specified second reference outlier.

[0076] In one embodiment, the post-processor (230) can synthesize the input image (201) onto the result image (205). In one embodiment, the post-processor (230) can synthesize the input image (201) onto the remaining portion of the result image (205) excluding the portion generated by the masking region. In one embodiment, the post-processor (230) can synthesize at least a portion of the input image (201) onto the remaining portion of the result image (205) excluding the portion generated by the masking region. The synthesis between the result image (205) and the input image (201) can be described with reference to FIGS. 6A, 6B, 6C, and 8D.

[0077] FIG. 2b is a block diagram of components within an electronic device (101) for determining whether to fill an image based on conditional judgment.

[0078] Referring to FIG. 2b, the electronic device (101) may include one or more detectors (211, 231) and one or more condition determination units (213, 233).

[0079] In one embodiment, the detector (211) of the preprocessor (210) can identify an object in the input image (201). In one embodiment, the detector (211) can identify a region including the object. In one embodiment, the detector (211) can identify a bounding box including the object. In one embodiment, the bounding box may be a virtual box formed in a substantially rectangular shape including the identified object. In one embodiment, the bounding box may be a box of the smallest size that can include the identified object. In one embodiment, the bounding box is not limited to a rectangular shape and may be formed in various shapes. In one embodiment, the bounding box may be referred to as a segmentation or a convex hull including the identified object.

[0080] In one embodiment, the detector (211) can identify a designated portion of an object in an input image (201). For example, if the object is a human, the designated portion of the object may be a body part of the human (e.g., face, eyes, nose, mouth, ears, arms, hands). For example, if the object is an animal, the designated portion of the object may be a body part of the animal (e.g., head, eyes, nose, mouth, ears, legs). For example, if the object is an object, the designated portion of the object may be a part of a component of the object (e.g., in the case of a car, a hood, wheels, windows, doors). However, the present invention is not limited thereto.

[0081] In one embodiment, the detector (211) may include one or more detectors for identifying different types of objects. In one embodiment, the detector (211) may identify an object based on a detector for identifying a type of object other than a person or a body part of a person, based on whether the type of the identified object is different from a person or a body part of a person (e.g., a face).

[0082] In one embodiment, the detector (211) can identify an object and / or a designated portion of an object based on one or more landmarks of the object in an input image (201). In one embodiment, the identification of the object and / or the designated portion of the object can be described with reference to FIGS. 3d and 3e.

[0083] In one embodiment, the detector (211) can identify an area that includes a designated portion of an object. In one embodiment, the detector (211) can identify a bounding box that includes at least a portion of the designated portion of the object.

[0084] In one embodiment, the condition judgment unit (213) of the preprocessor (210) may determine whether to perform filling processing on the input image (201) based on the bounding box. In one embodiment, the condition judgment unit (213) may determine whether to perform filling processing on the input image (201) based on the type of object included in the bounding box.

[0085] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on a first number of bounding boxes for the identified one or more objects and a second number of bounding boxes for a designated portion of each of the identified one or more objects. In one embodiment, the condition judgment unit (213) may determine to perform the filling process on the input image (201) if the second number of bounding boxes of the face is greater than or equal to the first number of bounding boxes of the body. In one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (201) if the second number of bounding boxes of the face is less than the first number of bounding boxes of the body.

[0086] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on the state of the bounding box for a designated portion of each of one or more identified objects. In one embodiment, the state of the bounding box may include the size of the bounding box and / or the ratio of the bounding box. For example, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the region of the bounding box located in the input image (201). For example, the ratio of the bounding box may mean the ratio between the size of the bounding box located in the masking region and the size of the region of the bounding box located in the input image (201). For example, the ratio of the bounding box may mean the ratio of landmarks included in the bounding box among designated landmarks of an object or a designated portion of an object (or included in the input image (201)). For example, if the object is a person, the designated landmarks of the person may be a nose, eyes, ears, mouth, shoulder, elbow, wrist, finger, hip, knee, ankle, heel, and / or foot. For example, if the designated part of the object is a person's face, the designated landmarks of the face may be a nose, eyes, ears, and / or mouth, but are not limited thereto.

[0087] In one embodiment, the condition determination unit (213) may determine whether to perform a filling process on the input image (201) based on comparing the ratio of the bounding box for each of one or more identified objects with a specified first ratio. In one embodiment, the condition determination unit (213) may determine to perform the filling process on the input image (201) if the ratio of the bounding box for all objects is greater than or equal to the specified first ratio. In one embodiment, the condition determination unit (213) may determine not to perform the filling process on the input image (201) if the ratio of the bounding box for at least one of the objects is less than the specified first ratio. In one embodiment, the first ratio may be set differently depending on the object. In one embodiment, the first ratio for a person, an animal, or an object may be different from each other. For example, the first ratio for a person, a dog, or a car may be 80%, 40%, or 20%. In one embodiment, the first ratio may be predetermined based on the difficulty of the filling process. For example, if the difficulty of the filling process is high, a higher first ratio may be used.

[0088] In one embodiment, the condition determination unit (213) may determine whether to perform a filling process on the input image (201) based on comparing the ratio of the bounding box for each designated portion of one or more identified objects with a designated first ratio. In one embodiment, the condition determination unit (213) may determine to perform the filling process on the input image (201) if the ratio of the bounding box for all designated portions is greater than or equal to the designated first ratio. In one embodiment, the condition determination unit (213) may determine not to perform the filling process on the input image (201) if the ratio of the bounding box for at least one of the designated portions is less than the designated first ratio. In one embodiment, the first ratio may be set differently depending on the object and / or the designated portion of the object. In one embodiment, the first ratios for the designated portions of a person, an animal, or an object may be different from each other. For example, the first ratio for a human face, a human hand, or a human shoulder may be 60%, 90%, or 20%. However, the present invention is not limited thereto. In one embodiment, the first ratio may be predetermined according to the difficulty of the filling process. In one embodiment, the difficulty of the filling process may depend on whether the identity between the image before the filling process and the image after the filling process is recognized. For example, in the case of an object (e.g., a human face) whose identity is recognized as low as the ratio of the newly generated area according to the filling process is high, it may be understood that the difficulty of the filling process is high. For example, when the difficulty of the filling process is high, the first ratio may have a higher ratio. For example, the first ratio for a person, a dog, or a car may be 50%, 60%, or 20%.

[0089] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on a comparison result between a first number of bounding boxes for the identified one or more objects and a second number of bounding boxes for a designated portion of each of the identified one or more objects and a state of the bounding boxes for the designated portion of each of the identified one or more objects. In one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (201) if the second number of bounding boxes for the face is less than the first number of bounding boxes for the body. In one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (201) if the ratio of bounding boxes for at least one of the objects is less than the designated first ratio. In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (201) when the second number of bounding boxes of the face is greater than or equal to the first number of bounding boxes of the body and the ratio of the bounding boxes of the face is greater than or equal to the specified first ratio.

[0090] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on whether the bounding box of each of the identified one or more objects is located on the boundary (or edge) of the input image (201). In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on whether the bounding box of a designated object (e.g., a head) among the identified one or more objects is located on the boundary (or edge) of the input image (201). For example, if the bounding box of a designated object (e.g., a head) is located on a designated boundary (or edge) of the input image (201) (e.g., the upper, left, and right boundaries of the input image (201), it may be determined not to perform a filling process on the input image (201).

[0091] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on whether the bounding box of each of the identified one or more objects is located on the boundary (or edge) of a masking area inside the input image (201). In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on whether the bounding box of a designated object (e.g., a head) among the identified one or more objects is located on the boundary (or edge) of a masking area inside the input image (201). For example, if the bounding box of the designated object (e.g., a head) is located on the designated boundary (or edge) of a masking area inside the input image (201), it may be determined not to perform a filling process on the input image (201).

[0092] In one embodiment, the condition determination unit (213) may first identify whether the bounding box of each of the identified one or more objects is located on the boundary (or edge) of the input image (201) in order to reduce latency. If the bounding box of one of the identified one or more objects is located on the boundary (or edge) of the input image (201), the condition determination unit (213) may identify whether one bounding box includes a designated object (e.g., a face). In one embodiment, the condition determination unit (213) may determine not to perform a filling process on the input image (201) if one bounding box located on the boundary that includes the designated object (e.g., a face) is identified.

[0093] In one embodiment, the condition determination unit (213) may first identify whether the bounding box of each of the identified one or more objects is located on the boundary (or edge) of the masking region of the input image (201) in order to reduce latency. If the bounding box of one of the identified one or more objects is located on the boundary (or edge) of the masking region of the input image (201), the condition determination unit (213) may identify whether one bounding box includes a designated object (e.g., a face). In one embodiment, the condition determination unit (213) may determine not to perform filling processing on the masking region inside the input image (201) when one bounding box located on the boundary that includes the designated object (e.g., a face) is identified.

[0094] In one embodiment, the detector (231) of the post-processor (230) can identify an object in the result image (205). In one embodiment, the detector (231) can identify an area containing the object. In one embodiment, the detector (231) can identify a bounding box containing the object.

[0095] In one embodiment, the detector (231) can identify a designated portion of an object in the result image (205). In one embodiment, the detector (231) can identify an area including the designated portion of the object. In one embodiment, the detector (231) can identify a bounding box including the designated portion of the object. For example, if the object is a human, the designated portion of the object may be a body part of the human (e.g., face, eyes, nose, mouth, ears, arms, hands). For example, if the object is an animal, the designated portion of the object may be a body part of the animal (e.g., head, eyes, nose, mouth, ears, legs). For example, if the object is an object, the designated portion of the object may be a part of a component of the object (e.g., in the case of a car, a hood, wheels, windows, doors). However, the present invention is not limited thereto.

[0096] In one embodiment, the condition judgment unit (233) of the post-processor (230) can determine whether to output the result image (205) based on the bounding box.

[0097] In one embodiment, the condition judgment unit (233) may determine whether to output the result image (205) based on the state of the bounding box for one or more identified objects and / or a designated portion of each of the objects. In one embodiment, the state of the bounding box may include the size of the bounding box and / or the ratio of the bounding box. For example, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the bounding box located in the partial image (or filled image) generated by the masking area.

[0098] In one embodiment, the condition determination unit (233) may determine whether to output the result image (205) based on comparing the ratio of the bounding box for each of one or more identified objects with a specified second ratio. In one embodiment, the condition determination unit (233) may determine to output the result image (205) if the ratio of the bounding box for all objects is less than or equal to the specified second ratio. In one embodiment, the condition determination unit (233) may determine not to output the result image (205) if the ratio of the bounding box for at least one object among the objects exceeds the specified second ratio. Depending on the embodiment, the first ratio and the second ratio may be the same. For example, the first ratio and the second ratio may be 50%. However, the present invention is not limited thereto. For example, the first ratio may be higher or lower than the second ratio.

[0099] In one embodiment, the condition determination unit (233) may determine whether to output the result image (205) based on comparing the ratio of the bounding box for each designated portion of one or more identified objects with a designated second ratio. In one embodiment, the condition determination unit (233) may determine to output the result image (205) if the ratio of the bounding box for all designated portions is less than or equal to the designated second ratio. In one embodiment, the condition determination unit (233) may determine not to output the result image (205) if the ratio of the bounding box for at least one of the designated portions exceeds the designated second ratio. Depending on the embodiment, the first ratio and the second ratio may be the same. For example, the first ratio and the second ratio may be 50%. However, the present invention is not limited thereto. For example, the first ratio may be higher or lower than the second ratio. The higher the ratio of the newly created area according to the filling process, the lower the identity may be recognized. Accordingly, the identity between the objects included in the input image (201) before the filling process and the objects included in the result image (205) after the filling process can be maintained by different second ratios depending on the type of objects, and accordingly, the user can avoid feeling awkwardness between the input image (201) and the output image (205) due to the filling process.

[0100] FIG. 2c is a block diagram of components within an electronic device (101) for determining whether to fill an image based on whether an anomaly is detected.

[0101] Referring to FIG. 2c, the electronic device (101) may include one or more abnormality detection units (215, 235).

[0102] In one embodiment, the anomaly detection unit (215) may include an auto-encoder. In one embodiment, the anomaly detection unit (215) may identify a first anomaly based on the auto-encoder. In one embodiment, the anomaly detection unit (215) may identify the first anomaly based on a difference value between an input image (201) and an output image of the auto-encoder. In one embodiment, the auto-encoder may be trained based on one or more first images that satisfy a specified condition. In one embodiment, the auto-encoder may include one or more encoding layers for encoding the input image (201) and one or more decoding layers for decoding the encoding result. In one embodiment, the auto-encoder may calculate a difference value between the input image (201) and an output image of the auto-encoder. In one embodiment, the auto-encoder may identify the difference value as the first anomaly. However, the present invention is not limited thereto.

[0103] In one embodiment, the one or more first images satisfying a specified condition may be images in which a second number of bounding boxes for a designated portion of each of the one or more objects identified in the image is greater than a first number of bounding boxes for the one or more objects identified in the image. In one embodiment, the one or more first images satisfying a specified condition may be images in which a ratio of bounding boxes for a designated portion of each of the one or more objects identified in the image is greater than or equal to a specified first ratio. In one embodiment, the one or more first images satisfying a specified condition may be images in which a second number of bounding boxes for a designated portion of each of the one or more objects identified in the image is greater than a first number of bounding boxes for the one or more objects identified in the image, and a ratio of bounding boxes for a designated portion of each of the one or more objects identified in the image is greater than or equal to a specified first ratio. However, the present invention is not limited thereto.

[0104] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on the first outlier. In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on comparing the first outlier with the first reference outlier.

[0105] In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (201) if the first outlier is greater than or equal to the first reference outlier. In one embodiment, the condition judgment unit (213) may determine not to perform a filling process on the input image (201) if the first outlier is less than the first reference outlier.

[0106] In one embodiment, the anomaly detection unit (235) may include an auto-encoder. In one embodiment, the anomaly detection unit (235) may identify a second anomaly based on the auto-encoder. In one embodiment, the anomaly detection unit (235) may identify the second anomaly based on a difference value between the result image (205) and an output image of the auto-encoder. In one embodiment, the auto-encoder may be trained based on one or more second images that satisfy a specified condition. In one embodiment, the auto-encoder may include one or more encoding layers for encoding the result image (205) and one or more decoding layers for decoding the encoding result. In one embodiment, the auto-encoder may calculate a difference value between the result image (205) and an output image of the auto-encoder. In one embodiment, the auto-encoder may identify the difference value as the second anomaly. However, the present invention is not limited thereto.

[0107] In one embodiment, the one or more second images satisfying the specified condition may be images in which the ratio of bounding boxes for the specified portions of each of the one or more objects identified in the image is greater than or equal to the specified second ratio.

[0108] In one embodiment, the condition judgment unit (233) may determine whether to output the result image (205) based on the first outlier. In one embodiment, the condition judgment unit (233) may determine whether to output the result image (205) based on comparing the first outlier with the first reference outlier.

[0109] In one embodiment, the condition judgment unit (233) may determine to output the result image (205) if the second abnormal value is greater than or equal to the second reference abnormal value. In one embodiment, the condition judgment unit (233) may determine not to output the result image (205) if the second abnormal value is less than the second reference abnormal value. Depending on the embodiment, the first reference abnormal value and the second reference abnormal value may be the same. For example, the first reference abnormal value and the second reference abnormal value may be 0.9. However, the present invention is not limited thereto. For example, the first reference abnormal value may be higher or lower than the second reference abnormal value.

[0110] According to an embodiment, the electronic device (101) may determine whether to perform filling processing on the input image (201) through the preprocessor (210), the detector (e.g., 211 of FIG. 2B) and / or the anomaly detection unit (215). In one embodiment, the electronic device (101) may determine whether to perform filling processing on the input image (201) based on a judgment result based on the detector (e.g., 211 of FIG. 2B) and / or a judgment result based on the anomaly detection unit (215).

[0111] According to an embodiment, the electronic device (101) may determine whether to perform filling processing on the input image (201) through the post-processor (230), the detector (e.g., 231 of FIG. 2B) and / or the anomaly detection unit (235). In one embodiment, the electronic device (101) may determine whether to output the result image (205) based on a judgment result based on the detector (e.g., 231 of FIG. 2B) and / or the anomaly detection unit (235).

[0112] Fig. 3a is a diagram illustrating an example of an input image. Fig. 3b is a diagram illustrating an example of an input image. Fig. 3c is a diagram illustrating an example of an input image.

[0113] FIGS. 3a, 3b, and 3c can be described with reference to FIGS. 1, 2a, 2b, and 2c.

[0114] Referring to FIG. 3A, the detector (211) can identify objects (321, 322) in an input image (311). In one embodiment, the detector (211) can identify bounding boxes (331, 332) for the objects (321, 322) in the input image (311). In one embodiment, the detector (211) may not identify a designated portion (e.g., a face) for the objects (321, 322) in the input image (311).

[0115] In one embodiment, the detector (211) may set a masking area (351) related to the input image (311). In one embodiment, the detector (211) may set a masking area (351) outside the input image (311) based on the input. In one embodiment, the detector (211) may set the masking area (351) to enlarge the size of the input image (311). For example, the masking area (351) may be an area extended in the direction of a selected side among one or more sides (361, 362, 363, 364) of the input image (311). For example, the masking area (351) may be an area above the upper side (361) of the input image (311). For example, the masking area (351) may be an area above the upper edge (361) and an area below the lower edge (363) of the input image (311). For example, the masking area (351) may be an area above the upper edge (361) and an area to the left of the left edge (364) of the input image (311). For example, the masking area (351) may be an area outside each of the four edges (361, 362, 363, 364) of the input image (311).

[0116] Referring to FIG. 3B, the detector (211) can identify objects (323, 324) in the input image (313). In one embodiment, the detector (211) can identify bounding boxes (333, 334) for the objects (323, 324) in the input image (313). In one embodiment, the detector (211) can identify a designated portion (e.g., a face) for some objects (324) among the objects (323, 324) in the input image (313). In one embodiment, the detector (211) can identify bounding boxes (343, 344) for a designated portion (e.g., a face) for the objects (323, 324) in the input image (313).

[0117] In one embodiment, the detector (211) may set a masking area (353) related to the input image (313). In one embodiment, the detector (211) may set a masking area (353) outside the input image (313) based on the input. In one embodiment, the detector (211) may set the masking area (353) to enlarge the size of the input image (313). For example, the masking area (353) may be an area extended in the direction of a selected side among one or more sides (361, 362, 363, 364) of the input image (313).

[0118] Referring to FIG. 3c, the detector (211) can identify objects (325, 326) in the input image (315). In one embodiment, the detector (211) can identify bounding boxes (335, 336) for the objects (325, 326) in the input image (315). In one embodiment, the detector (211) can identify designated portions (e.g., faces) of the objects (325, 326) in the input image (315). In one embodiment, the detector (211) can identify bounding boxes (345, 346) for designated portions (e.g., faces) of the objects (325, 326) in the input image (315).

[0119] In one embodiment, the detector (211) may set a masking area (355) related to the input image (315). In one embodiment, the detector (211) may set a masking area (355) outside the input image (315) based on the input. In one embodiment, the detector (211) may set the masking area (355) to enlarge the size of the input image (315). For example, the masking area (355) may be an area extended in the direction of a selected side among one or more sides (361, 362, 363, 364) of the input image (315).

[0120] In one embodiment, the condition judgment unit (213) may determine whether to perform filling processing for each of the input images (311, 313, 315) based on the bounding boxes (331, 332, 333, 334, 335, 336, 343, 344, 345, 346).

[0121] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process for each of the input images (311, 313, 315) based on a first number of bounding boxes (331, 332, 333, 334, 335, 336) for one or more identified objects (321, 322, 323, 324, 325, 326) and a second number of bounding boxes (343, 344, 345, 346) for a designated portion (e.g., head) of each of the one or more identified objects (321, 322, 323, 324, 325, 326). In one embodiment, the condition judgment unit (213) may determine to perform filling processing on each of the input images (311, 313, 315) when the second number of bounding boxes (343, 344, 345, 346) of the face is greater than or equal to the first number of bounding boxes (331, 332, 333, 334, 335, 336) of the body (321, 322, 323, 324, 325, 326). In one embodiment, the condition judgment unit (213) may determine not to perform filling processing on each of the input images (311, 313, 315) when the second number of bounding boxes (343, 344, 345, 346) of the face is less than the first number of bounding boxes (331, 332, 333, 334, 335, 336) of the body (321, 322, 323, 324, 325, 326).

[0122] Referring to FIG. 3a, in one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (311) because the second number (i.e., 0) of the input image (311) is less than the first number (i.e., 2). Referring to FIG. 3b, in one embodiment, the condition judgment unit (213) may determine to perform the filling process on the input image (313) because the second number (i.e., 2) of the input image (313) is greater than or equal to the first number (i.e., 2). Referring to FIG. 3c, in one embodiment, the condition judgment unit (213) may determine to perform the filling process on the input image (315) because the second number (i.e., 2) of the input image (315) is greater than or equal to the first number (i.e., 2).

[0123] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process for each of the input images (311, 313, 315) based on the ratio of the bounding box (344, 345, 346) to a designated portion (e.g., head) of each of the identified one or more objects (321, 322, 323, 324, 325, 326). In one embodiment, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the area of ​​the bounding box located in the input image (311, 313, 315). For example, the ratio of the bounding box (343) may be 60%. For example, the ratio of the bounding box (344) may be 30%. For example, the ratio of the bounding box (345) may be 90%. For example, the ratio of the bounding box (346) may be 70%.

[0124] Referring to FIG. 3A, in one embodiment, the condition judgment unit (213) may determine not to perform filling processing on the input image (311) because there is no bounding box for a specified portion (e.g., head) of the input image (311). Referring to FIG. 3B, in one embodiment, the condition judgment unit (213) may determine not to perform filling processing on the input image (315) because the ratio of the bounding box (344) of the input image (315) (i.e., 60%) is greater than or equal to a first reference ratio (e.g., 50%), but the ratio of the bounding box (344) (i.e., 30%) is less than the first reference ratio (e.g., 50%). Referring to FIG. 3c, in one embodiment, the condition judgment unit (213) may determine to perform filling processing on the input image (315) because the ratios of the bounding boxes (345, 346) of the input image (315) (i.e., 90%, 70%) are greater than or equal to the first reference ratio (e.g., 50%).

[0125] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (201) based on a comparison result between the first number and the second number and a ratio of bounding boxes for a designated portion. In one embodiment, the condition judgment unit (213) may determine not to perform a filling process on each of the images (311, 313, 315) when the second number of bounding boxes (343, 344, 345, 346) of the face is less than the first number of bounding boxes (331, 332, 333, 334, 335, 336) of the body (321, 322, 323, 324, 325, 326). In one embodiment, the condition judgment unit (213) may determine not to perform filling processing on each of the images (311, 313, 315) when the ratio of the bounding box (343, 344, 345, 346) of the face is less than a specified first ratio. In one embodiment, the condition judgment unit (213) may determine to perform filling processing on each of the images (311, 313, 315) when the second number of bounding boxes (343, 344, 345, 346) of the face is greater than or equal to the first number of bounding boxes (331, 332, 333, 334, 335, 336) of the body (321, 322, 323, 324, 325, 326) and the ratio of the bounding boxes (343, 344, 345, 346) of the face is greater than or equal to the specified first ratio.

[0126] Referring to FIG. 3A, in one embodiment, the condition judgment unit (213) may determine not to perform a filling process on the input image (311) because the second number (i.e., 0) of the input image (311) is less than the first number (i.e., 2), and there is no bounding box for a specified portion (e.g., head) of the input image (311). Referring to FIG. 3B, in one embodiment, the condition judgment unit (213) may determine not to perform a filling process on the input image (313) because the ratio (i.e., 60%) of the bounding box (343) of the input image (315) is greater than or equal to the first reference ratio (e.g., 50%), and the second number (i.e., 2) of the input image (313) is greater than or equal to the first number (i.e., 2), but the ratio (i.e., 30%) of the bounding box (344) is less than the first reference ratio (e.g., 50%). Referring to FIG. 3c, in one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (315) because the second number (i.e., 2) of the input image (315) is greater than or equal to the first number (i.e., 2), and the ratios (i.e., 90%, 70%) of the bounding boxes (345, 346) of the input image (315) are greater than or equal to the first reference ratio (e.g., 50%).

[0127] Figure 3d is a drawing illustrating an example of a landmark. Figure 3e is a drawing illustrating an example of a landmark.

[0128] FIG. 3d and FIG. 3e can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0129] Referring to FIG. 3D, the image (301) may represent a human face. In one embodiment, the electronic device (101) may identify one or more landmarks (303, 304, 305, 306, 307, 308) included in the human face. In one embodiment, the electronic device (101) may generate a bounding box (302) based on one or more landmarks (303, 304, 305, 306, 307, 308) included in the human face. In one embodiment, the electronic device (101) may generate a bounding box (302) that surrounds one or more landmarks (303, 304, 305, 306, 307, 308) included in the human face. In one embodiment, the electronic device (101) can identify the proportion of a human face included in a bounding box based on the number of identified landmarks among one or more landmarks (303, 304, 305, 306, 307, 308) included in a human face.

[0130] In one embodiment, the electronic device (101) can identify the degree of rotation of the human face based on landmarks identified among one or more landmarks (303, 304, 305, 306, 307, 308) included in the human face. For example, if the left eye (304), the left mouth, the left ear (306), the base of the nose, and the left cheek are identified, the electronic device (101) can identify that the human face is rotated to the right by about 36 degrees. For example, if the left eye (304), the right eye (305), the left mouth, a portion of the left ear (306), the base of the nose, and the left cheek are identified, the electronic device (101) can identify that the human face is rotated to the right by about 12 to 36 degrees.

[0131] In Figure 3D, a human face is shown as an example of a designated part of an object, but this is merely an example. Depending on the embodiment, the object may be an animal or an object other than a human. Furthermore, the designated part of the object may be a head, a body part, or a part of an object other than a face. Furthermore, depending on the object, landmarks may be set for the designated part.

[0132] In Figure 3D, a person is shown as an example of an object, but this is merely an example. Depending on the embodiment, the object may be an animal or an object other than a person. Additionally, depending on the object, landmarks may be set for the object.

[0133] The image (360) of FIG. 3E may represent locations of landmarks. In one embodiment, the electronic device (101) may identify one or more landmarks included in the person. In one embodiment, the electronic device (101) may generate a bounding box based on one or more landmarks included in the person. In one embodiment, the electronic device (101) may generate a bounding box enclosing one or more landmarks included in the person. In one embodiment, the electronic device (101) may identify the proportion of the person included in the bounding box based on the number of identified landmarks among one or more landmarks included in the person. For example, in FIG. 3E, only landmarks for the right arm among the landmarks of the person may be identified. Accordingly, a bounding box (365) enclosing the landmarks for the right arm may be generated. In one embodiment, the electronic device (101) can identify the proportion of a person included in the bounding box (365) based on the proportion (e.g., number proportion) of landmarks for the identified right arm among the person's landmarks.

[0134] In one embodiment, the electronic device (101) can identify the degree of rotation of the person based on landmarks identified among one or more landmarks included in the person.

[0135] Figure 3f is a diagram illustrating an example of an input image.

[0136] FIG. 3f can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0137] Referring to FIG. 3F, the detector (211) can identify objects (327, 328) in the input image (317). In one embodiment, the detector (211) can identify bounding boxes (337, 338) for the objects (327, 328) in the input image (317). In one embodiment, the detector (211) can identify designated portions for the objects (327, 328) in the input image (317).

[0138] In one embodiment, the detector (211) may set a masking area (357) related to the input image (317). In one embodiment, the detector (211) may set the masking area (357) outside the input image (317) based on the input. In one embodiment, the detector (211) may set the masking area (357) to enlarge the size of the input image (317). For example, the masking area (357) may be an area extended in the direction of a selected side among one or more sides (361, 362, 363, 364) of the input image (317). For example, the masking area (357) may be an area above the upper side (361) of the input image (317). For example, the masking area (357) may be an area above the upper edge (361) and an area below the lower edge (363) of the input image (317). For example, the masking area (357) may be an area above the upper edge (361) and an area to the left of the left edge (364) of the input image (317). For example, the masking area (357) may be an area outside each of the four edges (361, 362, 363, 364) of the input image (317).

[0139] In one embodiment, the condition judgment unit (213) may determine whether to perform filling processing on the input image (317) based on the bounding boxes (337, 338, 347).

[0140] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (317) based on a first number of bounding boxes (337, 338, 347) for one or more identified objects (327, 328) and a second number of bounding boxes (347) for a designated portion (e.g., head) of each of the one or more identified objects (337, 338, 347). In one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (317) if the second number of bounding boxes for the face is less than the first number of bounding boxes for the body. In one embodiment, the condition judgment unit (213) may determine not to perform the filling process on the input image (317) if the ratio of bounding boxes for at least one of the objects is less than the designated first ratio. In one embodiment, the condition judgment unit (213) may determine to perform filling processing on the input image (317) when the second number of bounding boxes of the face is greater than or equal to the first number of bounding boxes of the body and the ratio of the bounding boxes of the face is greater than or equal to the specified first ratio.

[0141] In one embodiment, the condition judgment unit (213) may identify the second number as 1 based on the number of bounding boxes of the face of the input image (317). In one embodiment, the condition judgment unit (213) may identify the first number as 1 based on the number of bounding boxes of the body of the input image (317). In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (317) because the second number (i.e., 1) of the input image (317) is greater than or equal to the first number (i.e., 1).

[0142] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (317) based on the ratio of the bounding boxes (337, 338) for each of the identified one or more objects (327, 328). In one embodiment, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the area of ​​the bounding box located in the input image (317). For example, the ratio of the bounding box may mean the ratio between the size of the bounding box located in the masking area and the size of the area of ​​the bounding box located in the input image (317). For example, the ratio of the bounding box may mean the ratio of landmarks included in the bounding box among the designated landmarks of the object or the designated part of the object (or included in the input image (317)). For example, since the bounding box (337) includes only the upper part of the human body (327) of the legs, the ratio of the bounding box (337) may be 60%. For example, the bounding box (337) may include only 60% of all landmarks of the human body (327) (e.g., landmarks of the upper body). For example, since the bounding box (338) includes only the front part of the car, the ratio of the bounding box (338) may be 30%.

[0143] In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (317) because the ratio of the bounding box (337) to the object (327) of the input image (317) (i.e., 60%) is greater than or equal to a first reference ratio for the object (e.g., 50%). In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (317) because the ratio of the bounding box (338) to the object (328) of the input image (317) (i.e., 30%) is greater than or equal to a first reference ratio for the object (e.g., 20%).

[0144] In one embodiment, the condition judgment unit (213) may determine whether to perform a filling process on the input image (317) based on the ratio of the bounding box (347) for a designated portion (e.g., head) of each of the identified one or more objects (327, 328). In one embodiment, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the area of ​​the bounding box located in the input image (317). For example, the ratio of the bounding box (347) may be 100%. In one embodiment, the bounding box (347) may include all landmarks of the head.

[0145] In one embodiment, the condition judgment unit (213) may determine to perform filling processing on the input image (315) because the ratio (i.e., 100%) of the bounding box (347) for a specified portion (e.g., head) of the input image (317) is greater than or equal to a first reference ratio (e.g., 50%).

[0146] In one embodiment, the condition judgment unit (213) may determine whether to perform filling processing on the input image (207) based on the comparison result between the first number and the second number and the ratio of the bounding box for the specified portion.

[0147] In one embodiment, the condition judgment unit (213) may identify the second number as 1 based on the number of bounding boxes of the face of the input image (317). In one embodiment, the condition judgment unit (213) may identify the first number as 1 based on the number of bounding boxes of the body of the input image (317). In one embodiment, the condition judgment unit (213) may determine to perform a filling process on the input image (317) because the second number (i.e., 1) of the input image (317) is greater than or equal to the first number (i.e., 1), the ratio (i.e., 100%) of the bounding box (347) for a specified portion (e.g., head) of the input image (317) is greater than or equal to the first reference ratio (e.g., 50%), and the ratio (i.e., 60%, 30%) of the bounding boxes (337, 338) of the objects (327, 328) of the input image (317) is greater than or equal to the first reference ratio (e.g., 50%, 20%).

[0148] In FIG. 3F, two objects (327, 328) and one designated portion (347) are described, but this is only an example. According to an embodiment, the electronic device (101) may determine whether to perform a filling process on the input image (207) based on the number of bounding boxes and / or the ratio of bounding boxes for all objects included in the image (317).

[0149] Figure 4a is a diagram illustrating an example of a result image. Figure 4b is a diagram illustrating an example of a result image. Figure 4c is a diagram illustrating an example of a result image.

[0150] FIG. 4a, FIG. 4b, and FIG. 4c can be described with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0151] Referring to FIG. 4A, the detector (231) can identify objects (421, 422) in the result image (411). In one embodiment, the detector (231) can identify designated portions (e.g., faces) of the objects (421, 422) in the result image (411). In one embodiment, the detector (231) can identify bounding boxes (441, 442) for designated portions (e.g., faces) of the objects (421, 422) in the result image (411).

[0152] Referring to FIG. 4b, the detector (231) can identify objects (423, 424) in the result image (413). In one embodiment, the detector (231) can identify a designated portion (e.g., a face) of the objects (423, 424) in the result image (413). In one embodiment, the detector (231) can identify bounding boxes (443, 444) for the designated portion (e.g., a face) of the objects (423, 424) in the result image (413). In FIG. 4a, the upper part of the nose of the face included in the bounding box (441) is newly generated, whereas in FIG. 4b, the upper part of the eyes of the face included in the bounding box (443) may be newly generated.

[0153] Referring to FIG. 4c, the detector (231) can identify objects (425, 426) in the result image (415). In one embodiment, the detector (231) can identify designated portions (e.g., faces) of the objects (425, 426) in the result image (415). In one embodiment, the detector (231) can identify bounding boxes (445, 446) for designated portions (e.g., faces) of the objects (425, 426) in the result image (415).

[0154] In one embodiment, the condition judgment unit (233) may determine whether to output each of the result images (411, 413, 415) based on the bounding boxes (441, 442, 443, 444, 445, 446).

[0155] In one embodiment, the condition judgment unit (233) may determine whether to output each of the result images (411, 413, 415) based on the ratio of the bounding box (441, 442, 443, 444, 445, 446) for a designated portion (e.g., head) of each of the identified one or more objects (421, 422, 423, 424, 425, 426). In one embodiment, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the bounding box located in the partial image (451, 453, 455) generated by the masking area located in the result image (411, 413, 415). For example, the ratio of bounding boxes may refer to the ratio of bounding boxes located in the masking region in the result images (411, 413, 415). In one embodiment, the masking region may be an area excluding the input images (401, 403, 405) in the result images (411, 413, 415). For example, the bounding box (441) may include a head whose lower part is present in the input image (401) and whose upper part is newly created. Accordingly, the ratio of landmarks included in the newly created region among all landmarks of the head in the bounding box (441) may be 60%. For example, the bounding box (443) may include a head whose lower part is present in the input image (403) and whose upper part is newly created. Accordingly, the ratio of landmarks included in the newly created region among all landmarks of the head in the bounding box (443) may be 50%. For example, the bounding box (445) may include a head whose lower part exists in the input image (405) and whose upper part includes a newly generated head. Accordingly, the proportion of the area of ​​landmarks included in the newly generated area among all landmarks of the head in the bounding box (445) may be 10%.For example, the ratios of each of the bounding boxes (441, 442, 443, 444, 445, 446) can be 60, 80, 50, 60, 10, 50%.

[0156] In one embodiment, the condition judgment unit (233) may determine not to output the result image (411) because the ratio (i.e., 100%, 80%) of the bounding boxes (441, 442) for a specified portion (e.g., head) of the result image (411) exceeds a second reference ratio (e.g., 50%). In one embodiment, the condition judgment unit (233) may determine not to output the result image (415) because the ratio (i.e., 80%) of the bounding box (443) among the bounding boxes (443, 444) of the result image (415) exceeds a second reference ratio (e.g., 50%). In one embodiment, the condition judgment unit (233) may determine to output the result image (415) because the ratios (i.e., 50%, 30%) of the bounding boxes (445, 446) of the result image (415) are less than or equal to the second reference ratio (e.g., 50%).

[0157] Figure 4d is a drawing showing an example of the result image.

[0158] FIG. 4d can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0159] Referring to FIG. 4d, the detector (231) can identify objects (427, 428) in the result image (417). In one embodiment, the detector (231) can identify bounding boxes (437, 438) for the objects (427, 428) in the result image (417).

[0160] In one embodiment, the detector (231) can identify a designated portion (e.g., a face) of objects (427, 428) in the result image (417). In one embodiment, the detector (231) can identify a bounding box (447) for a designated portion (e.g., a face) of objects (427, 428) in the result image (417).

[0161] In one embodiment, the condition judgment unit (233) can determine whether to output the result image (417) based on the bounding boxes (437, 438, 447).

[0162] In one embodiment, the condition judgment unit (233) may determine whether to output the result image (417) based on the ratio of the bounding boxes (437, 438) for each of the identified one or more objects (427, 428). In one embodiment, the ratio of the bounding box may mean the ratio between the overall size of the bounding box and the size of the bounding box located in the partial image (457) generated by the masking area located in the result image (417). For example, the ratio of the bounding box may mean the ratio of the bounding box located in the masking area in the result image (417). In one embodiment, the masking area may be an area excluding the input image (407) in the result image (417). For example, the bounding box (437) may include a body in which the upper part of the leg exists in the input image (407) and the part between the leg and the knee is newly generated. Accordingly, the ratio of landmarks included in the newly generated area among all landmarks of the body in the bounding box (437) may be 20%. For example, the bounding box (438) may include a car in which the front part of the car exists in the input image (407) and the middle part of the car is newly generated. Accordingly, the ratio of landmarks included in the newly generated area among all landmarks of the car in the bounding box (438) may be 20%. For example, the bounding box (447) may include a car in which the entire head exists in the input image (407) and no newly generated part exists. Accordingly, the ratio of landmarks included in the newly generated area among all landmarks of the head in the bounding box (447) may be 0%. For example, the ratios of the bounding boxes (437, 438, 447) may be 20, 20, and 0%, respectively.

[0163] In one embodiment, the condition judgment unit (233) may determine to output the result image (417) because the ratio (i.e., 20%, 70%) of the bounding boxes (437, 438) for each of the identified one or more objects (427, 428) of the result image (417) is less than or equal to the second reference ratio (e.g., 50%, 80%).

[0164] In one embodiment, the condition judgment unit (233) may determine whether to output the result image (417) based on the ratio of the bounding box (447) to a designated portion (e.g., head) of each of one or more identified objects (427, 428).

[0165] In one embodiment, the condition judgment unit (233) may determine to output the result image (417) because the ratio (i.e., 0%) of the bounding box (447) for the specified portion (e.g., head) of the result image (417) is less than or equal to the second reference ratio (e.g., 50%).

[0166] In FIG. 4d, two objects (427, 428) and one designated portion (447) are described, but this is only an example. According to an embodiment, the electronic device (101) can determine whether to output the result image (417) based on the ratio of the bounding box for all objects included in the image (417).

[0167] Figure 5 is a block diagram of components of an electronic device (101) and a server (108) for generating a result image.

[0168] FIG. 5 may illustrate an embodiment in which some functions of the electronic device (101) are implemented by the server (108), compared to FIG. 2A. For example, the filling processing of the electronic device (101) illustrated in FIG. 2A may be performed by the server (108). However, this is not limited thereto. For example, at least one of the preprocessing operation, the filling processing, or the postprocessing operation of the electronic device (101) illustrated in FIG. 2A may be performed by (108).

[0169] Below, the description of the configurations illustrated in FIG. 5 and those described through FIG. 2a may be simplified.

[0170] Referring to FIG. 5, the electronic device (101) may include a preprocessor (210), a postprocessor (230), and a communication circuit (510). In one embodiment, the preprocessor (210) may correspond to the preprocessor (210) of FIG. 2a, FIG. 2b, or FIG. 2c. In one embodiment, the postprocessor (230) may correspond to the postprocessor (230) of FIG. 2a, FIG. 2b, or FIG. 2c. In one embodiment, the communication circuit (510) may correspond to the communication module (190) of FIG. 1.

[0171] In one embodiment, the preprocessor (210) may determine whether to perform filling processing on the input image (201). In one embodiment, the preprocessor (210) may set a masking area associated with the input image (201).

[0172] In one embodiment, the preprocessor (210) may transmit data about the input image (201) and the masking area to the server (108). In one embodiment, the server (108) may correspond to the server (108) of FIG. 1.

[0173] In one embodiment, the server (108) may include an image filler (220) and a communication circuit (520). In one embodiment, the image filler (220) may correspond to the image filler (220) of FIG. 2A. In one embodiment, the communication circuit (520) may correspond to the communication module (190) of FIG. 1.

[0174] In one embodiment, the image filler (220) may include a generative artificial intelligence (AI) model. In one embodiment, the image filler (220) may generate a result image (205) for the input image (201) based on the generative AI model. In one embodiment, the result image (205) may include a partial image (or filled image) generated by a masking area. In one embodiment, the result image (205) may include the partial image (or filled image) and the input image (201).

[0175] In one embodiment, the server (108) may transmit the result image (205) to the communication circuit (510) of the electronic device (101) via the communication circuit (520). In one embodiment, the electronic device (101) may determine whether to output the result image (205) via the post-processor (230).

[0176] FIG. 6a is a diagram illustrating an example of an input image and a masking area. FIG. 6b is a diagram illustrating an example of an image for a masking area transmitted from a server (108) to an electronic device (101). FIG. 6c is a diagram illustrating an example of a result image synthesized in an electronic device (101).

[0177] FIGS. 6A, 6B, and 6C may be described with reference to FIGS. 1 and 5. FIGS. 6A and 6C may relate to operations performed in an electronic device (101), and FIG. 6B may relate to operations performed in a server (108).

[0178] Referring to FIG. 6A, the electronic device (101) can identify a masking area (651) for an input image (611). In one embodiment, the masking area (651) may be an area to be filled in for the input image (611).

[0179] In one embodiment, the electronic device (101) can identify an area (612) corresponding to the input image (611) in the entire area (652) of the result image to be generated. Here, the result image to be generated may be an image generated by filling in a masking area (651) for the input image (611).

[0180] In one embodiment, the electronic device (101) may transmit data about an input image (611), a masking area (651), and an area (612) to the server (108). The data about the area (612) may include location information of the area (612) among the entire area (652) of the resulting image to be generated.

[0181] Referring to FIG. 6B, the server (108) may generate a result image (623). In one embodiment, the server (108) may generate the result image (623) based on the input image (611) and the masking area (651). In one embodiment, the server (108) may generate the result image (623) using an image filler (220 of FIG. 5). In one embodiment, the server (108) may generate the result image (623) for the input image (611) based on a generative AI model of the image filler (220 of FIG. 5). In one embodiment, the result image (623) may include a partial image (or filled image) generated by the masking area (651). In one embodiment, the result image (623) may include a partial image (or filled image) for the masking region (651) and the input image (611). In one embodiment, the server (108) may identify an object and / or a designated portion of the object based on one or more landmarks of the object identified in the input image (611). In one embodiment, the server (108) may identify an area (or bounding box) that includes the designated portion of the object. In one embodiment, the server (108) may determine whether to perform a filling process on the input image (201) based on the bounding box. In one embodiment, the server (108) may determine whether to perform a filling process on the input image (611) based on the number of bounding boxes and / or a ratio of bounding boxes for one or more identified objects. However, the present invention is not limited thereto. In one embodiment, the server (108) may determine whether to perform a filling process on the input image (611) based on whether an anomaly is detected.

[0182] In one embodiment, the server (108) may generate an image (624) by removing (or excluding) a portion (634) corresponding to the area (612) from the result image (623). In one embodiment, the server (108) may transmit the image (624) to the electronic device (101).

[0183] Referring to FIG. 6c, the electronic device (101) can synthesize the input image (611) onto the image (624). In one embodiment, the electronic device (101) can synthesize the input image (611) onto the remaining portion (634) of the result image (623) excluding the portion generated by the masking area (651).

[0184] In one embodiment, the electronic device (101) may adjust the image (624) so ​​that the transparency decreases (or the opacity increases) as it gets farther away from the outer portion of the remaining portion (634), and may adjust the input image (611) so that the transparency decreases (or the opacity increases) as it gets farther away from the outer portion of the remaining portion (634). Here, in the image (624), the direction away from the outer portion of the remaining portion (634) may be the outward direction of the remaining portion (634). In the input image (611), the direction away from the outer portion of the remaining portion (634) may be the inward direction of the remaining portion (634). In one embodiment, by the electronic device (101) adjusting the transparency of the image (624) and / or the input image (611), the image (624) and the input image (611) may be synthesized more smoothly.

[0185] In one embodiment, the electronic device (101) can synthesize an input image (611) with reduced transparency (or increased opacity) onto an image (624) with reduced transparency (or increased opacity).

[0186] As described above, the electronic device (101) can use less data than when directly receiving the result image (623) by receiving an image (624) from the server (108) in which a portion (634) corresponding to the size of the input image (611) is excluded from the result image (623).

[0187] Figure 7a is a diagram illustrating an example of an input for an object included in an input image. Figure 7b is a diagram illustrating an example of a masking area and a resulting image determined based on the input. Figure 7c is a diagram illustrating an example of a masking area and a resulting image determined based on the input.

[0188] FIGS. 7a, 7b, and 7c can be described with reference to FIGS. 1, 2a, 2b, and 2c.

[0189] In one embodiment, the electronic device (101) can display an input image (711) through a display module (160).

[0190] In one embodiment, the electronic device (101) may obtain an input (740) for an input image (711). In one embodiment, the electronic device (101) may obtain an input (740) for an object (721) included in the input image (711). In one embodiment, the electronic device (101) may obtain an input (740) for changing the position and / or size of the object (721) included in the input image (711).

[0191] In one embodiment, the electronic device (101) can separate (or segment) a selected object (721) from an input image (711) based on an input (740).

[0192] In one embodiment, the electronic device (101) may move an object (725) separated from an object (721) on an input image (711) based on an input (740). In one embodiment, the electronic device (101) may display an object (725) moved (and / or reduced) from an object (721) on an input image (711) based on the input (740).

[0193] In one embodiment, the electronic device (101) may identify a masking area in which a filling process is to be performed, based on the input (740), in relation to the input image (711). The masking area may include an area of ​​an object (721) selected in the input image (711). The masking area may include an area (735) adjacent to an area of ​​an object (725) moved in the input image (711). In one embodiment, the adjacent area (735) may include an area extending in the direction of an edge from an area to which the selected object (721) has moved, when the selected object (721) is located on the edge (e.g., edge) of the input image (711). In one embodiment, the size of the adjacent area (735) may be set based on the type of the selected object (721). In one embodiment, the size of the neighboring region (735) may include an area of ​​an unidentified portion of an object identified based on the type of the selected object (721). For example, if the type of the selected object (721) is a person and legs are not identifiable from the person, the neighboring region (735) may include the area of ​​the legs. In one embodiment, the unidentified portion may be identified by one or more landmarks set for the type of the selected object (721), but is not limited thereto. For example, the neighboring region (735) may be identified by additional user input.

[0194] In one embodiment, the electronic device (101) may perform filling processing on an identified masking area (e.g., an area of ​​a selected object (721)). In one embodiment, when it is determined to perform filling processing on an input image (711), the electronic device (101) may perform filling processing on an identified masking area (e.g., an area of ​​a selected object (721)). For example, referring to FIG. 7B, the electronic device (101) may perform filling processing on a masking area (731) associated with the selected object (721). By performing filling processing on the masking area (731), the electronic device (101) may generate a result image (713). Referring to FIG. 7b, the result image (713) may include an object (725) moved by the input (740) and a filled portion (751) for a masking area (731). Referring to FIG. 7b, since no filling process is performed on an area (735) adjacent to the moved object (725), a portion (755) below the moved object (725) of the result image (713) may not be filled with portions (e.g., legs) that are not identified in the moved object (725).

[0195] In one embodiment, the electronic device (101) can perform filling processing on a masking area associated with a selected object (721) and a neighboring area (735). In one embodiment, when it is determined to perform filling processing on an input image (711), the electronic device (101) can perform filling processing on a masking area (731) associated with a selected object (721) and a neighboring area (735). For example, referring to FIG. 7C, the electronic device (101) can perform filling processing on a masking area (731) associated with a selected object (721) and a neighboring area (735). By performing filling processing on the area (731) and the neighboring area (735), the electronic device (101) can generate a result image (715). Referring to FIG. 7c, the result image (715) may include an object (725) moved by the input (740) and a filled portion (761) for an area (731). Referring to FIG. 7c, by performing a filling process on an area (735) adjacent to the moved object (725), an unidentified portion (e.g., a leg) of the moved object (725) in the lower portion (765) of the result image (715) may be filled.

[0196] Figure 8a is a diagram illustrating an example of an input for an object included in an input image. Figure 8b is a diagram illustrating an example of a masking area and a resulting image determined based on the input. Figure 8c is a diagram illustrating an example of a masking area and a resulting image determined based on the input. Figure 8d is a diagram illustrating an example of a masking area and a resulting image determined based on the input.

[0197] FIGS. 8a, 8b, 8c, and 8d can be described with reference to FIGS. 1, 2a, 2b, and 2c.

[0198] In one embodiment, the electronic device (101) can display an input image (811) through a display module (160).

[0199] In one embodiment, the electronic device (101) may obtain an input (840) for an input image (811). In one embodiment, the electronic device (101) may obtain an input (840) for an object (821) included in the input image (811). In one embodiment, the electronic device (101) may obtain an input (840) for changing the position and / or size of the object (821) included in the input image (811).

[0200] In one embodiment, the electronic device (101) may display an object (825) moved (and / or reduced) from an object (821) in an input image (811) based on an input (840).

[0201] In one embodiment, the electronic device (101) may identify a masking area in which a filling process is to be performed, based on the input (840), in relation to the input image (811). The masking area may include an area of ​​an object (821) selected in the input image (811). The masking area may be set based on an area of ​​the object (821) selected in the input image (811) and / or a moved (and / or reduced) object (825). In one embodiment, when an object (825) at least partially overlaps an area of ​​the selected object (821), the electronic device (101) may identify an area remaining in the area of ​​the selected object (821) excluding an area overlapped by the object (825) as a masking area.

[0202] According to an embodiment, when an object (825) overlaps at least partly with an area of ​​a selected object (821), the electronic device (101) may identify the area of ​​the selected object (821) as a masking area, or may identify the remaining area of ​​the area of ​​the selected object (821) excluding the area overlapped by the object (825) as a masking area. Depending on whether the area overlapped by the object (825) is excluded, the result of performing the filling process may vary.

[0203] FIG. 8B may illustrate result images (813, 814) generated according to a situation in which the electronic device (101) identifies the area (831) of the selected object (821) as a masking area when the object (825) at least partially overlaps the area of ​​the selected object (821). FIG. 8C may illustrate result images (816, 817) generated according to a situation in which the electronic device (101) identifies the remaining area of ​​the area (831) of the selected object (821) excluding the area (835) overlapped by the object (825) when the object (825) at least partially overlaps the area of ​​the selected object (821) as a masking area. In one embodiment, by excluding the overlapping area (835), the original area can be maintained in the generated result images (816, 817).

[0204] Referring to FIG. 8b, in the result images (813, 814), leaves (851, 852) related to the landscape, not people, can be filled in as the filling process is performed on the masked area without information about the object (825). Referring to FIG. 8c, in the result images (816, 817), people (853, 854) can be filled in as the filling process is performed on the masked area based on information about the object (825).

[0205] Referring to FIG. 8C, as the filling process is performed on the masking area, the resulting images (816, 817) may have a depiction of the object (829) that may be somewhat different from the object (825) of the input image (811). According to an embodiment, the electronic device (101) may synthesize the input image (811) into the resulting images (816, 817) to maintain the appearance of the object (825).

[0206] Referring to FIG. 8D, the electronic device (101) can synthesize the input image (811) onto the result image (816, 817). In one embodiment, the electronic device (101) can synthesize the object (825) of the input image (811) onto the area (835) overlapping the result image (816, 817).

[0207] In one embodiment, the electronic device (101) may adjust the result images (816, 817) so that the transparency decreases (or the opacity increases) as it moves away from the outer portion of the overlapping area (835), and may adjust the input image (811) so that the transparency decreases (or the opacity increases) as it moves away from the outer portion of the overlapping area (835). Here, in the result images (816, 817), the direction away from the outer portion of the overlapping area (835) may be an outward direction of the overlapping area (835). In the input image (811), the direction away from the outer portion of the overlapping area (835) may be an inward direction of the overlapping area (835).

[0208] In one embodiment, the electronic device (101) can synthesize an input image (811) with reduced transparency (or increased opacity) onto a result image (816, 817) with reduced transparency (or increased opacity). In one embodiment, the electronic device (101) can synthesize an area of ​​an object (825) of an input image (811) with reduced transparency (or increased opacity) onto a result image (816, 817) with reduced transparency (or increased opacity).

[0209] As described above, the electronic device (101) can prevent an object included in an input image (811) from disappearing from a result image by setting a masking area based on the movement of the object (821). In addition, as described above, the electronic device (101) can prevent the depiction of a moved object (825) from changing in the result image by inserting the moved object (825) within the masking area.

[0210] FIG. 9 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0211] FIG. 9 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0212] Referring to FIG. 9, in operation 910, the electronic device (101) may identify a filling processing condition for an input image. In one embodiment, the filling processing condition may be related to a state of one or more objects included in the input image. For example, the state of the object may include a size and / or a ratio of the object in the input image. In one embodiment, the filling processing condition may be based on an outlier of the input image. In one embodiment, the electronic device (101) may include one or more detectors for identifying different types of objects. In one embodiment, the electronic device (101) may identify an object based on a detector for identifying a type of object other than a person or a body part of a person, based on whether the type of the identified object is different from a person or a body part of a person (e.g., a face).

[0213] In one embodiment, the electronic device (101) may identify a filling processing condition for the input image based on the number of objects identified in the input image. For example, the electronic device (101) may identify a filling processing condition for the input image based on comparing a first number of objects identified in the input image with a second number of designated parts (e.g., a head) of the objects. For example, if the object is a human, the designated parts of the object may be body parts of the human (e.g., a face, eyes, a nose, a mouth, ears, arms, hands). For example, if the object is an animal, the designated parts of the object may be body parts of the animal (e.g., a head, eyes, a nose, a mouth, ears, legs). For example, if the object is an object, the designated parts of the object may be parts of a component of the object (e.g., a hood, wheels, windows, doors in the case of a car). However, the present invention is not limited thereto.

[0214] In one embodiment, the electronic device (101) may identify a filling processing condition for the input image based on a ratio of objects identified in the input image. For example, the ratio of objects may refer to a ratio of objects identified in the input image among all objects. For example, the ratio of objects may refer to a ratio of landmarks identified in the input image among designated landmarks of the object. For example, if the object is a person, designated landmarks of the person may be a nose, an eye, an ear, a mouth, a shoulder, an elbow, a wrist, a finger, a hip, a knee, an ankle, a heel, and / or a foot.

[0215] In one embodiment, the electronic device (101) may identify a filling processing condition for the input image based on a ratio of a designated portion of an object identified in the input image. For example, the ratio of the designated portion of the object may refer to a ratio of the designated portion identified in the input image to the entire designated portion. For example, the ratio of the designated portion may refer to a ratio of landmarks identified in the input image to designated landmarks of the designated portion. For example, if the designated portion of the object is a human face, the designated landmarks of the face may be a nose, eyes, ears, and / or a mouth. However, the present invention is not limited thereto.

[0216] In operation 920, the electronic device (101) can determine whether the filling processing condition is satisfied.

[0217] In one embodiment, the electronic device (101) may determine to perform a filling process on the input image if the second number of designated portions of objects identified in the input image is greater than or equal to the first number of objects identified in the input image. In one embodiment, the electronic device (101) may determine not to perform a filling process on the input image if the second number of designated portions of objects identified in the input image is less than the first number of objects identified in the input image.

[0218] In one embodiment, the electronic device (101) may determine to perform a filling process on the input image if the ratio of all objects is greater than or equal to a specified first ratio. In one embodiment, the electronic device (101) may determine not to perform a filling process on the input image if the ratio of at least one object among the objects is less than the specified first ratio. In one embodiment, the first ratio may be set differently depending on the object. In one embodiment, the first ratio for a person, an animal, or an object may be different. For example, the first ratio for a person, a dog, or a car may be 80%, 40%, or 20%. In one embodiment, the first ratio may be predetermined according to the difficulty of the filling process. In one embodiment, the difficulty of the filling process may depend on whether the sameness between the image before the filling process and the image after the filling process is recognized. For example, for objects (e.g., a human face) that are perceived as having a lower identity as the proportion of newly generated areas through the filling process increases, the difficulty of the filling process can be understood to be high. In one embodiment, the difficulty of the filling process may vary depending on the type (or class) of the object to be filled. For example, a higher difficulty of the filling process may have a higher first proportion.

[0219] In one embodiment, the electronic device (101) may determine to perform a filling process on the input image if the ratio of all designated portions is greater than or equal to a designated first ratio. In one embodiment, the electronic device (101) may determine not to perform a filling process on the input image if the ratio of at least one designated portion among the designated portions is less than the designated first ratio. In one embodiment, the first ratio may be set differently depending on the designated portion of the object. In one embodiment, the first ratios for designated portions of a person, an animal, or an object may be different. For example, the first ratios for a person's face, a person's hand, or a person's shoulder may be 60%, 90%, or 20%. However, the present invention is not limited thereto. In one embodiment, the first ratio may be predetermined depending on the difficulty of the filling process. For example, if the difficulty of the filling process is high, a higher first ratio may be used.

[0220] In operation 920, in response to identifying that the filling processing condition is satisfied, the electronic device (101) may perform operation 930. In operation 920, in response to identifying that the filling processing condition is not satisfied, the electronic device (101) may perform operation 940.

[0221] In operation 930, the electronic device (101) may perform filling processing on a masking area for an input image.

[0222] In one embodiment, the electronic device (101) may set a masking area associated with an input image. In one embodiment, the electronic device (101) may set a masking area outside the input image and / or a masking area inside the input image based on an input. In one embodiment, the electronic device (101) may set a masking area associated with an object selected based on a user input for the input image. For example, the masking area may be an area of ​​an object selected in the input image. For example, the masking area may be set based on an area of ​​an object selected in the input image and / or a moved (and / or reduced) object. For example, the masking area may be an area excluding an area overlapped by an object in the area of ​​the selected object, if the object at least partially overlaps the area of ​​the object selected. For example, the masking area may include an area adjacent to an area of ​​an object moved in the input image. In one embodiment, the neighboring region may include an area extending in the direction of the edge from an area to which the selected object has moved, when the selected object is located on the edge (e.g., edge) of the input image. In one embodiment, the size of the neighboring region may be set based on the type of the selected object. In one embodiment, the size of the neighboring region may include an area of ​​an unidentified part of an object identified based on the type of the selected object. For example, when the type of the selected object is a person and legs are not identifiable from the person, the neighboring region may include an area of ​​the legs. However, the present invention is not limited thereto. In one embodiment, the electronic device (101) may set a masking region to enlarge the size of the input image. For example, the masking region may be an area extending in the direction of a selected edge among one or more edges of the input image.

[0223] In one embodiment, the electronic device (101) may generate a result image for an input image based on a generative AI (artificial intelligence) model. In one embodiment, the generative AI model may be a model that can draw a new image in a masking area based on the input image. For example, the generative AI model may draw a new image in a masking area related to the input image based on at least one prompt. Here, the prompt may include at least one keyword related to the input image. The prompt may include at least one keyword that can be extracted from the input image. Here, the keyword may be at least one object included in the input image and / or a word (or sentence) to describe the input image. In one embodiment, the result image may include a partial image (or a filled image) generated by the masking area. In one embodiment, the result image may include the partial image (or the filled image) and the input image.

[0224] In operation 940, the electronic device (101) may stop filling processing for the input image. Here, stopping filling processing for the input image may include not performing filling processing for the input image. Stopping filling processing for the input image may include refraining from filling processing for the input image.

[0225] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0226] FIG. 10 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0227] Referring to FIG. 10, in operation 1010, the electronic device (101) may identify an output condition for a result image on which a filling process has been performed. In one embodiment, the output condition may be related to a state of one or more objects included in an input image. For example, the state of an object may include a size and / or a ratio of the object within the output image. In one embodiment, the output condition may be based on an outlier of the output image.

[0228] In one embodiment, the electronic device (101) may identify a filling processing condition for the output image based on a ratio of an object identified in the output image. For example, the ratio of an object may refer to a ratio of an object identified in a masked area of ​​the output image among all objects. For example, the ratio of an object may refer to a ratio of a landmark identified in a masked area of ​​the output image among designated landmarks of the object. For example, if the object is a human, designated landmarks of the human may be a nose, an eye, an ear, a mouth, a shoulder, an elbow, a wrist, a finger, a hip, a knee, an ankle, a heel, and / or a foot.

[0229] In one embodiment, the electronic device (101) may identify a filling processing condition for the output image based on a ratio of a designated portion of an object identified in the output image. For example, the ratio of the designated portion of the object may refer to a ratio of the designated portion identified in a masked area of ​​the output image among the entire designated portion. For example, the ratio of the designated portion may refer to a ratio of landmarks identified in a masked area of ​​the output image among designated landmarks of the designated portion. For example, if the designated portion of the object is a human face, the designated landmarks of the face may be a nose, eyes, ears, and / or a mouth. However, the present invention is not limited thereto.

[0230] In operation 1020, the electronic device (101) can determine whether the output condition is satisfied.

[0231] In one embodiment, the electronic device (101) may determine to output an output image if the ratio of all objects is less than a specified second ratio. In one embodiment, the electronic device (101) may determine to stop outputting the output image if the ratio of at least one object among the objects is greater than or equal to the specified second ratio. In one embodiment, the second ratio may be set differently depending on the object. In one embodiment, the second ratio for a person, an animal, or an object may be different. For example, the second ratio for a person, a dog, or a car may be 20%, 60%, or 80%. In one embodiment, the second ratio may be predetermined depending on the difficulty of the filling process. For example, if the difficulty of the filling process is high, a higher first ratio may be used.

[0232] In one embodiment, the electronic device (101) may determine to output an output image if the ratio of all designated portions is less than a designated second ratio. In one embodiment, the electronic device (101) may determine to stop outputting the output image if the ratio of at least one designated portion among the designated portions is greater than or equal to the designated second ratio. In one embodiment, the second ratio may be set differently depending on the designated portion of the object. In one embodiment, the second ratios for designated portions of a person, an animal, or an object may be different. For example, the second ratios for a person's face, a person's hand, or a person's shoulder may be 40%, 10%, or 80%. However, the present invention is not limited thereto. In one embodiment, the second ratio may be predetermined depending on the difficulty of the filling process. For example, if the difficulty of the filling process is high, a higher second ratio may be used.

[0233] In operation 1020, in response to identifying that the output condition is satisfied, the electronic device (101) may perform operation 1030. In operation 1020, in response to identifying that the output condition is not satisfied, the electronic device (101) may perform operation 1040.

[0234] In operation 1030, the electronic device (101) may display a result image. The electronic device (101) may display the result image through a display module (160). According to an embodiment, the electronic device (101) may store the result image in the memory (130).

[0235] In operation 1040, the electronic device (101) may stop displaying the result image. In some embodiments, the electronic device (101) may delete the result image from the memory (130).

[0236] FIG. 11 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0237] FIG. 11 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0238] Referring to FIG. 11, in operation 1110, an electronic device (101) may select an object from an image. The electronic device (101) may select an object from among one or more objects included in the image based on an input. The electronic device (101) may separate (or segment) the selected object from the image. In one embodiment, the object may be a person, an animal, and / or an object.

[0239] In one embodiment, the electronic device (101) may move an object separated from an object on an input image based on an input. In one embodiment, the electronic device (101) may display an object moved (and / or reduced) from an object on an input image based on the input.

[0240] In operation 1120, the electronic device (101) may set a masking area based on the selected object. In one embodiment, the electronic device (101) may identify a masking area in an input image where filling processing is to be performed based on the input. The masking area may include an area of ​​an object selected by the input in the input image.

[0241] In operation 1130, the electronic device (101) can identify whether addition of a masking area is required by the selected object.

[0242] In one embodiment, the electronic device (101) may identify that addition of a masking region is required based on the fact that at least one landmark among the designated landmarks in the selected object is not identified. For example, if the selected object is a person and no landmark related to a leg is identified in the selected object, the electronic device (101) may identify that addition of a masking region for a leg is required.

[0243] In operation 1130, in response to identifying that the addition of a masking area is required by the selected object, the electronic device (101) may perform operation 1140. In operation 1130, in response to identifying that the addition of a masking area is not required by the selected object, the electronic device (101) may perform operation 1150.

[0244] In operation 1140, the electronic device (101) may set an additional masking area. In one embodiment, the electronic device (101) may identify an additional masking area in which filling processing is to be performed, based on the input, in relation to the input image. The additional masking area may include an area adjacent to an area of ​​an object moved in the input image. In one embodiment, the adjacent area may include an area extending in the direction of an area of ​​an area to which the selected object has moved, when the selected object is located at an edge (e.g., an edge) of the input image. In one embodiment, the size of the adjacent area may be set based on the type of the selected object. In one embodiment, the size of the adjacent area may include an area of ​​an unidentified portion of an object identified based on the type of the selected object. For example, if the type of the selected object is a person and a leg is not identifiable from the person, the neighboring area may include an area of ​​the leg. In one embodiment, the unidentified portion may be identified by one or more landmarks set for the type of the selected object. However, this is not limited to this. For example, neighboring areas can be identified by additional user input.

[0245] In operation 1150, the electronic device (101) may perform filling processing on a masking area for an input image.

[0246] In one embodiment, the electronic device (101) may generate a result image for an input image based on a generative AI (artificial intelligence) model. In one embodiment, the electronic device (101) may fill an image in a masking area based on a selected object based on the generative AI model. In one embodiment, the electronic device (101) may fill an image in an additional masking area adjacent to an object moved from the selected object based on the generative AI model.

[0247] FIG. 12 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0248] FIG. 12 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0249] Referring to FIG. 12, in operation 1210, the electronic device (101) may select an object from an image. The electronic device (101) may select an object from among one or more objects included in the image based on an input. The electronic device (101) may separate (or segment) the selected object from the image. In one embodiment, the object may be a person, an animal, and / or an object.

[0250] In one embodiment, the electronic device (101) may move an object separated from an object on an input image based on an input. In one embodiment, the electronic device (101) may display an object moved (and / or reduced) from an object on an input image based on the input.

[0251] In operation 1220, the electronic device (101) may set a masking area based on the selected object. In one embodiment, the electronic device (101) may identify a masking area in an input image where filling processing is to be performed based on the input. The masking area may include an area of ​​an object selected by the input in the input image.

[0252] In operation 1230, the electronic device (101) can identify whether the selected object is moved into the masking area.

[0253] In operation 1230, in response to identifying that the selected object has moved into the masking area, the electronic device (101) may perform operation 1240. In operation 1230, in response to identifying that the selected object has not moved into the masking area, the electronic device (101) may perform operation 1250.

[0254] In operation 1240, the electronic device (101) may adjust the masking area. The electronic device (101) may set the area excluding the area to which the selected object has moved as the masking area based on the selected object. In one embodiment, if the moved object at least partially overlaps the area of ​​the selected object, the electronic device (101) may identify the remaining area excluding the area overlapped by the moved object in the area of ​​the selected object as the masking area.

[0255] In operation 1250, the electronic device (101) may perform filling processing on a masking area for an input image.

[0256] In one embodiment, the electronic device (101) may generate a result image for an input image based on a generative artificial intelligence (AI) model. In one embodiment, the electronic device (101) may fill an image in a masked area where some areas are excluded from an area based on a selected object based on the generative AI model. Here, the some areas may be areas overlapped by objects moved from the area of ​​the selected object.

[0257] FIG. 13 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0258] FIG. 13 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0259] Referring to FIG. 13, in operation 1310, the electronic device (101) can obtain a filled image.

[0260] In operation 1320, the electronic device (101) can select the transparency of the boundary of the masking area.

[0261] In one embodiment, the electronic device (101) may acquire a filled image so that the transparency decreases (or the opacity increases) as it moves away from the outer portion of the masking area, and may adjust an input image so that the transparency decreases (or the opacity increases) as it moves away from the outer portion of the masking area. Here, in the filled image, the direction away from the outer portion of the masking area may be an outer direction of the masking area. In the input image, the direction away from the outer portion of the masking area may be an inner direction of the masking area.

[0262] In operation 1330, the electronic device (101) can synthesize the input image into the filled image based on transparency.

[0263] In one embodiment, the electronic device (101) can composite an input image with reduced transparency (or increased opacity) onto a filled image with reduced transparency (or increased opacity). In one embodiment, the electronic device (101) can composite an area of ​​an object of an input image with reduced transparency (or increased opacity) onto a filled image with reduced transparency (or increased opacity).

[0264] FIG. 14 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0265] FIG. 14 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c.

[0266] Referring to FIG. 14, in operation 1410, the electronic device (101) can obtain a filled image.

[0267] In operation 1420, the electronic device (101) can identify whether a specified portion exists in the input image. For example, the specified portion may be a face (or head) and / or a skeleton point (e.g., a portion corresponding to a landmark).

[0268] In operation 1420, in response to identifying that the specified part exists, the electronic device (101) may perform operation 1430. In operation 1420, in response to identifying that the specified part does not exist, the electronic device (101) may terminate the operation of FIG. 14.

[0269] In operation 1430, the electronic device (101) can select the transparency of the boundary of the specified portion.

[0270] In one embodiment, the electronic device (101) may acquire a filled image so that the transparency decreases (or the opacity increases) as it moves away from the boundary of the designated portion, and may adjust an input image so that the transparency decreases (or the opacity increases) as it moves away from the boundary of the designated portion. Here, in the filled image, the direction away from the boundary of the designated portion may be an inward direction of the area of ​​the designated portion. In the input image, the direction away from the boundary of the designated portion may be an outward direction of the area of ​​the designated portion.

[0271] In operation 1440, the electronic device (101) can synthesize a specified portion into a filled image based on transparency.

[0272] In one embodiment, the electronic device (101) can composite a specified portion of an input image with reduced transparency (or increased opacity) onto a filled image with reduced transparency (or increased opacity).

[0273] As described above, the electronic device (101) may include a display (160). The electronic device (101) may include a processor (120). The electronic device (101) may include a memory (130) that stores instructions. The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a result image (411, 413, 415, 417) generated by filling processing on the input image (311, 313, 315, 317) based on a first ratio of an object (321, 322) present in the input image (311, 313, 315, 317) being greater than or equal to a first reference ratio. The instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio of the newly filled portion of the object (321, 322) exceeding a second reference ratio. The instructions, when executed by the processor (120), may cause the electronic device (101) to display the result image (411, 413, 415, 417) through the display (160) based on the second ratio being less than or equal to the second reference ratio.

[0274] The above instructions, when executed by the processor (120), may cause the electronic device (101) to acquire the result image (411, 413, 415, 417) in response to the first ratio being greater than or equal to the first reference ratio. The above instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from acquiring the result image (411, 413, 415, 417) in response to the first ratio being less than the first reference ratio.

[0275] The instructions, when executed by the processor (120), may cause the electronic device (101) to acquire the result image (411, 413, 415, 417) in response to a first number of designated portions of the objects (321, 322) included in the input image (311, 313, 315, 317) being greater than a second number of the objects (321, 322). The instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from acquiring the result image (411, 413, 415, 417) in response to the first number being less than or equal to the second number.

[0276] The electronic device (101) may further include a communication circuit (190). The instructions, when executed by the processor (120), may cause the electronic device (101) to identify a masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed. The instructions, when executed by the processor (120), may cause the electronic device (101) to transmit data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) to a server (108) via the communication circuit (190). The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a result image (411, 413, 415, 417) generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108).

[0277] The electronic device (101) may further include a communication circuit (190). The instructions, when executed by the processor (120), may cause the electronic device (101) to identify a masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed. The instructions, when executed by the processor (120), may cause the electronic device (101) to transmit data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) to a server (108) via the communication circuit (190). The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain another image generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108). The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain the result image (411, 413, 415, 417) by synthesizing the other image and the input image.

[0278] The above masking area (351, 353, 355, 357) may include a first area of ​​another object (321, 322) selected from the input image (311, 313, 315, 317). The above masking area (351, 353, 355, 357) may exclude a second area in which the other object (321, 322) overlapping the first area has moved on the input image (311, 313, 315, 317).

[0279] The above masking area (351, 353, 355, 357) may include a first area adjacent to a second area where another object (321, 322) selected from the input image (311, 313, 315, 317) has moved on the input image (311, 313, 315, 317). The first area may be an area of ​​a portion of the input image (311, 313, 315, 317) that is not identified from the other object (321, 322).

[0280] As described above, the electronic device (101) may include a display (160). The electronic device (101) may include a processor (120). The electronic device (101) may include a memory (130) that stores instructions. The instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a first anomaly for an input image (311, 313, 315, 317) through the first anomaly detection module (215). The first anomaly detection module (215) may be trained by images in which a first ratio of an object (321, 322) is greater than or equal to a first reference ratio. The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a result image (411, 413, 415, 417) generated by filling processing on the input image (311, 313, 315, 317) based on the first outlier being less than or equal to a first reference outlier. The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a second outlier for the result image (411, 413, 415, 417) through a second anomaly detection module (235). The second anomaly detection module (235) may be trained by images in which a second ratio of an object (321, 322) is greater than or equal to a designated second reference ratio. The above instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from displaying the result images (411, 413, 415, 417) through the display (160) based on the second abnormality being greater than or equal to the second reference abnormality.The above instructions, when executed by the processor (120), may cause the electronic device (101) to display the result image (411, 413, 415, 417) through the display (160) based on the second abnormality being less than the second reference abnormality.

[0281] As described above, the method can be performed in an electronic device (101) including a display (160). The method can include an operation of obtaining a result image (411, 413, 415, 417) generated by filling processing on an input image (311, 313, 315, 317) based on a first ratio of an object (321, 322) present in the input image (311, 313, 315, 317) being greater than or equal to a first reference ratio. The method can include an operation of refraining from displaying the result image (411, 413, 415, 417) through the display (160) based on a second ratio of a newly filled portion of the object (321, 322) exceeding a second reference ratio. The above method may include an operation of displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio being less than or equal to the second reference ratio.

[0282] The method may include an operation of acquiring the result image (411, 413, 415, 417) in response to the first ratio being greater than or equal to the first reference ratio. The method may include an operation of refraining from acquiring the result image (411, 413, 415, 417) in response to the first ratio being less than the first reference ratio.

[0283] The method may include an operation of acquiring the result image (411, 413, 415, 417) in response to a first number of designated portions of the objects (321, 322) included in the input image (311, 313, 315, 317) being greater than a second number of the objects (321, 322). The method may include an operation of refraining from acquiring the result image (411, 413, 415, 417) in response to the first number being less than or equal to the second number.

[0284] The method may include an operation of identifying a masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed. The method may include an operation of transmitting data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) to a server (108) via a communication circuit (190) of the electronic device (101). The above method may include an operation of obtaining a result image (411, 413, 415, 417) generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108).

[0285] The method may include an operation of identifying a masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed. The method may include an operation of transmitting data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) to a server (108) via a communication circuit (190) of the electronic device (101). The method may include an operation of obtaining another image generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108). The above method may include an operation of obtaining the result image (411, 413, 415, 417) by synthesizing the other image and the input image.

[0286] The above masking area (351, 353, 355, 357) may include a first area of ​​another object (321, 322) selected from the input image (311, 313, 315, 317). The above masking area (351, 353, 355, 357) may exclude a second area in which the other object (321, 322) overlapping the first area has moved on the input image (311, 313, 315, 317).

[0287] The above masking area (351, 353, 355, 357) may include a first area adjacent to a second area where another object (321, 322) selected from the input image (311, 313, 315, 317) has moved on the input image (311, 313, 315, 317). The first area may be an area of ​​a portion of the input image (311, 313, 315, 317) that is not identified from the other object (321, 322).

[0288] As described above, the method can be performed in an electronic device (101) including a display (160). The method can include an operation of obtaining a first outlier for an input image (311, 313, 315, 317) through a first outlier detection module (215). The first outlier detection module (215) can be trained by images in which a first ratio of an object (321, 322) is greater than or equal to a first reference ratio. The method can include an operation of obtaining a result image (411, 413, 415, 417) generated by filling processing for the input image (311, 313, 315, 317) based on the first outlier being less than or equal to the first reference outlier. The method may include an operation of obtaining a second outlier for the result images (411, 413, 415, 417) through a second outlier detection module (235). The second outlier detection module (235) may be trained by images in which a second ratio of an object (321, 322) is greater than or equal to a designated second reference ratio. The method may include an operation of refraining from displaying the result images (411, 413, 415, 417) through the display (160) based on the second outlier being greater than or equal to the second reference outlier. The method may include an operation of displaying the result images (411, 413, 415, 417) through the display (160) based on the second outlier being less than the second reference outlier.

[0289] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when executed by a processor (120) of an electronic device (101) including a display (160), can cause the electronic device (101) to obtain a result image (411, 413, 415, 417) generated by filling processing on an input image (311, 313, 315, 317) based on a first ratio of an object (321, 322) present in the input image (311, 313, 315, 317) being greater than or equal to a first reference ratio. The instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio of the newly filled portion of the object (321, 322) exceeding a second reference ratio. The instructions, when executed by the processor (120), may cause the electronic device (101) to display the result image (411, 413, 415, 417) through the display (160) based on the second ratio being less than or equal to the second reference ratio.

[0290] As described above, a non-transitory computer-readable recording medium can store a program including instructions. When the instructions are executed by the processor (120) of the electronic device (101) including the display (160), the electronic device (101) can cause the first anomaly detection module (215) to acquire a first anomaly for an input image (311, 313, 315, 317). The first anomaly detection module (215) can be trained by images in which the first ratio of the object (321, 322) is greater than or equal to a first reference ratio. The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a result image (411, 413, 415, 417) generated by filling processing on the input image (311, 313, 315, 317) based on the first outlier being less than or equal to a first reference outlier. The above instructions, when executed by the processor (120), may cause the electronic device (101) to obtain a second outlier for the result image (411, 413, 415, 417) through a second anomaly detection module (235). The second anomaly detection module (235) may be trained by images in which a second ratio of an object (321, 322) is greater than or equal to a designated second reference ratio. The above instructions, when executed by the processor (120), may cause the electronic device (101) to refrain from displaying the result images (411, 413, 415, 417) through the display (160) based on the second abnormality being greater than or equal to the second reference abnormality.The above instructions, when executed by the processor (120), may cause the electronic device (101) to display the result image (411, 413, 415, 417) through the display (160) based on the second abnormality being less than the second reference abnormality.

[0291] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0292] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0293] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0294] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0295] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0296] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), Display (160), processor (120), and It includes a memory (130) that stores instructions, and when the instructions are executed by the processor (120), the electronic device (101) Based on the fact that the first ratio of the object (321, 322) existing in the input image (311, 313, 315, 317) is greater than or equal to the first reference ratio, a result image (411, 413, 415, 417) generated by filling processing for the input image (311, 313, 315, 317) is obtained, Refrain from displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio of the newly filled portion of the above object (321, 322) exceeding the second reference ratio, and Based on the above second ratio being less than or equal to the second reference ratio, causing the result image (411, 413, 415, 417) to be displayed through the display (160). Electronic devices.

2. In claim 1, The above instructions, when executed by the processor (120), cause the electronic device (101) to: In response to the above first ratio being greater than or equal to the above first reference ratio, the result image (411, 413, 415, 417) is obtained, In response to the first ratio being less than the first reference ratio, causing acquisition of the result image (411, 413, 415, 417) to be avoided. Electronic devices.

3. In claim 2, The above instructions, when executed by the processor (120), cause the electronic device (101) to: In response to the first number of designated portions of the object (321, 322) included in the input image (311, 313, 315, 317) being greater than the second number of the object (321, 322), the result image (411, 413, 415, 417) is obtained, In response to the first number being less than or equal to the second number, causing acquisition of the result image (411, 413, 415, 417) to be avoided. Electronic devices.

4. In any one of claims 1 to 3, Further comprising a communication circuit (190), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Identify the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed, Through the above communication circuit (190), data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) are transmitted to the server (108), Causing to obtain a result image (411, 413, 415, 417) generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108). Electronic devices.

5. In any one of claims 1 to 4, Further comprising a communication circuit (190), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Identify the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed, Through the above communication circuit (190), data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) are transmitted to the server (108), Obtain another image generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108), By synthesizing the above-mentioned other image and the above-mentioned input image, the result image (411, 413, 415, 417) is obtained. Electronic devices.

6. In claim 5, The above masking area (351, 353, 355, 357) includes a first area of ​​another object (321, 322) selected from the input image (311, 313, 315, 317), The above masking area (351, 353, 355, 357) is a second area where the other object (321, 322) overlapping the first area is moved on the input image (311, 313, 315, 317) excluding the second area. Electronic devices.

7. In claim 5, The above masking area (351, 353, 355, 357) includes a first area adjacent to a second area in which another object (321, 322) selected from the input image (311, 313, 315, 317) has moved on the input image (311, 313, 315, 317), The above first region is an area of ​​a part of the input image (311, 313, 315, 317) that is not identified from the other object (321, 322). Electronic devices.

8. In the electronic device (101), Display (160), processor (120), and It includes a memory (130) that stores instructions, and when the instructions are executed by the processor (120), the electronic device (101) The first anomaly detection module (215) obtains the first anomaly for the input image (311, 313, 315, 317), and the first anomaly detection module (215) is learned by images in which the first ratio of the object (321, 322) is greater than or equal to the first reference ratio. Based on the above first outlier being less than or equal to the first reference outlier, a result image (411, 413, 415, 417) generated by filling processing for the input image (311, 313, 315, 317) is obtained, The second abnormality detection module (235) obtains the second abnormality for the result images (411, 413, 415, 417), and the second abnormality detection module (235) is learned by images in which the second ratio of the object (321, 322) is greater than or equal to the specified second reference ratio. Based on the above second outlier being greater than or equal to the second reference outlier, refrain from displaying the result images (411, 413, 415, 417) through the display (160), and Based on the above second abnormal value being less than the second reference abnormal value, causing the result image (411, 413, 415, 417) to be displayed through the display (160). Electronic devices.

9. In claim 8, Further comprising a communication circuit (190), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Identify the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed, Through the above communication circuit (190), data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) are transmitted to the server (108), Causing to obtain a result image (411, 413, 415, 417) generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108). Electronic devices.

10. In claim 8 or claim 9, Further comprising a communication circuit (190), The above instructions, when executed by the processor (120), cause the electronic device (101) to: Identify the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) on which the filling process is to be performed, Through the above communication circuit (190), data representing the input image (311, 313, 315, 317) and information about the masking area (351, 353, 355, 357) are transmitted to the server (108), Obtain another image generated by the filling process for the masking area (351, 353, 355, 357) of the input image (311, 313, 315, 317) from the server (108), By synthesizing the above-mentioned other image and the above-mentioned input image, the result image (411, 413, 415, 417) is obtained. Electronic devices.

11. In claim 10, The above masking area (351, 353, 355, 357) includes a first area of ​​another object (321, 322) selected from the input image (311, 313, 315, 317), The above masking area (351, 353, 355, 357) is a second area where the other object (321, 322) overlapping the first area is moved on the input image (311, 313, 315, 317) excluding the second area. Electronic devices.

12. In claim 10, The above masking area (351, 353, 355, 357) includes a first area adjacent to a second area in which another object (321, 322) selected from the input image (311, 313, 315, 317) has moved on the input image (311, 313, 315, 317), The above first region is an area of ​​a part of the input image (311, 313, 315, 317) that is not identified from the other object (321, 322). Electronic devices.

13. A method performed in an electronic device (101) including a display (160), An operation of obtaining a result image (411, 413, 415, 417) generated by filling processing on the input image (311, 313, 315, 317) based on the fact that a first ratio of an object (321, 322) existing in the input image (311, 313, 315, 317) is greater than or equal to a first reference ratio. An operation of refraining from displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio of the newly filled portion of the above object (321, 322) exceeding the second reference ratio, and An operation of displaying the result image (411, 413, 415, 417) through the display (160) based on the second ratio being less than or equal to the second reference ratio. method.

14. In claim 13, In response to the first ratio being greater than or equal to the first reference ratio, an operation of obtaining the result image (411, 413, 415, 417), and In response to the first ratio being less than the first reference ratio, an action is included to refrain from acquiring the result image (411, 413, 415, 417). method.

15. In claim 13 or claim 14, An operation of obtaining the result image (411, 413, 415, 417) in response to a first number of designated portions of the object (321, 322) included in the input image (311, 313, 315, 317) being greater than a second number of the object (321, 322), and In response to the first number being less than or equal to the second number, an operation of refraining from acquiring the result image (411, 413, 415, 417) is included. method.

Images