Electronic device comprising camera and operation method thereof

Abstract

Disclosed are an electronic device comprising a camera and an operation method thereof. The electronic device may acquire a first image through a camera module. The electronic device may identify a target object requiring additional imaging among a plurality of objects included in the first image, and adjust the imaging range of the camera module from a first imaging range to a second imaging range in order to additionally image the target object. The electronic device may acquire an additional image of the target object through the camera module. The electronic device may generate and display a second image including a basic area and an extended area related to the target object on the basis of the first image and the additional image. Various other embodiments understood through the present document are also possible.

Description

Electronic device including a camera and method of operating the same

[0001] The present disclosure relates to an electronic device including a camera and a method of operating the same.

[0002] Electronic devices have become equipped with complex functions, such as taking photos and videos, playing music and videos, playing games, receiving broadcasts, and supporting wireless Internet, and are being implemented as comprehensive multimedia players. Accordingly, electronic devices are evolving in new ways, both in hardware and software, to satisfy user needs while enhancing portability and convenience. One example of this development is the emergence of foldable electronic devices.

[0003] Additionally, electronic devices (e.g., smartphones, mobile terminals, digital cameras, or wearable devices) can provide various functions utilizing cameras or images (e.g., shooting functions, filter functions, photo editing functions, album functions, transmission / reception functions, video calling functions, or messenger functions). Various attempts are being made to improve camera performance or image quality (e.g., photographs) in electronic devices.

[0004] An electronic device according to one embodiment may include a display, a camera module, at least one processor, and a memory storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to acquire a first image through the camera module, identify at least one target object requiring additional shooting from among a plurality of objects included in the first image, adjust a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification, acquire an additional image of the at least one target object through the camera module, generate a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image, and display the second image through the display.

[0005] An operating method of an electronic device according to one embodiment may include an operation of acquiring a first image through a camera module, an operation of identifying at least one target object requiring additional shooting among a plurality of objects included in the first image, an operation of adjusting a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification, an operation of acquiring an additional image of the at least one target object through the camera module, an operation of generating a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image, and an operation of displaying the second image through a display.

[0006] A computer-readable storage medium according to one embodiment may store a program for executing a method, the method including the steps of: acquiring a first image through a camera module; identifying at least one target object requiring additional shooting among a plurality of objects included in the first image; adjusting a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification; acquiring an additional image of the at least one target object through the camera module; generating a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image; and displaying the second image through a display.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0009] Figure 3 is a block diagram of an electronic device according to one embodiment.

[0010] Figure 4 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0011] FIG. 5 is a diagram illustrating an example of an output image generated by an electronic device according to one embodiment.

[0012] FIG. 6 is a flowchart illustrating a method of an electronic device according to one embodiment of the present invention for capturing an image based on object completeness and / or object-specific weights.

[0013] FIG. 7A is a diagram illustrating an example of an input image captured by an electronic device according to one embodiment.

[0014] FIG. 7b is a diagram illustrating an example of an additional image captured by an electronic device according to one embodiment.

[0015] FIG. 7c is a diagram illustrating an example of a primary output image generated by an electronic device according to one embodiment.

[0016] FIG. 7d is a diagram illustrating an example of a secondary output image generated by an electronic device according to one embodiment.

[0017] FIG. 7e is a diagram illustrating another example of a secondary output image generated by an electronic device according to one embodiment.

[0018] FIG. 8A is a drawing showing an unfolded state of a foldable electronic device according to one embodiment.

[0019] FIG. 8b is a diagram showing a folded state of a foldable electronic device according to one embodiment.

[0020] FIGS. 9A and 9B are drawings for explaining changes in the shooting range according to the folding angle of a foldable electronic device according to one embodiment.

[0021] FIG. 10 is a diagram illustrating an example of a user interface for providing an image capturing function according to one embodiment.

[0022] FIG. 11 is a diagram illustrating an example of a user interface for guiding additional shooting according to one embodiment.

[0023] FIG. 12a is a diagram for explaining a multi-object detection artificial intelligence model according to one embodiment.

[0024] FIG. 12b is a diagram for explaining an image generation artificial intelligence model according to one embodiment.

[0025] FIG. 13 is a diagram for explaining an image correction operation of an image generation artificial intelligence model according to one embodiment.

[0026] FIG. 14 is a diagram illustrating a system including a generative artificial intelligence model according to one embodiment.

[0027] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0028] Electronic devices may offer a camera-based photography function. For example, a user can view a preview screen displayed on the device and press a shutter button to take a photo. Simple photography functions may only provide the user with the results of their manual shooting. In such cases, captured photos may lack essential details or be incomplete, reducing user satisfaction with the resulting images. Furthermore, capturing an appropriate scene can be inconvenient, requiring the user to move repeatedly or change the angle of view.

[0029] Various embodiments of the present disclosure can provide an electronic device including a camera and an operating method thereof that can increase user satisfaction with the captured image result while reflecting the user's intention when capturing an image.

[0030] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0031] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0032] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0033] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0034] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0035] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0036] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0037] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0038] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0039] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0040] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0041] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0043] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0045] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). The corresponding communication module within these communication modules can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0048] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0050] At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0052] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments.

[0053] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0054] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0055] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0056] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0057] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0058] FIG. 3 is a block diagram of an electronic device (300) according to one embodiment.

[0059] Referring to FIG. 3, an electronic device (300) according to one embodiment may include a camera module (310), a display (320), a processor (330), and / or a memory (340). The electronic device (300) of FIG. 3 may correspond to the electronic device (101) illustrated in FIG. 1. The camera module (310), the display (320), the processor (330), and the memory (340) included in the electronic device (300) of FIG. 3 may each correspond to the camera module (180), the display module (160), the processor (120), and the memory (130) of FIG. 1. In some embodiments, at least one of the illustrated components of the electronic device (300) may be omitted, integrated with each other, or additionally provided with another component. The camera module (310), display (320), processor (330), and memory (340) included in the electronic device (300) may be electrically and / or operationally connected to each other to exchange signals (e.g., commands or data) with each other.

[0060] According to one embodiment, an electronic device (300) (e.g., electronic device (101) of FIG. 1) may provide an image capturing function. For example, the electronic device (300) may provide an image capturing function based on artificial intelligence (AI). For example, the electronic device (300) may execute an application (e.g., a camera application, a gallery application) and use the application to provide an image capturing function. The application executed in the electronic device (300) may operate independently or in conjunction with an external electronic device (e.g., electronic device (102), electronic device (104) of FIG. 1, or server (108) of FIG. 1).

[0061] According to one embodiment, the processor (330) (e.g., the processor (120) of FIG. 1) may include at least one processor. The processor (330) may execute various functions supported by the electronic device (300). The processor (330) may control the camera module (310), the display (320), the processor (330), and / or the memory (340). The processor (330) may execute an application and control various hardware by executing code written in a programming language stored in the memory (340) of the electronic device (300).

[0062] According to one embodiment, the memory (340) (e.g., the memory (130) of FIG. 1) may store instructions that, when executed by the processor (330), cause the electronic device (300) to perform various operations. For example, the processor (330) of the electronic device (300) may provide an image capturing function by executing instructions stored in the memory (340).

[0063] According to one embodiment, the display (320) (e.g., the display module (160) of FIG. 1) may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), organic light emitting diodes (OLED), a light emitting diode (LED), an active matrix organic LED (AMOLED), a flexible display, and a 3-dimensional display. In some embodiments, some of these displays may be configured as transparent or light-transmitting so that the outside may be viewed therethrough. For example, the transparent or light-transmitting display may be configured as a transparent display including a TOLED (transparent OLED).

[0064] According to one embodiment, the camera module (310) (e.g., the camera module (180) of FIG. 1) may include at least one camera. For example, each camera may correspond to the camera module (180) of FIG. 1 or include at least a portion of the camera module (180). Portions of each camera may be implemented in an integrated form. For example, each camera may include only some components of the camera module (180) (e.g., the lens assembly (210)), and other components (e.g., the image sensor (230) or the image signal processor (260)) may be integrated into one.

[0065] According to one embodiment, the camera module (310) can acquire an image (e.g., a still image, a moving image) by performing a shooting (or capturing) operation according to a user's operation. The captured image can be displayed through the display (320) or stored in the memory (340). For example, the camera module (310) can acquire a preview image in frame units as the camera application is executed, and can capture (or capture) the image when a user input for shooting (e.g., shutter button input) is received while displaying the preview image. The display (320) can display the preview image acquired through the camera module (310) in real time. The image captured by the camera module (310) can be stored in the memory (340). For example, when the gallery application is executed, the captured image stored in the memory (340) can be displayed through the display (320).

[0066] According to one embodiment, the processor (330) can control the camera module (310). The processor (330) can support various functions using the camera module (310). For example, the processor (330) can store images (e.g., preview images and / or captured images) acquired through the camera module (310) in the memory (340) or display the images through the display (320). For example, while a camera application is running, the processor (330) can acquire a preview image in real time through the camera module (310) and display the preview image on the display (320). For example, while a gallery application is running, the processor (330) can load captured images stored in the memory (340) and display the captured images through the display (320).

[0067] According to one embodiment, the processor (330) can obtain an input image through the camera module (310). The processor (330) can store the input image and / or an output image generated from the input image in the memory (340) or display it through the display (320).

[0068] According to one embodiment, the processor (330) may obtain a first image (input image) through the camera module (310). The processor (330) may identify at least one target object requiring additional shooting among a plurality of objects included in the first image. Based on the identification, the processor (330) may adjust the shooting range of the camera module (310) from the first shooting range to a second shooting range in order to additionally shoot the at least one target object. The processor (330) may obtain an additional image of the at least one target object through the camera module (310). The processor (330) may generate a second image (output image) including a basic region and an extended region related to the at least one target object based on the first image and the additional image. The processor (330) may display the second image through the display (320).

[0069] According to one embodiment, at least one target object requiring additional shooting may correspond to at least one of a second object around a focused first object among a plurality of objects included in a first image (input image), a second object around a first object located at the center of the first image, a second object located at the periphery of the first image, a second object including only a partial shape without an entire shape, or a second object related to location information of the electronic device (300).

[0070] According to one embodiment, the extended region associated with at least one target object in the second image (output image) may include at least one of an actual shooting region or a virtual region. The actual shooting region may be cropped and / or warped from an additionally captured additional image, and may include a partial shape of the entire shape of at least one target object that is not included in the first image (input image). The virtual region may be generated based on at least one of the additional image or the actual shooting region.

[0071] According to one embodiment, the processor (330) can detect a plurality of objects from a first image (input image). The processor (330) can identify an object among the plurality of objects whose object completeness is lower than a specified ratio as a target object requiring additional shooting.

[0072] According to one embodiment, the processor (330) can detect a plurality of objects from a first image (input image). The processor (330) can assign weights to the plurality of objects. Based on the assigned weights, the processor (330) can identify a target object requiring additional imaging among the plurality of objects.

[0073] According to one embodiment, the processor (330) may classify each of the plurality of objects into one of a first object having a first weight (e.g., a key object), a second object having a second weight lower than the first weight (e.g., a key object), and a third object having a third weight lower than the second weight (e.g., a non-key object) based on the assigned weights. The processor (330) may identify the second object having the second weight as a target object requiring additional imaging.

[0074] According to one embodiment, the camera module (310) of the electronic device (300) may include a plurality of cameras having different angles of view. The processor (330) of the electronic device (300) may acquire the first image for the first shooting range through the activated first camera when the first camera having the first angle of view among the plurality of cameras is activated. The processor (330) may activate a second camera having the second angle of view among the plurality of cameras. The processor (330) may acquire an additional image for the second shooting range through the activated second camera.

[0075] According to one embodiment, the electronic device (300) may be a foldable electronic device (e.g., the foldable electronic device (800) of FIGS. 8A and 8B). The processor (330) of the electronic device (300) may acquire a first image for a first shooting range through a first camera in a camera module (310). The processor (330) may change the shooting range of the first camera from the first shooting range to a second shooting range by changing a folding angle of the foldable electronic device. The processor (330) may acquire an additional image for the second shooting range through the first camera. In one embodiment, the processor (330) may automatically change the shooting range of the first camera by changing the folding angle of the electronic device (300) while the electronic device (300) (e.g., the foldable electronic device (800) of FIGS. 8A and 8B) operates in a flex mode.

[0076] According to one embodiment, the processor (330) may obtain object-related information about at least one target object among a plurality of objects included in the first image by transmitting first image information about the first image (input image, first captured image) to a first artificial intelligence model (e.g., the first artificial intelligence model (1210) of FIG. 12A) trained to identify at least one target object requiring additional capturing. The electronic device (300) may identify at least one target object based on the object-related information.

[0077] According to one embodiment, the processor (330) may obtain the second image (output image, AI extended image) generated based on the first image and the additional image by transmitting first image information for the first image (input image, first captured image) and additional image information for the additional image (additional captured image) to a second artificial intelligence model (e.g., the second artificial intelligence model (1250) of FIG. 12a) trained to generate the second image.

[0078] Hereinafter, operating methods of an electronic device (300) according to various embodiments will be described with reference to FIGS. 4, 5, 6, 7a, 7b, 7c, 7d, and 7e. The illustrated operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. In some embodiments, some of the illustrated operations may be omitted, some operations may be combined, the order of some operations may be changed, or other operations may be added. At least some of the operations of the operating methods of the electronic device (300) according to various embodiments described below may be performed in correspondence with each other or in combination with each other.

[0079] FIG. 4 is a flowchart illustrating an operation method of an electronic device (300) according to one embodiment.

[0080] According to one embodiment, the method of operating the electronic device (300) may be for providing an image capturing function.

[0081] Referring to FIG. 4, an operation method of an electronic device (300) for providing an image capturing function may include operations 410, 420, 430, 440, 450, and 460.

[0082] In operation 410, the electronic device (300) can obtain a first image (e.g., input image (510) of FIG. 5) through the camera module (310).

[0083] According to one embodiment, the first image may be an image initially photographed (or captured) according to the user's intention. The first image may be an image photographed (or captured) by the camera module (310) when the photographing range (or camera photographing range) of the camera module (310) is set to the first photographing range. For example, the camera photographing range may correspond to at least one of a folding angle, a camera angle of view, or a camera position.

[0084] In operation 420, the electronic device (300) can identify at least one target object requiring additional shooting among the plurality of objects included in the first image acquired through operation 410.

[0085] According to one embodiment, at least one target object requiring additional shooting may correspond to at least one of a second object around a focused first object among a plurality of objects included in the first image, a second object around a first object located at the center of the first image, a second object located at the outskirts of the first image, a second object including only a partial shape without an entire shape, or a second object related to location information of the electronic device (300).

[0086] According to one embodiment, the electronic device (300) can identify at least one target object requiring additional shooting based on at least one of object completeness, object-specific weight, object-specific location, or location information of the electronic device (300).

[0087] According to one embodiment, the electronic device (300) may identify at least one target object requiring additional photography based on the object completeness. For example, the electronic device (300) may identify an object (e.g., a second object including only a partial shape without a full shape) among a plurality of objects in the first image acquired through operation 410, wherein the object completeness is lower than a specified ratio, as a target object requiring additional photography.

[0088] According to one embodiment, the electronic device (300) may identify at least one target object requiring additional shooting based on weights for each object. For example, the electronic device (300) may detect a plurality of objects from the first image acquired through operation 410. The electronic device (300) may assign a weight (or priority) to each of the detected plurality of objects. The electronic device (300) may identify a target object requiring additional shooting among the plurality of objects based on the weights for each object assigned to the plurality of objects. For example, the electronic device (300) may classify each of the plurality of objects in the first image into one of a first object having a first weight (e.g., a key object), a second object having a second weight lower than the first weight (e.g., a key object), and a third object having a third weight lower than the second weight (e.g., a non-key object). The electronic device (300) can identify the second object (e.g., main object) having the second weight based on the assigned object-specific weight as a target object requiring additional shooting.

[0089] According to one embodiment, the electronic device (300) may identify at least one target object requiring additional shooting based on the location of each object. For example, the electronic device (300) may identify a second object around a focused first object (e.g., a first object located at a vanishing point location) among a plurality of objects in the first image acquired through operation 410, a second object around a first object located at the center of the first image, or a second object located at the periphery (boundary) of the first image as a target object requiring additional shooting.

[0090] According to one embodiment, the electronic device (300) can identify at least one target object requiring additional shooting based on the location information (or place information) of the electronic device (300). For example, the electronic device (300) can obtain the location information (or place information) of the electronic device (300) through at least one sensor (e.g., a GNSS communication module, a time of flight (TOF) sensor, an ultra wideband (UWB) sensor). For example, the electronic device (300) can determine a second object (e.g., a landmark terrain, facility, or character related to the current location) having the highest correlation with the location information among a plurality of objects in a first image, and identify the second object as a target object requiring additional shooting.

[0091] According to one embodiment, the electronic device (300) may transmit first image information about the first image acquired through operation 410 as an input to a first artificial intelligence model (e.g., the first artificial intelligence model (1210) of FIG. 12A) trained to identify at least one target object requiring additional shooting, thereby acquiring object-related information about the at least one target object among a plurality of objects included in the first image. The electronic device (300) may identify the at least one target object based on the object-related information.

[0092] In operation 430, the electronic device (300) can adjust the shooting range (or camera shooting range) of the camera module (310) from the first shooting range to the second shooting range to additionally shoot at least one target object based on the identification result of operation 420.

[0093] According to one embodiment, the electronic device (300) can adjust the camera shooting range by changing at least one of the folding angle, the camera angle of view, or the camera position.

[0094] In operation 440, the electronic device (300) may acquire an additional image of the at least one target object through the camera module (310). The additional image may be an image secondarily captured (or taken) by the camera module while the shooting range (or camera shooting range) of the camera module (310) is adjusted to a second shooting range.

[0095] According to one embodiment, the shooting range (or camera shooting range) of the camera module (310) may vary as the folding angle of the electronic device (300) changes. The electronic device (300) may acquire both the first image and the additional images using the same camera. For example, the electronic device (300) may be a foldable electronic device (e.g., the foldable electronic device (800) of FIGS. 8A and 8B ). The foldable electronic device may acquire a first image for the first shooting range through the first camera (e.g., the front camera (804) of FIG. 8A) in a state in which the foldable electronic device is folded at a first folding angle (e.g., 45 degrees). Here, the first shooting range may correspond to the first folding angle. The foldable electronic device can change the shooting range of the first camera (e.g., the front camera (804) of FIG. 8A) from the first shooting range to the second shooting range by changing the folding angle from the first folding angle to the second folding angle (e.g., 30 degrees). Here, the second shooting range can correspond to the second folding angle. The foldable electronic device can acquire additional images for the second shooting range through the first camera (e.g., the front camera (804) of FIG. 8A) when folded at the second folding angle (e.g., 30 degrees).

[0096] According to one embodiment, the shooting range (or camera shooting range) of the camera module (310) may vary as the camera that is activated (or selected) among a plurality of cameras having different angles of view is changed from a first camera to a second camera. The camera module (310) of the electronic device (300) may include a plurality of cameras having different angles of view. The electronic device (300) may acquire a first image and an additional image using the cameras having different angles of view in the camera module (310). For example, the electronic device (300) may acquire a first image for a first shooting range through the activated first camera (e.g., a wide-angle camera with an angle of view of 85 degrees) having a first angle of view among the plurality of cameras when the first camera is activated. Here, the first shooting range may correspond to the first angle of view of the first camera. The electronic device (300) may activate a second camera (e.g., an ultra-wide-angle camera with a 120-degree field of view) among the plurality of cameras. The electronic device (300) may acquire additional images of a second shooting range through the activated second camera. Here, the second shooting range may correspond to the second angle of view of the second camera.

[0097] According to one embodiment, the shooting range (or camera shooting range) of the camera module (310) may vary as the camera that is activated (or selected) among the plurality of cameras arranged at different locations changes from the first camera to the second camera. The electronic device (300) may acquire the first image and the additional images, respectively, using the cameras arranged at different locations within the camera module (310). For example, the electronic device (300) may acquire the first image for the first shooting range through the activated first camera when the first camera (e.g., the front center camera) at the first location among the plurality of cameras is activated. Here, the first shooting range may correspond to the first location (or first direction) of the first camera. The electronic device (300) may activate the second camera (e.g., the front left camera and / or the front right camera) at the second location among the plurality of cameras. The electronic device (300) may acquire the additional image for the second shooting range through the activated second camera. Here, the second shooting range may correspond to the second position (or second direction) of the second camera.

[0098] In operation 450, the electronic device (300) may generate a second image (e.g., an output image (520) of FIG. 5) based on the first image acquired through operation 410 and the additional image acquired through operation 440. The second image (e.g., the output image (520) of FIG. 5) may include a basic region (e.g., a basic region (521) of FIG. 5) and an extended region (e.g., at least one of the extended regions (522, 523) of FIG. 5) related to at least one target object. For example, the basic region may be a region corresponding to the first image. The extended region may be a region added to the basic region related to the at least one target object.

[0099] According to one embodiment, the extended area associated with at least one target object may include at least one of an actual shooting area (e.g., the first extended area (522) of FIG. 5) or a virtual area (e.g., the second extended area (523) of FIG. 5).

[0100] In one embodiment, the actual capture area (e.g., the first extended area (522) of FIG. 5) may be a cropped and / or warped area from an additional image acquired through operation 440. For example, the actual capture area may include a partial shape of at least one target object requiring additional capture that is not included in the first image.

[0101] In one embodiment, the virtual area (e.g., the second extended area (523) of FIG. 5) may be an area generated based on at least one of the additional image or the actual captured area. For example, the virtual area may be an area out-painted (or upscaled, rendered) from a region synthesized by cropping and / or warping an actual captured area from an additional image (e.g., the first extended area (522) of FIG. 5) and a basic area corresponding to the first image (e.g., the basic area (521) of FIG. 5).

[0102] According to one embodiment, the electronic device (300) may transfer first image information for the first image (input image) acquired through operation 410 and additional image information for the additional image acquired through operation 440 to a second artificial intelligence model (e.g., the second artificial intelligence model (1250) of FIG. 12b) trained to generate a second image (output image) including an extended area, thereby obtaining a second image (e.g., the output image (520) of FIG. 5) generated based on the first image and the additional image.

[0103] In operation 460, the electronic device (300) may display a second image generated through operation 450 (e.g., the output image (520) of FIG. 5) and / or a user interface related to the second image through the display (320).

[0104] FIG. 5 is a diagram illustrating an example of an output image generated by an electronic device (300) according to one embodiment.

[0105] In Fig. 5, reference number 510 represents an input image, reference number 520 represents an output image, reference number 521 represents a basic area, reference number 522 represents a first extended area, and reference number 523 represents a second extended area.

[0106] Referring to FIG. 5, the electronic device (300) can capture an input image (510) according to a user operation. For example, the input image (510) may be a primary capture result intended by the user. For example, the input image (510) may be an image captured for the first time by a camera module (310) within the electronic device (300) (e.g., a front camera at a first folding angle, a wide-angle camera with a field of view of 85 degrees).

[0107] In one embodiment, the electronic device (300) may generate an output image (520) using an input image (510). For example, the output image (520) may be a secondary photographing result generated using an artificial intelligence-based image capturing function.

[0108] In one embodiment, the output image (520) may include a base region (521) corresponding to the input image (510). The output image (520) may further include one or more extended regions (522, 523) in addition to the base region (521).

[0109] In one embodiment, the first extended area (522) may be an actual captured area. For example, the first extended area (522) may be a cropped area from an image additionally captured by a camera module (310) within the electronic device (300) (e.g., a front camera at a second folding angle, an ultra-wide-angle camera with a field of view of 120 degrees). For example, the first extended area (522) may be an area to which a warping technique is applied from an image additionally captured by the camera module within the electronic device (300). When applying the warping technique to the additionally captured image, the warping operation may be performed based on the image information of the basic area (521).

[0110] In one embodiment, the second extended area (523) may be a virtual area generated based on at least one of an image additionally captured by a camera module (310) within the electronic device (300) (e.g., a front camera at a second folding angle, an ultra-wide-angle camera with a field of view of 120 degrees) and a first extended area (522) cropped and / or warped from the additionally captured image. For example, the second extended area (523) may be an area out-painted (or upscaled, rendered) from an area synthesized by cropping and / or warping the first extended area (522) from the additionally captured image and the basic area (521) corresponding to the input image (510).

[0111] FIG. 6 is a flowchart illustrating a method for an electronic device (300) to capture an image based on object completeness and / or object-specific weights according to one embodiment.

[0112] Referring to FIG. 6, a method of capturing an image by an electronic device (300) may include operations 610, 620, 630, 640, 650, 660, 670, and 680.

[0113] For example, operation 610 of FIG. 6 may correspond to operation 410 of FIG. 4. Operations 620, 630, and 640 of FIG. 6 may correspond to operation 420 of FIG. 4. Operation 650 of FIG. 6 may correspond to operation 430 of FIG. 4. Operation 660 of FIG. 6 may correspond to operation 440 of FIG. 4. Operation 670 of FIG. 6 may correspond to operation 450 of FIG. 4. Operation 680 of FIG. 6 may correspond to operation 460 of FIG. 4.

[0114] In operation 610, the electronic device (300) may acquire a first image through the camera module (310). The first image may be an image that is first photographed (or captured) according to the user's intention. The first image may be an image that is first photographed by the camera module (310) when the photographing range (or camera photographing range) of the camera module (310) is set to the first photographing range. For example, the first image may be the input image (710) of FIG. 7A. The input image (710) of FIG. 7A may be an image photographed at a first folding angle and / or an image photographed by a wide-angle camera.

[0115] According to one embodiment, the electronic device (300) may identify at least one target object requiring additional shooting based on object completeness and / or object-specific weights. Operations 620, 630, and 640 may be for identifying an object requiring additional shooting among a plurality of objects in the first image.

[0116] In operation 620, the electronic device (300) can detect a plurality of objects included in the first image acquired through operation 610. For example, the electronic device (300) can detect object A (711), object B (712), object C (713), object D (714), and object E (715) of FIG. 7A from the input image (710) of FIG. 7A.

[0117] In operation 630, the electronic device (300) may assign a weight to each of the plurality of objects detected from the first image through operation 620. Based on the object-specific weights assigned to the plurality of objects, the electronic device (300) may identify at least one target object requiring additional shooting among the plurality of objects.

[0118] In one embodiment, the electronic device (300) can classify each of the plurality of objects (711, 712, 713, 714, 715) included in the first image (710) of FIG. 7A into one of a core object, a primary object, and a non-core object.

[0119] For example, the electronic device (300) may classify object B (712) and object D (714) among object A (711), object B (712), object C (713), object D (714), and object E (715) as key objects, classify object A (711) and object E (715) as key objects, and classify object C (713) as non-key objects.

[0120] For example, a core object may refer to an object with the highest weight (or highest priority) among a plurality of objects. A primary object may refer to an object with the second-highest weight (or middle priority) among the plurality of objects after the core object. A non-primary object may refer to an object with the lowest weight (or lowest priority) among the plurality of objects. Non-primary objects may be any object other than the core object and the primary object.

[0121] According to one embodiment, the electronic device (300) may identify a key object among key objects, primary objects, and non-primary objects as a target object requiring additional shooting. The key object may refer to an object with a second highest weight (or medium priority) after the key object.

[0122] For example, the electronic device (300) can identify object A (711) and object E (715), which are classified as main objects among a plurality of objects (711, 712, 713, 714, 715) in the first image (710) of FIG. 7a, as target objects requiring additional shooting.

[0123] In operation 640, the electronic device (300) can determine whether the object completeness (or object accuracy) of the primary object identified through operation 630 exceeds a specified percentage (a%, e.g., a=40).

[0124] In one embodiment, the electronic device (300) estimates the size of the overall shape of the main object based on the features (e.g., location, size, shape, etc.) of a portion of the shape included in the first image that was first captured, and determines the object completeness of the main object based on the estimation. In one embodiment, the electronic device (300) collects reference images that include objects similar to the portion of the shape through a web search, and determines the object completeness of the main object through a comparison with the reference images. In one embodiment, the electronic device (300) may collect reference images or determine the object completeness of the main object by additionally considering the location information of the electronic device (300).

[0125] If the judgment result of operation 640 shows that the object completion rate of the main object is below a specified ratio, the electronic device (300) can proceed to operation 650.

[0126] In operation 650, the electronic device (300) can adjust the shooting range (or camera shooting range) of the camera module (310) from the first shooting range to the second shooting range to perform additional shooting of the main object. For example, the electronic device (300) can adjust the camera shooting range for additional shooting through an operation of changing the folding angle of the electronic device (300) and / or an operation of changing a shooting camera (an activated or selected camera) among a plurality of cameras having different angles of view or positions. For example, the electronic device (300) can change the shooting camera from a wide-angle camera to an ultra-wide-angle camera so that the additional shooting area becomes wider than the primary shooting area, or can change the camera angle of view or folding angle so that the additional shooting direction faces the main object.

[0127] In operation 660, the electronic device (300) may acquire an additional image of the at least one target object through the camera module (310). The additional image may be an image additionally photographed (or captured) by the camera module (310) while the photographing range of the camera module (310) is adjusted to a second photographing range. For example, the additional image may be the first additional image (720) and / or the second additional image (730) of FIG. 7B. The first additional image (720) of FIG. 7B may be an image photographed at a second folding angle. The second additional image (730) of FIG. 7B may be an image photographed by an ultra-wide-angle camera.

[0128] In operation 670, the electronic device (300) may generate a second image based on the first image acquired through operation 610 and the additional image acquired through operation 660. For example, the second image may be the secondary output image (760) of FIG. 7d.

[0129] In operation 680, the electronic device (300) may display the second image generated through operation 670 and / or a user interface related to the second image through the display (320).

[0130] FIG. 7a is a diagram illustrating an example of an input image captured by an electronic device (300) according to one embodiment.

[0131] In Fig. 7a, reference number 710 represents an input image, reference number 711 represents object A, reference number 712 represents object B, reference number 713 represents object C, reference number 714 represents object D, and reference number 715 represents object E.

[0132] Referring to FIG. 7A, the electronic device (300) can capture an input image (710) according to a user operation. For example, the electronic device (300) can acquire the input image (710) by performing a first capture using a wide-angle camera while the camera angle and / or folding angle have a first angle.

[0133] The electronic device (300) can detect object A (711), object B (712), object C (713), object D (714), and object E (715) from the input image (710).

[0134] The electronic device (300) may determine that among the five objects (711, 712, 713, 714, 715) included in the input image (710), object B (712) and object D (714) are key objects that are in focus or are person objects, and that their entire shapes are included without object clipping. The electronic device (300) may determine that object A (711) and object E (715) are key objects related to the current location (or place) of the electronic device (300), and that only partial shapes are included due to object clipping, and that additional shooting of object A (711) and object E (715) is necessary. The electronic device (300) may determine that object C (713) is a non-key object that has a general shape.

[0135] FIG. 7b is a diagram illustrating an example of an additional image captured by an electronic device (300) according to one embodiment.

[0136] In Fig. 7b, reference numeral 710 represents an input image, reference numeral 720 represents a first additional image, and reference numeral 730 represents a second additional image.

[0137] Referring to FIG. 7b, the electronic device (300) can perform additional shooting of object A (711) and object E (715) among five objects (711, 712, 713, 714, 715) included in the input image (710). The electronic device (300) can obtain one or more additional images, for example, a first additional image (720) and / or a second additional image (730), through the additional shooting.

[0138] For example, the electronic device (300) can acquire a first additional image (720) by performing additional shooting (or secondary shooting) after changing the camera angle of view and / or folding angle from the first angle to the second angle. The first additional image (720) can include more of the first shooting area (721) for object A (711) and the second shooting area (725) for object E (715) than the input image (710). The first shooting area (721) for object A (711) can include a partial shape of object A (711) that is not included in the input image (710). The second shooting area (725) for object E (715) can include a partial shape of object E (715) that is not included in the input image (710).

[0139] For example, the electronic device (300) can acquire a second additional image (730) by performing additional shooting after changing the shooting camera from a wide-angle camera to an ultra-wide-angle camera. The second additional image (730) can include more of the first shooting area (731) for object A (711) and the second shooting area (735) for object E (715) than the input image (710). The first shooting area (731) for object A (711) can include a partial shape of object A (711) that is not included in the input image (710). The second shooting area (735) for object E (715) can include a partial shape of object E (715) that is not included in the input image (710).

[0140] FIG. 7c is a diagram illustrating an example of a primary output image generated by an electronic device (300) according to one embodiment.

[0141] In Fig. 7c, reference numeral 710 represents an input image, reference numeral 730 represents a second additional image, and reference numeral 740 represents a first output image.

[0142] Referring to FIG. 7c, the electronic device (300) can generate a first output image (740) by synthesizing (or combining) the input image (710) and the second additional image (730) (and / or the first additional image (720)).

[0143] FIG. 7d is a diagram illustrating an example of a secondary output image generated by an electronic device (300) according to one embodiment.

[0144] In Fig. 7d, reference numeral 740 represents the first output image, and reference numeral 760 represents the second output image.

[0145] Referring to FIG. 7d, the primary output image (740) may have a different aspect ratio (image ratio or image size) than the input image (710). To match the aspect ratio of the primary output image (740) with that of the input image (710), addition of new regions (751, 752) may be required.

[0146] The electronic device (300) can generate a secondary output image (760) having the same aspect ratio as the input image (710) by adding a first region (761) and a second region (762) to the primary output image (740). The secondary output image (760) can include the primary output image (740), the first region (761), and the second region (762). For example, the primary output image (740) in the secondary output image (760) can be an actual captured region. The first region (761) and the second region (762) in the secondary output image (760) can be virtual regions generated based on the input image (710) and / or the primary output image (740).

[0147] FIG. 7e is a diagram illustrating another example of a secondary output image generated by an electronic device (300) according to one embodiment.

[0148] In Fig. 7e, reference numeral 710 represents an input image, reference numerals 770 and 780 represent secondary output images.

[0149] Referring to FIG. 7e, the electronic device (300) can apply various visual effects (e.g., object resizing, filter effects, post-processing for natural image synthesis, adding augmented objects) to the secondary output images (770, 780).

[0150] According to one embodiment, the electronic device (300) can maintain or adjust the sizes of object B (712) and object D (714), which are determined as key objects, according to the aspect ratio of the secondary output images (770, 780). For example, the electronic device (300) can reduce the sizes of object B (712) and object D (714) to match the aspect ratio of the secondary output image (770), and generate a secondary output image (780) including the reduced object B (772) and the reduced object D (774). For example, the electronic device (300) can maintain the sizes of object B (782) and object D (784) in the secondary output image (780) to be the same as the sizes of object B (712) and object D (714) in the input image (710).

[0151] According to one embodiment, the electronic device (300) may be an electronic device of a type whose form factor can be transformed. According to one embodiment, the electronic device (300) may be a foldable electronic device (800). FIG. 8A is a diagram illustrating an unfolded state of a foldable electronic device (800) according to one embodiment. FIG. 8B is a diagram illustrating a folded state of a foldable electronic device (800) according to one embodiment.

[0152] Referring to FIGS. 8A and 8B, a foldable electronic device (800) may include a foldable housing (801) and a flexible (or foldable) display (830) disposed within a space formed by the foldable housing (801).

[0153] According to one embodiment, the foldable housing (801) may include a first housing (810) and a second housing (820).

[0154] According to one embodiment, the first housing (810) and / or the second housing (820) may form at least a portion of the exterior of the foldable electronic device (800). According to one embodiment, the surface on which the flexible display (830) is visually exposed is defined as the front surface of the foldable electronic device (800) (e.g., the first front surface (810a) and the second front surface (820a)). And, the surface opposite to the front surface is defined as the back surface of the foldable electronic device (800) (e.g., the first back surface (810b) and the second back surface (820b)). In addition, the surface surrounding at least a portion of the space between the front surface and the back surface is defined as the side surface of the foldable electronic device (800) (e.g., the first side surface (810c) and the second side surface (820c)).

[0155] According to one embodiment, the first housing (810) and the second housing (820) are connected by a hinge in the hinge housing (840) and can rotate relative to each other about a folding axis (A) by the hinge. Accordingly, the first housing (810) and / or the second housing (820) of the foldable electronic device (800) can be folded or unfolded about the folding axis (A). In addition, the folding angle of the foldable electronic device (800) and / or the state of the foldable electronic device (800) (e.g., the unfolded state of FIG. 8A, the folded state of FIG. 8B, or an intermediate state between the unfolded state and the folded state) can be changed.

[0156] In one embodiment, the first housing (810) can provide relative motion to the second housing (820), and the second housing (820) can provide relative motion to the first housing (810).

[0157] According to one embodiment, the first housing (810) can rotate relative to the second housing (820) using a hinge. Accordingly, the foldable electronic device (800) can be changed into a folded state (e.g., FIG. 8b) or an unfolded state (e.g., FIG. 8a). In the folded state, the first front surface (810a) can face the second front surface (820a), and in the unfolded state, the direction in which the first front surface (810a) faces can be the same as the direction in which the second front surface (820a) faces. For example, in the unfolded state, the first front surface (810a) can be positioned on substantially the same plane as the second front surface (820a).

[0158] According to one embodiment, the first housing (810) and the second housing (820) are arranged on both sides with respect to the folding axis (A) as the center, and may have a shape that is overall symmetrical with respect to the folding axis (A). According to one embodiment, the angle between the first housing (810) and the second housing (820) may change depending on whether the state of the foldable electronic device (800) is an unfolded state, a folded state, or an intermediate state between the unfolded state and the folded state. The angle between the first housing (810) and the second housing (820) may be interpreted as a folding angle of the foldable electronic device (800).

[0159] According to one embodiment, the foldable electronic device (800) may include a hinge housing (840). The hinge housing (840) may be disposed between the first housing (810) and the second housing (820). According to one embodiment, the hinge housing (840) may be covered by a portion of the first housing (810) and the second housing (820), or may be exposed to the outside of the foldable electronic device (800), depending on the state of the foldable electronic device (800). According to one embodiment, the hinge housing (840) may protect a hinge that rotatably connects the first housing (810) and the second housing (810) from an external impact of the foldable electronic device (800). The hinge housing (840) may be interpreted as a hinge cover for protecting the hinge.

[0160] According to one embodiment, the angle or distance between the first housing (810) and the second housing (820) may vary depending on whether the state of the foldable electronic device (800) is an unfolded state (extended state, flat state) (or opened state), a folded state (or closed state), or an intermediate state.

[0161] According to one embodiment, the degree of external exposure of the hinge housing (840) may vary depending on the state of the foldable electronic device (800). For example, as illustrated in FIG. 8A, when the foldable electronic device (800) is in an unfolded state, the hinge housing (840) may be covered by the first housing (810) and the second housing (820) and may not be exposed. For example, as illustrated in FIG. 8B, when the foldable electronic device (800) is in a folded state (e.g., a fully folded state), the hinge housing (840) may be exposed to the outside between the first housing (810) and the second housing (820). For example, when the first housing (810) and the second housing (820) are in an intermediate state where they are folded at a certain angle, the hinge housing (840) may be partially exposed to the outside between the first housing (810) and the second housing (820). However, in this case, the exposed area may be less than that in the fully folded state. In one embodiment, the hinge housing (840) may include a curved surface.

[0162] In one embodiment, the flexible display (830) may have a structure in which at least some areas can be transformed into a flat or curved surface. For example, the flexible display (830) may be formed to vary in response to the relative movement of the second housing (820) with respect to the first housing (810). According to one embodiment, the flexible display (830) may include a folding area (833), a first display area (831) disposed on one side (e.g., above (in the +Y direction) of the folding area (833) as shown in FIG. 8A) with respect to the folding area (833), and a second display area (832) disposed on the other side (e.g., below (in the -Y direction) of the folding area (833) as shown in FIG. 8A). According to one embodiment, the folding area (833) may be positioned on a hinge that rotatably connects the first housing (810) and the second housing (820). According to one embodiment, the first display area (831) may be disposed on the first housing (810), and the second display area (832) may be disposed on the second housing (820). According to one embodiment, the flexible display (830) may include the first housing (810) and the second It can be accommodated in the housing (820).

[0163] However, the division of the regions of the flexible display (830) illustrated in FIG. 8A is exemplary, and the flexible display (830) may be divided into a plurality of regions (for example, four or more or two) depending on the structure or function. For example, in the embodiment illustrated in FIG. 8A, the regions of the flexible display (830) may be divided by a folding region (833) extending parallel to the X-axis or a folding axis (A), but in other embodiments, the regions of the flexible display (830) may be divided based on another folding axis (for example, a folding axis parallel to the Y-axis). According to one embodiment, the flexible display (830) may be combined with or disposed adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a digitizer configured to detect a magnetic field-type stylus pen.

[0164] According to one embodiment, the flexible display (830) of the foldable electronic device (800) may be a front display. The foldable electronic device (800) may further include a rear display (834). The rear display (834) may be arranged to face a different direction than the flexible display (830). For example, the flexible display (830) may be visually exposed through the front side (e.g., the first front side (810a) and / or the second front side (820a)) of the foldable electronic device (800), and the rear display (834) may be visually exposed through the rear side (e.g., the first rear side (810b)) of the foldable electronic device (800).

[0165] According to one embodiment, the foldable electronic device (800) may include at least one camera (804, 806). According to one embodiment, the foldable electronic device (800) may include a front camera (804) exposed through the front (e.g., the first front (810a)) and / or a rear camera (806) exposed through the rear (e.g., the first rear (810b)). The cameras (804, 806) may correspond to the camera module (180) of FIG. 2 or the camera module (310) of FIG. 3.

[0166] According to some embodiments, the photographing range of at least one camera (804, 806) may vary depending on the folding angle of the foldable electronic device (800). For example, when the first housing (810) and / or the second housing (820) rotates around the folding axis (A), the rotation may cause physical movement of at least one camera (804, 806). Accordingly, the photographing range of at least one camera (804, 806) of the foldable electronic device (800) may vary.

[0167] Hereinafter, the operation of the first housing (810) and the second housing (820) and each area of ​​the flexible display (830) according to the operating state (e.g., unfolded state and folded state) of the foldable electronic device (800) will be described.

[0168] In one embodiment, when the foldable electronic device (800) is in an unfolded state (e.g., a fully unfolded state in FIG. 8A), the first housing (810) and the second housing (820) form a substantially 180-degree angle, and the first front surface (810a) and the second front surface (820a) of the flexible display (830) may be arranged to face the same direction, for example, to display a screen in a direction parallel to each other. In addition, the folding area (833) may form the same plane as the first front surface (810a) and the second front surface (820a).

[0169] In one embodiment, when the foldable electronic device (800) is in a folded state (e.g., the folded state of FIG. 8B), the first housing (810) and the second housing (820) may be arranged to face each other. For example, when the foldable electronic device (800) is in a folded state (e.g., the folded state of FIG. 8B), the first front surface (810a) and the second front surface (820a) of the flexible display (830) may form a narrow angle (e.g., between 0 and 10 degrees) with each other and face each other. When the foldable electronic device (800) is in a folded state (e.g., the state of FIG. 2), the folding area (833) may form a curved surface having at least a predetermined curvature.

[0170] In one embodiment, when the foldable electronic device (800) is in an intermediate state (e.g., an intermediate state between the fully unfolded state of FIG. 8A and the folded state of FIG. 8B), the first housing (810) and the second housing (820) may be arranged to form a certain angle with respect to each other, for example, a 90 degree or 120 degree angle. For example, in the intermediate state, the first front surface (810a) and the second front surface (820a) of the flexible display (830) may form an angle that is greater than the angle in the folded state and less than the angle in the unfolded state. The folding area (833) may be formed as a curved surface having at least a certain curvature, and the curvature at this time may be less than that in the folded state.

[0171] FIGS. 9A and 9B are drawings for explaining changes in the shooting range according to the folding angle of a foldable electronic device (800) according to one embodiment.

[0172] According to one embodiment, the foldable electronic device (800) may include a sensor (910) for detecting a folding angle and an active hinge (920) for adjusting the folding angle. In one embodiment, the folding angle may refer to an angle (or intersection angle) formed when the first housing (810) and the second housing (820) meet.

[0173] In one embodiment, the sensor (910) may be included in the sensor module (176) of FIG. 1. For example, the sensor (910) may include at least one of an acceleration sensor, a gyro sensor, or a Hall sensor.

[0174] In one embodiment, the active hinge (920) may be configured to support physical movement when the foldable electronic device (800) is folded or unfolded. The active hinge (920) may implement automatic folding or automatic unfolding of the foldable electronic device (800) under the control of a processor (e.g., the processor (330) of FIG. 3 ). The active hinge (920) may include an actuator. The actuator may operate within the active hinge (920) to control the movement of the hinge, thereby adjusting the folding angle.

[0175] According to one embodiment, the foldable electronic device (800) (e.g., the processor (330) of FIG. 3) can capture (first capture) a first image (901) for a first capturing range through at least one camera (e.g., a rear camera (806)) while being folded at a first folding angle (α). The first image (901) may be an image for a first capturing range corresponding to the first folding angle (α) (e.g., a first angle for wide-angle shooting or first-direction shooting, a folding angle matching an angle of view of 85 degrees). The foldable electronic device (800) can identify a target object requiring additional shooting from the first image (901). The foldable electronic device (800) can change the folding angle from the first folding angle (α) to the second folding angle (β) by driving the active hinge (920) to capture additional shots of the target object. The foldable electronic device (800) can capture an additional image (902) (secondary capture or additional capture) while folded at a second folding angle (β). The additional image (902) can be an image for a second capturing range corresponding to the second folding angle (α) (e.g., a first angle for ultra-wide-angle capture or second-direction capture, a folding angle matching a 120-degree field of view).

[0176] In one embodiment, the additional image (902) captured at the second folding angle (β) may be an image whose base line is warped (e.g., corrected, rotated, or image post-processed) by an offset angle (α-β) compared to the first image (901) captured at the first folding angle (α). The foldable electronic device (800) may perform warping so that the base line of the additional image (902) matches the base line of the first image (901) based on the offset angle (α-β).

[0177] According to one embodiment, the foldable electronic device (800) can operate in a flex mode. According to one embodiment, when the state of the foldable electronic device (800) is an intermediate state between an unfolded state (e.g., the unfolded state of FIG. 8A) and a folded state (e.g., the folded state of FIG. 8B), or when the folding angle of the foldable electronic device (800) falls within a specified angle range (e.g., more than 30 degrees and less than or equal to 150 degrees), the foldable electronic device (800) can operate in the flex mode. For example, the flex mode may be a mode for supporting convenient shooting in a stationary state in which the foldable electronic device (800) is folded at a specific angle within the specified angle range. While operating in the flex mode, the foldable electronic device (800) can automatically adjust the shooting range of at least one camera (e.g., the rear camera (806)) by changing the folding angle through the active hinge (920).

[0178] FIG. 10 is a diagram illustrating an example of a user interface for providing an image capturing function according to one embodiment.

[0179] Referring to FIG. 10, the electronic device (300) may display a user interface such as a first screen (1010). In one embodiment, the first screen (1010) may be configured to provide an artificial intelligence-based image capturing function. For example, the first screen (1010) may be an execution screen of an application (e.g., a gallery application, a camera application).

[0180] The first screen (1010) may include a first screen area (1030), a second screen area (1020), a third screen area (1040), and a fourth screen area (1050).

[0181] The electronic device (300) can display a second image (AI extended image) generated according to an artificial intelligence-based image capturing function through the first screen area (1030).

[0182] The electronic device (300) can display thumbnail images of a first image (original image) and a third image captured according to a user's intention through a second screen area (1020). The third image may be another version of the second image displayed on the first screen area (1030), for example, an image (another AI-enhanced image) that has visual effects (e.g., object resizing, filter effects, post-processing for natural image synthesis, addition of augmented objects) applied to the second image.

[0183] The first screen area (1030) may display an AI shooting icon (1031) for turning on / off an artificial intelligence-based image shooting function. For example, when the user touches the shooting button or runs the gallery application while the AI ​​shooting icon (1031) is activated, the electronic device (300) may provide a second image (AI expanded image) that includes an extended area in addition to the first image (original image) captured according to the user's intention.

[0184] When the AI ​​shooting icon (1031) displayed on the first screen area (1030) is deactivated or a swipe input (1032) is detected, the electronic device (300) can move the first image (original image) or the third image (another AI extended image) displayed as a thumbnail on the second screen area (1020) to the first screen area (1030) and display it.

[0185] The electronic device (300) can display information about visual effects applied to the second image (AI extended image) displayed on the first screen area (1030) through the third screen area (1040). The electronic device (300) can display an input window for artificial intelligence prompt input on the fourth screen area (1050). The electronic device (300) can receive a user command through the input window and additionally apply a visual effect according to the user command to the second image (AI extended image) displayed on the first screen area (1030).

[0186] FIG. 11 is a diagram illustrating an example of a user interface for guiding additional shooting according to one embodiment.

[0187] In one embodiment, the electronic device (300) may be a fixed-form-factor type electronic device. In this case, the electronic device (300) may display a user interface to guide additional shooting through the display (320).

[0188] For example, the electronic device (300) may sequentially display a first preview screen (1110), a second preview screen (1120), and a third preview screen (1130) to provide an artificial intelligence-based image capturing function.

[0189] In operation 1101, the electronic device (300) may display a first image (original image) captured initially according to the user's intention on a first preview screen (1110). The first image may be an image corresponding to a first angle of view.

[0190] In operation 1102, the electronic device (300) may display an interface element (1121) on the second preview screen (1120) that induces a change in the angle of view for additional shooting (e.g., a guide message saying, "Please raise the phone higher to change the angle of view for AI selfie shooting."). The angle of view may change from the first angle of view to the second angle of view depending on the movement of the electronic device (300). Through additional shooting, an additional image corresponding to the second angle of view may be acquired.

[0191] In operation 1103, the electronic device (300) may generate a second image (AI extended image) based on the first image captured initially and the additional image captured additionally (or secondarily). The electronic device (300) may display the second image (AI extended image) through the third preview screen (1130).

[0192] FIG. 12a is a diagram for explaining a first artificial intelligence model (1210) according to one embodiment. FIG. 12b is a diagram for explaining a second artificial intelligence model (1250) according to one embodiment.

[0193] According to one embodiment, the electronic device (300) may provide an artificial intelligence-based image capturing function using the first artificial intelligence model (1210) and / or the second artificial intelligence model (1250).

[0194] Referring to FIG. 12a, the first artificial intelligence model (1210) may be an artificial intelligence model trained for the purpose of detecting multiple objects within a single image and / or identifying at least one target object requiring additional shooting among a plurality of objects within the image.

[0195] In one embodiment, the electronic device (300) may transmit first image information (1220) regarding the first image as an input to a first artificial intelligence model (1210) trained to identify at least one target object requiring additional shooting. For example, the first image information (1220) may include at least a portion of the first image, an image corresponding to the first image (e.g., a thumbnail, a preview), partial images of objects in the first image, and metadata (e.g., links, formats, properties, etc.) regarding the first image and / or the partial images.

[0196] In one embodiment, the first artificial intelligence model (1210) can generate object-related information (1230) based on first image information (1220) for the first image. For example, the first artificial intelligence model (1210) can detect a background of the first image and multiple objects distinct from the background using the first image information (1220), analyze the detected objects, and generate object-related information (1230) based on the analysis results. The first artificial intelligence model (1210) can provide a response including the object-related information (1230).

[0197] In one embodiment, the object-related information (1230) may include at least some of object-specific identification information (e.g., object-specific name or ID, identification number), object-specific location (or area) information within the image, object-specific weight information, object-specific category information, and object-specific completeness information. The object-specific category information may be information indicating which category each object belongs to among a plurality of categories (e.g., core object, main object, non-main object). The object-specific completeness information may be information indicating whether each object is a complete object (or a cropped object) in which a substantial entire shape (e.g., 70% or more) is included within the image.

[0198] In one embodiment, the object-related information (1230) may include information about a target object that requires additional shooting among a plurality of objects included in the first image (e.g., category information indicating that it is a main object, object completeness information indicating that it is a cropped object, and information indicating that it is highly related to the current location).

[0199] According to one embodiment, the electronic device (300) may perform an operation (e.g., operation 420 of FIG. 4) of identifying at least one target object that requires additional shooting among a plurality of objects included in the first image based on object-related information (1230) provided from the first artificial intelligence model (1210).

[0200] For example, the electronic device (300) may identify, based on object-related information (1230), an object belonging to a main object category among a plurality of objects in the first image, an object having an object completeness ratio below a specified ratio (e.g., 70%), and / or an object having a high relevance to the current location, and determine the identified object(s) as a target object requiring additional shooting.

[0201] Referring to FIG. 12b, the second artificial intelligence model (1250) may be a generative artificial intelligence model trained for the purpose of generating an output image including an extended region using an input image.

[0202] In one embodiment, the second artificial intelligence model (1250) can generate a second image (1270), which is an output image, based on a prompt input (or request) (1260) that includes first image information (1261) for the first image, which is an input image. For example, the second artificial intelligence model (1250) can receive a prompt input (1260) that includes first image information (1261) for the first image, which is an input image (e.g., input image (510) of FIG. 5 and input image (710) of FIG. 7A), and an input script (1262). In one embodiment, the input script (1262) can include the first input script and / or the second input script. The first input script can include a request for generating an extended area of ​​the first image (e.g., size, ratio, resolution, format for an area to be out-painted). The second input script may include object-related information necessary for generating an expanded region of the first image (e.g., information about the main object, cropped objects, or partial images), and / or examples of reference images for generating the expanded region.

[0203] According to one embodiment, the second artificial intelligence model (1250) may generate an output image of a specified type (or specified format) (e.g., the output image (520) of FIG. 5, the secondary output image (760) of FIG. 7d) based on the received prompt input (1260), and provide a response including the output image.

[0204] In one embodiment, the second AI model (1250) can generate a second image (1270) that is an extension from the first image by performing outpainting (or upscaling or rendering for the extension region) based on a prompt input (1260) that includes first image information (1261) and an input script (1262) for the first image. The second image (1270) can include an extension region generated based on the first image information (1261) for the first image. For example, the extension region can be a virtual region generated based on the first image information (1261) for the first image. The second AI model (1250) can generate the second image (1270) that is an extended version of the first image so as to be continuous with the first image and satisfy the request of the input script (1262).

[0205] In one embodiment, the second artificial intelligence model (1250) may be a generative artificial intelligence model trained for the purpose of generating an output image using an input image and an additional image (e.g., two images each containing a portion of the shape of the same object and another portion of the shape).

[0206] In one embodiment, the electronic device (300) transmits a prompt input (1260) including first image information (1261) for a first image (an input image for a target object) and additional image information (not shown) for an additional image (an additional image for the same target object) to a second artificial intelligence model (1250), thereby obtaining a second image (1270) generated based on the first image information (1261) and the additional image information from the second artificial intelligence model (1250).

[0207] According to one embodiment, the prompt input (1260) for the second artificial intelligence model (1250) may further include additional image information other than the first image information (1261). For example, the second artificial intelligence model (1250) may generate a more sophisticated extended version of the second image (1270) by using the first image information (1261) for the first image and the additional image information for the additional image together to generate an extended area. For example, if the primary object captured in the first image corresponding to the first image information (1261) is a cropped object and the object completeness is below a specified ratio (e.g., 70%), the second artificial intelligence model (1250) may request additional image information for an additional image captured in a second time of the primary object. For example, the additional image corresponding to the additional image information may include a partial shape of the entire shape of the primary object that is not included in the first image. The second artificial intelligence model (1250) may receive additional image information and generate an extended area to be included in the second image (1270) based on the first image information (1261) and the additional image information. For example, the extended area may include an actual shooting area and / or a virtual area. The actual shooting area may be an area cropped from the additional image and including a partial shape of the entire shape of the main object that is not included in the first image that was first captured. The virtual area may be an area processed based on at least one of a second captured additional image or an actual shooting area cropped from the second captured additional image. The second artificial intelligence model (1250) may generate a second image (1270) that includes the extended area in the first image and provide a response that includes the second image (1270).

[0208] According to one embodiment, the electronic device (300) may perform an operation (e.g., operation 450 of FIG. 4, operation 670 of FIG. 6) of generating a second image (1270) extended from a first image using a second artificial intelligence model (1250).

[0209] According to one embodiment, the first artificial intelligence model (1210) of FIG. 12a and / or the second artificial intelligence model (1250) of FIG. 12b may be implemented in hardware and / or software to perform a predetermined function.

[0210] According to one embodiment, the electronic device (300) of FIG. 3 may be configured to include at least a portion of the first artificial intelligence model (1210) of FIG. 12A or the second artificial intelligence model (1250) of FIG. 12B. The first artificial intelligence model (1210) and / or the second artificial intelligence model (1250) may be included in the electronic device (300) (e.g., memory (340)) in the form of on-device artificial intelligence models, but is not limited thereto. For example, at least a portion of the first artificial intelligence model (1210) or the second artificial intelligence model (1250) may be included in a generative AI server (e.g., server (108) of FIG. 1) and / or an electronic device of another user (e.g., electronic devices (102, 104) of FIG. 1).

[0211] FIG. 13 is a diagram for explaining an image correction operation of an artificial intelligence model (e.g., the second artificial intelligence model (1250) of FIG. 12b) according to one embodiment.

[0212] According to one embodiment, in the process of generating a second image (output image, AI extended image) using a first image (input image) that was initially captured and an additional image that was additionally captured, warping (e.g., image correction) may be required. For example, when performing a warping operation to align the reference line of the additional image with the reference line of the first image, a blank area (1311) may be generated in the first output image (1310) due to a difference in the field of view of the additional image and the second image.

[0213] In this case, the electronic device (300) can generate a secondary output image (1320) by performing out-painting to fill in the blank area (1311) using an artificial intelligence model (e.g., the second artificial intelligence model (1250) of FIG. 12b).

[0214] FIG. 14 is a diagram illustrating a system including a generative artificial intelligence model according to one embodiment.

[0215] Referring to FIG. 14, the User Query / Response Interface (1410) can receive a user's input. The user's input may be in the form of natural language, images, and / or videos. Furthermore, context information may also be transmitted when the user's input is transmitted. Context information may include various additional information at the time of user input. For example, information on the application currently being used by the user or information on the user's location. Furthermore, the user's input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user's input may also be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (1410) can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query Interface can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user.

[0216] The AI ​​framework (1420) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0217] User input received from the User Query / Response Interface (1410) can be transmitted to the Prompt design component (1421). The Prompt design component (1421) can be used to generate prompts suitable for inputting the user input into a Large Language Model (LLM) or a Large Multimodal Model (LMM). The Prompt design component (1421) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The Prompt design component (1421) can access a knowledge component (e.g., knowledge repositories (1440)) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0218] The API / Plug-in management component (1423) can communicate with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (1423) can establish a channel for communicating with the outside of the AI ​​Interface through the API, and can enable access to various data sources (e.g., knowledge repositories (1440)) through the established channel. In addition, if the API / Plug-in management component (1423) needs to perform an action that performs the user input as a final result rather than an intermediate result in an application or service, it can request the action to the application / service component (1430) through the API. Information obtained from an external source can be used to generate a prompt in the prompt design component (1421) together with the user input, or can be passed as input to the generative model.

[0219] The Refiner component (e.g., the output modification component (1425)) can fine-tune the output from a generative model. For example, the Refiner component can verify that the content generated by the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the Refiner component can determine the degree to which the output matches the user's desired result and, if necessary, perform additional processing. The Refiner component can also configure and provide users with hints to avoid undesirable output.

[0220] Generative AI Model (1450) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. Generative AI Model (1450) can include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0221] According to one embodiment, the electronic device (101) of FIG. 1 and / or the electronic device (300) of FIG. 3 may be configured to include at least a portion of the User Query / Response Interface (1410), the AI ​​framework (1420), the application / service component (1430), the knowledge repositories (1440), or the Generative AI Model (1450) of FIG. 14. According to one embodiment, at least a portion of the User Query / Response Interface (1410), the AI ​​framework (1420), the application / service component (1430), the knowledge repositories (1440), or the Generative AI Model (1450) of FIG. 14 may be included in another electronic device (e.g., another user's electronic device (e.g., the electronic device (102, 104) of FIG. 1) and / or a server (e.g., the server (108) of FIG. 1).

[0222] An electronic device according to one embodiment (e.g., electronic device (300) of FIG. 3) may include a display (e.g., display (320) of FIG. 3), a camera module (e.g., camera module (310) of FIG. 3), at least one processor (e.g., processor (330) of FIG. 3), and a memory (e.g., memory (340) of FIG. 3) that stores instructions. The above instructions, when executed by the at least one processor, may cause the electronic device to acquire a first image through the camera module, identify at least one target object requiring additional shooting among a plurality of objects included in the first image, adjust a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification, acquire an additional image of the at least one target object through the camera module, generate a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image, and display the second image through the display.

[0223] According to one embodiment, the at least one target object may correspond to at least one of a second object around a focused first object among the plurality of objects included in the first image, a second object around a first object located at the center of the first image, a second object located at the periphery of the first image, a second object including only a partial shape without an entire shape, or a second object related to location information of the electronic device.

[0224] According to one embodiment, the extended area associated with the at least one target object may include at least one of an actual captured area cropped from the additional image and including a partial shape of the entire shape of the at least one target object that is not included in the first image, or a virtual area generated based on at least one of the additional image or the actual captured area.

[0225] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to detect the plurality of objects from the first image and identify an object among the plurality of objects having an object completeness lower than a specified ratio as a target object requiring additional shooting.

[0226] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to detect the plurality of objects from the first image, assign weights to the plurality of objects, and identify a target object requiring additional imaging among the plurality of objects based on the assigned weights.

[0227] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to classify each of the plurality of objects into one of a first object having a first weight, a second object having a second weight lower than the first weight, and a third object having a third weight lower than the second weight, based on the assigned weights, and to identify the second object as a target object requiring additional imaging.

[0228] According to one embodiment, the electronic device may be a foldable electronic device. The instructions, when executed by the at least one processor, may cause the electronic device to acquire the first image for the first shooting range through the first camera in the camera module, change the shooting range of the first camera from the first shooting range to the second shooting range by changing the folding angle, and acquire the additional image for the second shooting range through the first camera.

[0229] According to one embodiment, the foldable electronic device may automatically change the shooting range of the first camera by changing the folding angle while operating in the flex mode.

[0230] According to one embodiment, the camera module may include a plurality of cameras having different angles of view. The instructions, when executed by the at least one processor, may cause the electronic device to acquire the first image for the first shooting range through the activated first camera while a first camera having a first angle of view among the plurality of cameras is activated, activate a second camera having a second angle of view among the plurality of cameras, and acquire the additional image for the second shooting range through the activated second camera.

[0231] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to obtain object-related information about at least one target object among a plurality of objects included in the first image by transferring first image information about the first image to a first artificial intelligence model trained to identify the at least one target object, and to obtain the second image generated based on the first image and the additional image by transferring the first image information about the first image and additional image information about the additional image to a second artificial intelligence model trained to generate the second image.

[0232] According to one embodiment, a method of operating an electronic device (e.g., the electronic device (300) of FIG. 3) may include an operation of acquiring a first image through a camera module (e.g., the camera module (310) of FIG. 3), an operation of identifying at least one target object requiring additional shooting among a plurality of objects included in the first image, an operation of adjusting a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification, an operation of acquiring an additional image of the at least one target object through the camera module, an operation of generating a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image, and an operation of displaying the second image through a display (e.g., the display (320) of FIG. 3).

[0233] According to one embodiment, the at least one target object may correspond to at least one of a second object around a focused first object among the plurality of objects included in the first image, a second object around a first object located at the center of the first image, a second object located at the periphery of the first image, a second object including only a partial shape without an entire shape, or a second object related to location information of the electronic device.

[0234] According to one embodiment, the extended area associated with the at least one target object may include at least one of an actual captured area cropped from the additional image and including a partial shape of the entire shape of the at least one target object that is not included in the first image, or a virtual area generated based on at least one of the additional image or the actual captured area.

[0235] According to one embodiment, the operation of identifying at least one target object requiring additional shooting among the plurality of objects may include the operation of detecting the plurality of objects from the first image, and the operation of identifying an object among the plurality of objects, the object completeness of which is lower than a specified ratio, as the target object requiring additional shooting.

[0236] According to one embodiment, the operation of identifying at least one target object requiring additional shooting among the plurality of objects may include the operation of detecting the plurality of objects from the first image, the operation of assigning weights to the plurality of objects, and the operation of identifying the target object requiring additional shooting among the plurality of objects based on the assigned weights.

[0237] According to one embodiment, the operation of identifying the target object may include an operation of classifying each of the plurality of objects into one of a first object having a first weight, a second object having a second weight lower than the first weight, and a third object having a third weight lower than the second weight, based on the assigned weights, and an operation of identifying the second object as a target object requiring additional shooting.

[0238] According to one embodiment, the electronic device may be a foldable electronic device. The operation of adjusting the shooting range of the camera module may include an operation of acquiring the first image for the first shooting range through a first camera in the camera module, an operation of changing the shooting range of the first camera from the first shooting range to the second shooting range by changing a folding angle, and an operation of acquiring the additional image for the second shooting range through the first camera.

[0239] According to one embodiment, the foldable electronic device may automatically change the shooting range of the first camera by changing the folding angle while operating in the flex mode.

[0240] According to one embodiment, the camera module may include a plurality of cameras having different angles of view. The operation of adjusting the shooting range of the camera module may include an operation of acquiring the first image for the first shooting range through the activated first camera while a first camera having a first angle of view among the plurality of cameras is activated, and an operation of activating a second camera having a second angle of view among the plurality of cameras and acquiring the additional image for the second shooting range through the activated second camera.

[0241] According to one embodiment, the operation of identifying at least one target object requiring additional shooting among the plurality of objects may include an operation of obtaining object-related information about the at least one target object among the plurality of objects included in the first image by transferring first image information about the first image to a first artificial intelligence model trained to identify the at least one target object. The operation of generating the second image may include an operation of obtaining the second image generated based on the first image and the additional image by transferring the first image information about the first image and the additional image information about the additional image to a second artificial intelligence model trained to generate the second image.

[0242] A computer-readable storage medium (e.g., memory (340) of FIG. 3) according to one embodiment may store a program for executing a method, including the steps of: acquiring a first image through a camera module (e.g., camera module (310) of FIG. 3); identifying at least one target object requiring additional shooting among a plurality of objects included in the first image; adjusting a shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot the at least one target object based on the identification; acquiring an additional image of the at least one target object through the camera module; generating a second image including a basic region and an extended region related to the at least one target object based on the first image and the additional image; and displaying the second image through a display (e.g., display (320) of FIG. 3).

[0243] According to various embodiments of the present disclosure, it is possible to increase user satisfaction with the captured image results while reflecting the user's intention when capturing an image.

[0244] According to various embodiments of the present disclosure, by generating a shooting result based on multiple actually captured images, a more realistic shooting result can be provided compared to a case where only one shooting is performed.

[0245] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the above description.

[0246] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0247] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0248] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0249] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0250] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0251] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; camera module; at least one processor; and When executed by at least one processor, the electronic device causes: Acquire a first image through the above camera module, Identifying at least one target object that requires additional shooting among a plurality of objects included in the first image; Adjusting the shooting range of the camera module from the first shooting range to the second shooting range to additionally shoot at least one target object based on the above identification, Acquire additional images of at least one target object through the camera module, Generating a second image including a basic region and an extended region related to at least one target object based on the first image and the additional image, An electronic device comprising a memory storing instructions for displaying the second image through the display.

2. In claim 1, At least one target object is, A second object surrounding a focused first object among the plurality of objects included in the first image; A second object around a first object located at the center of the first image; A second object located on the outskirts of the first image; A second object containing only partial shapes without a full shape; or An electronic device corresponding to at least one of the second objects related to the location information of the electronic device.

3. In claim 1, The extended area associated with at least one target object is: An actual shooting area cropped from the additional image and including a partial shape of the entire shape of at least one target object that is not included in the first image; or An electronic device comprising at least one of the virtual areas generated based on at least one of the additional images or the actual photographed area.

4. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Detecting the plurality of objects from the first image, An electronic device that identifies an object among the above multiple objects whose object completion rate is lower than a specified ratio as a target object requiring additional shooting.

5. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Detecting the plurality of objects from the first image, Assign weights to the above multiple objects, An electronic device that identifies a target object requiring additional shooting among the plurality of objects based on the assigned weights.

6. In claim 5, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the assigned weights, each of the plurality of objects is classified into one of a first object having a first weight, a second object having a second weight lower than the first weight, and a third object having a third weight lower than the second weight, An electronic device that identifies the second object as a target object requiring additional shooting.

7. In claim 1, The above electronic device is a foldable electronic device, The above instructions, when executed by the at least one processor, cause the electronic device to: Acquire the first image for the first shooting range through the first camera in the camera module, By changing the folding angle, the shooting range of the first camera is changed from the first shooting range to the second shooting range, An electronic device that acquires the additional image for the second shooting range through the first camera.

8. In claim 7, The above foldable electronic device, An electronic device that automatically changes the shooting range of the first camera by changing the folding angle while operating in flex mode.

9. In claim 1, The above camera module includes multiple cameras having different angles of view, The above instructions, when executed by the at least one processor, cause the electronic device to: Acquire the first image for the first shooting range through the activated first camera while the first camera having the first angle of view among the plurality of cameras is activated, An electronic device that activates a second camera having a second angle of view among the plurality of cameras and acquires the additional image for the second shooting range through the activated second camera.

10. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: By transmitting first image information about the first image to a first artificial intelligence model trained to identify at least one target object, object-related information about at least one target object among a plurality of objects included in the first image is obtained, An electronic device that obtains the second image generated based on the first image and the additional image by transmitting the first image information for the first image and the additional image information for the additional image to a second artificial intelligence model trained to generate the second image.

11. In the method of operating an electronic device, An action of acquiring a first image through a camera module; An operation of identifying at least one target object that requires additional shooting among a plurality of objects included in the first image; An operation of adjusting the shooting range of the camera module from a first shooting range to a second shooting range to additionally shoot at least one target object based on the identification; An operation of acquiring an additional image of at least one target object through the camera module; An operation of generating a second image including a base region and an extended region related to at least one target object based on the first image and the additional image; and A method comprising the action of displaying the second image through a display.

12. In claim 11, At least one target object is, A second object surrounding a focused first object among the plurality of objects included in the first image; A second object around a first object located at the center of the first image; A second object located on the outskirts of the first image; A second object containing only partial shapes without a full shape; or A method corresponding to at least one of the second objects related to the location information of the electronic device.

13. In claim 11, The extended area associated with at least one target object is: An actual shooting area cropped from the additional image and including a partial shape of the entire shape of at least one target object that is not included in the first image; or A method comprising at least one of the virtual areas generated based on at least one of the additional images or the actual photographed area.

14. In claim 11, The operation of identifying at least one target object requiring additional shooting among the above plurality of objects is: An operation of detecting the plurality of objects from the first image; and A method comprising an action of identifying an object among the plurality of objects having an object completion rate lower than a specified ratio as a target object requiring additional shooting.

15. In claim 11, The operation of identifying at least one target object requiring additional shooting among the above plurality of objects is: An operation of detecting the plurality of objects from the first image; An operation of assigning weights to the above plurality of objects; and A method comprising an operation of identifying a target object requiring additional shooting among the plurality of objects based on the assigned weights.

Images