Apparatus and method for generating an image in which a subject has been captured

Abstract

A device and a method of generating an output image in which a subject has been captured are provided. A method of generating an output image in which a subject has been captured from an original image, performed by a device, includes: obtaining, by using a first processor configured to control the device, an original image corresponding to the subject by controlling a camera sensor in the device; inputting, by using a second processor, the original image corresponding to the subject to a first artificial intelligence (AI) model trained to scale a luminance of an image, the second processor being configured to perform image processing on the original image through the AI model; obtaining, by using the second processor, tone map data output from the first AI model; and storing an output image generated based on the tone map data.

Description

Technical Field

[0001] This disclosure relates to an apparatus and method for generating an output image in which a subject has been captured using an artificial intelligence (AI) model. Background Technology

[0002] Due to the widespread use of mobile devices and the development of communication network technologies, consumer demands related to these mobile devices have diversified, leading to the installation of various types of add-ons in mobile devices. Furthermore, mobile devices have become smaller and support camera functions for capturing subjects.

[0003] However, due to the reduced size of mobile devices, high-performance camera sensors may not be easy to install in mobile devices, and mobile devices require a lot of resources to process raw data.

[0004] This creates an increasing need for technologies that use the camera sensors of mobile devices to generate images with a variety of image features, as well as artificial intelligence (AI) technologies that can generate high dynamic range (HDR) images. Summary of the Invention

[0005] Technical issues

[0006] Embodiments of this disclosure may provide an apparatus and method for generating an output image in which a subject has been captured using an artificial intelligence (AI) model.

[0007] Embodiments of this disclosure may also provide an apparatus and method for obtaining an output image of a subject captured by inputting a raw image generated from a camera sensor into an AI model.

[0008] Embodiments of this disclosure may also provide an apparatus and method for obtaining an output image of a subject that has been captured using an AI model for generating a tone map.

[0009] Embodiments of this disclosure may also provide an apparatus and method for obtaining an output image in which a subject has been captured from an original image by sequentially using multiple AI models.

[0010] Embodiments of this disclosure may also provide an apparatus and method for obtaining a real-time view image of a subject and an output image of the subject that has been captured using at least some of a plurality of AI models trained together.

[0011] Embodiments of this disclosure may also provide an apparatus and method for obtaining a real-time view image and an output image, the real-time view image being used to capture a subject, and the output image having been captured by an AI model trained in association with at least one of the user's preferences or the circumstances of capturing the subject.

[0012] Technical solution

[0013] According to a first aspect of this disclosure, a method for generating an output image in which a subject has been captured from an original image, performed by a device, is provided, the method comprising: obtaining an original image corresponding to a subject in front of the camera sensor in the device by controlling a camera sensor using a first processor configured to control the device; inputting the original image corresponding to the subject in front of the camera sensor into a first artificial intelligence (AI) model trained to scale the brightness of an image using a second processor, the second processor being configured to perform image processing on the original image using the AI ​​model; obtaining tone map data output from the first AI model using the second processor; and storing the output image generated based on the tone map data.

[0014] According to a second aspect of this disclosure, an apparatus for generating an output image in which a subject has been captured from an original image can be provided, the apparatus comprising a camera sensor, a display, a first memory storing first instructions for controlling the apparatus, a first processor configured to execute the first instructions stored in the first memory, a second memory storing at least one artificial intelligence (AI) model for performing image processing on the original image and second instructions related to the execution of the AI ​​model, and a second processor configured to execute the at least one AI model and the second instructions stored in the second memory, wherein the first processor is further configured to acquire an original image corresponding to a subject in front of the camera sensor using the camera sensor, wherein the second processor is further configured to input the original image corresponding to the subject in front of the camera sensor into a first AI model trained to scale the brightness of an image, wherein the second processor is further configured to acquire tone map data output from the first AI model, and wherein the first processor is further configured to store an output image generated based on the tone map data in the first memory.

[0015] According to a second aspect of this disclosure, a computer-readable recording medium having a program recorded thereon for performing the method of the first aspect on a computer can be provided. Attached Figure Description

[0016] Figure 1 This is a view illustrating an example of a device 1000 generating an output image by capturing a subject according to an embodiment of the present disclosure.

[0017] Figure 2 This is a block diagram of a device 1000 according to an embodiment of the present disclosure.

[0018] Figure 3 This is a view used to describe the process of generating an output image in which a subject has been captured from an original image, according to embodiments of the present disclosure.

[0019] Figure 4A This is a view used to describe the process of generating a tone map from an original image 30 performed by a device 1000 according to an embodiment of the present disclosure.

[0020] Figure 4B This is a view used to describe the process of extracting features from an original image 30 performed by a device 1000 according to an embodiment of the present disclosure.

[0021] Figure 4C This is a view used to describe the process of modifying a feature image performed by device 1000 according to an embodiment of the present disclosure.

[0022] Figure 4D This is a view used to describe the process performed by device 1000 to generate an output image from a modified feature image according to an embodiment of the present disclosure.

[0023] Figure 5 This is a view illustrating an example of the structure of a feature extraction model 1732 according to an embodiment of the present disclosure.

[0024] Figure 6 This is a flowchart of a method performed by device 1000 to generate an output image by capturing a subject, according to an embodiment of the present disclosure.

[0025] Figure 7 This is a view illustrating an example of training an AI model 1620 according to an embodiment of this disclosure.

[0026] Figure 8 This is a flowchart of a method for training an AI model 1620 according to an embodiment of the present disclosure.

[0027] Figure 9 This is a flowchart of a method for outputting a real-time viewfinder image performed by device 1000 according to an embodiment of the present disclosure.

[0028] Figure 10A This is a view illustrating an example of deactivating the tone map generation model 1731 in the AI ​​model 1730 to generate a live view image according to an embodiment of the present disclosure.

[0029] Figure 10BThis is a view illustrating an example of deactivating the feature extraction model 1732 and the image regression model 1734 in the AI ​​model 1730 according to an embodiment of the present disclosure to generate a real-time view image.

[0030] Figure 10C This is a view illustrating an example of deactivating the tone map generation model 1731, feature extraction model 1732, and image regression model 1734 in the AI ​​model 1730 according to embodiments of the present disclosure to generate a real-time view image.

[0031] Figure 11 This is a flowchart of a method performed by device 1000 according to an embodiment of the present disclosure to update AI model 1620 by receiving a retrained AI model 1620 from a server (not shown).

[0032] Figure 12 This is a flowchart of a method for retraining and updating an AI model 1620 performed by device 1000 according to an embodiment of this disclosure.

[0033] Figure 13A This is a view illustrating an example of a graphical user interface (GUI) for camera settings of a device 1000 for capturing a subject according to an embodiment of this disclosure.

[0034] Figure 13B This is a view illustrating an example of a GUI for camera settings of a device 1000 for capturing a subject according to an embodiment of this disclosure. Detailed Implementation

[0035] In the following, embodiments of this disclosure will be described in detail with reference to the accompanying drawings in a manner readily performed by those skilled in the art. However, this disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, for clarity, parts unrelated to this disclosure are not shown, and like elements are always indicated by the same reference numerals.

[0036] Throughout this specification, when an element is referred to as being “connected to” another element, the element is capable of being “directly connected to” the other element or “electrically connected to” the other element via an intermediate element. As used herein, the terms “comprising,” “consisting of,” “is composed of,” “includes,” and / or “is comprising” specify the presence of the stated element but do not preclude the presence or addition of one or more other elements.

[0037] As used herein, an artificial intelligence (AI) processing unit can be a processing unit configured to process images using AI technology. For example, an AI processing unit is a processing unit designed to perform image processing using an AI model, and can be a processing unit dedicated to image processing. Alternatively, for example, an AI processing unit can be implemented by configuring a neural processing unit (NPU) for image processing using an AI model.

[0038] As used in this paper, an AI model is a model trained to generate an output image from a raw image as a result of capturing a subject, and may include multiple sub-AI models. These sub-AI models may include a tone map generation model, a feature extraction model, an image modification model, and an image regression model. The tone map generation model may be an AI model trained to generate a tone map from a raw image; the feature extraction model may be an AI model trained to extract features from the raw image input to the feature extraction model; the image modification model may be an AI model trained to modify the feature image output from the feature extraction model; and the image regression model may be an AI model trained to generate an output image in which the subject has been captured from the feature image.

[0039] As used herein, a tone map can be graph data that includes information for scaling the brightness of pixels in the original image. A tone map can be graph data of at least one of a local tone map for scaling the brightness of pixels in a portion of the original image or a global tone map for scaling the brightness of the entire original image.

[0040] As used in this article, a live view image can be an image output to the monitor screen, allowing the user who wants to capture the subject to examine the subject.

[0041] Reference will now be made in detail to embodiments, examples of which are shown in the accompanying drawings.

[0042] Figure 1 This is a view illustrating an example of a device 1000 generating an output image by capturing a subject according to an embodiment of the present disclosure.

[0043] refer to Figure 1The device 1000 can capture a subject by using an AI model to generate an output image in which the subject has been captured. The device 1000 may include an AI processing unit 1700 configured to generate an output image in which the subject has been captured. The AI ​​processing unit 1700 of the device 1000 can input an original image generated using the camera sensor in the device 1000 and preset settings for image modification into the AI ​​model, and obtain an output image output from the AI ​​model. The AI ​​model used by the AI ​​processing unit 1700 is a model trained to generate an output image from the original image as a result of capturing the subject in the original image, and may include multiple sub-AI models. The multiple sub-AI models included in the AI ​​model may include an AI model for generating a tone map, an AI model for extracting features from the image, an AI model for modifying a feature image representing the extracted features, and an AI model for generating the output image.

[0044] The AI ​​processing unit 1700 of the device 1000 can generate a real-time framing image for capturing a subject by using at least one of a plurality of sub-AI models.

[0045] Device 1000 may be a smartphone, personal computer (PC), tablet PC, smart TV (TV), cellular phone, personal digital assistant (PDA), laptop computer, media player, global positioning system (GPS), e-book reader, digital broadcast receiver, navigation system, self-service terminal, digital camera, home appliance, or other mobile or non-mobile computing device, but is not limited thereto. Device 1000 may be a wearable device with communication and data processing capabilities, such as a watch, glasses, hairband, or ring. However, device 1000 is not limited to these and may include any type of device capable of capturing a subject.

[0046] Device 1000 can communicate with a server (not shown) via a network to obtain an output image of the subject already captured therein. The network can be implemented as a wired network such as a Local Area Network (LAN), Wide Area Network (WAN), or Value-Added Network (VAN), or any wireless network such as a mobile radio communication network or a satellite communication network. The network can include combinations of two or more of LANs, WANs, VANs, mobile radio communication networks, and satellite communication networks, broadly referring to data communication networks capable of enabling seamless communication between network entities, and can include wired internet, wireless internet, or mobile wireless communication networks. Wireless communication can include, but is not limited to, wireless LAN (or Wi-Fi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), or Near Field Communication (NFC).

[0047] Figure 2 This is a block diagram of a device 1000 according to an embodiment of the present disclosure.

[0048] refer to Figure 2 The device 1000 according to embodiments of this disclosure may include a user input device 1100, a display 1200, a communication interface 1300, a camera sensor 1400, a first processor 1500, a first memory 1600, and an AI processing unit 1700. The AI ​​processing unit 1700 may include a second processor 1710 configured to perform image processing using AI technology, and a second memory 1720. For example, the AI ​​processing unit 1700 is a processing unit designed for image processing using an AI model, and may be a processing unit dedicated to image processing. Alternatively, for example, the AI ​​processing unit 1700 may be implemented by configuring a neural processing unit (NPU) for image processing using an AI model.

[0049] User input device 1100 refers to a component through which a user inputs data for controlling device 1000. For example, user input device 1100 may include, but is not limited to, at least one of a keypad, dome switch, touchpad (e.g., capacitive overlay, resistive overlay, infrared beam, surface acoustic wave, integrated strain gauge, or piezoelectric touchpad), scroll wheel, or push switch. User input device 1100 can receive user input from a user of device 1000 for capturing photographs.

[0050] The display 1200 displays information processed in the device 1000. For example, the display 1200 may display a graphical user interface (GUI) for capturing photos, a live view image, and an output image as a result of capturing photos.

[0051] Simultaneously, when the display 1200 is stacked with the touchpad to configure a touchscreen, the display 1200 can function not only as an output device but also as an input device. The display 1200 may include at least one of a liquid crystal display (LCD), a thin-film transistor liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED), a flexible display, a three-dimensional (3D) display, or an electrophoretic display. Depending on the implementation of the device 1000, the device 1000 may include two or more displays 1200.

[0052] The communication interface 1300 may include one or more elements capable of communicating with another device (not shown) and a server (not shown). For example, the communication interface 1300 may include a short-range wireless communication unit, a mobile communication unit, and a broadcast receiver.

[0053] The short-range wireless communication unit may include, but is not limited to, a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication (NFC) unit, a Wireless Local Area Network (WLAN) communication unit, a Zigbee communication unit, an Infrared Data Association (IrDA) communication unit, a Wi-Fi Direct (WFD) communication unit, an Ultra Wideband (UWB) communication unit, or an Ant+ communication unit. The mobile communication unit transmits wireless signals to at least one of a base station, external device, or server on the mobile communication network and receives wireless signals from at least one of the base station, external device, or server on the mobile communication network. Here, the wireless signals may include various types of data transmitted and received based on voice call signals, video call signals, or text / multimedia messages. The broadcast receiver receives broadcast signals and / or broadcast information from the outside via a broadcast channel. The broadcast channel may include satellite channels and terrestrial channels. Depending on the embodiment, device 1000 may not include a broadcast receiver.

[0054] The communication interface 1300 can send and receive information from another device (not shown) and a server (not shown) that is required to manage the AI ​​model 1620 used to capture photos.

[0055] Camera sensor 1400 may include a color filter 1410 and an image sensor 1420, and generates a raw image corresponding to the subject under the control of a first processor 1500 or a second processor 1710, as described below. Camera sensor 1400 may generate the raw image corresponding to the subject under the control of the second processor 1710 or independently of the second processor 1710, based on control signals from the first processor 1500. For example, when the second processor 1710 is included in camera sensor 1400, and when the first processor 1500 sends a shooting control request signal to camera sensor 1400, the second processor 1710 in camera sensor 1400 may receive the control request signal and generate the raw image by controlling camera sensor 1400. Alternatively, for example, when the second processor 1710 is provided externally to camera sensor 1400, and when the first processor 1500 sends a shooting control request signal to second processor 1710, the second processor 1710 may receive the control request signal and generate the raw image by controlling camera sensor 1400. Alternatively, for example, the first processor 1500 can acquire the raw image by controlling the camera sensor 1400 and provide the acquired raw image to the second processor 1710.

[0056] Because the image obtained by the image sensor 1420, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD), is a monochrome image, a color filter 1410 configured to allow only a specific frequency band in the visible light region can be provided at the front end of the image sensor 1420, and a color image can be obtained from the light provided to the image sensor 1420 through the color filter 1410. The color filter 1410 allows only a specific frequency band of light from the subject to pass through. The color filter 1410 can be divided into multiple regions corresponding to multiple colors, and each of the multiple regions can allow only light of the same frequency band as one of three colors, such as red, green, and blue. The light passing through the color filter 1410 is provided to the image sensor 1420, such as a CMOS or CCD, and the image sensor 1420 can convert the received light into an electrical signal. For example, the electrical signal converted by the image sensor 1420 can include red, green, and blue values, and a raw image including an array of red, green, and blue values ​​can be generated. The original image can have a pattern based on the array pattern of the color filter 1410, such as any of the Bayer pattern, RGBE pattern, RYYB pattern, CYYM pattern, CYGM pattern, RGBW Bayer pattern and X-trans pattern.

[0057] The first memory 1600 may store programs for processing and control by at least one of the first processor 1500 or the second processor 1710 described below, and store data input to or output from the device 1000.

[0058] The first memory 1600 may include at least one type of storage medium selected from flash memory, hard disk, micro multimedia card, memory card (e.g., Secure Digital (SD) or Extreme Digital (XD) memory card), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, and optical disk.

[0059] The program stored in the first memory 1600 may be categorized into multiple modules depending on its function, and includes, for example, an image processing module 1610, at least one AI model 1620, and a model management module 1630. The image processing module 1610 may include a preprocessing module 1611, an output image generation module 1612, and a real-time view generation module 1613. The AI ​​model 1620 may include multiple sub-AI models, such as a tone map generation model, a feature extraction model, an image modification model, and an image regression model. The model management module 1630 may include a model selection module 1634, a download module 1631, an update module 1632, and a retraining module 1633. The first processor 1500 typically controls the overall operation of the device 1000. For example, the first processor 1500 can control the user input device 1100, the display 1200, the communication interface 1300, the camera sensor 1400, and the first memory 1600 by executing the program stored in the first memory 1600. The first processor 1500 may include one or more processors. In this case, one or more processors may include at least one of a general-purpose processor (e.g., a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP)) or a dedicated graphics processor (e.g., a graphics processing unit (GPU) or a visual processing unit (VPU)).

[0060] The AI ​​processing unit 1700 may include a second processor 1710 and a second memory 1720. The second processor 1710 and the second memory 1720 may be designed to perform image processing using an AI model, but are not limited thereto.

[0061] The second memory 1720 may store programs for processing and control by the second processor 1710. For example, the second memory 1720 may store an AI model 1730 selected from one or more AI models 1620 stored in the first memory 1600, and the AI ​​model 1730 stored in the second memory 1720 may be executed by the second processor 1710. The AI ​​model 1730 stored in the second memory 1720 may include multiple sub-AI models, such as a tone map generation model 1731, a feature extraction model 1732, an image modification model 1733, and an image regression model 1734.

[0062] The second memory 1720 may include at least one type of storage medium selected from flash memory, hard disk, micro multimedia card, memory card (e.g., SD or XD memory card), RAM, SRAM, ROM, EEPROM, PROM, magnetic storage, magnetic disk, and optical disk.

[0063] The image processing module 1610, stored in the first memory 1600, can be executed by at least one of the first processor 1500 or the second processor 1710 to generate a live view image for capturing a subject and an output image in which the subject has been captured. The live view image may be an image output on the display 1200, allowing a user who wishes to capture a subject to examine the subject to be captured, and the output image in which the subject has been captured may be an image stored in the device 1000 as a result of capturing the subject.

[0064] The first processor 1500 can generate a raw image corresponding to the subject by executing a preprocessing module 1611 stored in the first memory 1600. When user input for capturing the subject is received, the first processor 1500 can generate a raw image based on the light provided to the image sensor 1420 of the camera sensor 1400 by the color filter 1410 when the user input is received. The first processor 1500 can generate a raw image including, for example, an array of red, green, and blue values.

[0065] The first processor 1500 can generate an output image in which the subject has been captured by executing the output image generation module 1612. The first processor 1500 can request the second processor 1710 to generate the output image using the original image. In this way, the second processor 1710 can input the original image into the AI ​​model 1720, which includes a tone map generation model 1731, a feature extraction model 1732, an image modification model 1733, and an image regression model 1734, and obtain the output image output from the AI ​​model 1730. The second processor 1710 can obtain a tone map by inputting the original image into the tone map generation model 1731, obtain multiple feature images related to multiple features of the subject by inputting the generated tone map and the original image into the feature extraction model 1732, modify the multiple feature images by inputting the multiple feature images into the image modification model 1733, and obtain the output image to be stored in the device 1000 by inputting the multiple modified feature images into the image regression model 1734. The output image from the AI ​​model 1730 can be displayed on the display 1200 and stored in the first memory 1600 by the first processor 1500.

[0066] An AI model 1730 related to the user's capture of the subject can be selected from one or more AI models 1620 stored in the first memory 1600 and loaded into the second memory 1720, and its description will be provided below.

[0067] To generate a tone map for scaling the brightness of red, green, and blue values ​​in the original image, the second processor 1710 can input the original image into the tone map generation model 1731. For example, when the second processor 1710 is included in the camera sensor 1400, and when the first processor 1500 sends a shooting control request signal to the camera sensor 1400, the second processor 1710 in the camera sensor 1400 can receive the control request signal, control the camera sensor 1400 to generate the original image, and input the original image into the tone map generation model 1731. Alternatively, for example, when the second processor 1710 is provided externally to the camera sensor 1400, and when the first processor 1500 sends a shooting control request signal to the second processor 1710, the second processor 1710 can receive the control request signal, control the camera sensor 1400 to generate the original image, and input the original image into the tone map generation model 1731. For example, a first processor 1500 can acquire a raw image by controlling a camera sensor 1400, provide the acquired raw image to a second processor 1710, and the second processor 1710 can input the raw image into a tone map generation model 1731. The tone map can be graph data including information for scaling the brightness of pixels in the raw image. The tone map can be graph data of at least one of a local tone map for scaling the brightness of pixels in a portion of the raw image or a global tone map for scaling the brightness of the entire raw image. For example, the tone map can be generated to scale the brightness of dark areas more than the bright areas in the raw image. The tone map generation model 1731 can be an AI model trained to generate tone maps from raw images. The tone map generation model 1731 can include multiple neural network layers, and each of the multiple neural network layers can have multiple weight values, performing neural network computations by calculating the results of previous layers with the multiple weight values. The results of training the tone map generation model 1731 can be used to optimize multiple weight values ​​of multiple neural network layers, and the tone map generation model 1731 can be trained together with, for example, a feature extraction model 1732, an image modification model 1733, and an image regression model 1734. For example, the tone map generation model 1731 may include, but is not limited to, a convolutional neural network (CNN).

[0068] The second processor 1710 can combine the original image and the tone map and input them into the feature extraction model 1732, obtaining multiple feature images output from the feature extraction model 1732. For example, the second processor 1710 can use the tone map to scale the brightness of pixels in the original image and input the brightness-scaled original image into the feature extraction model 1732. The original image input into the feature extraction model 1732 can be scaled to increase the brightness values ​​of pixels in dark areas, and thus subjects located in dark areas of the original image can be identified more effectively.

[0069] The feature extraction model 1732 can extract features from the original image input to the feature extraction model 1732. Multiple feature images, each representing a different feature in the original image, can be output from the feature extraction model 1732. For example, the multiple feature images may include, but are not limited to, feature images representing features related to edges in the original image, feature images representing features related to lines in the original image, feature images representing features related to space in the original image, feature images representing features related to the shape and depth of objects in the original image, feature images representing features related to people in the original image, and feature images representing features related to objects in the original image.

[0070] Feature extraction model 1732 can be an AI model trained to extract features from the original image input to feature extraction model 1732. Feature extraction model 1732 can include multiple neural network layers, and each of the multiple neural network layers can have multiple weight values, and perform neural network computation by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the results of feature extraction model 1732, and for example, feature extraction model 1732 can be trained together with tone map generation model 1731, image modification model 1733, and image regression model 1734. For example, feature extraction model 1732 can be composed of a model with the following characteristics: Figure 5 The U-NET implementation describes, but is not limited to, the end-to-end fully convolutional network architecture.

[0071] The second processor 1710 can modify the feature image output from the feature extraction model 1732. The second processor 1710 can modify the feature image based on preset settings related to image attributes. For example, the second processor 1710 can modify the feature image using the image modification model 1733 based on preset criteria for white balance adjustment and color correction. The second processor 1710 can input the feature image output from the feature extraction model 1732 and preset attribute values ​​related to image attributes into the image modification model 1733, and obtain the modified feature image output from the image modification model 1733.

[0072] In this case, the second processor 1710 can determine the image attributes used to modify the feature image. For example, the image attributes used to modify the feature image can be determined based on the shooting environment of the device 1000. The second processor 1710 can obtain sensing data representing the surrounding environment of the device 1000 when the subject is captured through the camera sensor 1400, and input the white balance matrix value and color correction matrix value used to modify the feature image together with the feature image into the image modification model 1733 based on the criteria preset according to the sensing data.

[0073] Alternatively, for example, the first processor 1500 may display a GUI for camera settings on the device 1000 and preset settings for white balance adjustment and color correction based on user input via the GUI.

[0074] For example, device 1000 can be used in situations such as Figure 13A and Figure 13B The monitor 1200 displays a GUI showing the camera settings of the device 1000 used to capture the subject. In this way, the user can input settings related to ISO, shutter speed, white balance, color temperature, hue, contrast, saturation, highlight effect, shadow effect, etc., into the device 1000 through the GUI displayed on the monitor 1200.

[0075] In this case, the first processor 1500 can extract the white balance matrix value and the color correction matrix value from the first memory 1600 based on user-preset criteria, and input the extracted white balance matrix value and color correction matrix value together with the feature image into the image modification model 1733.

[0076] Image modification model 1733 can be an AI model trained to modify image attributes of a feature image input to image modification model 1733. Image modification model 1733 can include multiple neural network layers, and each of the multiple neural network layers can have multiple weight values, and neural network computation is performed by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the results of image modification model 1733, and for example, image modification model 1733 can be trained together with tone map generation model 1731, feature extraction model 1732, and image regression model 1734.

[0077] Meanwhile, as described above, the white balance matrix values ​​and color correction matrix values ​​are input to the image modification model 1733, but this is not the only possibility. Multiple AI models 1620 can be trained independently based on settings related to white balance adjustment and color correction. In this case, to generate an output image, an AI model 1730 corresponding to specific settings related to white balance adjustment and color correction can be loaded into the second memory 1720 and used by the second processor 1710. Although the white balance matrix values ​​and color correction matrix values ​​are not input to the AI ​​model 1730 loaded in the second memory 1720, an output image considering the specific settings related to white balance adjustment and color correction can be output from the AI ​​model 1730 loaded in the second memory 1720.

[0078] The second processor 1710 can input the modified feature image into the image regression model 1734 and obtain the output image from the image regression model 1734. The output image from the image regression model 1734 can be an image to be stored in the device 1000 as a result of capturing a subject.

[0079] When user input for capturing a subject is received, the image output by the AI ​​model 1730 can be compressed based on specific criteria and stored in the first memory 1600, but is not limited thereto.

[0080] Image regression model 1734 can be an AI model trained to generate an output image in which the subject has been captured from a feature image. Image regression model 1734 can include multiple neural network layers, and each of the multiple neural network layers can have multiple weight values, and neural network computation is performed by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the results of image regression model 1734, and for example, image regression model 1734 can be trained together with tone map generation model 1731, feature extraction model 1732, and image modification model 1733. For example, image regression model 1734 can include, but is not limited to, a CNN.

[0081] The first processor 1500 generates a live view image by executing the live view generation module 1613. When user input for activating camera functions for capturing a subject is received, the first processor 1500 can generate the live view image and display it on the display 1200. For example, when user input for executing a camera application installed in the device 1000 is received, the first processor 1500 can execute the camera application, generate a live view image, and display it on the display 1200, allowing the user to check the subject to be captured.

[0082] When the camera function of device 1000 is activated, the first processor 1500 can generate a raw image for generating a live view image based on the light input through the camera sensor 1400. To reduce the time spent generating the live view image, the second processor 1710 may not use at least one of the AI ​​models 1730. The first processor 1500 may not use at least one of the tone map generation model 1731, feature extraction model 1732, image modification model 1733, or image regression model 1734. In this case, the model among tone map generation model 1731, feature extraction model 1732, and image regression model 1734 that is not used to generate the live view image can be preset.

[0083] The second processor 1710 can input the raw image into the AI ​​model 1730, which includes at least one deactivated model, and provide the real-time view image output from the AI ​​model 1730 to the first processor 1500, which can then display the real-time view image on the display 1200. In this case, the AI ​​model 1730 can be pre-trained to output a satisfactory real-time view image, while at least one model in the AI ​​model 1730 is deactivated.

[0084] Thus, when the camera application of device 1000 is executed to activate the camera function, a live view image generated using the limited resources of device 1000 can be displayed on the screen of device 1000, and when a capture input is received from a user who wants to capture a subject, an output image generated by AI model 1730 from the original image generated when the capture input is received can be stored in the first memory 1600.

[0085] The first processor 1500 can select an AI model 1730 to be used by the second processor 1710, and update the AI ​​model 1620 stored in the first memory 1600 by executing the model management module 1630.

[0086] The first processor 1500 can select an AI model 1730 to be used by the second processor 1710 from one or more AI models 1620 stored in the first memory 1600 by executing the model selection module 1634. The AI ​​model 1730 selected by the first processor 1500 can be loaded into the second memory 1720 of the AI ​​processing unit 1700.

[0087] The first memory 1600 can store multiple AI models 1620, and the multiple AI models 1620 can be AI models trained individually based on multiple conditions. For example, the multiple AI models 1620 can be AI models trained based on at least one of the following conditions: camera filter, camera lens, camera manufacturer, device model, multiple images captured in burst mode, shooting environment, subject type, or image attribute.

[0088] For example, an AI model trained on a camera filter may include, but is not limited to, an AI model trained on an image captured using a passivation mask, an AI model trained on an image captured using a contrast adjustment mask, and an AI model trained on an image captured using a color filter mask. For example, an AI model trained on a camera lens may include, but is not limited to, an AI model trained on an image captured using a telephoto lens, an AI model trained on an image captured using a wide-angle lens, and an AI model trained on an image captured using a fisheye lens. For example, an AI model trained on a camera manufacturer may include, but is not limited to, an AI model trained on an image captured using a camera from manufacturer A, an AI model trained on an image captured using a camera from manufacturer B, and an AI model trained on an image captured using a camera from manufacturer C. For example, an AI model trained on a device may include, but is not limited to, an AI model trained on an image captured using a Galaxy S10, an AI model trained on an image captured using a Galaxy S20, and an AI model trained on an image captured using a Galaxy Note 20.

[0089] For example, AI models trained based on the shooting environment can include, but are not limited to, AI models trained on images captured in indoor environments, AI models trained on images captured in outdoor environments, and AI models trained on images captured within a specific brightness range. For example, AI models trained based on subject type can include, but are not limited to, AI models trained on images of people, AI models trained on images of food, and AI models trained on images of buildings. For example, AI models trained based on image attributes can include, but are not limited to, AI models trained on images captured by applying a specific white balance value, AI models trained on images captured by applying a specific ISO value, and AI models trained on images captured at a specific shutter speed.

[0090] When the camera application of device 1000 is executed to activate the camera function, the first processor 1500 can identify at least one of the following conditions related to camera filter, camera lens, camera manufacturer, device model, multiple images captured in burst mode, shooting environment, subject type, or image attributes by executing the model selection module 1634. For example, the first processor 1500 can identify at least one condition based on sensing values ​​obtained by sensing the surrounding environment of device 1000 and the settings of the camera application.

[0091] The first processor 1500 can display a specific GUI for camera settings on the display 1200, and identify at least one condition based on values ​​set by user input through the GUI. For example, the first processor 1500 can... Figure 13A and Figure 13B The monitor 1200 displays a GUI showing the camera settings of the device 1000 used to capture the subject. This allows the user to input settings related to ISO, shutter speed, white balance, color temperature, hue, contrast, saturation, highlight effects, and shadow effects into the device 1000 via the GUI displayed on the monitor 1200.

[0092] The first processor 1500 can extract an AI model 1730 corresponding to at least one identified condition from one or more AI models 1620 stored in the first memory 1600 and load it into the second memory 1720. For example, a user can set settings related to ISO, white balance, color temperature, hue, saturation, contrast, etc., through a GUI displayed on the display 1200 of the device 1000, and the device 1000 can select the AI ​​model 1730 corresponding to the user-set settings from a plurality of AI models 1620 stored in the first memory 1600 and load the selected AI model 1730 into the second memory 1720.

[0093] The first processor 1500 may extract an AI model 1730 corresponding to the user's preferences from the first memory 1600 and load it into the second memory 1720 based on information such as, for example, camera filters used by the user more than a certain number of times, camera lenses used by the user more than a certain number of times, camera manufacturers used by the user more than a certain number of times, device models used by the user more than a certain number of times, shooting environments provided more than a certain number of times, types of subjects captured by the user more than a certain number of times, or image attributes used by the user more than a certain number of times.

[0094] The first processor 1500 can display, for example, a preset image and load an AI model 1730 trained to output images having features similar to those of at least one image selected by the user from the displayed images into a second memory 1720. For example, when the user frequently selects an edge-enhanced image, the AI ​​model 1730 trained to output an edge-enhanced image can be loaded into the second memory 1720.

[0095] When the AI ​​model 1730 corresponding to at least one identified condition is not stored in the first memory 1600, the first processor 1500 may request the AI ​​model 1730 from a server (not shown) and receive the AI ​​model 1730 from the server (not shown) to store it in the second memory 1720.

[0096] The first processor 1500 can receive a retrained AI model 1620 or data for retraining the AI ​​model 1620 from a server (not shown) by executing the download module 1631. The first processor 1500 can request the retrained AI model 1620 from the server (not shown). The AI ​​model 1620 can be retrained by the server (not shown), and the server (not shown) can provide the device 1000 with notification information indicating the existence of the retrained AI model 1620. The first processor 1500 can display the notification information received from the server (not shown) on the screen and receive user input for updating the AI ​​model 1620. The first processor 1500 can provide the server (not shown) with information about photo attributes preferred by the user to request the retrained AI model 1620 from the server (not shown). Information regarding user-preferred photo attributes may include, for example, information about camera filters used more than a certain number of times, camera lenses used more than a certain number of times, camera manufacturers used more than a certain number of times, device models used more than a certain number of times, shooting environments provided more than a certain number of times, types of subjects captured more than a certain number of times, or image attributes used more than a certain number of times. When the first processor 1500 has provided information about user-preferred photo attributes to a server (not shown), the first processor 1500 can download a retrained AI model 1620 based on the user-preferred photo attributes from the server (not shown). The first processor 1500 can update the AI ​​model 1620 in device 1000 by executing update module 1632, replacing the AI ​​model 1620 in device 1000 with the retrained AI model 1620 received from the server (not shown).

[0097] The first processor 1500 can download a reference original image for retraining and a corresponding reference image from a server (not shown) via the download module 1631. The reference image can be an image generated from the reference original image for retraining. The first processor 1500 can request and receive the reference original image and the reference image for retraining the AI ​​model 1620 from the server (not shown). In this case, the first processor 1500 can provide the server (not shown) with information about user-preferred photo attributes. The information about user-preferred photo attributes can include, for example, information about camera filters used more than a certain number of times, camera lenses used more than a certain number of times, camera manufacturers used more than a certain number of times, device models used more than a certain number of times, at least one of the shooting environments provided more than a certain number of times, the type of subject captured more than a certain number of times by the user, or image attributes used more than a certain number of times by the user. In this case, the server (not shown) can provide the device 1000 with a reference image and a reference original image generated based on user-preferred photo attributes.

[0098] The first processor 1500 can obtain a reference original image for retraining and a reference image corresponding to the reference original image by executing the retraining module 1633, and the second processor 1710 can retrain the AI ​​model 1620. The second processor 1710 can use the reference image received from the server (not shown) as the ground truth (GT) image, and retrain the AI ​​model 1620 by inputting the reference original image received from the server (not shown) into the AI ​​model 1620 and comparing the output image output from the AI ​​model 1620 with the reference image.

[0099] Meanwhile, as described above, device 1000 generates an output image in which a subject has been captured using AI model 1620 within device 1000, but is not limited to this. Device 1000 can generate the output image in which a subject has been captured together with a server (not shown). For example, device 1000 can generate a raw image and send the generated raw image to a server (not shown) to request an output image from the server (not shown). Device 1000 can provide the raw image along with setting information for image modification to the server (not shown). In this case, AI model 1620 can be included in the server (not shown), and the server (not shown) can generate the output image from the raw image using AI model 1620 in the server (not shown). The server (not shown) can then provide the generated output image to device 1000.

[0100] Alternatively, for example, device 1000 may provide the original image and the tone map output from tone map generation model 1731 to a server (not shown) to request an output image from the server (not shown). Device 1000 may provide the original image, the tone map, and setting information for image modification together to the server (not shown). In this case, feature extraction model 1732, image modification model 1733, and image regression model 1734 in AI model 1620 may be included in the server (not shown), and the server (not shown) may generate the output image by using feature extraction model 1732, image modification model 1733, and image regression model 1734 in the server (not shown). The server (not shown) may provide the generated output image to device 1000.

[0101] Alternatively, for example, device 1000 may provide the feature image output from feature extraction model 1732 to a server (not shown) to request an output image from the server (not shown). Device 1000 may provide the feature image along with setting information for image modification to the server (not shown). In this case, image modification model 1733 and image regression model 1734 in AI model 1620 may be included in the server (not shown), and the server (not shown) may generate the output image by using image modification model 1733 and image regression model 1734 in the server (not shown). The server (not shown) may provide the generated output image to device 1000.

[0102] Device 1000 can determine, based on its own circumstances, whether to generate an output image independently or in conjunction with a server (not shown). For example, considering battery level, resource usage, or communication status, device 1000 can determine whether to generate the output image independently or in conjunction with a server (not shown). For example, when device 1000's battery level is below a threshold, device 1000 can request the server (not shown) to generate an output image. For example, when device 1000's resource usage exceeds a threshold, device 1000 can request the server (not shown) to generate an output image. For example, when device 1000's communication status is good, device 1000 can request the server (not shown) to generate an output image. In this case, when device 1000 requests an output image from the server (not shown), it can set, based on various criteria, whether to provide the original image, provide the original image and tone map, or provide a feature image, depending on device 1000's circumstances. For example, when the camera function is activated, device 1000 can identify battery level, resource usage, or communication status, and determine whether to provide a raw image, a raw image and tone map, or a feature image to a server (not shown). Based on the determination to request an output image from the server (not shown), device 1000 can provide at least one of the raw image, tone map, or feature image to the server (not shown).

[0103] Figure 3 This is a view used to describe the process of generating an output image in which a subject has been captured from an original image, according to embodiments of the present disclosure.

[0104] refer to Figure 3 Using the AI ​​processing unit 1700, the device 1000 can input the original image 30 corresponding to the subject into the tone map generation model 1731 and obtain the tone map output from the tone map generation model 1731. The device 1000 can combine the original image 30 and the tone map and input them into the feature extraction model 1732, and obtain multiple feature images output from the feature extraction model 1732. Then, the device 1000 can input the multiple feature images into the image modification model 1733 and obtain the modified feature images output from the image modification model 1733. Afterwards, the device 1000 can input the modified feature images into the image regression model 1734, and thus obtain the output image 38 output from the image regression model 1734.

[0105] Each of the tone map generation model 1731, feature extraction model 1732, image modification model 1733, and image regression model 1734 may include multiple neural network layers. Each of the multiple neural network layers may have multiple weight values, and neural network computation is performed by computation between the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the AI ​​model. For example, the multiple weight values ​​can be updated to reduce or minimize the loss or cost values ​​obtained by the AI ​​model during the training process. Artificial neural networks may include, for example, convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q networks, but are not limited to the examples above. The tone map generation model 1731, feature extraction model 1732, image modification model 1733, and image regression model 1734 can be trained together using a reference original image and a reference output image, as described below. Figure 7 As stated above.

[0106] Figure 4A This is a view used to describe the process of generating a tone map from an original image 30 performed by a device 1000 according to an embodiment of the present disclosure.

[0107] refer to Figure 4A When the camera application of device 1000 is executed to activate the camera function, device 1000 can generate a raw image 30. When the function for capturing a subject is activated, device 1000 can generate a raw image based on light provided from the subject using camera sensor 1400, and when user input for capturing a subject is received, device 1000 can obtain the raw image 30 generated by camera sensor 1400 when user input is received. The raw image 30 may include, for example, an array of red, green, and blue values.

[0108] When the camera function is activated, device 1000 can identify at least one condition for capturing a subject, retrieve the AI ​​model 1730 corresponding to the identified condition from first memory 1600, and load it into second memory 1720 in AI processing unit 1700. When the AI ​​model 1730 corresponding to the identified condition is not stored in first memory 1600, device 1000 can request AI model 1730 from server (not shown) and receive AI model 1730 from server (not shown) to load it into second memory 1720.

[0109] Then, in order to generate a tone map 40 for scaling the brightness of the red, green, and blue values ​​in the original image 30, the device 1000 may receive user input for capturing a subject after the camera application is executed, and input the original image 30 into the tone map generation model 1731 in response to the received user input. The tone map 40 may include information for scaling the brightness of pixels in the original image 30. For example, the tone map 40 may be generated to scale the brightness of dark areas more than the bright areas in the original image 30.

[0110] The tone map generation model 1731 may include multiple neural network layers, each of which may have multiple weight values, and neural network computation is performed by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the tone map generation model 1731, and for example, the tone map generation model 1731 can be trained together with a feature extraction model 1732, an image modification model 1733, and an image regression model 1734. For example, the tone map generation model 1731 may include, but is not limited to, a CNN.

[0111] Figure 4B This is a view used to describe the process of extracting features from an original image 30 performed by a device 1000 according to an embodiment of the present disclosure.

[0112] refer to Figure 4B The device 1000 can combine the original image 30 and the tone map 40 and input them into the feature extraction model 1732, and obtain multiple feature images 42 output from the feature extraction model 1732. For example, the device 1000 can use the tone map 40 to scale the brightness of pixels in the original image 30 and input the brightness-scaled original image 30 into the feature extraction model 1732. The original image 30 input into the feature extraction model 1732 can be scaled to increase the brightness values ​​of pixels in dark areas, and therefore subjects located in dark areas of the original image 30 can be identified more effectively.

[0113] The feature extraction model 1732 can extract features from the original image 30 input to the feature extraction model 1732. Multiple feature images 42, each representing a different feature in the original image 30, can be output from the feature extraction model 1732. For example, the multiple feature images 42 may include, but are not limited to, feature images representing features related to edges in the original image 30, feature images representing features related to lines in the original image 30, feature images representing features related to space in the original image 30, feature images representing features related to the shape and depth of objects in the original image 30, feature images representing features related to people in the original image 30, and feature images representing features related to things in the original image 30.

[0114] Feature extraction model 1732 may include multiple neural network layers, each of which may have multiple weight values, and neural network computation is performed by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the results of feature extraction model 1732, and for example, feature extraction model 1732 can be trained together with tone map generation model 1731, image modification model 1733, and image regression model 1734. For example, feature extraction model 1732 may be composed of layers with the following characteristics... Figure 5 The U-NET implementation describes, but is not limited to, the end-to-end fully convolutional network architecture.

[0115] Figure 4C This is a view used to describe the process of modifying a feature image performed by device 1000 according to an embodiment of the present disclosure.

[0116] refer to Figure 4C The device 1000 can modify the feature image 42 output from the feature extraction model 1732. The device 1000 can modify the feature image 42 by inputting the feature image 42 output from the feature extraction model 1732 and setting values ​​related to image attributes into the image modification model 1733. For example, the device 1000 can modify the feature image 42 based on preset criteria used for white balance adjustment and color correction.

[0117] For example, the first processor 1500 of device 1000 can generate sensing data representing the surrounding environment of device 1000 when the subject is captured using sensors in device 1000, automatically identify the surrounding environment of device 1000 based on the generated sensing data, and extract a white balance matrix 44 and a color correction matrix 45 for modifying the feature image from the first memory 1600 based on preset criteria. The first processor 1500 can provide the extracted white balance matrix 44 and color correction matrix 45 to the second processor 1710. The second processor 1710 can input the white balance matrix 44 and color correction matrix 45 received from the first processor 1500 together with the feature image 42 into the image modification model 1733.

[0118] Alternatively, for example, when the camera function is activated, the first processor 1500 of device 1000 can display a GUI on the display 1200 of device 1000 for white balance adjustment and color correction settings for feature image 42, as well as preset settings for white balance adjustment and color correction based on user input via the GUI. The first processor 1500 can retrieve the white balance matrix 44 and color correction matrix 45 from the first memory 1600 based on the settings input by the user, and provide the white balance matrix 44 and color correction matrix 45 to the second processor 1710. The second processor 1710 can input the white balance matrix 44 and color correction matrix 45 received from the first processor 1500, together with the feature image 42, into the image modification model 1733.

[0119] In the above description, device 1000 modifies feature image 42 using image modification model 1733, but is not limited thereto. Device 1000 can modify feature image 42 without using image modification model 1733. In this case, device 1000 can modify feature image 42 by using matrices for modifying image attributes. For example, device 1000 can adjust the white balance of feature image 42 by multiplying each element in feature image 42 by a white balance matrix 44 for white balance adjustment. Device 1000 can correct the color of feature image 42 by multiplying each element in feature image 42 by a color correction matrix 45 for color correction.

[0120] Meanwhile, as described above, the white balance matrix values ​​and color correction matrix values ​​are input to the image modification model 1733, but this is not the only possibility. Multiple AI models 1620 can be trained independently based on settings related to white balance adjustment and color correction. In this case, to generate an output image, an AI model 1730 corresponding to specific settings related to white balance adjustment and color correction can be loaded into the second memory 1720 and used by the second processor 1710. Although the white balance matrix values ​​and color correction matrix values ​​are not input to the AI ​​model 1730 loaded in the second memory 1720, an output image considering the specific settings related to white balance adjustment and color correction can be output from the AI ​​model 1730 loaded in the second memory 1720.

[0121] Figure 4D This is a view used to describe the process performed by device 1000 to generate an output image from a modified feature image according to an embodiment of the present disclosure.

[0122] Device 1000 can input the modified feature image 46 into image regression model 1734 and obtain output image 38 from image regression model 1734. Output image 38 from image regression model 1734 can be an image to be stored in device 1000 as a result of capturing a subject. Image regression model 1734 can include multiple neural network layers, and each of the multiple neural network layers can have multiple weight values, and neural network computation is performed by calculating the results of previous layers and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized by training the results of image regression model 1734, and for example, image regression model 1734 can be trained together with tone map generation model 1731 and feature extraction model 1732. For example, image regression model 1734 can include CNN, but is not limited to this.

[0123] Figure 5 This is a view illustrating an example of the structure of a feature extraction model 1732 according to an embodiment of the present disclosure.

[0124] refer to Figure 5 The feature extraction model 1732 can be implemented using U-NET, which has an end-to-end fully convolutional network architecture. The feature extraction model 1732 may include: layers for a shrinking path, configured to capture the context of the input image; and layers for an expanding path, used to perform upsampling to obtain high-resolution results from the feature images of the shrinking path. The layers for the shrinking path may be symmetrical to the layers for the expanding path.

[0125] Figure 6 This is a flowchart of a method performed by device 1000 to generate an output image by capturing a subject, according to an embodiment of the present disclosure.

[0126] In operation S600, device 1000 can receive user input for capturing a subject near device 1000, and in operation S605, device 1000 can generate a raw image based on the light input through camera sensor 1400. When the function for capturing a subject is activated, device 1000 can generate a raw image based on the light provided from the subject using camera sensor 1400, and when user input for capturing a subject is received, device 1000 can obtain the raw image generated by camera sensor 1400 when the user input is received.

[0127] In operation S610, device 1000 can input the original image into tone map generation model 1731. Device 1000 can input the original image into tone map generation model 1731 and obtain the tone map output from tone map generation model 1731.

[0128] In operation S615, device 1000 can input the generated tone map and the original image into feature extraction model 1732. Device 1000 can use the tone map to scale the brightness of pixels in the original image and input the brightness-scaled original image into the feature extraction model. Device 1000 can obtain the feature image output from feature extraction model 1732.

[0129] In operation S620, device 1000 can modify the feature image generated by feature extraction model 1732 based on preset criteria. Device 1000 can modify the feature image based on preset settings related to image attributes. For example, device 1000 can modify the feature image based on preset criteria used for white balance adjustment and color correction, but the preset criteria used to modify the feature image are not limited to these.

[0130] In operation S625, device 1000 can input the modified feature image into image regression model 1734, and in operation S630, device 1000 can store the output image output from image regression model 1734.

[0131] Figure 7 This is a view illustrating an example of training an AI model 1620 according to an embodiment of this disclosure.

[0132] refer to Figure 7 The reference image 72, generated by performing image signal processing (ISP) on a reference original image, can be used as the ground truth (GT) image for training the AI ​​model 1620. For example, the reference original image can be generated by combining multiple original images 70, which are generated by capturing a subject in burst mode within a short time period, and the reference image 72 can be output by performing ISP processing on the reference original image. Existing ISP processing can be existing image processing performed on the original image without using an AI model, such as image processing including preprocessing, white balance adjustment, depigmentation, gamma correction, color correction, etc., performed on the original image without using an AI model. The output reference image 72 can be used as the GT image for the AI ​​model. Alternatively, one of multiple original images 70 generated by capturing a subject in burst mode within a short time period can be selected, and the image output by performing existing ISP processing on the selected original image can be used as the GT image.

[0133] To train the AI ​​model 1620, a reference original image or one of a plurality of original images 70 captured in burst mode can be input into the AI ​​model 1620. For example, when eight original images are generated in burst mode, a reference original image can be generated by combining the eight original images, and the AI ​​model 1620 can be trained using the reference original image and a reference image generated from the reference original image through existing ISP processing. Alternatively, for example, when eight original images are generated in burst mode, eight output images can be generated individually from the eight original images through existing ISP processing. In this case, one of the eight original images can be selected, and the AI ​​model 1620 can be trained using the selected original image and the output image corresponding to the selected original image.

[0134] Image attribute settings for modifying feature images can be input into the AI ​​model 1620. These settings may include, for example, preset matrices for white balance adjustment and preset matrices for color correction. Various settings can be input into the AI ​​model 1620 to train it based on various image attributes.

[0135] A specific reference original image, including noise, can be input into AI model 1620 to train AI model 1620. In this case, a reference original image including progressively increasing noise can be input into AI model 1620 to train AI model 1620. For example, a server (not shown) or device 1000 can generate n input images using reference original images including noise levels one through n, and input these n input images into AI model 1620 to train AI model 1620. In this way, AI model 1620 can be trained to output a denoised output image from the noisy original image.

[0136] By comparing the reference image 72 with the output image 74 from the AI ​​model 1620, the weights of the neural network layers in the AI ​​model 1620 can be adjusted to reduce the loss between the reference image 72 and the output image 74. In this case, the AI ​​model 1620 may include a tone map generation model 1731, a feature extraction model 1732, and an image regression model 1734, and therefore the weights of the neural network layers in the tone map generation model 1731, the feature extraction model 1732, and the image regression model 1734 can be adjusted together.

[0137] Figure 8 This is a flowchart of a method for training an AI model 1620 according to an embodiment of the present disclosure.

[0138] In operation S800, a server (not shown) can obtain multiple raw images generated by capturing a subject in burst mode. The server (not shown) can obtain multiple raw images generated by capturing a subject in burst mode within a short time period. Because the multiple raw images are generated in burst mode, the multiple raw images can have similar image information.

[0139] In operation S805, a server (not shown) can generate a reference original image by combining multiple original images. Alternatively, a server (not shown) can generate a reference original image by combining multiple original images using image fusion.

[0140] In operation S810, the server (not shown) can obtain a reference image generated from the original reference image through ISP processing. ISP processing can be existing image processing performed on the original image without using an AI model, such as image processing performed on the original image without using an AI model, including preprocessing, white balance adjustment, demosaicing, gamma correction, color correction, etc. The reference image generated by ISP processing can be used as the GT image of AI model 1620.

[0141] In operations S815 and S820, the server (not shown) can obtain a first output image from the AI ​​model 1620. The server (not shown) can input a reference original image or one of multiple original images into the AI ​​model 1620 (S815) and obtain the first output image from the AI ​​model 1620 (S820). The AI ​​model 1620 to be trained by the server (not shown) may include... Figure 3 The elements of AI model 1620 shown.

[0142] In operation S825, the server (not shown) can analyze the loss between the reference image and the first output image. The server (not shown) can use the reference image as the ground truth image and compare the reference image with the first output image.

[0143] In operation S830, the server (not shown) can change the weight values ​​of the AI ​​model 1620 based on the analyzed loss. The server (not shown) can adjust the weight values ​​of the neural network layers in the AI ​​model 1620 to reduce the loss between the reference image and the first output image. In this case, the AI ​​model 1620 may include a tone map generation model 1731, a feature extraction model 1732, and an image regression model 1734, and therefore the weight values ​​of the neural network layers in the tone map generation model 1731, the feature extraction model 1732, and the image regression model 1734 can be adjusted together.

[0144] In operation S835, the server (not shown) can input a reference original image or at least one original image from the original images into the AI ​​model 1620 with modified weight values. In operation S840, the server (not shown) can obtain a second output image from the AI ​​model 1620 with modified weight values. In operation S845, the server (not shown) can analyze the loss between the reference image and the second output image.

[0145] In operation S850, the server (not shown) can determine whether to terminate the training of AI model 1620. When the loss analyzed in operation S845 is less than a preset threshold, the server (not shown) can determine to terminate the training of AI model 1620, and when the loss analyzed in operation S845 is greater than the preset threshold, the server (not shown) can repeat the operation used to change the weight values ​​of AI model 1620.

[0146] At the same time, multiple AI models 1620 can be trained based on various conditions. For example, multiple AI models 1620 can be trained based on at least one of the following conditions: camera filter, camera lens, camera manufacturer, device model, multiple images captured in burst mode, shooting environment, subject type, or image attribute.

[0147] For example, AI model 1620 can be trained using a reference original image and a reference image captured using a passivation mask, a reference original image and a reference image captured using a contrast adjustment mask, and a reference image captured using a color filter mask, but is not limited to these. For example, AI model 1620 can be trained using a reference original image and a reference image captured based on a camera lens, and the camera lens can include, for example, a telephoto lens, a wide-angle lens, and a fisheye lens, but is not limited to these. For example, AI model 1620 can be trained using a reference original image and a reference image captured based on a camera manufacturer, or a reference image captured based on a device model, but is not limited to these. For example, AI model 1620 can be trained using a reference original image and a reference image captured in an indoor environment, a reference original image and a reference image captured in an outdoor environment, and a reference image captured within a specific brightness range, but is not limited to these. For example, AI model 1620 can be trained using a reference original image and a reference image of a person, a reference original image and a reference image of food, and a reference image of a building, but is not limited to these. For example, an AI model 1620 can be trained using a reference original image and a reference image captured based on image attributes, and image attributes can include, but are not limited to, white balance, ISO, and shutter speed.

[0148] At the same time, in relation to Figure 8The AI ​​model 1620 is trained by a server (not shown) in the relevant description, but is not limited to this. The AI ​​model 1620 can be trained by a device 1000. In this case, the device 1000 can directly generate reference original images and reference images to be used for training the AI ​​model 1620, or request and receive reference original images and reference images from a server (not shown).

[0149] For example, device 1000 may provide images captured by device 1000 to a server (not shown) to request reference original images and reference images for training AI model 1620. The server (not shown) may analyze the images received from device 1000 and provide the reference original images and reference images to device 1000 in relation to user preferences. For example, the server (not shown) may identify user preferences by analyzing the images received from device 1000, such as the type of device 1000 used by the user, the type of subject in the images captured by device 1000, the user's preferred image style, or the environment (e.g., indoor, outdoor, brightness, or weather) of places where the user typically takes photos. Alternatively, for example, device 1000 may provide information about the camera type or lens type of device 1000 to the server (not shown) to request reference original images and reference images for training AI model 1620. The server (not shown) may provide the reference original images and reference images to device 1000 in relation to the camera type or lens type received from device 1000.

[0150] Alternatively, device 1000 may provide a server (not shown) with user preference information of the subject, indicating image style, shooting environment, and user preferences, to request reference original images and reference images for training AI model 1620. The server (not shown) may provide device 1000 with reference original images and reference images related to the user preference information received from device 1000.

[0151] Figure 9 This is a flowchart of a method for outputting a real-time viewfinder image performed by device 1000 according to an embodiment of the present disclosure.

[0152] In operation S900, device 1000 can receive user input for activating camera functions to capture a subject. For example, device 1000 can receive user input for executing a camera application installed on device 1000.

[0153] In operation S905, device 1000 can generate a raw image based on the light input through camera sensor 1400. When the camera function is activated, device 1000 can generate a raw image for generating a live view image.

[0154] In operation S910, device 1000 can deactivate at least one model in AI model 1730. Device 1000 can deactivate at least one of tone map generation model 1731, feature extraction model 1732, image modification model 1733, or image regression model 1734. The model deactivated from tone map generation model 1731, feature extraction model 1732, image modification model 1733, or image regression model 1734 can be preset for generating a real-time view image.

[0155] In operation S915, device 1000 can input the raw image into AI model 1730, which has at least one model deactivated, and in operation S920, device 1000 can display the real-time view image output from AI model 1730 on the screen. In this case, AI model 1620 can be pre-trained to output a satisfactory real-time view image while deactivating at least one model in AI model 1620.

[0156] Furthermore, in the above description, at least one of the tonal map generation model 1731, feature extraction model 1732, image modification model 1733, or image regression model 1734 is deactivated in the AI ​​model 1730 to generate real-time view images, but this is not a limitation. Real-time view images can be generated using the AI ​​model 1730 in which all of the tonal map generation model 1731, feature extraction model 1732, image modification model 1733, and image regression model 1734 are activated.

[0157] Simultaneously, at least one of the deactivated AI models 1730, namely tone map generation model 1731, feature extraction model 1732, image modification model 1733, or image regression model 1734, can be used to generate and store an output image in which the subject has been captured.

[0158] Figure 10A This is a view illustrating an example of deactivating the tone map generation model 1731 in the AI ​​model 1730 to generate a live view image according to an embodiment of the present disclosure.

[0159] refer to Figure 10A When the tone map generation model 1731 in AI model 1730 is deactivated, the original image can be input into AI model 1730. The original image can be input into feature extraction model 1732, and the feature image output from feature extraction model 1732 can be modified. The modified feature image can be input into image regression model 1734, and a real-time view image can be output from image regression model 1734. In this case, AI model 1730 can be... Figure 7The model was trained in the same way as in the previous model, while the tone map generation model 1731 was deactivated.

[0160] Figure 10B This is a view illustrating an example of deactivating the feature extraction model 1732 and the image regression model 1734 in the AI ​​model 1730 according to an embodiment of the present disclosure to generate a real-time view image.

[0161] refer to Figure 10B While the feature extraction model 1732 and image regression model 1734 in AI model 1730 are deactivated, the original image can be input into AI model 1730. The original image can be input into tone map generation model 1731, where the brightness of the original image can be scaled based on the tone map output from tone map generation model 1731, and the brightness-scaled original image can be modified to generate a real-time view image. In this case, AI model 1730 can be... Figure 7 The model was trained in the same way as in the previous model, while the feature extraction model 1732 and the image regression model 1734 were deactivated.

[0162] Figure 10C This is a view illustrating an example of deactivating the tone map generation model 1731, feature extraction model 1732, and image regression model 1734 in the AI ​​model 1730 according to embodiments of the present disclosure to generate a real-time view image.

[0163] refer to Figure 10C When the tone map generation model 1731, feature extraction model 1732 and image regression model 1734 in AI model 1730 are deactivated, the original image can be modified to generate a real-time view image.

[0164] At the same time, the white balance matrix value and the color correction matrix value are input to Figures 10A to 10C The image modification model 1733 is used, but is not limited to it. Multiple AI models 1620 can be trained separately based on settings related to white balance adjustment and color correction. In this case, to generate a real-time view image, an AI model 1730 corresponding to specific settings related to white balance adjustment and color correction can be loaded into a second memory 1720 and used by a second processor 1710. Although the white balance matrix values ​​and color correction matrix values ​​are not input to the AI ​​model 1730 loaded in the second memory 1720, a real-time view image considering the specific settings related to white balance adjustment and color correction can be output from the AI ​​model 1730 loaded in the second memory 1720.

[0165] Figure 11This is a flowchart of a method executed by device 1000 to update AI model 1620 by receiving a retrained AI model 1620 from a server (not shown).

[0166] In operation S1100, device 1000 may request a retrained AI model 1620 from a server (not shown). The AI ​​model 1620 can be retrained by the server (not shown), and the server (not shown) may provide device 1000 with notification information indicating the existence of the retrained AI model 1620. Device 1000 may display the notification information received from the server (not shown) on a screen and receive user input for updating the AI ​​model 1620. Device 1000 may provide information about photo attributes preferred by the user to the server (not shown) in order to request a retrained AI model 1620 from the server (not shown).

[0167] Device 1000 can provide images captured by device 1000 to a server (not shown) to request a retrained AI model 1620. The server (not shown) can analyze the images received from device 1000 and retrain the AI ​​model 1620 using reference original images and reference images related to user preferences associated with device 1000. For example, the server (not shown) can identify user preferences by analyzing the images received from device 1000, such as the type of device 1000 used by the user, the type of subject in the images captured by device 1000, the user's preferred image style, or the environment (e.g., indoor, outdoor, brightness, or weather) of places where the user typically takes photos. The server (not shown) can retrain the AI ​​model 1620 using reference original images and reference images related to user preferences.

[0168] Alternatively, for example, device 1000 may provide information about the camera type or lens type of device 1000 to a server (not shown) to request a retrained AI model 1620. The server (not shown) may retrain the AI ​​model 1620 using a reference original image and a reference image associated with the camera type or lens type received from device 1000.

[0169] Alternatively, device 1000 may provide a server (not shown) with user preference information indicating user preferences, such as image style, shooting environment, and subject, to request a retrained AI model 1620 from the server (not shown). The server (not shown) may retrain the AI ​​model 1620 using a reference original image and a reference image associated with the user preference information received from device 1000.

[0170] In operation S1110, device 1000 can receive a retrained AI model 1620 from a server (not shown). When device 1000 has provided information about the photo attributes of user preferences to the server (not shown), the server (not shown) can provide device 1000 with the AI ​​model 1620 that has been retrained about the photo attributes of user preferences.

[0171] In operation S1120, device 1000 can update the AI ​​model 1620 in device 1000 based on the retrained AI model 1620. Device 1000 can update the AI ​​model 1620 in device 1000, for example, by replacing the AI ​​model 1620 in device 1000 with the retrained AI model 1620 received from a server (not shown).

[0172] Figure 12 This is a flowchart of the method for retraining and updating the AI ​​model 1620, performed by device 1000.

[0173] In operation S1200, device 1000 can obtain a reference original image for retraining and a reference image corresponding to the reference original image. The reference image may be an image generated from the reference original image used for retraining. Device 1000 can request and receive the reference original image and the reference image for retraining AI model 1620 from a server (not shown). In this case, device 1000 can provide the server (not shown) with information about photos captured by device 1000 or photo attributes related to user preferences, and the server (not shown) can provide device 1000 with a reference image and a reference original image generated in relation to user preferences.

[0174] For example, device 1000 may provide images captured by device 1000 to a server (not shown) to request reference original images and reference images for retraining AI model 1620. The server (not shown) may analyze the images received from device 1000 and provide the reference original images and reference images to device 1000 in relation to user preferences. For example, the server (not shown) may identify user preferences by analyzing the images received from device 1000, such as the type of device 1000 used by the user, the type of subject in the images captured by device 1000, the user's preferred image style, or the environment (e.g., indoor, outdoor, brightness, or weather) of places where the user typically takes photos. The server (not shown) may provide the reference original images and reference images to device 1000 in relation to user preferences.

[0175] Alternatively, for example, device 1000 may provide information about the camera type or lens type of device 1000 to a server (not shown) to request reference original images and reference images for retraining AI model 1620. The server (not shown) may provide device 1000 with reference original images and reference images related to the camera type or lens type received from device 1000.

[0176] Alternatively, device 1000 may provide a server (not shown) with user preference information indicating user preferences, such as image style, shooting environment, and subject, to request reference original images and reference images for retraining AI model 1620. The server (not shown) may provide device 1000 with reference original images and reference images related to the user preference information received from device 1000.

[0177] In operation S1210, device 1000 can update AI model 1620 in device 1000 by using a reference original image for retraining and a reference image corresponding to the reference original image. Device 1000 can use a reference image received from a server (not shown) as a ground truth image, and retrain AI model 1620 by inputting the reference original image received from the server (not shown) into AI model 1620 and comparing the output image output from AI model 1620 with the reference image.

[0178] Processors and memory are used to perform AI-related functions according to this disclosure. The processor may include one or more processors. In this case, one or more processors may include general-purpose processors (e.g., CPUs, APs, and DSPs), dedicated graphics processors (e.g., GPUs and VPUs), or dedicated AI processors (e.g., NPUs). One or more processors control the input data to be processed based on predefined operating rules or AI models stored in memory. Alternatively, when one or more processors include a dedicated AI processor, the dedicated AI processor may be designed in a hardware architecture specifically designed to process a particular AI model.

[0179] Predefined operating rules or AI models are completed through training. When a predefined operating rule or AI model is completed through training, it means that a basic AI model is trained using multiple training data sets according to a learning algorithm, thereby completing the predefined operating rule or AI model configured to perform desired characteristics (or purposes). Training can be performed by a device for performing AI operations according to this disclosure, or by a server and / or system. The learning algorithm can include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples above.

[0180] AI models can include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network computation is performed by calculating the results of previous layers and these multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized using the results of training the AI ​​model. For example, multiple weight values ​​can be updated to reduce or minimize the loss or cost values ​​acquired by the AI ​​model during the training process. Artificial neural networks can include, but are not limited to, examples such as CNNs, DNNs, RNNs, RBMs, DBNs, BRDNNs, or deep Q-networks.

[0181] In embodiments of this disclosure, the original image can be used as input data for an AI model, and the output image data can be output from the AI ​​model. The AI ​​model can be completed through training. When an AI model is completed through training, this means that a basic AI model is trained using multiple training data sets according to a learning algorithm, and thus a predefined operating rule or AI model configured to perform desired characteristics (or purposes) is completed. The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network computation is performed through calculations between the computation results of previous layers and the multiple weight values. The AI ​​model can be used for object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, image enhancement, etc.

[0182] Embodiments of this disclosure can be implemented in the form of a recording medium including computer-executable instructions, such as a program module to be executed by a computer. A computer-readable medium can be any available medium accessible by a computer and includes volatile and non-volatile media, as well as removable and non-removable media. Computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile media, as well as removable and non-removable media, implemented using any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals.

[0183] Computer-readable storage media may be provided in the form of non-transitory storage media. When a storage medium is "non-transitory," it means that the storage medium is tangible and does not contain signals (e.g., electromagnetic waves), and does not limit the data to being stored semi-permanently or temporarily in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0184] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product may be traded as a commercial product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM This can be done electronically (e.g., by downloading or uploading) directly between two user devices (e.g., smartphones). For electronic distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0185] As used herein, the term “unit” can refer to a hardware component such as a processor or circuit, and / or a software component that can be executed by a hardware component such as a processor.

[0186] Throughout this disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0187] The above description of this disclosure is provided for illustrative purposes, and those skilled in the art will understand that various changes in form and detail can be readily made without departing from the scope of this disclosure. Therefore, it should be understood that the above embodiments should be considered descriptive only and not for limiting purposes. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a composable manner.

[0188] The scope of this disclosure is defined by the appended claims rather than by the specific embodiments, and it should be understood that all modifications derived from the claims and their equivalents are included within the scope of this disclosure.

Claims

1. A method for generating an output image performed by a device, wherein a subject has been captured from an original image in the output image, the method comprising: By using a first processor configured to control the device, a raw image corresponding to the subject in front of the camera sensor in the device is obtained by controlling the camera sensor. By using a second processor, the original image corresponding to the subject in front of the camera sensor is input into a first artificial intelligence (AI) model. The first AI model is trained to scale the brightness of the image, and the second processor is configured to perform image processing on the original image by using the AI ​​model. By using a second processor, tone map data output from the first AI model is obtained; The brightness of the original image is scaled based on the tone map data by using a second processor; By using a second processor, the original image with scaled brightness is input into a second AI model, which is then trained to analyze the features of the image. By using a second processor, multiple feature images output from a second AI model are obtained; By using a second processor, a third AI model modifies multiple feature images output from the second AI model based on at least one setting value related to image attributes; By using a second processor, an output image including the subject is generated based on multiple feature images modified according to at least one setting; as well as The output image is stored using the first processor.

2. The method according to claim 1, wherein, Generating the output image involves using a second processor to input multiple modified feature images into a fourth AI model to generate the output image.

3. The method according to claim 2, wherein, The first AI model was pre-trained to generate a tone map for scaling the brightness of each pixel in the original image. The second AI model is pre-trained to analyze multiple preset features in an image. The third AI model is pre-trained to modify multiple feature images based on at least one setting value related to image attributes. The fourth AI model is a model pre-trained to regress the output image from multiple feature images.

4. The method according to claim 2, wherein, Based on the original reference image and the reference output image generated from the original reference image through a preset image signal processing (ISP), the first AI model, the second AI model, and the third AI model are trained together.

5. The method according to claim 3, wherein, Based on the loss between the reference output image and the output image output from the reference original image by the first AI model, the second AI model and the third AI model, the first AI model and the third AI model are trained together. The reference output image is generated from the reference original image generated by combining multiple original images generated in burst mode through preset ISP processing.

6. The method according to claim 1, wherein, The original image is an image generated by the image sensor and color filter in the camera sensor, and has any one of the following patterns: Bayer pattern, RGBE pattern, RYYB pattern, CYYM pattern, CYGM pattern, RGBW Bayer pattern, and X-trans pattern.

7. The method according to claim 1, wherein, At least one setting includes either a setting for white balance adjustment or a setting for color correction.

8. The method according to claim 2, wherein, The first, second, and third AI models are selected by reflecting the preferences of the device's users.

9. The method according to claim 2, further comprising: Generate a real-time view image from the original image without using at least one of the first AI model, the second AI model, or the third AI model; and Displays the generated live view image.

10. The method according to claim 2, further comprising: Retrain the first AI model, the second AI model, and the third AI model.

11. The method according to claim 10, wherein, Retraining includes: Obtain a reference image for retraining and a corresponding original reference image; and The first, second, and third AI models were retrained using the obtained reference images and the original reference images.

12. An apparatus for generating an output image in which a subject has been captured from an original image, the apparatus comprising: Camera sensor; monitor; The first memory stores the first instructions for controlling the device; A first processor is configured to execute a first instruction stored in a first memory; The second memory stores at least one artificial intelligence (AI) model for performing image processing on the original image, and second instructions related to the execution of the AI ​​model. and The second processor is configured to execute at least one AI model and second instructions stored in a second memory. The first processor is also configured to acquire a raw image corresponding to the subject in front of the camera sensor by using the camera sensor. The second processor is also configured to input the raw image corresponding to the subject in front of the camera sensor into the first AI model, which is trained to scale the brightness of the image. The second processor is also configured to acquire tone map data output from the first AI model. The second processor is also configured to execute a second instruction to: The brightness of the original image is scaled based on tone map data; The original image, after brightness scaling, is input into a second AI model, which is then trained to analyze the features of the image. Obtain multiple feature images output from the second AI model; The third AI model modifies multiple feature images output from the second AI model based on at least one setting value related to image attributes; and An output image including the subject is generated based on multiple feature images modified according to at least one setting. The first processor is also configured to store the output image in a first memory.

13. A computer-readable recording medium having a program recorded thereon for performing the method of claim 1 on a computer.

Images