Electronic device comprising display for receiving external light, method, and non-transitory computer-readable storage medium

Abstract

This electronic device comprises: a memory which stores instructions and which includes one or more storage media; a display for receiving external light; and at least one processor including a central processing unit (CPU) and a neural processing unit (NPU), wherein, when executed individually or collectively by the at least one processor, the instructions can instruct the electronic device to: receive sensing data for the external light from the display; execute the sensing data on the NPU; provide the sensing data to a trained model; acquire, through the trained model, information related to an external material in contact with the display; and, in order to display the information through the display, control the display such that receiving of the external light through the display is stopped.

Description

Electronic device, method, and non-transitory computer-readable storage medium including a display configured to receive external light

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium including a display configured to receive external light.

[0002] An electronic device can receive external light from a display. For example, the external light can be received through pixels included in the display. The electronic device can receive external light from the display while the display is not displaying visual information. The electronic device can display visual information through the display while the display is not receiving external light.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] An electronic device is described. The electronic device may include a memory storing instructions and including one or more storage media, a display configured to receive external light, and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensing data regarding the external light from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensing data to a trained model executed on the NPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, through the trained model, information regarding an external substance contacting the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop receiving the external light through the display so as to display the information through the display.

[0005] A method is described. The method can be performed in an electronic device comprising at least one processor, the processor comprising a display configured to receive external light, a central processing unit (CPU) including processing circuitry, and a neural processing unit (NPU) including processing circuitry. The method can include receiving sensing data regarding the external light from the display. The method can include providing the sensing data to a trained model executed on the NPU. The method can include obtaining information related to an external substance contacting the display through the trained model. The method can include controlling the display to stop receiving the external light through the display so as to display the information through the display.

[0006] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display configured to receive external light and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry, cause the electronic device to receive sensing data regarding the external light from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to a trained model that is executed on the NPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain information related to an external substance that has come into contact with the display through the trained model. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop receiving the external light through the display to display the information through the display.

[0007] An electronic device is described. The electronic device may include a memory storing instructions and including one or more storage media, a display configured to receive external light, and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensing data regarding the external light from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensing data to a trained model executed on the NPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, through the trained model, a visual object representing an external substance contacted on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop receiving the external light through the display so as to display the visual object through the display.

[0008] A method is described. The method can be performed in an electronic device comprising at least one processor, the processor comprising a display configured to receive external light, a central processing unit (CPU) including processing circuitry, and a neural processing unit (NPU) including processing circuitry. The method can include receiving sensing data regarding the external light from the display. The method can include providing the sensing data to a trained model executed on the NPU. The method can include obtaining, through the trained model, a visual object representing an external substance contacted on the display. The method can include controlling the display to stop receiving the external light through the display so as to display the visual object through the display.

[0009] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display configured to receive external light and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry, cause the electronic device to receive sensing data regarding the external light from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to a trained model that is executed on the NPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain, through the trained model, a visual object representing an external substance that has come into contact on the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop receiving the external light through the display to display the visual object through the display.

[0010] Figure 1 illustrates an example of an external substance coming into contact with a display.

[0011] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0012] FIG. 3 is a flowchart illustrating exemplary operations of an electronic device for obtaining information related to an external substance contacted on a display.

[0013] Figure 4 illustrates an example of receiving external light from a display.

[0014] FIG. 5 is a flowchart illustrating exemplary operations of an electronic device for obtaining a visual object representing an external material and other visual objects.

[0015] Figure 6 illustrates an example of displaying a visual object or other visual object.

[0016] Figure 7 illustrates an example of a user interface (UI) related to an external substance contacting the display.

[0017] FIG. 8 is a flowchart illustrating exemplary operations of an electronic device for acquiring a visual object using a model.

[0018] FIG. 9 is a block diagram of an electronic device within a network environment according to various embodiments.

[0019] FIG. 10 illustrates an example of a generative artificial intelligence system according to one embodiment.

[0020] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0021] Figure 1 illustrates an example of an external substance coming into contact with a display.

[0022] Referring to FIG. 1, the electronic device (100) may be described as a rollable type smartphone. For example, the electronic device (100) may include a first housing part, a second housing part, and a flexible display (105) (e.g., the display (230) of FIG. 2).

[0023] For example, an external material may come into contact with the flexible display (105). For example, the external material may include an external object and a foreign body. When the external material comes into contact with the flexible display (105), the electronic device (100) may experience an error in receiving a touch input through the flexible display (105). For example, when the electronic device (100) experiences an error in receiving a touch input, the user may experience discomfort.

[0024] For example, the second housing part of the electronic device (100) is movable relative to the first housing part between the collapsed state and the expanded state of the electronic device (100).

[0025] Can be combined. The flexible display (105) can include a first display area and a second display area. For example, the first display area of ​​the flexible display (105) can be arranged on the first housing part. For example, the second display area of ​​the flexible display (105) can be configured to extend from the first display area and be rolled into the first housing part along the second housing part that moves relative to the first housing part, or be exposed outside the first housing part.

[0026] The electronic device (100) can be switched from an expanded state to a collapsed state, or from a collapsed state to an expanded state. The electronic device (100) may include a cover (115) to prevent an external substance (110) that comes into contact with the flexible display (105) from entering the first housing when switching from the expanded state to the collapsed state. Even if the electronic device (100) physically prevents the entry of the external substance (110) using the cover (115), the external substance (110) may enter the first housing. The electronic device (100) may be damaged or broken as the external substance (110) enters the first housing. The user may feel discomfort due to damage or breakage of the electronic device (100) caused by the external substance (110).

[0027] A method for alleviating inconveniences caused by a foreign substance (110) coming into contact with a flexible display (105) may be required. To alleviate such user inconvenience, the electronic device (100) may detect or identify the foreign substance (110) coming into contact with the flexible display (105). The electronic device (100) may request the user to remove the foreign substance (110) by displaying information about the foreign substance (110) through the flexible display (105).

[0028] The electronic device (100) may perform the operations exemplified in the descriptions of FIGS. 3 to 8 to display information about an external material (110). The electronic device (100) may include components for performing the operations. The components may be exemplified in the description of FIG. 2.

[0029] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0030] Referring to FIG. 2, the electronic device (200) may be one of various forms of mobile devices, such as smartphones having various form factors (e.g., bar-type smartphones, foldable-type smartphones, or rollable-type smartphones), tablets, wearable devices, cellular phones, laptops, and / or other similar computing devices. For example, the electronic device (200) may include the electronic device (100) of FIG. 1 or may correspond to the electronic device (100) of FIG. 1. For example, the electronic device (200) may include at least a portion of the electronic device (901) of FIG. 9 or may correspond to at least a portion of the electronic device (901) of FIG. 9. For example, the electronic device (200) may include at least one processor (210), a memory (220), and a display (230).

[0031] At least one processor (210) may include a processing circuit. At least one processor (210) may include a central processing unit (CPU) (210-1) (e.g., including a processing circuit). At least one processor (210) may include a neural processing unit (NPU) (210-2) (e.g., including a processing circuit) and a graphic processing unit (GPU) (210-3) (e.g., including a processing circuit). For example, the NPU (210-2) may be used to execute a trained model and process data obtained from the trained model. For example, the GPU (210-3) may be used to control the system, manage the system, and execute applications. At least one processor (210) may include a digital signal processor (DSP) and an image signal processor (ISP). For example, the DSP may be used to process signals. For example, the ISP may be used to perform preprocessing of image data. For example, at least one processor (210) may be configured to control the memory (220) and the display (230). The at least one processor (210) may be configured to individually or collectively execute instructions stored in the memory (220) to cause the electronic device (200) (or the electronic device (100)) to perform at least some of the operations illustrated in the description of FIG. 1. The at least one processor (210) may be configured to individually or collectively execute instructions stored in the memory (220) to cause the electronic device (200) to perform at least some of the operations illustrated in the description of FIGS. 3 to 8.

[0032] The memory (220) may include one or more storage media. The memory (220) may store various data used by at least one component of the electronic device (200) (e.g., at least one processor (210), the memory (220), and / or the display (230)). The memory (220) may include a unified frame buffer. The unified frame buffer may be connected to the NPU (210-2) and the GPU (210-3). The unified frame buffer may be used to increase the efficiency of data transfer by sharing data of the NPU (210-2) with the GPU (210-3). For example, the data may include input data or output data for software and commands related thereto. The memory (220) may include volatile memory or non-volatile memory.

[0033] The display (230) may be configured to visualize information (or signals) provided from at least one processor (210). The display (230) may include a display driver integrated circuitry (DDIC) (230-1) and a display panel. For example, the display (230) may include a pixel sensor. For example, the display (230) may be configured to receive external light through the pixel sensor. For example, the display (230) may be configured to display visual information by emitting light. For example, the pixel sensor may include a perovskite pixel. For example, the display (230) composed of perovskite pixels may be configured to be bent by stably changing the shape of the display (230). The structure of the perovskite pixel is exemplified in the description of FIG. 4. For example, the display (230) may include a timing controller (TCON). For example, the TCON may be used to control the timing of the operation of the display panel, to control the display (230) to receive external light from the display (230), or to control the display (230) to display visual information through the display (230). For example, the display (230) may include an analog-to-digital converter (ADC) and a digital-to-analogue converter (DAC). For example, the ADC may be used to convert an analog signal acquired from a pixel sensor into digital data. For example, the DAC may be used to convert digital brightness data into an analog voltage signal. For example, the display (230) may include a gate driver and a source driver.For example, a gate driver may be used to select a row of pixels and activate the pixels. For example, a source driver may be used to transmit data and signals to the pixels and control the brightness of the pixels.

[0034] The electronic device (200) illustrated in the description of FIG. 2 can execute at least some of the operations illustrated in the description of FIGS. 3 to 8. The operations illustrated in the description of FIGS. 3 to 8 can be caused by (or within) the electronic device (200) under the control of at least one processor (210) (or DDIC (230-1), or display (230)).

[0035] FIG. 3 is a flowchart illustrating exemplary operations of an electronic device for obtaining information related to an external substance contacted on a display.

[0036] Referring to FIG. 3, in operation 300, at least one processor (210) (or DDIC (230-1)) may receive sensing data on external light from the display (230). For example, at least one processor (210) (or DDIC (230-1)) may receive sensing data on external light from the display (230) while stopping displaying visual information through the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may sequentially receive sensing data from pixels of the display (230) using a gate driver and a source driver. For example, the operation of stopping displaying visual information through the display (230) in order to receive sensing data on external light from the display (230) may be performed by a TCON of the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can receive sensing data from the display (230) by converting an analog signal for external light generated from the display (230) into digital data using an ADC.

[0037] For example, at least one processor (210) (or DDIC (230-1)) may stop displaying visual information through the display (230) based on a user input and receive sensing data about external light from the display (230). In another embodiment, at least one processor (210) (or DDIC (230-1)) may stop displaying visual information through the display (230) based on a cycle and receive sensing data about external light from the display (230). In another embodiment, at least one processor (210) (or DDIC (230-1)) may stop displaying visual information through the display (230) based on a first cycle and receive sensing data about external light from the display (230). At least one processor (210) (or DDIC (230-1), or display (230)) can detect an external substance that has come into contact with the display (230) using sensing data about external light. At least one processor (210) (or DDIC (230-1), or display (230)) can, based on the detection, stop displaying visual information through the display (230) according to a second period shorter than the first period, and receive sensing data about the external light from the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can relatively accurately identify an external substance that has come into contact with the display (230) by receiving sensing data about the external light from the display (230) according to a second period shorter than the first period. Receiving sensing data about the external light from the display (230) is exemplified in the description of FIG. 4.

[0038] In operation 310, at least one processor (210) (or DDIC (230-1), or display (230)) may provide sensing data for external light to a trained model running on an NPU (210-2). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may load weights and parameters of the trained model from a dynamic random access memory (DRAM).

[0039] For example, the model can be trained to identify a foreign object in contact with the display (230) using sensing data on external light. For example, the model can be trained to simultaneously detect and classify foreign objects through multi-task learning. For example, the trained model can include an artificial intelligence (AI) model, a deep learning model, and / or a machine learning model. For example, the trained model can include a convolutional neural network (CNN) algorithm specialized for extracting images from sensing data and detecting foreign objects in contact with the display (230). For example, the trained model can include the generative AI model (1030) of FIG. 10.

[0040] For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data before providing the sensing data to the trained model. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data by performing normalization of the sensing data and noise removal of the sensing data. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data by extracting features of an external material in contact with the display (230) from the sensing data.

[0041] In operation 320, at least one processor (210) (e.g., NPU (210-2)) may acquire information related to an external substance contacting the display (230) through a trained model. For example, at least one processor (210) (e.g., NPU (210-2)) may store the acquired information in DRAM. Information related to an external substance contacting the display (230) is exemplified in the description of FIG. 4.

[0042] Figure 4 illustrates an example of receiving external light from a display.

[0043] Referring to FIG. 4, state (400) can be described as a state of receiving sensing data for external light (420, 425) from the display (230). For example, within state (400), at least one processor (210) (or DDIC (230-1)) can stop displaying visual information through the display (230) in order to receive sensing data for external light (420, 425) from the display (230). For example, at least one processor (210) (or DDIC (230-1)) can receive sensing data for external light (420, 425) from the display (230) while stopping displaying visual information through the display (230). For example, a mode of the electronic device (200) that stops displaying visual information through the display (230) and receives sensing data for external light (420, 425) from the display (230) may be referred to as a light-receiving mode.

[0044] Pixels (e.g., pixel (410) and pixel (415)) can receive external light or emit light by having a structure (401). For example, the structure (401) can include a cathode layer (401-1), an electron transport layer (ETL) (401-2), a perovskite layer (401-3), a hole transport layer (HTL) (401-4), and an anode layer (401-5). For example, the perovskite layer (401-3) can be composed of perovskite. For example, the perovskite can have a crystal structure such as ABX3. For example, A and B can be different cations having a relatively large size difference, and X can be defined as an anion bonded to A and B. For example, perovskite can absorb external light and separate charges by having the above crystal structure.

[0045] For example, the perovskite layer (401-3) can generate electrons and holes based on external light. For example, the electrons can move to the electron transport layer (401-2), and the holes can move to the hole transport layer (401-4). For example, as the electrons are collected in the cathode layer (401-1), and the holes are collected in the anode layer (401-5), current can be generated. For example, at least one processor (210) (or DDIC (230-1), or display (230)) can obtain sensing data for external light based on the generated current. For example, the anode layer (401-5) can be made of a reflective metal and can reflect external light that is not absorbed by the perovskite layer (401-3).

[0046] For example, at least one processor (210) (or DDIC (230-1)) can apply a voltage to the cathode layer (401-1) and the anode layer (401-5) to emit light using the pixels (e.g., pixel (410) and pixel (415)). For example, when the cathode layer (401-1) and the anode layer (401-5) inject electrons and holes into the electron transport layer (401-2) and the hole transport layer (401-4), the electrons and holes can pass through the electron transport layer (401-2) and the hole transport layer (401-4) and combine in the perovskite layer (401-3). For example, the combination of electrons and holes in the perovskite layer (401-3) can cause the perovskite to transition to an excited state. For example, pixels (e.g., pixel (410) and pixel (415)) can emit light by transitioning perovskite from an excited state to a ground state within the perovskite layer (401-3). For example, the perovskite layer (401-3) can adjust the wavelength of the emitted light by changing the composition of the halogen element. For example, the perovskite layer (401-3) can express light within the visible light range by adjusting the wavelength of the emitted light.

[0047] The display (230) may include pixels (e.g., pixels (410) and pixels (415)). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may receive sensing data for external light (420, 425) from each of the pixels (e.g., pixels (410) and pixels (415)). For example, an external material (405) may be positioned on a path through which at least some of the pixels (e.g., pixels (410) and pixels (415)) included in the display (230) receive external light (420). For example, at least some pixels (e.g., pixels (410)) may be unable to receive external light (420) or may receive relatively little external light (420) due to the presence of an external material (405) on a path through which at least some pixels (e.g., pixels (410)) receive external light (420). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may receive sensing data for relatively little external light (420) from at least some pixels (e.g., pixels (410)) where an external material (405) is positioned on a path through which at least some pixels (e.g., pixels (410)) receive external light (420).

[0048] For example, at least one processor (210) (or DDIC (230-1), or display (230)) can receive sensing data about external light (425) from pixels (e.g., pixels (415)) that do not have an external object (405) positioned on a path through which at least some pixels (e.g., pixels (415)) receive external light (425). For example, pixels (e.g., pixels (415)) that do not have an external object (405) positioned on a path through which at least some pixels (e.g., pixels (415)) receive external light (425) can receive relatively more external light (425) than pixels (e.g., pixels (410)) that have an external object (405) positioned on a path through which at least some pixels (e.g., pixels (410)) receive external light (420). For example, sensing data received from at least some pixels (e.g., pixels (410)) in which an external material (405) is positioned on a path through which at least some pixels (e.g., pixels (410)) receive external light (420) may be different from sensing data received from pixels (e.g., pixels (415)) in which an external material (405) is not positioned on a path through which at least some pixels (e.g., pixels (410)) receive external light (420). For example, sensing data for external light may include data about coordinates of pixels of the display (230) (e.g., pixel (410) and pixel (415)), data about response times of the pixels (e.g., pixel (410) and pixel (415)), data about the amount of light of the pixels (e.g., pixel (410) and pixel (415)), data about spectra of the pixels (e.g., pixel (410) and pixel (415)), and / or data about a screen displayed on the display (230).

[0049] For example, at least one processor (210) (or DDIC (230-1), or display (230)) may run on the NPU (210-2) and provide the acquired sensing data to a trained model. For example, the trained model may identify a location of a pixel on the display (230) where a foreign substance is likely to be located based on sensing data for external light. For example, the trained model may identify an area on the display (230) where a foreign substance is located by grouping pixels having similar characteristics. For example, the trained model may obtain information on the shape of the foreign substance (405) and the material of the foreign substance (405) based on the relationship and time series information between pixels (e.g., pixel (410) and pixel (415)). For example, the trained model can obtain information about an external substance (405) that has come into contact with the display (230) based on the difference between sensing data received from at least some pixels (e.g., pixels (410)) in which the external substance (405) is located on a path through which at least some pixels (e.g., pixels (410)) receive external light (420) and sensing data received from pixels (e.g., pixels (415)) that have not come into contact with the external substance (405) on a path through which at least some pixels (e.g., pixels (415)) receive external light (425).

[0050] For example, at least one processor (210) (e.g., NPU (210-2)) can obtain information related to a foreign substance (405) that has come into contact with the display (230) through a trained model. For example, the information related to the foreign substance (405) can include information about a probability that the foreign substance (405) is located on pixels (e.g., pixels (410) and pixels (415)), information about a movement of the foreign substance (405) on pixels (e.g., pixels (410) and pixels (415)), information about a type of the foreign substance (405), information about a material of the foreign substance (405), information about a size of the foreign substance (405), and / or information about a shape of the foreign substance (405). For example, at least one processor (210) (e.g., NPU (210-2)) may use information related to the acquired external substance (405) to identify the type, size, shape, texture, and / or material of the acquired external substance (405). For example, at least one processor (210) (e.g., NPU (210-2)) may store information related to the acquired external substance (405) in DRAM.

[0051] In order to request the user to remove the foreign substance (405), it may be required to provide the user with information related to the foreign substance (405). For example, at least one processor (210) (or DDIC (230-1)) may display the information related to the foreign substance (405) to the user via the display (230). For example, at least one processor (210) (or DDIC (230-1)) may control the display (230) to stop receiving external light (420, 425) via the display (230) in order to display the information related to the foreign substance (405) via the display (230). For example, in order to display the information related to the foreign substance (405), it may be required to visualize the information related to the foreign substance (405) (or acquire a visual object representing the foreign substance (405). Visualizing information related to an external substance (405) (or obtaining a visual object representing an external substance (405)) is exemplified within the description of FIG. 5.

[0052] FIG. 5 is a flowchart illustrating exemplary operations of an electronic device for obtaining a visual object representing an external material and other visual objects.

[0053] Referring to FIG. 5, in operation 500, at least one processor (210) (or DDIC (230-1), or display (230)) may provide information about an external substance in contact with the display (230), obtained from a trained model, to another trained model running on the NPU (210-2). For example, the information about the external substance may be provided to the other model from DRAM. For example, at least one processor (210) (e.g., NPU (210-2)) may provide an image of a screen being displayed on the display (230) from a unified frame buffer to the other model.

[0054] For example, another model may be trained to visualize an external object using information about the external object, or to obtain a visual object representing the external object. For example, the trained other model may include an artificial intelligence (AI) model, a deep learning model, and / or a machine learning model. For example, the trained other model may include a CNN algorithm. For example, the trained other model may include the generative AI model (1030) of FIG. 10.

[0055] For example, at least one processor (210) (or DDIC (230-1), or display (230)) may preprocess information about an external object before providing the information about the external object to another trained model. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may preprocess information about an external object by generating a mask for the external object or by vectorizing an image of a screen being displayed on the display (230).

[0056] In operation 510, at least one processor (210) (or DDIC (230-1), or display (230)) may obtain a visual object representing an external substance contacted on the display (230) through another trained model. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may synthesize, through another trained model, a visual object representing an external substance into an image of a screen being displayed on the display (230), or display it on an image of a screen being displayed on the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may store the visual object in an integrated frame buffer. Displaying a visual object representing an external substance is exemplified in the description of FIG. 6.

[0057] For example, rendering processing may be required on visual objects to effectively display external objects to the user. For example, rendering processing on visual objects may be performed on the GPU (210-3).

[0058] In operation 520, at least one processor (210) (e.g., GPU (210-3)) may obtain a visual object representing an external material from the unified frame buffer. For example, at least one processor (210) (e.g., GPU (210-3)) may obtain another visual object that visually emphasizes the visual object by performing rendering processing on the visual object. For example, the other visual object may be stored in the unified frame buffer. For example, the other visual object may represent an external material that has come into contact on the display (230). For example, at least one processor (210) (e.g., GPU (210-3)) may obtain another visual object by visually emphasizing the visual object by using color space conversion, image filtering, mask operation, and / or blending. For example, at least one processor (210) (e.g., GPU (210-3)) may composite or display another visual object into an image for a screen being displayed on a display (230).

[0059] As a non-limiting example, at least one processor (210) (e.g., GPU (210-3)) may acquire another visual object for augmented reality (AR) by performing three-dimensional (3D) modeling, texture mapping, viewpoint change, and / or projection processing on the visual object, but is not limited thereto.

[0060] At operation 530, at least one processor (210) (e.g., GPU (210-3)) may provide another visual object stored in the unified frame buffer to the DDIC (230-1) or the display (230). The at least one processor (210) (or the DDIC (230-1), or the display (230)) may display the other visual object through the display (230). The at least one processor (210) (or the DDIC (230-1), or the display (230)) may control the display (230) to stop receiving external light through the display (230) in order to display the other visual object through the display (230). For example, the operation of controlling the display (230) to stop receiving external light through the display (230) in order to display the other visual object through the display (230) may be performed by the TCON. Displaying other visual objects is exemplified within the description of Figure 6.

[0061] Figure 6 illustrates an example of displaying a visual object or other visual object.

[0062] Referring to FIG. 6, a state (600) may be described as a state in which a visual object (605) is displayed through a display (230). For example, within the state (600), at least one processor (210) (or DDIC (230-1), or display (230)) may stop receiving sensing data for external light from the display (230) in order to display the visual object (605). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may display the visual object (605) through the display (230) while stopping receiving sensing data for external light from the display (230). For example, a mode of the electronic device (200) that stops receiving sensing data about external light from the display (230) and displays visual information through the display (230) may be referred to as a light-emitting mode.

[0063] At least one processor (210) (or DDIC (230-1), or display (230)) can display a visual object (605) representing an external substance at a location corresponding to a location of the external substance contacted on the display (230) through the display (230). For example, the size of the visual object (605) can correspond to the size of the external substance contacted on the display (230). For example, the shape of the visual object (605) can correspond to the shape of the external substance contacted on the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can display the visual object (605) by overlaying it on an image of a screen being displayed on the display (230). For example, by displaying a visual object (605) by overlaying it on an image of a screen being displayed on a display (230), an area on the image where the visual object (605) is located can be emphasized.

[0064] At least one processor (210) (or DDIC (230-1), or display (230)) can provide the user with information about an external substance contacted on the display (230) by displaying a visual object (605). For example, by having the user view the visual object (605) and remove the external substance from the display (230), inconvenience due to errors associated with touch input can be resolved. For example, when the electronic device (200) is a rollable type smartphone, inconvenience due to damage or breakage of the electronic device (200) caused by the external substance can be resolved by having the user view the visual object (605) and remove the external substance from the display (230). For example, if the electronic device (200) is a foldable type smartphone, the user can eliminate inconvenience caused by damage or breakage of the hinge structure of the electronic device (200) by removing external materials that have come into contact with the display area configured to be bent of the flexible display.

[0065] State (610) can be described as a state in which another visual object (615) is displayed through the display (230). For example, within state (610), at least one processor (210) (or DDIC (230-1), or display (230)) can stop receiving sensing data for external light from the display (230) in order to display another visual object (615). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can display another visual object (615) through the display (230) in a light-emitting mode.

[0066] At least one processor (210) (or DDIC (230-1), or display (230)) can display another visual object (615) representing the foreign substance on the display (230) at a location corresponding to the location of the foreign substance contacted on the display (230). For example, the other visual object (615) can be visually emphasized by applying a visual effect for color space transformation, image filtering, mask operation, and / or blending to the visual object (605). For example, the other visual object (615) can have a different color depending on the type of the foreign substance (e.g., dust, fingerprints, and / or liquid) (or the severity of the foreign substance). For example, the other visual object (615) can have a 3D shape representing the foreign substance based on the rendering processing for the visual object (605). For example, since the other visual object (615) has a 3D shape, the other visual object (615) can be viewed from various angles. For example, the other visual object (615) can have an outline with a visual effect applied. For example, the visual style of the other visual object (615) (e.g., the color, transparency, and / or animation effect of the other visual object (615)) can be set by the user. For example, the other visual object (615) can be acquired by the GPU (210-3) and provided to the DDIC (230-1) from the unified frame buffer.

[0067] For example, at least one processor (210) (or DDIC (230-1), or display (230)) can display another visual object (615) by overlaying it on an image of a screen being displayed on the display (230). For example, by displaying another visual object (615) by overlaying it on an image of a screen being displayed on the display (230), an area of ​​the image where the other visual object (615) is located can be emphasized. For example, at least one processor (210) (or DDIC (230-1), or display (230)) can provide the user with information about an external substance contacted on the display (230) by displaying another visual object (615). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can enlarge another visual object (615) based on a user's pinch input (or zoom input), or reduce another visual object (615) based on an unpinch input (or zoom out input). For example, another visual object (615) can blink on the display (230) periodically. For example, when a user looks at another visual object (615) and removes a foreign substance from the display (230), inconvenience due to an error related to a touch input can be resolved. For example, when the electronic device (200) is a rollable type smartphone, inconvenience due to damage or breakage of the electronic device (200) caused by a foreign substance can be resolved when a user looks at the visual object (605) and removes the foreign substance from the display (230).For example, if the electronic device (200) is a foldable type smartphone, the user can eliminate inconvenience caused by damage or breakage of the hinge structure of the electronic device (200) by removing external materials that have come into contact with the display area configured to be bent of the flexible display.

[0068] At least one processor (210) (or DDIC (230-1), or display (230)) can control the display (230) to stop displaying the other visual object (615) (or visual object (605)) to display another visual object (615) (or visual object (605)) through the display (230) and receive sensing data about external light from the display (230). For example, the at least one processor (210) (or DDIC (230-1), or display (230)) can use the sensing data to identify whether a foreign substance that has come into contact with the display (230) has been removed. For example, the at least one processor (210) (or DDIC (230-1), or display (230)) can stop (or refrain from, or not display) the other visual object (615) (or visual object (605)) based on the removed foreign substance.

[0069] For example, at least one processor (210) (or DDIC (230-1), or display (230)) may additionally display a user interface related to an external material together with another visual object (615) (or visual object (605)). The user interface related to the external material is exemplified within the description of FIG. 7.

[0070] Figure 7 illustrates an example of a user interface (UI) related to an external substance contacting the display.

[0071] Referring to FIG. 7, a state (700) may be described as a state in which a foreign substance in contact with the display (230) is detected. For example, within the state (700), at least one processor (210) (or DDIC (230-1), or display (230)) may additionally display a pop-up window (705) together with other visual objects (615) based on detecting a foreign substance using sensing data for external light received from the display (230). For example, the pop-up window (705) may include text indicating the type of the foreign substance and text notifying the detection of the foreign substance. For example, the pop-up window (705) may include an executable object (710) for requesting that a method for removing the foreign substance be provided. According to another embodiment, at least one processor (210) (or DDIC (230-1), or display (230)) may, based on detecting a foreign substance, display a pop-up window (705) and additionally output audio through a speaker of the electronic device (200) to indicate detection of the foreign substance contacting the display (230).

[0072] At least one processor (210) (or DDIC (230-1), or display (230)) can receive an input (715) for an executable object (710). For example, the input (715) can include a touch input having a contact point on the executable object (710). For example, the input (715) can be received via the display (230) (e.g., a touchscreen). For example, at least one processor (210) (or DDIC (230-1), or display (230)) can stop (or refrain from, or not display) the pop-up window (705) based on not receiving an input (715) for the executable object (710) until a reference time elapses from the time the pop-up window (705) is displayed. In another embodiment, the input (715) may include a touch input having a contact point on another visual object (615).

[0073] Based on the input (715), the electronic device (200) can transition from state (700) to state (720). For example, within state (720), at least one processor (210) (or DDIC (230-1), or display (230)) can display a user interface screen (725) for removing a foreign substance that has come into contact on the display (230) based on the input (715). For example, the screen (725) can include an image (730) of the electronic device (200) with another visual object (615) superimposed thereon. For example, the screen (725) can include an image (735) representing a method for removing a foreign substance that has come into contact on the display (230) and guidance (740) describing a method for removing a foreign substance that has come into contact on the display (230). For example, an image (735) representing a method for removing a foreign substance may include an image representing a scene of removing a foreign substance that has come into contact with the display (230). For example, guidance (740) may include text explaining a method for removing a foreign substance according to the type of foreign substance.

[0074] At least one processor (210) (or DDIC (230-1), or display (230)) can provide a method for the user to remove a foreign substance that has come into contact on the display (230) by displaying a user interface screen (725). For example, by allowing the user to view images (730, 735) and guidance (740) within the screen (725) and remove the foreign substance from the display (230), inconvenience due to errors associated with touch input can be resolved. For example, when the electronic device (200) is a rollable type smartphone, by allowing the user to view images (730, 735) and guidance (740) within the screen (725) and remove the foreign substance from the display (230), inconvenience due to damage or breakage of the electronic device (200) caused by the foreign substance can be resolved. For example, if the electronic device (200) is a foldable type smartphone, the user can eliminate inconvenience caused by damage or breakage of the hinge structure of the electronic device (200) by removing external materials that have come into contact with the display area configured to be bent of the flexible display.

[0075] In another embodiment, at least one processor (210) (or DDIC (230-1), or display (230)) may obtain another visual object (615) representing an external material through a model that is an integrated model and another model. Obtaining another visual object (615) through a single model that is an integrated model and another model is exemplified in the description of FIG. 8.

[0076] FIG. 8 is a flowchart illustrating exemplary operations of an electronic device for acquiring a visual object using a model.

[0077] Referring to FIG. 8, in operation 800, at least one processor (210) (or DDIC (230-1)) may receive sensing data on external light from the display (230). For example, at least one processor (210) (or DDIC (230-1)) may receive sensing data on external light from the display (230) while stopping displaying visual information through the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may sequentially receive sensing data from pixels using a gate driver and a source driver. For example, the operation of stopping displaying visual information through the display (230) to receive sensing data on external light from the display (230) may be performed by a TCON of the display (230). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may receive sensing data from the display (230) by converting an analog signal for external light generated from the display (230) into sensing data using an ADC. For example, operation 800 may correspond to operation 300 of FIG. 3.

[0078] In operation 810, at least one processor (210) (or DDIC (230-1), or display (230)) may provide sensing data for external light to a trained model running on an NPU (210-2). For example, at least one processor (210) (or DDIC (230-1), or display (230)) may load weights and parameters of the trained model from a dynamic random access memory (DRAM).

[0079] For example, the model can be trained to identify a foreign substance in contact with the display (230) using sensing data about external light, visualize the foreign substance, or obtain a visual object representing the foreign substance. For example, the model can be trained to detect and classify foreign substances simultaneously through multi-task learning. For example, the trained model can include an artificial intelligence (AI) model, a deep learning model, and / or a machine learning model. For example, the trained model can include a convolutional neural network (CNN) algorithm specialized for extracting images and detecting foreign objects in contact with the display (230). For example, the trained model can include the generative AI model (1030) of FIG. 10. For example, the trained model can be described as an integrated model of the trained model in operation 310 of FIG. 3 and another model trained in operation 500 of FIG. 5.

[0080] For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data before providing the sensing data to the trained model. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data by performing normalization of the sensing data and noise removal of the sensing data. For example, at least one processor (210) (or DDIC (230-1), or display (230)) may perform preprocessing of the sensing data by extracting features for detecting a foreign substance in contact with the display (230) from the sensing data.

[0081] For example, the trained model can convert the sensing data into a vector by performing an embedding process on the sensing data. For example, the trained model can add location information of pixels of the display (230) to the sensing data on which the embedding process has been performed. For example, the trained model can model the relationship between pixels and extract the characteristics of an external material. For example, the trained model can identify the type of the external material based on the extracted characteristics of the external material and determine a visualization style for visualizing the external material. For example, the trained model can generate header information for visualizing the external material based on information about the external material.

[0082] For example, at least one processor (210) (or DDIC (230-1), or display (230)) can process sensing data relatively quickly and reduce the complexity of the processing of sensing data by using a model that integrates the trained model in operation 310 of FIG. 3 and another trained model in operation 500 of FIG. 5.

[0083] In operation 820, at least one processor (210) (e.g., NPU (210-2)) may obtain a visual object representing an external substance that has come into contact on the display (230) through a trained model. For example, at least one processor (210) (e.g., NPU (210-2)) may synthesize or display a visual object representing an external substance into an image of a screen being displayed on the display (230) through the trained model. For example, at least one processor (210) (e.g., NPU (210-2)) may store the visual object in an integrated frame buffer. The display of the visual object representing the external substance may refer to the description of FIG. 6.

[0084] For example, rendering processing may be required on visual objects to effectively display external objects to the user. For example, rendering processing on visual objects may be performed on the GPU (210-3).

[0085] In operation 830, at least one processor (210) (e.g., GPU (210-3)) may obtain a visual object representing an external material from the unified frame buffer. For example, at least one processor (210) (e.g., GPU (210-3)) may obtain another visual object that visually emphasizes the visual object by performing rendering processing on the visual object. For example, the other visual object may be stored in the unified frame buffer. For example, the other visual object may represent an external material that has come into contact on the display (230). For example, at least one processor (210) (e.g., GPU (210-3)) may visually emphasize the visual object by using color space conversion, image filtering, masking operations, and / or blending. For example, at least one processor (210) (e.g., GPU (210-3)) may composite or display the other visual object into an image for a screen being displayed on the display (230).

[0086] As a non-limiting example, at least one processor (210) (e.g., GPU (210-3)) may acquire another visual object for augmented reality (AR) by performing three-dimensional (3D) modeling, texture mapping, viewpoint change, and / or projection processing on the visual object, but is not limited thereto.

[0087] At operation 840, at least one processor (210) (e.g., GPU (210-3)) may provide another visual object stored in the unified frame buffer to the DDIC (230-1) or the display (230). The at least one processor (210) (or the DDIC (230-1), or the display (230)) may display the other visual object through the display (230). The at least one processor (210) (or the DDIC (230-1), or the display (230)) may control the display (230) to stop receiving external light through the display (230) in order to display the other visual object through the display (230). For example, the operation of controlling the display (230) to stop receiving external light through the display (230) in order to display the other visual object through the display (230) may be performed by the TCON. For displaying other visual objects, reference may be made to the description in Fig. 6.

[0088] FIG. 9 is a block diagram of an electronic device within a network environment according to various embodiments.

[0089] Referring to FIG. 9, in a network environment (900), an electronic device (901) may communicate with an electronic device (902) via a first network (998) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (904) or a server (908) via a second network (999) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (901) may communicate with the electronic device (904) via the server (908). According to one embodiment, the electronic device (901) may include a processor (920), a memory (930), an input module (950), an audio output module (955), a display module (960), an audio module (970), a sensor module (976), an interface (977), a connection terminal (978), a haptic module (979), a camera module (980), a power management module (988), a battery (989), a communication module (990), a subscriber identification module (996), or an antenna module (997). In some embodiments, the electronic device (901) may omit at least one of these components (e.g., the connection terminal (978)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (976), the camera module (980), or the antenna module (997)) may be integrated into one component (e.g., the display module (960)).

[0090] The processor (920) may, for example, execute software (e.g., a program (940)) to control at least one other component (e.g., a hardware or software component) of the electronic device (901) connected to the processor (920) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (920) may store commands or data received from other components (e.g., a sensor module (976) or a communication module (990)) in a volatile memory (932), process the commands or data stored in the volatile memory (932), and store result data in a non-volatile memory (934). According to one embodiment, the processor (920) may include a main processor (921) (e.g., a central processing unit or an application processor) or an auxiliary processor (923) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (921). For example, when the electronic device (901) includes the main processor (921) and the auxiliary processor (923), the auxiliary processor (923) may be configured to use less power than the main processor (921) or to be specialized for a given function. The auxiliary processor (923) may be implemented separately from the main processor (921) or as a part thereof.

[0091] The auxiliary processor (923) may control at least a portion of functions or states associated with at least one component (e.g., a display module (960), a sensor module (976), or a communication module (990)) of the electronic device (901), for example, on behalf of the main processor (921) while the main processor (921) is in an inactive (e.g., sleep) state, or together with the main processor (921) while the main processor (921) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (923) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (980) or a communication module (990)). In one embodiment, the auxiliary processor (923) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (901) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (908)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0092] The memory (930) can store various data used by at least one component (e.g., the processor (920) or the sensor module (976)) of the electronic device (901). The data can include, for example, software (e.g., the program (940)) and input data or output data for commands related thereto. The memory (930) can include a volatile memory (932) or a non-volatile memory (934).

[0093] The program (940) may be stored as software in the memory (930) and may include, for example, an operating system (942), middleware (944), or an application (946).

[0094] The input module (950) can receive commands or data to be used in a component of the electronic device (901) (e.g., a processor (920)) from an external source (e.g., a user) of the electronic device (901). The input module (950) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0095] The audio output module (955) can output audio signals to the outside of the electronic device (901). The audio output module (955) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0096] The display module (960) can visually provide information to an external party (e.g., a user) of the electronic device (901). The display module (960) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0097] The audio module (970) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (970) can acquire sound through the input module (950), output sound through the sound output module (955), or an external electronic device (e.g., electronic device (902)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (901).

[0098] The sensor module (976) can detect the operating status (e.g., power or temperature) of the electronic device (901) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (976) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0099] The interface (977) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (901) with an external electronic device (e.g., the electronic device (902)). In one embodiment, the interface (977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0100] The connection terminal (978) may include a connector through which the electronic device (901) may be physically connected to an external electronic device (e.g., the electronic device (902)). In one embodiment, the connection terminal (978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0101] The haptic module (979) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (979) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0102] The camera module (980) can capture still images and videos. According to one embodiment, the camera module (980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0103] The power management module (988) can manage the power supplied to the electronic device (901). According to one embodiment, the power management module (988) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0104] A battery (989) may power at least one component of the electronic device (901). In one embodiment, the battery (989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0105] The communication module (990) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (901) and an external electronic device (e.g., electronic device (902), electronic device (904), or server (908)), and the performance of communication through the established communication channel. The communication module (990) may operate independently from the processor (920) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (990) may include a wireless communication module (992) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (994) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (904) via a first network (998) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (999) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (992) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (996) to identify or authenticate the electronic device (901) within a communication network such as the first network (998) or the second network (999).

[0106] The wireless communication module (992) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (992) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (992) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (992) can support various requirements specified in the electronic device (901), an external electronic device (e.g., the electronic device (904)), or a network system (e.g., the second network (999)). According to one embodiment, the wireless communication module (992) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 94 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0107] The antenna module (997) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (997) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (998) or the second network (999), may be selected from the plurality of antennas, for example, by the communication module (990). A signal or power may be transmitted or received between the communication module (990) and the external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (997).

[0108] According to various embodiments, the antenna module (997) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0109] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0110] According to one embodiment, commands or data may be transmitted or received between the electronic device (901) and an external electronic device (904) via a server (908) connected to a second network (999). Each of the external electronic devices (902 or 904) may be the same or a different type of device as the electronic device (901). According to one embodiment, all or part of the operations executed in the electronic device (901) may be executed in one or more of the external electronic devices (902, 904, or 908). For example, when the electronic device (901) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (901) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (901). The electronic device (901) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (901) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (904) may include an Internet of Things (IoT) device. The server (908) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (904) or the server (908) may be included in the second network (999).The electronic device (901) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0111] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0112] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0113] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0114] Various embodiments of the present document may be implemented as software (e.g., a program (940)) including one or more instructions stored in a storage medium (e.g., an internal memory (936) or an external memory (938)) readable by a machine (e.g., an electronic device (901)). For example, a processor (e.g., a processor (920)) of the machine (e.g., an electronic device (901)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0115] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0116] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0117] FIG. 10 illustrates an example of a generative artificial intelligence system according to one embodiment.

[0118] Referring to FIG. 10, a generative artificial intelligence system (1000) illustrates an example of a system including a generative AI model (1030). For example, the generative artificial intelligence system (1000) may be included in an electronic device (200). For example, the generative artificial intelligence system (1000) may be included in a server. For example, the generative AI model (1030) may run on an NPU (210-2) and may include a trained model and other models.

[0119] A user query / response interface (1010) can receive user input. The user input may be in the form of natural language, images, and / or videos. Furthermore, context information may also be transmitted when the user input is transmitted. The context information may include various additional information at the time of user input. For example, the various additional information may include information about the application currently being used by the user or information about the user's location. Furthermore, the user input may be in a mixed form of natural language, images, sounds, and context information. Furthermore, the user input may also be in a non-natural language form, such as selecting a menu. The user query / response interface (1010) can output the results of the generative artificial intelligence system (1000) to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query / response interface (1010) can output the results of the generative artificial intelligence system (1000) to the user. The output can be in natural language form, in the form of specific content, or in the form of actions requested by the user.

[0120] The AI ​​framework (1020) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0121] User input received from the user question response interface (1010) can be transmitted to a prompt design component (1021). The prompt design component (1021) can be used to generate a prompt suitable for inputting the user input into an LLM or LMM. The prompt design component (1021) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (1021) can access knowledge repositories (1040) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0122] The API / plug-in management component (1022) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / plug-in management component (1022) can establish a channel for communicating with the outside of the AI ​​Interface through the API, and can enable access to various data sources through the established channel. In addition, the API / plug-in management component (1022) can request an action through the API when the application / service component (1050) needs to perform an action that performs the user input as a final result rather than an intermediate result. Information obtained from the outside can be used to generate a prompt in the prompt design component (1021) together with the user input, or can be passed as an input to the generative AI model (1030).

[0123] The refiner component (1023) can fine-tune the output from the generative model. For example, the refiner component (1023) can verify that the content generated through the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the refiner component (1023) can determine the degree to which the content matches the user's desired result and, if necessary, perform additional processing. The refiner component (1023) can additionally configure and provide hints to the user to avoid undesirable output.

[0124] A generative AI model (1030) may generally refer to an artificial intelligence neural network that creates new types of data based on user input information. The generative AI model (1030) may include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational auto encoder (VAE), and examples include a Diffusion-based generative model that uses a VAE and a Transformer structure. A language-generating model is a model trained to output the most statistically appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4.

[0125] For example, a language-generating model can refer to a language model that can perform inference without fine-tuning using methods such as few-shot learning, and can have more than 10 times more parameters (e.g., about 100 billion parameters) than existing general language models. For example, large-scale language models such as GPT-3 (generative pre-trained transformer 3) and GPT-4 (generative pre-trained transformer 4) are excellent few-shot learners that can be controlled through natural text prompts. They can solve NLP (natural language processing) problems by understanding patterns with only a small amount of data through prompts, which is possible with in-context learning. For example, a language-generating model can also be an LMM (large multimodal model) that can recognize various types of data input such as text, images, and speech and generate new corresponding data.

[0126] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0127] The electronic device (e.g., the electronic device (200) of FIG. 2) described above may include a memory (e.g., the memory (220) of FIG. 2) that stores instructions and includes one or more storage media, a display (e.g., the display (230) of FIG. 2) configured to receive external light, and at least one processor (e.g., at least one processor (210) of FIG. 2) that includes a central processing unit (CPU) (e.g., the CPU (210-1) of FIG. 2) that includes processing circuitry and a neural processing unit (NPU) (e.g., the NPU (210-2) of FIG. 2) that includes processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensing data regarding the external light (e.g., the external light (420, 425) of FIG. 4)) from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensed data to a trained model running on the NPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, through the trained model, information related to a foreign substance (e.g., foreign substance (405) of FIG. 4) in contact with the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop receiving the external light through the display so as to display the information through the display.

[0128] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the information to another model running on the NPU and trained. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a visual object representing the external material through the other trained model. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the information by displaying the visual object through the display.

[0129] For example, the at least one processor may further include a GPU (graphics processing unit) including a processing circuit (e.g., GPU (210-3) of FIG. 2). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the information to another model that is executed on the NPU and trained. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a visual object representing the external material (e.g., visual object (605) of FIG. 6) through the trained another model. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain another visual object that visually emphasizes the visual object (e.g., another visual object (615) of FIG. 6) by performing a rendering process on the visual object on the GPU. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the information by displaying the other visual object through the display.

[0130] For example, the electronic device may further include a buffer connected to the NPU and connected to the GPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to store the visual object acquired through the other model in the buffer. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire the other visual object by performing the rendering processing of the visual object stored in the buffer on the GPU.

[0131] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display guidance to remove the foreign substance from the display along with the information on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the information on the display to receive sensing data about the foreign substance from the display based on a period. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to use the sensing data to identify whether the foreign substance has been removed from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to stop displaying the information and the guidance based on the foreign substance being removed from the display.

[0132] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a screen through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to detect the external substance in contact with the display using the sensing data. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection in a second period shorter than the first period. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensing data to the model.

[0133] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0134] For example, the information may include information about the probability that the external substance will come into contact with pixels of the display, information about the type of the external substance, information about the material of the external substance, information about the size of the external substance, and / or information about the shape of the external substance.

[0135] The method as described above can be performed in an electronic device including at least one processor including a display configured to receive external light and a central processing unit (CPU) including a processing circuit and a neural processing unit (NPU) including a processing circuit. The method can include an operation of receiving sensing data regarding the external light from the display. The method can include an operation of providing the sensing data to a model executed on the NPU and trained. The method can include an operation of obtaining information related to an external substance contacting the display through the trained model. The method can include an operation of controlling the display to stop receiving the external light through the display so as to display the information through the display.

[0136] For example, the method may include an operation of providing the information to another model running on the NPU and trained. The method may include an operation of obtaining a visual object representing the external material through the other trained model. The method may include an operation of displaying the information by displaying the visual object through the display.

[0137] For example, the at least one processor may further include a GPU (graphics processing unit) including a processing circuit. The method may include an operation of providing the information to another model that is executed and trained on the NPU. The method may include an operation of obtaining a visual object representing the external material through the other trained model. The method may include an operation of obtaining another visual object that visually emphasizes the visual object by performing a rendering process on the visual object on the GPU. The method may include an operation of displaying the information by displaying the other visual object through the display.

[0138] For example, the electronic device may be connected to the NPU and further include a buffer connected to the GPU. The method may include an operation of storing the visual object acquired through the other model in the buffer. The method may include an operation of acquiring the other visual object by performing the rendering processing of the visual object stored in the buffer on the GPU.

[0139] For example, the method may include an operation of displaying guidance to remove the foreign substance from the display along with the information through the display. The method may include an operation of controlling the display to stop displaying the information through the display in order to receive sensing data regarding the external light from the display based on a period. The method may include an operation of identifying whether the foreign substance has been removed from the display using the sensing data. The method may include an operation of stopping displaying the information and the guidance based on the foreign substance being removed from the display.

[0140] For example, the method may include an operation of displaying a screen through the display. The method may include an operation of controlling the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The method may include an operation of detecting the external substance in contact with the display using the sensing data. The method may include an operation of controlling the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection according to a second period shorter than the first period. The method may include an operation of providing the sensing data to the model.

[0141] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0142] For example, the information may include information about the probability that the external substance will come into contact with pixels of the display, information about the type of the external substance, information about the material of the external substance, information about the size of the external substance, and / or information about the shape of the external substance.

[0143] The non-transitory computer-readable storage medium described above may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display configured to receive external light and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry, cause the electronic device to receive sensing data regarding the external light from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to a trained model that is executed on the NPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain information related to an external substance that has come into contact with the display through the trained model. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop receiving the external light through the display to display the information through the display.

[0144] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the information to another model that is executed on the NPU and trained. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain a visual object representing the external material through the other trained model. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display the information by displaying the visual object through the display.

[0145] For example, the at least one processor may further include a GPU (graphics processing unit) including a processing circuit. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the information to another model that is executed on the NPU and trained. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain a visual object representing the external material through the other trained model. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain another visual object that visually emphasizes the visual object by performing a rendering process on the visual object on the GPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display the information by displaying the other visual object through the display.

[0146] For example, the electronic device may further include a buffer connected to the NPU and connected to the GPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to store the visual object acquired through the other model in the buffer. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to acquire the other visual object by performing the rendering processing of the visual object stored in the buffer on the GPU.

[0147] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display guidance to remove the foreign substance from the display along with the information on the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the information on the display to receive sensing data regarding the foreign substance from the display based on a period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, using the sensing data, whether the foreign substance has been removed from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to stop displaying the information and the guidance based on the foreign substance being removed from the display.

[0148] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a screen through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to detect the external substance in contact with the display using the sensing data. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection in a second period shorter than the first period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to the model.

[0149] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0150] For example, the information may include information about the probability that the external substance will come into contact with pixels of the display, information about the type of the external substance, information about the material of the external substance, information about the size of the external substance, and / or information about the shape of the external substance.

[0151] The electronic device as described above may include at least one processor including a memory storing instructions and including one or more storage media, a display configured to receive external light, and a central processing unit (CPU) including a processing circuit and a neural processing unit (NPU) including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensing data regarding the external light from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensing data to a trained model executed on the NPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, through the trained model, a visual object representing an external substance contacted on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop receiving the external light through the display so as to display the visual object through the display.

[0152] For example, the at least one processor may further include a graphics processing unit (GPU) including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain another visual object that visually emphasizes the visual object by performing a rendering process on the visual object on the GPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop receiving the external light through the display so as to display the other visual object through the display.

[0153] For example, the electronic device may further include a buffer connected to the NPU and connected to the GPU. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to store the visual object obtained through the model in the buffer. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain the other visual object by performing the rendering process on the visual object stored in the buffer on the GPU.

[0154] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display guidance on the display to remove the foreign matter from the display along with the visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the information on the display to receive sensing data about the foreign matter from the display based on a period. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to use the sensing data to identify whether the foreign matter has been removed from the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to stop displaying the visual object and the guidance based on the foreign matter being removed from the display.

[0155] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a screen through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to detect the external substance in contact with the display using the sensing data. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection in a second period shorter than the first period. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide the sensing data to the model.

[0156] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0157] The method as described above can be performed in an electronic device including at least one processor including a display configured to receive external light and a central processing unit (CPU) including a processing circuit and a neural processing unit (NPU) including a processing circuit. The method can include an operation of receiving sensing data regarding the external light from the display. The method can include an operation of providing the sensing data to a trained model executed on the NPU. The method can include an operation of obtaining, through the trained model, a visual object representing an external substance contacted on the display. The method can include an operation of controlling the display to stop receiving the external light through the display so as to display the visual object through the display.

[0158] For example, the at least one processor may further include a graphics processing unit (GPU) including a processing circuit. The method may include an operation of obtaining another visual object that visually emphasizes the visual object by performing a rendering process on the visual object on the GPU. The method may include an operation of controlling the display to stop receiving the external light through the display in order to display the other visual object through the display.

[0159] For example, the electronic device may be connected to the NPU and further include a buffer connected to the GPU. The method may include an operation of storing the visual object obtained through the model in the buffer. The method may include an operation of obtaining the other visual object by performing the rendering process on the visual object stored in the buffer on the GPU.

[0160] For example, the method may include an operation of displaying guidance to remove the external substance from the display together with the visual object through the display. The method may include an operation of controlling the display to stop displaying the information through the display in order to receive sensing data about the external light from the display based on a period. The method may include an operation of identifying, using the sensing data, whether the external substance has been removed from the display. The method may include an operation of stopping displaying the visual object and the guidance based on the external substance being removed from the display.

[0161] For example, the method may include an operation of displaying a screen through the display. The method may include an operation of controlling the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The method may include an operation of detecting the external substance in contact with the display using the sensing data. The method may include an operation of controlling the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection according to a second period shorter than the first period. The method may include an operation of providing the sensing data to the model.

[0162] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0163] The non-transitory computer-readable storage medium described above may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display configured to receive external light and at least one processor including a central processing unit (CPU) including processing circuitry and a neural processing unit (NPU) including processing circuitry, cause the electronic device to receive sensing data regarding the external light from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to a trained model that is executed on the NPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain, through the trained model, a visual object representing an external substance that has come into contact on the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop receiving the external light through the display to display the visual object through the display.

[0164] For example, the at least one processor may further include a GPU (graphics processing unit) including a processing circuit. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain another visual object that visually emphasizes the visual object by performing a rendering process on the visual object on the GPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop receiving the external light through the display so as to display the other visual object through the display.

[0165] For example, the electronic device may further include a buffer connected to the NPU and connected to the GPU. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to store the visual object obtained through the model in the buffer. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain the other visual object by performing the rendering process on the visual object stored in the buffer on the GPU.

[0166] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display guidance to remove the foreign substance from the display along with the visual object on the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the information on the display to receive sensing data about the foreign substance from the display based on a period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, using the sensing data, whether the foreign substance has been removed from the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to stop displaying the visual object and the guidance based on the foreign substance being removed from the display.

[0167] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a screen through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on a first period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to detect the external substance in contact with the display using the sensing data. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to control the display to stop displaying the screen through the display in order to receive the sensing data about the external light from the display based on the detection in a second period shorter than the first period. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to provide the sensing data to the model.

[0168] For example, the sensing data may include data about coordinates of pixels of the display, data about response times of the pixels, data about the amount of light of the pixels, data about spectra of the pixels, and / or data about a screen displayed on the display.

[0169] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

Claims

In an electronic device (200), A memory (220) storing instructions and including one or more storage media; a display (230) configured to receive external light (420, 425); and At least one processor (210) including a central processing unit (CPU) (210-1) including a processing circuit and a neural processing unit (NPU) (210-2) including a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), From the above display (230), sensing data for the external light (420, 425) is received, The above sensing data is provided to the trained model running on the NPU (210-2), Through the above trained model, information related to the external material (405) in contact with the display (230) is obtained, and To control the display (230) to stop receiving the external light (420, 425) through the display (230) in order to display the information through the display (230). causing the above electronic device (200), Electronic device (200). In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), Provide the above information to another model that is run and trained on the NPU (210-2), Through the other trained model, a visual object (605) representing the external material (405) is obtained, and By displaying the above visual object (605) through the display (230), the above information is displayed. causing the above electronic device (200), Electronic device (200). In claim 1, At least one processor (210) above, Further comprising a GPU (graphic processing unit) (210-3) including a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), Provide the above information to another model that is run and trained on the NPU (210-2), Through the other trained model, a visual object (605) representing the external material (405) is obtained, By performing rendering processing on the visual object (605) on the GPU (210-3), another visual object (615) that visually emphasizes the visual object (605) is obtained, and By displaying the above other visual object (615) through the display (230), the above information is displayed. causing the above electronic device (200), Electronic device (200). In claim 3, further including a buffer connected to the NPU (210-2) and connected to the GPU (210-3), The above instructions, when executed individually or collectively by the at least one processor (210), Store the visual object (605) obtained through the other model in the buffer, and By performing the rendering processing of the visual object (605) stored in the buffer on the GPU (210-3), the other visual object (615) is obtained. causing the above electronic device (200), Electronic device (200). In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), A guidance to remove the external material (405) from the display (230) is displayed along with the information through the display (230), Based on a period, controlling the display (230) to stop displaying the information through the display (230) in order to receive sensing data for the external light (420, 425) from the display (230), Using the above sensing data, it is identified whether the external material (405) has been removed from the display (230), and To stop displaying the information and the guidance based on the foreign material (405) removed from the display (230). causing the above electronic device (200), Electronic device (200). In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), Display the screen through the above display (230), Based on the first cycle, controlling the display (230) to stop displaying the screen through the display (230) in order to receive the sensing data for the external light (420, 425) from the display (230), Using the above sensing data, the external material (405) in contact with the display (230) is detected, Based on the above detection, controlling the display (230) to stop displaying the screen through the display (230) in order to receive the sensing data for the external light (420, 425) from the display (230) according to a second cycle shorter than the first cycle, and To provide the above sensing data to the above model, causing the above electronic device (200), Electronic device (200). In claim 1, The above sensing data is, Including data on coordinates of pixels of the display (230), data on response time of the pixels, data on light amount of the pixels, data on spectra of the pixels, and / or data on a screen displayed on the display (230). Electronic device (200). In claim 1, The above information is, Information about the probability that the external substance (405) will come into contact with the pixels of the display (230), information about the type of the external substance (405), information about the material of the external substance (405), information about the size of the external substance (405), and / or information about the shape of the external substance (405). Electronic device (200). In an electronic device (200), A memory (220) storing instructions and including one or more storage media; a display (230) configured to receive external light (420, 425); and At least one processor (210) including a central processing unit (CPU) (210-1) including a processing circuit and a neural processing unit (NPU) (210-2) including a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), From the above display (230), sensing data for the external light (420, 425) is received, The above sensing data is provided to the trained model running on the NPU (210-2), Through the above trained model, a visual object (605) representing an external material (405) contacted on the display (230) is obtained, and To control the display (230) to stop receiving the external light (420, 425) through the display (230) in order to display the visual object (605) through the display (230). causing the above electronic device (200), Electronic device (200). In claim 9, At least one processor (210) above, Further comprising a GPU (graphic processing unit) (210-3) including a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), By performing rendering processing on the visual object (605) on the GPU (210-3), another visual object (615) that visually emphasizes the visual object (605) is obtained, and To control the display (230) to stop receiving the external light (420, 425) through the display (230) in order to display the other visual object (615) through the display (230). causing the above electronic device (200), Electronic device (200). In claim 10, further including a buffer connected to the NPU (210-2) and connected to the GPU (210-3), The above instructions, when executed individually or collectively by the at least one processor (210), Store the visual object (605) obtained through the above model in the buffer, and On the GPU (210-3), by performing the rendering processing on the visual object (605) stored in the buffer, the other visual object (615) is obtained. causing the above electronic device (200), Electronic device (200). In claim 9, The above instructions, when executed individually or collectively by the at least one processor (210), A guidance is displayed to remove the external material (405) from the display (230) together with the visual object (605) through the display (230), Based on a period, controlling the display (230) to stop displaying the information through the display (230) in order to receive sensing data for the external light (420, 425) from the display (230), Using the above sensing data, it is identified whether the external material (405) has been removed from the display (230), and To stop displaying the visual object (605) and the guidance based on the foreign material (405) removed from the display (230). causing the above electronic device (200), Electronic device (200). In claim 9, The above instructions, when executed individually or collectively by the at least one processor (210), Display the screen through the above display (230), Based on the first cycle, controlling the display (230) to stop displaying the screen through the display (230) in order to receive the sensing data for the external light (420, 425) from the display (230), Using the above sensing data, the external material (405) in contact with the display (230) is detected, Based on the above detection, controlling the display (230) to stop displaying the screen through the display (230) in order to receive the sensing data for the external light (420, 425) from the display (230) according to a second cycle shorter than the first cycle, and To provide the above sensing data to the above model, causing the above electronic device (200), Electronic device (200). In claim 9, The above sensing data is, Including data on coordinates of pixels of the display (230), data on response time of the pixels, data on light amount of the pixels, data on spectra of the pixels, and / or data on a screen displayed on the display (230). Electronic device (200). A method executed in an electronic device (200) comprising at least one processor (210) including a display (230) configured to receive external light (420, 425) and a central processing unit (CPU) (210-1) including a processing circuit and a neural processing unit (NPU) (210-2) including a processing circuit, the method comprising: An operation of receiving sensing data for the external light (420, 425) from the above display (230), An operation of providing the above sensing data to a trained model running on the NPU (210-2), An operation of obtaining information related to an external material (405) in contact with the display (230) through the above trained model, and In order to display the information through the display (230), the display (230) comprises an operation of controlling the display (230) to stop receiving the external light (420, 425) through the display (230). method.

Images